guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/testexec.16593.ini bucket_storage=magma,rerun=false,bucket_eviction_policy=fullEviction,randomize_value=True,enable_dp=True,log_level=debug,infra_log_level=debug,upgrade_version=7.0.0-5270 -t magma.magma_disk_full.MagmaDiskFull.test_disk_full_reduce_replica,nodes_init=4,num_items=5000000,doc_size=2048,sdk_timeout=60,replicas=3,GROUP=P0' Test Input params: {'doc_size': '2048', 'conf_file': 'conf/magma/disk_full.conf', 'upgrade_version': '7.0.0-5270', 'spec': 'disk_full', 'rerun': 'false', 'num_nodes': 4, 'GROUP': 'P0', 'enable_dp': 'True', 'sdk_timeout': '60', 'case_number': 3, 'cluster_name': 'testexec.16593', 'ini': '/tmp/testexec.16593.ini', 'replicas': '3', 'bucket_storage': 'magma', 'log_level': 'debug', 'bucket_eviction_policy': 'fullEviction', 'logs_folder': '/data/workspace/centos-p0-magma-vset00-00-disk_full_7.0_P0/logs/testrunner-21-Jun-08_05-04-46/test_3', 'nodes_init': '4', 'num_items': '5000000', 'randomize_value': 'True', 'infra_log_level': 'debug'} test_disk_full_reduce_replica (magma.magma_disk_full.MagmaDiskFull) ... 2021-06-08 06:27:31,842 | infra | INFO | MainThread | [task_manager:__init__:14] Initiating TaskManager with 30 threads 2021-06-08 06:27:31,842 | infra | DEBUG | MainThread | [cluster:__init__:29] Initiating ServerTasks 2021-06-08 06:27:31,842 | test | INFO | MainThread | [basetestcase:log_setup_status:546] ========= MagmaDiskFull setup started for test #3 test_disk_full_reduce_replica ========= 2021-06-08 06:27:31,854 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.196 with username: root, password: couchbase 2021-06-08 06:27:31,934 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 06:27:32,138 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: ls /etc/ 2021-06-08 06:27:32,194 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 06:27:32,194 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: ls /cygdrive/c/Windows 2021-06-08 06:27:32,207 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: uname -m 2021-06-08 06:27:32,219 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: cat /proc/cpuinfo 2021-06-08 06:27:32,230 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: df -Thl 2021-06-08 06:27:32,247 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: cat /proc/meminfo 2021-06-08 06:27:32,259 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: hostname 2021-06-08 06:27:32,272 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: hostname -d 2021-06-08 06:27:32,283 | infra | DEBUG | MainThread | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.196 2021-06-08 06:27:32,285 | test | INFO | MainThread | [basetestcase:log_setup_status:546] ========= BaseTestCase setup started for test #3 test_disk_full_reduce_replica ========= 2021-06-08 06:27:32,285 | test | DEBUG | MainThread | [basetestcase:setUp:294] Cleaning up cluster 2021-06-08 06:27:32,328 | test | DEBUG | MainThread | [bucket_ready_functions:delete_all_buckets:1571] Deleting existing buckets [] on 172.23.106.196 2021-06-08 06:27:32,345 | test | DEBUG | MainThread | [bucket_ready_functions:delete_all_buckets:1571] Deleting existing buckets [] on 172.23.106.199 2021-06-08 06:27:32,361 | test | DEBUG | MainThread | [bucket_ready_functions:delete_all_buckets:1571] Deleting existing buckets [] on 172.23.106.168 2021-06-08 06:27:32,378 | test | DEBUG | MainThread | [bucket_ready_functions:delete_all_buckets:1571] Deleting existing buckets [] on 172.23.106.198 2021-06-08 06:27:32,410 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:46] Checking if ns_server is running with timeout 120secs 2021-06-08 06:27:32,492 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:148] Waiting for ns_server @ 172.23.106.196:8091 2021-06-08 06:27:32,492 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:46] Checking if ns_server is running with timeout 360secs 2021-06-08 06:27:32,500 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:151] ns_server @ 172.23.106.196:8091 is running 2021-06-08 06:27:32,502 | test | INFO | MainThread | [basetestcase:setUp:320] Initializing cluster 2021-06-08 06:27:32,512 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.196 with username: root, password: couchbase 2021-06-08 06:27:32,582 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 06:27:32,783 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: ls /etc/ 2021-06-08 06:27:32,838 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 06:27:32,838 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: ls /cygdrive/c/Windows 2021-06-08 06:27:32,851 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: uname -m 2021-06-08 06:27:32,861 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: cat /proc/cpuinfo 2021-06-08 06:27:32,874 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: df -Thl 2021-06-08 06:27:32,888 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: cat /proc/meminfo 2021-06-08 06:27:32,900 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: hostname 2021-06-08 06:27:32,911 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: hostname -d 2021-06-08 06:27:32,953 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.196 with username: root, password: couchbase 2021-06-08 06:27:33,026 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 06:27:33,223 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: ls /etc/ 2021-06-08 06:27:33,276 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 06:27:33,276 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: ls /cygdrive/c/Windows 2021-06-08 06:27:33,289 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: uname -m 2021-06-08 06:27:33,299 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: cat /proc/cpuinfo 2021-06-08 06:27:33,312 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: df -Thl 2021-06-08 06:27:33,326 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: cat /proc/meminfo 2021-06-08 06:27:33,338 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: hostname 2021-06-08 06:27:33,351 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: hostname -d 2021-06-08 06:27:33,365 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:713] 172.23.106.196 - Checking if couchbase-server is installed 2021-06-08 06:27:33,384 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:746] 172.23.106.196 - The version file /opt/couchbase/ VERSION.txt exists 2021-06-08 06:27:33,420 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: cat /opt/couchbase/VERSION.txt 2021-06-08 06:27:33,434 | infra | DEBUG | MainThread | [remote_util:stop_couchbase:3770] 172.23.106.196 - Running systemd command 2021-06-08 06:27:33,434 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: systemctl stop couchbase-server.service 2021-06-08 06:27:35,151 | test | DEBUG | MainThread | [cluster_ready_functions:stop_server:417] Couchbase stopped on ip:172.23.106.196 port:8091 ssh_username:root 2021-06-08 06:27:35,151 | infra | DEBUG | MainThread | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.196 2021-06-08 06:27:35,151 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: rm -rf /data/* 2021-06-08 06:27:35,165 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: rm -rf /opt/couchbase/var/lib/couchbase/config//* 2021-06-08 06:27:35,181 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: rm -rf /opt/couchbase/var/lib/couchbase/crash//* 2021-06-08 06:27:35,194 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: rm -rf /opt/couchbase/var/lib/couchbase/config//* 2021-06-08 06:27:35,209 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.196 with username: root, password: couchbase 2021-06-08 06:27:35,282 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 06:27:35,483 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: ls /etc/ 2021-06-08 06:27:35,535 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 06:27:35,536 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: ls /cygdrive/c/Windows 2021-06-08 06:27:35,548 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: uname -m 2021-06-08 06:27:35,559 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: cat /proc/cpuinfo 2021-06-08 06:27:35,571 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: df -Thl 2021-06-08 06:27:35,585 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: cat /proc/meminfo 2021-06-08 06:27:35,598 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: hostname 2021-06-08 06:27:35,608 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: hostname -d 2021-06-08 06:27:35,621 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:713] 172.23.106.196 - Checking if couchbase-server is installed 2021-06-08 06:27:35,638 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:746] 172.23.106.196 - The version file /opt/couchbase/ VERSION.txt exists 2021-06-08 06:27:35,671 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: cat /opt/couchbase/VERSION.txt 2021-06-08 06:27:35,684 | infra | DEBUG | MainThread | [remote_util:start_couchbase:3867] 172.23.106.196 - Running systemd command 2021-06-08 06:27:35,684 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: systemctl start couchbase-server.service 2021-06-08 06:27:35,704 | test | DEBUG | MainThread | [cluster_ready_functions:start_server:431] Couchbase started on ip:172.23.106.196 port:8091 ssh_username:root 2021-06-08 06:27:35,704 | infra | DEBUG | MainThread | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.196 Got this failure java.net.ConnectException: Connection refused: /172.23.106.196:8091 during connect (<_realsocket at 0x169ca type=client open_count=1 channel=[id: 0x20169af8, 0.0.0.0/0.0.0.0:41794] timeout=300.0>) 2021-06-08 06:27:35,707 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.106.196:8091/nodes/self error [Errno 111] Connection refused 2021-06-08 06:27:35,714 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L762 2021-06-08 06:27:35,714 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None Got this failure java.net.ConnectException: Connection refused: /172.23.106.196:8091 during connect (<_realsocket at 0x169cb type=client open_count=1 channel=[id: 0x86d404f6, 0.0.0.0/0.0.0.0:49819] timeout=300.0>) 2021-06-08 06:27:37,717 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.106.196:8091/nodes/self error [Errno 111] Connection refused 2021-06-08 06:27:37,723 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L762 2021-06-08 06:27:37,723 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None Got this failure java.net.ConnectException: Connection refused: /172.23.106.196:8091 during connect (<_realsocket at 0x169cc type=client open_count=1 channel=[id: 0x413332f8, 0.0.0.0/0.0.0.0:57806] timeout=300.0>) 2021-06-08 06:27:39,726 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.106.196:8091/nodes/self error [Errno 111] Connection refused 2021-06-08 06:27:39,732 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L762 2021-06-08 06:27:39,732 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None Got this failure java.net.ConnectException: Connection refused: /172.23.106.196:8091 during connect (<_realsocket at 0x169cd type=client open_count=1 channel=[id: 0x66d767e5, 0.0.0.0/0.0.0.0:42164] timeout=300.0>) 2021-06-08 06:27:41,733 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.106.196:8091/nodes/self error [Errno 111] Connection refused 2021-06-08 06:27:41,740 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L762 2021-06-08 06:27:41,740 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-06-08 06:27:43,752 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:46] Checking if ns_server is running with timeout 360secs 2021-06-08 06:27:43,760 | infra | DEBUG | MainThread | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.196 2021-06-08 06:27:43,763 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.199 with username: root, password: couchbase 2021-06-08 06:27:43,838 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 06:27:44,046 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: ls /etc/ 2021-06-08 06:27:44,099 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 06:27:44,099 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: ls /cygdrive/c/Windows 2021-06-08 06:27:44,111 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: uname -m 2021-06-08 06:27:44,124 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: cat /proc/cpuinfo 2021-06-08 06:27:44,135 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: df -Thl 2021-06-08 06:27:44,151 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: cat /proc/meminfo 2021-06-08 06:27:44,161 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: hostname 2021-06-08 06:27:44,174 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: hostname -d 2021-06-08 06:27:44,214 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.199 with username: root, password: couchbase 2021-06-08 06:27:44,283 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 06:27:44,484 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: ls /etc/ 2021-06-08 06:27:44,536 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 06:27:44,536 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: ls /cygdrive/c/Windows 2021-06-08 06:27:44,548 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: uname -m 2021-06-08 06:27:44,559 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: cat /proc/cpuinfo 2021-06-08 06:27:44,571 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: df -Thl 2021-06-08 06:27:44,585 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: cat /proc/meminfo 2021-06-08 06:27:44,595 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: hostname 2021-06-08 06:27:44,607 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: hostname -d 2021-06-08 06:27:44,618 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:713] 172.23.106.199 - Checking if couchbase-server is installed 2021-06-08 06:27:44,635 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:746] 172.23.106.199 - The version file /opt/couchbase/ VERSION.txt exists 2021-06-08 06:27:44,668 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: cat /opt/couchbase/VERSION.txt 2021-06-08 06:27:44,680 | infra | DEBUG | MainThread | [remote_util:stop_couchbase:3770] 172.23.106.199 - Running systemd command 2021-06-08 06:27:44,680 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: systemctl stop couchbase-server.service 2021-06-08 06:27:46,279 | test | DEBUG | MainThread | [cluster_ready_functions:stop_server:417] Couchbase stopped on ip:172.23.106.199 port:8091 ssh_username:root 2021-06-08 06:27:46,279 | infra | DEBUG | MainThread | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.199 2021-06-08 06:27:46,279 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: rm -rf /data/* 2021-06-08 06:27:46,293 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: rm -rf /opt/couchbase/var/lib/couchbase/config//* 2021-06-08 06:27:46,306 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: rm -rf /opt/couchbase/var/lib/couchbase/crash//* 2021-06-08 06:27:46,319 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: rm -rf /opt/couchbase/var/lib/couchbase/config//* 2021-06-08 06:27:46,332 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.199 with username: root, password: couchbase 2021-06-08 06:27:46,413 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 06:27:46,618 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: ls /etc/ 2021-06-08 06:27:46,674 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 06:27:46,674 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: ls /cygdrive/c/Windows 2021-06-08 06:27:46,687 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: uname -m 2021-06-08 06:27:46,698 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: cat /proc/cpuinfo 2021-06-08 06:27:46,710 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: df -Thl 2021-06-08 06:27:46,724 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: cat /proc/meminfo 2021-06-08 06:27:46,736 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: hostname 2021-06-08 06:27:46,749 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: hostname -d 2021-06-08 06:27:46,762 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:713] 172.23.106.199 - Checking if couchbase-server is installed 2021-06-08 06:27:46,779 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:746] 172.23.106.199 - The version file /opt/couchbase/ VERSION.txt exists 2021-06-08 06:27:46,815 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: cat /opt/couchbase/VERSION.txt 2021-06-08 06:27:46,826 | infra | DEBUG | MainThread | [remote_util:start_couchbase:3867] 172.23.106.199 - Running systemd command 2021-06-08 06:27:46,826 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: systemctl start couchbase-server.service 2021-06-08 06:27:46,848 | test | DEBUG | MainThread | [cluster_ready_functions:start_server:431] Couchbase started on ip:172.23.106.199 port:8091 ssh_username:root 2021-06-08 06:27:46,848 | infra | DEBUG | MainThread | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.199 Got this failure java.net.ConnectException: Connection refused: /172.23.106.199:8091 during connect (<_realsocket at 0x169ce type=client open_count=1 channel=[id: 0xef533421, 0.0.0.0/0.0.0.0:53918] timeout=300.0>) 2021-06-08 06:27:46,851 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.106.199:8091/nodes/self error [Errno 111] Connection refused 2021-06-08 06:27:46,858 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L762 2021-06-08 06:27:46,858 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None Got this failure java.net.ConnectException: Connection refused: /172.23.106.199:8091 during connect (<_realsocket at 0x169cf type=client open_count=1 channel=[id: 0x33e7e52e, 0.0.0.0/0.0.0.0:34397] timeout=300.0>) 2021-06-08 06:27:48,859 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.106.199:8091/nodes/self error [Errno 111] Connection refused 2021-06-08 06:27:48,867 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L762 2021-06-08 06:27:48,867 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None Got this failure java.net.ConnectException: Connection refused: /172.23.106.199:8091 during connect (<_realsocket at 0x169d0 type=client open_count=1 channel=[id: 0xd3acb066, 0.0.0.0/0.0.0.0:42117] timeout=300.0>) 2021-06-08 06:27:50,868 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.106.199:8091/nodes/self error [Errno 111] Connection refused 2021-06-08 06:27:50,875 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L762 2021-06-08 06:27:50,875 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None Got this failure java.net.ConnectException: Connection refused: /172.23.106.199:8091 during connect (<_realsocket at 0x169d1 type=client open_count=1 channel=[id: 0xbe1ed112, 0.0.0.0/0.0.0.0:56819] timeout=300.0>) 2021-06-08 06:27:52,878 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.106.199:8091/nodes/self error [Errno 111] Connection refused 2021-06-08 06:27:52,885 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L762 2021-06-08 06:27:52,885 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-06-08 06:27:54,900 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:46] Checking if ns_server is running with timeout 360secs 2021-06-08 06:27:54,908 | infra | DEBUG | MainThread | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.199 2021-06-08 06:27:54,911 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.168 with username: root, password: couchbase 2021-06-08 06:27:54,986 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 06:27:55,191 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: ls /etc/ 2021-06-08 06:27:55,243 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 06:27:55,244 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: ls /cygdrive/c/Windows 2021-06-08 06:27:55,256 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: uname -m 2021-06-08 06:27:55,269 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: cat /proc/cpuinfo 2021-06-08 06:27:55,280 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: df -Thl 2021-06-08 06:27:55,295 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: cat /proc/meminfo 2021-06-08 06:27:55,306 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: hostname 2021-06-08 06:27:55,319 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: hostname -d 2021-06-08 06:27:55,358 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.168 with username: root, password: couchbase 2021-06-08 06:27:55,427 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 06:27:55,631 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: ls /etc/ 2021-06-08 06:27:55,683 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 06:27:55,683 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: ls /cygdrive/c/Windows 2021-06-08 06:27:55,696 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: uname -m 2021-06-08 06:27:55,707 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: cat /proc/cpuinfo 2021-06-08 06:27:55,719 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: df -Thl 2021-06-08 06:27:55,733 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: cat /proc/meminfo 2021-06-08 06:27:55,743 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: hostname 2021-06-08 06:27:55,756 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: hostname -d 2021-06-08 06:27:55,767 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:713] 172.23.106.168 - Checking if couchbase-server is installed 2021-06-08 06:27:55,786 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:746] 172.23.106.168 - The version file /opt/couchbase/ VERSION.txt exists 2021-06-08 06:27:55,819 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: cat /opt/couchbase/VERSION.txt 2021-06-08 06:27:55,831 | infra | DEBUG | MainThread | [remote_util:stop_couchbase:3770] 172.23.106.168 - Running systemd command 2021-06-08 06:27:55,831 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: systemctl stop couchbase-server.service 2021-06-08 06:27:57,486 | test | DEBUG | MainThread | [cluster_ready_functions:stop_server:417] Couchbase stopped on ip:172.23.106.168 port:8091 ssh_username:root 2021-06-08 06:27:57,486 | infra | DEBUG | MainThread | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.168 2021-06-08 06:27:57,486 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: rm -rf /data/* 2021-06-08 06:27:57,502 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: rm -rf /opt/couchbase/var/lib/couchbase/config//* 2021-06-08 06:27:57,516 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: rm -rf /opt/couchbase/var/lib/couchbase/crash//* 2021-06-08 06:27:57,526 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: rm -rf /opt/couchbase/var/lib/couchbase/config//* 2021-06-08 06:27:57,540 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.168 with username: root, password: couchbase 2021-06-08 06:27:57,611 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 06:27:57,815 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: ls /etc/ 2021-06-08 06:27:57,869 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 06:27:57,869 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: ls /cygdrive/c/Windows 2021-06-08 06:27:57,881 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: uname -m 2021-06-08 06:27:57,892 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: cat /proc/cpuinfo 2021-06-08 06:27:57,904 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: df -Thl 2021-06-08 06:27:57,920 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: cat /proc/meminfo 2021-06-08 06:27:57,931 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: hostname 2021-06-08 06:27:57,944 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: hostname -d 2021-06-08 06:27:57,956 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:713] 172.23.106.168 - Checking if couchbase-server is installed 2021-06-08 06:27:57,973 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:746] 172.23.106.168 - The version file /opt/couchbase/ VERSION.txt exists 2021-06-08 06:27:58,006 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: cat /opt/couchbase/VERSION.txt 2021-06-08 06:27:58,019 | infra | DEBUG | MainThread | [remote_util:start_couchbase:3867] 172.23.106.168 - Running systemd command 2021-06-08 06:27:58,019 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: systemctl start couchbase-server.service 2021-06-08 06:27:58,042 | test | DEBUG | MainThread | [cluster_ready_functions:start_server:431] Couchbase started on ip:172.23.106.168 port:8091 ssh_username:root 2021-06-08 06:27:58,043 | infra | DEBUG | MainThread | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.168 Got this failure java.net.ConnectException: Connection refused: /172.23.106.168:8091 during connect (<_realsocket at 0x169d2 type=client open_count=1 channel=[id: 0xd9893dc9, 0.0.0.0/0.0.0.0:47039] timeout=300.0>) 2021-06-08 06:27:58,046 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.106.168:8091/nodes/self error [Errno 111] Connection refused 2021-06-08 06:27:58,052 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L762 2021-06-08 06:27:58,052 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None Got this failure java.net.ConnectException: Connection refused: /172.23.106.168:8091 during connect (<_realsocket at 0x169d3 type=client open_count=1 channel=[id: 0xcec3aa7e, 0.0.0.0/0.0.0.0:37965] timeout=300.0>) 2021-06-08 06:28:00,055 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.106.168:8091/nodes/self error [Errno 111] Connection refused 2021-06-08 06:28:00,062 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L762 2021-06-08 06:28:00,062 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None Got this failure java.net.ConnectException: Connection refused: /172.23.106.168:8091 during connect (<_realsocket at 0x169d4 type=client open_count=1 channel=[id: 0x8194d795, 0.0.0.0/0.0.0.0:40910] timeout=300.0>) 2021-06-08 06:28:02,063 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.106.168:8091/nodes/self error [Errno 111] Connection refused 2021-06-08 06:28:02,071 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L762 2021-06-08 06:28:02,071 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None Got this failure java.net.ConnectException: Connection refused: /172.23.106.168:8091 during connect (<_realsocket at 0x169d5 type=client open_count=1 channel=[id: 0xdd269f65, 0.0.0.0/0.0.0.0:60921] timeout=300.0>) 2021-06-08 06:28:04,072 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.106.168:8091/nodes/self error [Errno 111] Connection refused 2021-06-08 06:28:04,082 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L762 2021-06-08 06:28:04,082 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-06-08 06:28:06,095 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:46] Checking if ns_server is running with timeout 360secs 2021-06-08 06:28:06,104 | infra | DEBUG | MainThread | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.168 2021-06-08 06:28:06,107 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.198 with username: root, password: couchbase 2021-06-08 06:28:06,170 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 06:28:06,375 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: ls /etc/ 2021-06-08 06:28:06,428 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 06:28:06,428 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: ls /cygdrive/c/Windows 2021-06-08 06:28:06,441 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: uname -m 2021-06-08 06:28:06,453 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: cat /proc/cpuinfo 2021-06-08 06:28:06,464 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: df -Thl 2021-06-08 06:28:06,480 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: cat /proc/meminfo 2021-06-08 06:28:06,490 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: hostname 2021-06-08 06:28:06,503 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: hostname -d 2021-06-08 06:28:06,543 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.198 with username: root, password: couchbase 2021-06-08 06:28:06,617 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 06:28:06,816 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: ls /etc/ 2021-06-08 06:28:06,869 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 06:28:06,869 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: ls /cygdrive/c/Windows 2021-06-08 06:28:06,882 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: uname -m 2021-06-08 06:28:06,894 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: cat /proc/cpuinfo 2021-06-08 06:28:06,907 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: df -Thl 2021-06-08 06:28:06,921 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: cat /proc/meminfo 2021-06-08 06:28:06,933 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: hostname 2021-06-08 06:28:06,944 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: hostname -d 2021-06-08 06:28:06,957 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:713] 172.23.106.198 - Checking if couchbase-server is installed 2021-06-08 06:28:06,976 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:746] 172.23.106.198 - The version file /opt/couchbase/ VERSION.txt exists 2021-06-08 06:28:07,010 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: cat /opt/couchbase/VERSION.txt 2021-06-08 06:28:07,022 | infra | DEBUG | MainThread | [remote_util:stop_couchbase:3770] 172.23.106.198 - Running systemd command 2021-06-08 06:28:07,023 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: systemctl stop couchbase-server.service 2021-06-08 06:28:08,678 | test | DEBUG | MainThread | [cluster_ready_functions:stop_server:417] Couchbase stopped on ip:172.23.106.198 port:8091 ssh_username:root 2021-06-08 06:28:08,678 | infra | DEBUG | MainThread | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.198 2021-06-08 06:28:08,680 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: rm -rf /data/* 2021-06-08 06:28:08,694 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: rm -rf /opt/couchbase/var/lib/couchbase/config//* 2021-06-08 06:28:08,709 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: rm -rf /opt/couchbase/var/lib/couchbase/crash//* 2021-06-08 06:28:08,720 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: rm -rf /opt/couchbase/var/lib/couchbase/config//* 2021-06-08 06:28:08,733 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.198 with username: root, password: couchbase 2021-06-08 06:28:08,821 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 06:28:09,026 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: ls /etc/ 2021-06-08 06:28:09,078 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 06:28:09,079 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: ls /cygdrive/c/Windows 2021-06-08 06:28:09,092 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: uname -m 2021-06-08 06:28:09,102 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: cat /proc/cpuinfo 2021-06-08 06:28:09,115 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: df -Thl 2021-06-08 06:28:09,128 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: cat /proc/meminfo 2021-06-08 06:28:09,141 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: hostname 2021-06-08 06:28:09,151 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: hostname -d 2021-06-08 06:28:09,164 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:713] 172.23.106.198 - Checking if couchbase-server is installed 2021-06-08 06:28:09,180 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:746] 172.23.106.198 - The version file /opt/couchbase/ VERSION.txt exists 2021-06-08 06:28:09,216 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: cat /opt/couchbase/VERSION.txt 2021-06-08 06:28:09,226 | infra | DEBUG | MainThread | [remote_util:start_couchbase:3867] 172.23.106.198 - Running systemd command 2021-06-08 06:28:09,227 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: systemctl start couchbase-server.service 2021-06-08 06:28:09,249 | test | DEBUG | MainThread | [cluster_ready_functions:start_server:431] Couchbase started on ip:172.23.106.198 port:8091 ssh_username:root 2021-06-08 06:28:09,250 | infra | DEBUG | MainThread | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.198 Got this failure java.net.ConnectException: Connection refused: /172.23.106.198:8091 during connect (<_realsocket at 0x169d6 type=client open_count=1 channel=[id: 0xa9ae391d, 0.0.0.0/0.0.0.0:39375] timeout=300.0>) 2021-06-08 06:28:09,252 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.106.198:8091/nodes/self error [Errno 111] Connection refused 2021-06-08 06:28:09,257 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L762 2021-06-08 06:28:09,257 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None Got this failure java.net.ConnectException: Connection refused: /172.23.106.198:8091 during connect (<_realsocket at 0x169d7 type=client open_count=1 channel=[id: 0xe8cd9623, 0.0.0.0/0.0.0.0:47827] timeout=300.0>) 2021-06-08 06:28:11,260 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.106.198:8091/nodes/self error [Errno 111] Connection refused 2021-06-08 06:28:11,267 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L762 2021-06-08 06:28:11,267 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None Got this failure java.net.ConnectException: Connection refused: /172.23.106.198:8091 during connect (<_realsocket at 0x169d8 type=client open_count=1 channel=[id: 0x78dec22d, 0.0.0.0/0.0.0.0:49454] timeout=300.0>) 2021-06-08 06:28:13,269 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.106.198:8091/nodes/self error [Errno 111] Connection refused 2021-06-08 06:28:13,276 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L762 2021-06-08 06:28:13,276 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None Got this failure java.net.ConnectException: Connection refused: /172.23.106.198:8091 during connect (<_realsocket at 0x169d9 type=client open_count=1 channel=[id: 0x53644c58, 0.0.0.0/0.0.0.0:50404] timeout=300.0>) 2021-06-08 06:28:15,279 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.106.198:8091/nodes/self error [Errno 111] Connection refused 2021-06-08 06:28:15,285 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L762 2021-06-08 06:28:15,286 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-06-08 06:28:17,298 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:46] Checking if ns_server is running with timeout 360secs 2021-06-08 06:28:17,306 | infra | DEBUG | MainThread | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.198 2021-06-08 06:28:17,312 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from couchbase_utils/cluster_utils/cluster_ready_functions.py -> reset_cluster():L507 2021-06-08 06:28:17,312 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-06-08 06:28:27,315 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.196 with username: root, password: couchbase 2021-06-08 06:28:27,380 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 06:28:27,585 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: ls /etc/ 2021-06-08 06:28:27,637 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 06:28:27,637 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: ls /cygdrive/c/Windows 2021-06-08 06:28:27,648 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: uname -m 2021-06-08 06:28:27,660 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: cat /proc/cpuinfo 2021-06-08 06:28:27,671 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: df -Thl 2021-06-08 06:28:27,686 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: cat /proc/meminfo 2021-06-08 06:28:27,697 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: hostname 2021-06-08 06:28:27,709 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: hostname -d 2021-06-08 06:28:27,723 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.199 with username: root, password: couchbase 2021-06-08 06:28:27,802 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 06:28:28,010 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: ls /etc/ 2021-06-08 06:28:28,065 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 06:28:28,065 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: ls /cygdrive/c/Windows 2021-06-08 06:28:28,078 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: uname -m 2021-06-08 06:28:28,089 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: cat /proc/cpuinfo 2021-06-08 06:28:28,101 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: df -Thl 2021-06-08 06:28:28,117 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: cat /proc/meminfo 2021-06-08 06:28:28,128 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: hostname 2021-06-08 06:28:28,141 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: hostname -d 2021-06-08 06:28:28,154 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.168 with username: root, password: couchbase 2021-06-08 06:28:28,240 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 06:28:28,451 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: ls /etc/ 2021-06-08 06:28:28,505 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 06:28:28,505 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: ls /cygdrive/c/Windows 2021-06-08 06:28:28,517 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: uname -m 2021-06-08 06:28:28,529 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: cat /proc/cpuinfo 2021-06-08 06:28:28,540 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: df -Thl 2021-06-08 06:28:28,555 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: cat /proc/meminfo 2021-06-08 06:28:28,568 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: hostname 2021-06-08 06:28:28,579 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: hostname -d 2021-06-08 06:28:28,594 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.198 with username: root, password: couchbase 2021-06-08 06:28:28,677 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 06:28:28,884 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: ls /etc/ 2021-06-08 06:28:28,937 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 06:28:28,937 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: ls /cygdrive/c/Windows 2021-06-08 06:28:28,950 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: uname -m 2021-06-08 06:28:28,960 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: cat /proc/cpuinfo 2021-06-08 06:28:28,973 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: df -Thl 2021-06-08 06:28:28,986 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: cat /proc/meminfo 2021-06-08 06:28:28,997 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: hostname 2021-06-08 06:28:29,009 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: hostname -d 2021-06-08 06:28:29,022 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: rm -rf /data/* 2021-06-08 06:28:29,036 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: chown -R couchbase:couchbase /data 2021-06-08 06:28:29,059 | test | DEBUG | MainThread | [rest_client:set_data_path:2138] /nodes/self/controller/settings: path=%2Fdata&index_path=%2Fdata 2021-06-08 06:28:29,065 | test | DEBUG | MainThread | [rest_client:set_data_path:2141] Setting paths: path=%2Fdata&index_path=%2Fdata: status 1 2021-06-08 06:28:29,066 | infra | INFO | MainThread | [task_manager:add_new_task:23] Added new task: node_init_task_172.23.106.196_8091 2021-06-08 06:28:29,066 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: rm -rf /data/* 2021-06-08 06:28:29,066 | infra | INFO | pool-5-thread-1 | [task:start_task:87] Thread 'node_init_task_172.23.106.196_8091' started 2021-06-08 06:28:29,082 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: chown -R couchbase:couchbase /data 2021-06-08 06:28:29,085 | infra | DEBUG | pool-5-thread-1 | [task:wait_until:131] Wait for expected condition to get satisfied 2021-06-08 06:28:29,085 | test | DEBUG | pool-5-thread-1 | [task:call:5740] server: ip:172.23.106.196 port:8091 ssh_username:root, nodes/self: {'memcached': 11210, 'memoryFree': 11745034240L, 'eventingMemoryQuota': 256, 'rest_password': '', 'storage': [], 'ports': [], 'cbasMemoryQuota': 1839, 'hostname': u'172.23.106.196:8091', 'ftsMemoryQuota': 512, 'clusterCompatibility': 458752, 'id': u'ns_1@172.23.106.196', 'mcdMemoryReserved': 9379, 'moxi': 11211, 'rest_username': '', 'os': u'x86_64-unknown-linux-gnu', 'ip': u'172.23.106.196', 'curr_items': 0, 'mcdMemoryAllocated': 9379, 'services': [u'kv'], 'version': u'7.0.0-5270-enterprise', 'indexMemoryQuota': 512, 'uptime': u'51', 'clusterMembership': u'active', 'storageTotalRam': 11724L, 'port': u'8091', 'availableStorage': [], 'memoryTotal': 12294176768L, 'memoryQuota': 6368, 'status': u'healthy'} 2021-06-08 06:28:29,086 | test | DEBUG | pool-5-thread-1 | [rest_client:set_service_mem_quota:859] Setting RAM quotas: {'memoryQuota': 8351} 2021-06-08 06:28:29,098 | test | DEBUG | pool-5-thread-1 | [rest_client:set_indexer_storage_mode:883] settings/indexes params: storageMode=plasma 2021-06-08 06:28:29,108 | test | DEBUG | MainThread | [rest_client:set_data_path:2138] /nodes/self/controller/settings: path=%2Fdata&index_path=%2Fdata 2021-06-08 06:28:29,115 | test | DEBUG | pool-5-thread-1 | [rest_client:init_node_services:837] /node/controller/setupServices params on 172.23.106.196: 8091:services=kv&hostname=172.23.106.196&password=password&user=Administrator 2021-06-08 06:28:29,118 | test | DEBUG | MainThread | [rest_client:set_data_path:2141] Setting paths: path=%2Fdata&index_path=%2Fdata: status 1 2021-06-08 06:28:29,118 | infra | INFO | MainThread | [task_manager:add_new_task:23] Added new task: node_init_task_172.23.106.199_8091 2021-06-08 06:28:29,118 | infra | INFO | pool-5-thread-2 | [task:start_task:87] Thread 'node_init_task_172.23.106.199_8091' started 2021-06-08 06:28:29,118 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: rm -rf /data/* 2021-06-08 06:28:29,132 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: chown -R couchbase:couchbase /data 2021-06-08 06:28:29,137 | infra | DEBUG | pool-5-thread-2 | [task:wait_until:131] Wait for expected condition to get satisfied 2021-06-08 06:28:29,137 | test | DEBUG | pool-5-thread-2 | [task:call:5740] server: ip:172.23.106.199 port:8091 ssh_username:root, nodes/self: {'memcached': 11210, 'memoryFree': 11736588288L, 'eventingMemoryQuota': 256, 'rest_password': '', 'storage': [], 'ports': [], 'cbasMemoryQuota': 1839, 'hostname': u'172.23.106.199:8091', 'ftsMemoryQuota': 512, 'clusterCompatibility': 458752, 'id': u'ns_1@172.23.106.199', 'mcdMemoryReserved': 9379, 'moxi': 11211, 'rest_username': '', 'os': u'x86_64-unknown-linux-gnu', 'ip': u'172.23.106.199', 'curr_items': 0, 'mcdMemoryAllocated': 9379, 'services': [u'kv'], 'version': u'7.0.0-5270-enterprise', 'indexMemoryQuota': 512, 'uptime': u'35', 'clusterMembership': u'active', 'storageTotalRam': 11724L, 'port': u'8091', 'availableStorage': [], 'memoryTotal': 12294176768L, 'memoryQuota': 6368, 'status': u'healthy'} 2021-06-08 06:28:29,137 | test | DEBUG | pool-5-thread-2 | [rest_client:set_service_mem_quota:859] Setting RAM quotas: {'memoryQuota': 8351} 2021-06-08 06:28:29,148 | test | DEBUG | pool-5-thread-1 | [rest_client:init_cluster:770] settings/web params on 172.23.106.196:8091:password=password&port=8091&username=Administrator 2021-06-08 06:28:29,148 | test | DEBUG | pool-5-thread-2 | [rest_client:set_indexer_storage_mode:883] settings/indexes params: storageMode=plasma 2021-06-08 06:28:29,157 | test | DEBUG | MainThread | [rest_client:set_data_path:2138] /nodes/self/controller/settings: path=%2Fdata&index_path=%2Fdata 2021-06-08 06:28:29,160 | test | DEBUG | pool-5-thread-2 | [rest_client:init_cluster:770] settings/web params on 172.23.106.199:8091:password=password&port=8091&username=Administrator 2021-06-08 06:28:29,164 | test | DEBUG | MainThread | [rest_client:set_data_path:2141] Setting paths: path=%2Fdata&index_path=%2Fdata: status 1 2021-06-08 06:28:29,164 | infra | INFO | MainThread | [task_manager:add_new_task:23] Added new task: node_init_task_172.23.106.168_8091 2021-06-08 06:28:29,164 | infra | INFO | pool-5-thread-3 | [task:start_task:87] Thread 'node_init_task_172.23.106.168_8091' started 2021-06-08 06:28:29,164 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: rm -rf /data/* 2021-06-08 06:28:29,178 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: chown -R couchbase:couchbase /data 2021-06-08 06:28:29,181 | infra | DEBUG | pool-5-thread-3 | [task:wait_until:131] Wait for expected condition to get satisfied 2021-06-08 06:28:29,181 | test | DEBUG | pool-5-thread-3 | [task:call:5740] server: ip:172.23.106.168 port:8091 ssh_username:root, nodes/self: {'memcached': 11210, 'memoryFree': 11730796544L, 'eventingMemoryQuota': 256, 'rest_password': '', 'storage': [], 'ports': [], 'cbasMemoryQuota': 1839, 'hostname': u'172.23.106.168:8091', 'ftsMemoryQuota': 512, 'clusterCompatibility': 458752, 'id': u'ns_1@172.23.106.168', 'mcdMemoryReserved': 9379, 'moxi': 11211, 'rest_username': '', 'os': u'x86_64-unknown-linux-gnu', 'ip': u'172.23.106.168', 'curr_items': 0, 'mcdMemoryAllocated': 9379, 'services': [u'kv'], 'version': u'7.0.0-5270-enterprise', 'indexMemoryQuota': 512, 'uptime': u'25', 'clusterMembership': u'active', 'storageTotalRam': 11724L, 'port': u'8091', 'availableStorage': [], 'memoryTotal': 12294176768L, 'memoryQuota': 6368, 'status': u'healthy'} 2021-06-08 06:28:29,183 | test | DEBUG | pool-5-thread-3 | [rest_client:set_service_mem_quota:859] Setting RAM quotas: {'memoryQuota': 8351} 2021-06-08 06:28:29,194 | test | DEBUG | pool-5-thread-3 | [rest_client:set_indexer_storage_mode:883] settings/indexes params: storageMode=plasma 2021-06-08 06:28:29,201 | test | DEBUG | MainThread | [rest_client:set_data_path:2138] /nodes/self/controller/settings: path=%2Fdata&index_path=%2Fdata 2021-06-08 06:28:29,204 | test | DEBUG | pool-5-thread-3 | [rest_client:init_cluster:770] settings/web params on 172.23.106.168:8091:password=password&port=8091&username=Administrator 2021-06-08 06:28:29,210 | test | DEBUG | MainThread | [rest_client:set_data_path:2141] Setting paths: path=%2Fdata&index_path=%2Fdata: status 1 2021-06-08 06:28:29,210 | infra | INFO | MainThread | [task_manager:add_new_task:23] Added new task: node_init_task_172.23.106.198_8091 2021-06-08 06:28:29,210 | infra | INFO | pool-5-thread-4 | [task:start_task:87] Thread 'node_init_task_172.23.106.198_8091' started 2021-06-08 06:28:29,210 | infra | DEBUG | MainThread | [task_manager:get_task_result:26] Getting task result for node_init_task_172.23.106.196_8091 2021-06-08 06:28:29,230 | infra | DEBUG | pool-5-thread-4 | [task:wait_until:131] Wait for expected condition to get satisfied 2021-06-08 06:28:29,230 | test | DEBUG | pool-5-thread-4 | [task:call:5740] server: ip:172.23.106.198 port:8091 ssh_username:root, nodes/self: {'memcached': 11210, 'memoryFree': 11734667264L, 'eventingMemoryQuota': 256, 'rest_password': '', 'storage': [], 'ports': [], 'cbasMemoryQuota': 1839, 'hostname': u'172.23.106.198:8091', 'ftsMemoryQuota': 512, 'clusterCompatibility': 458752, 'id': u'ns_1@172.23.106.198', 'mcdMemoryReserved': 9379, 'moxi': 11211, 'rest_username': '', 'os': u'x86_64-unknown-linux-gnu', 'ip': u'172.23.106.198', 'curr_items': 0, 'mcdMemoryAllocated': 9379, 'services': [u'kv'], 'version': u'7.0.0-5270-enterprise', 'indexMemoryQuota': 512, 'uptime': u'15', 'clusterMembership': u'active', 'storageTotalRam': 11724L, 'port': u'8091', 'availableStorage': [], 'memoryTotal': 12294176768L, 'memoryQuota': 6368, 'status': u'healthy'} 2021-06-08 06:28:29,232 | test | DEBUG | pool-5-thread-4 | [rest_client:set_service_mem_quota:859] Setting RAM quotas: {'memoryQuota': 8351} 2021-06-08 06:28:29,243 | infra | DEBUG | MainThread | [task_manager:get_task_result:26] Getting task result for node_init_task_172.23.106.199_8091 2021-06-08 06:28:29,244 | test | DEBUG | pool-5-thread-4 | [rest_client:set_indexer_storage_mode:883] settings/indexes params: storageMode=plasma 2021-06-08 06:28:29,250 | infra | DEBUG | MainThread | [task_manager:get_task_result:26] Getting task result for node_init_task_172.23.106.168_8091 2021-06-08 06:28:29,255 | test | DEBUG | pool-5-thread-4 | [rest_client:init_cluster:770] settings/web params on 172.23.106.198:8091:password=password&port=8091&username=Administrator 2021-06-08 06:28:29,290 | infra | DEBUG | MainThread | [task_manager:get_task_result:26] Getting task result for node_init_task_172.23.106.198_8091 2021-06-08 06:28:29,338 | test | WARNING | MainThread | [basetestcase:_initialize_nodes:598] RAM quota was defined less than 100 MB: 2021-06-08 06:28:29,338 | test | DEBUG | MainThread | [basetestcase:_initialize_nodes:601] Node: 172.23.106.196: RAM: ['MemTotal: 12006032 kB\n', 'MemFree: 10620340 kB\n', 'MemAvailable: 10954316 kB\n', 'Buffers: 0 kB\n', 'Cached: 843460 kB\n', 'SwapCached: 0 kB\n', 'Active: 671460 kB\n', 'Inactive: 434524 kB\n', 'Active(anon): 219496 kB\n', 'Inactive(anon): 351432 kB\n', 'Active(file): 451964 kB\n', 'Inactive(file): 83092 kB\n', 'Unevictable: 11792 kB\n', 'Mlocked: 11792 kB\n', 'SwapTotal: 3670012 kB\n', 'SwapFree: 3670012 kB\n', 'Dirty: 96 kB\n', 'Writeback: 0 kB\n', 'AnonPages: 274016 kB\n', 'Mapped: 90924 kB\n', 'Shmem: 304744 kB\n', 'Slab: 133460 kB\n', 'SReclaimable: 65936 kB\n', 'SUnreclaim: 67524 kB\n', 'KernelStack: 8576 kB\n', 'PageTables: 7672 kB\n', 'NFS_Unstable: 0 kB\n', 'Bounce: 0 kB\n', 'WritebackTmp: 0 kB\n', 'CommitLimit: 9673028 kB\n', 'Committed_AS: 1658720 kB\n', 'VmallocTotal: 34359738367 kB\n', 'VmallocUsed: 78304 kB\n', 'VmallocChunk: 34359612600 kB\n', 'HardwareCorrupted: 0 kB\n', 'AnonHugePages: 6144 kB\n', 'CmaTotal: 0 kB\n', 'CmaFree: 0 kB\n', 'HugePages_Total: 0\n', 'HugePages_Free: 0\n', 'HugePages_Rsvd: 0\n', 'HugePages_Surp: 0\n', 'Hugepagesize: 2048 kB\n', 'DirectMap4k: 135168 kB\n', 'DirectMap2M: 6043648 kB\n', 'DirectMap1G: 6291456 kB\n'] 2021-06-08 06:28:29,338 | test | DEBUG | MainThread | [basetestcase:_initialize_nodes:601] Node: 172.23.106.199: RAM: ['MemTotal: 12006032 kB\n', 'MemFree: 10265628 kB\n', 'MemAvailable: 10611020 kB\n', 'Buffers: 0 kB\n', 'Cached: 1192180 kB\n', 'SwapCached: 0 kB\n', 'Active: 708012 kB\n', 'Inactive: 731144 kB\n', 'Active(anon): 201240 kB\n', 'Inactive(anon): 699160 kB\n', 'Active(file): 506772 kB\n', 'Inactive(file): 31984 kB\n', 'Unevictable: 11868 kB\n', 'Mlocked: 11868 kB\n', 'SwapTotal: 3670012 kB\n', 'SwapFree: 3670012 kB\n', 'Dirty: 12 kB\n', 'Writeback: 0 kB\n', 'AnonPages: 257716 kB\n', 'Mapped: 84192 kB\n', 'Shmem: 649780 kB\n', 'Slab: 153924 kB\n', 'SReclaimable: 81364 kB\n', 'SUnreclaim: 72560 kB\n', 'KernelStack: 8560 kB\n', 'PageTables: 7528 kB\n', 'NFS_Unstable: 0 kB\n', 'Bounce: 0 kB\n', 'WritebackTmp: 0 kB\n', 'CommitLimit: 9673028 kB\n', 'Committed_AS: 2010844 kB\n', 'VmallocTotal: 34359738367 kB\n', 'VmallocUsed: 77280 kB\n', 'VmallocChunk: 34359553732 kB\n', 'HardwareCorrupted: 0 kB\n', 'AnonHugePages: 2048 kB\n', 'CmaTotal: 0 kB\n', 'CmaFree: 0 kB\n', 'HugePages_Total: 0\n', 'HugePages_Free: 0\n', 'HugePages_Rsvd: 0\n', 'HugePages_Surp: 0\n', 'Hugepagesize: 2048 kB\n', 'DirectMap4k: 178176 kB\n', 'DirectMap2M: 8097792 kB\n', 'DirectMap1G: 4194304 kB\n'] 2021-06-08 06:28:29,338 | test | DEBUG | MainThread | [basetestcase:_initialize_nodes:601] Node: 172.23.106.168: RAM: ['MemTotal: 12006032 kB\n', 'MemFree: 10586528 kB\n', 'MemAvailable: 10935652 kB\n', 'Buffers: 0 kB\n', 'Cached: 871728 kB\n', 'SwapCached: 0 kB\n', 'Active: 723348 kB\n', 'Inactive: 401736 kB\n', 'Active(anon): 211932 kB\n', 'Inactive(anon): 368016 kB\n', 'Active(file): 511416 kB\n', 'Inactive(file): 33720 kB\n', 'Unevictable: 12372 kB\n', 'Mlocked: 12372 kB\n', 'SwapTotal: 3670012 kB\n', 'SwapFree: 3670012 kB\n', 'Dirty: 200 kB\n', 'Writeback: 0 kB\n', 'AnonPages: 265316 kB\n', 'Mapped: 91108 kB\n', 'Shmem: 322928 kB\n', 'Slab: 147128 kB\n', 'SReclaimable: 76068 kB\n', 'SUnreclaim: 71060 kB\n', 'KernelStack: 8496 kB\n', 'PageTables: 7756 kB\n', 'NFS_Unstable: 0 kB\n', 'Bounce: 0 kB\n', 'WritebackTmp: 0 kB\n', 'CommitLimit: 9673028 kB\n', 'Committed_AS: 1681616 kB\n', 'VmallocTotal: 34359738367 kB\n', 'VmallocUsed: 78304 kB\n', 'VmallocChunk: 34359605960 kB\n', 'HardwareCorrupted: 0 kB\n', 'AnonHugePages: 6144 kB\n', 'CmaTotal: 0 kB\n', 'CmaFree: 0 kB\n', 'HugePages_Total: 0\n', 'HugePages_Free: 0\n', 'HugePages_Rsvd: 0\n', 'HugePages_Surp: 0\n', 'Hugepagesize: 2048 kB\n', 'DirectMap4k: 153600 kB\n', 'DirectMap2M: 6025216 kB\n', 'DirectMap1G: 6291456 kB\n'] 2021-06-08 06:28:29,338 | test | DEBUG | MainThread | [basetestcase:_initialize_nodes:601] Node: 172.23.106.198: RAM: ['MemTotal: 12006032 kB\n', 'MemFree: 10308336 kB\n', 'MemAvailable: 10656972 kB\n', 'Buffers: 0 kB\n', 'Cached: 1133648 kB\n', 'SwapCached: 0 kB\n', 'Active: 748044 kB\n', 'Inactive: 658540 kB\n', 'Active(anon): 233096 kB\n', 'Inactive(anon): 626672 kB\n', 'Active(file): 514948 kB\n', 'Inactive(file): 31868 kB\n', 'Unevictable: 11824 kB\n', 'Mlocked: 11824 kB\n', 'SwapTotal: 3670012 kB\n', 'SwapFree: 3670012 kB\n', 'Dirty: 352 kB\n', 'Writeback: 0 kB\n', 'AnonPages: 284480 kB\n', 'Mapped: 90136 kB\n', 'Shmem: 583192 kB\n', 'Slab: 141472 kB\n', 'SReclaimable: 71732 kB\n', 'SUnreclaim: 69740 kB\n', 'KernelStack: 8528 kB\n', 'PageTables: 8456 kB\n', 'NFS_Unstable: 0 kB\n', 'Bounce: 0 kB\n', 'WritebackTmp: 0 kB\n', 'CommitLimit: 9673028 kB\n', 'Committed_AS: 1979436 kB\n', 'VmallocTotal: 34359738367 kB\n', 'VmallocUsed: 77280 kB\n', 'VmallocChunk: 34359637100 kB\n', 'HardwareCorrupted: 0 kB\n', 'AnonHugePages: 4096 kB\n', 'CmaTotal: 0 kB\n', 'CmaFree: 0 kB\n', 'HugePages_Total: 0\n', 'HugePages_Free: 0\n', 'HugePages_Rsvd: 0\n', 'HugePages_Surp: 0\n', 'Hugepagesize: 2048 kB\n', 'DirectMap4k: 149504 kB\n', 'DirectMap2M: 4980736 kB\n', 'DirectMap1G: 7340032 kB\n'] 2021-06-08 06:28:29,338 | infra | DEBUG | MainThread | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.196 2021-06-08 06:28:29,339 | infra | DEBUG | MainThread | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.199 2021-06-08 06:28:29,339 | infra | DEBUG | MainThread | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.168 2021-06-08 06:28:29,339 | infra | DEBUG | MainThread | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.198 2021-06-08 06:28:29,342 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.196 with username: root, password: couchbase 2021-06-08 06:28:29,426 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 06:28:29,631 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: ls /etc/ 2021-06-08 06:28:29,680 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 06:28:29,681 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: ls /cygdrive/c/Windows 2021-06-08 06:28:29,693 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: uname -m 2021-06-08 06:28:29,704 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: cat /proc/cpuinfo 2021-06-08 06:28:29,716 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: df -Thl 2021-06-08 06:28:29,730 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: cat /proc/meminfo 2021-06-08 06:28:29,743 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: hostname 2021-06-08 06:28:29,753 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: hostname -d 2021-06-08 06:28:29,765 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: grep ^COUCHBASE_NUM_VBUCKETS /opt/couchbase/bin/couchbase-server | cut -d "=" -f 2 2021-06-08 06:28:29,776 | test | DEBUG | MainThread | [cluster_ready_functions:change_env_variables:301] Current vBuckets on node ip:172.23.106.196 port:8091 ssh_username:root: [] 2021-06-08 06:28:29,776 | infra | DEBUG | MainThread | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.196 2021-06-08 06:28:29,778 | test | DEBUG | MainThread | [cluster_ready_functions:change_env_variables:318] ========= CHANGED ENVIRONMENT SETTING =========== 2021-06-08 06:28:29,779 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.199 with username: root, password: couchbase 2021-06-08 06:28:29,858 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 06:28:30,059 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: ls /etc/ 2021-06-08 06:28:30,111 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 06:28:30,112 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: ls /cygdrive/c/Windows 2021-06-08 06:28:30,125 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: uname -m 2021-06-08 06:28:30,137 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: cat /proc/cpuinfo 2021-06-08 06:28:30,148 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: df -Thl 2021-06-08 06:28:30,163 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: cat /proc/meminfo 2021-06-08 06:28:30,174 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: hostname 2021-06-08 06:28:30,187 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: hostname -d 2021-06-08 06:28:30,198 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: grep ^COUCHBASE_NUM_VBUCKETS /opt/couchbase/bin/couchbase-server | cut -d "=" -f 2 2021-06-08 06:28:30,210 | test | DEBUG | MainThread | [cluster_ready_functions:change_env_variables:301] Current vBuckets on node ip:172.23.106.199 port:8091 ssh_username:root: [] 2021-06-08 06:28:30,210 | infra | DEBUG | MainThread | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.199 2021-06-08 06:28:30,211 | test | DEBUG | MainThread | [cluster_ready_functions:change_env_variables:318] ========= CHANGED ENVIRONMENT SETTING =========== 2021-06-08 06:28:30,213 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.168 with username: root, password: couchbase 2021-06-08 06:28:30,285 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 06:28:30,489 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: ls /etc/ 2021-06-08 06:28:30,542 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 06:28:30,542 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: ls /cygdrive/c/Windows 2021-06-08 06:28:30,555 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: uname -m 2021-06-08 06:28:30,566 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: cat /proc/cpuinfo 2021-06-08 06:28:30,578 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: df -Thl 2021-06-08 06:28:30,592 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: cat /proc/meminfo 2021-06-08 06:28:30,605 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: hostname 2021-06-08 06:28:30,617 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: hostname -d 2021-06-08 06:28:30,630 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: grep ^COUCHBASE_NUM_VBUCKETS /opt/couchbase/bin/couchbase-server | cut -d "=" -f 2 2021-06-08 06:28:30,642 | test | DEBUG | MainThread | [cluster_ready_functions:change_env_variables:301] Current vBuckets on node ip:172.23.106.168 port:8091 ssh_username:root: [] 2021-06-08 06:28:30,642 | infra | DEBUG | MainThread | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.168 2021-06-08 06:28:30,642 | test | DEBUG | MainThread | [cluster_ready_functions:change_env_variables:318] ========= CHANGED ENVIRONMENT SETTING =========== 2021-06-08 06:28:30,644 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.198 with username: root, password: couchbase 2021-06-08 06:28:30,723 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 06:28:30,924 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: ls /etc/ 2021-06-08 06:28:30,977 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 06:28:30,977 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: ls /cygdrive/c/Windows 2021-06-08 06:28:30,990 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: uname -m 2021-06-08 06:28:31,002 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: cat /proc/cpuinfo 2021-06-08 06:28:31,013 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: df -Thl 2021-06-08 06:28:31,028 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: cat /proc/meminfo 2021-06-08 06:28:31,039 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: hostname 2021-06-08 06:28:31,049 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: hostname -d 2021-06-08 06:28:31,062 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: grep ^COUCHBASE_NUM_VBUCKETS /opt/couchbase/bin/couchbase-server | cut -d "=" -f 2 2021-06-08 06:28:31,075 | test | DEBUG | MainThread | [cluster_ready_functions:change_env_variables:301] Current vBuckets on node ip:172.23.106.198 port:8091 ssh_username:root: [] 2021-06-08 06:28:31,075 | infra | DEBUG | MainThread | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.198 2021-06-08 06:28:31,075 | test | DEBUG | MainThread | [cluster_ready_functions:change_env_variables:318] ========= CHANGED ENVIRONMENT SETTING =========== 2021-06-08 06:28:31,075 | test | DEBUG | MainThread | [cluster_ready_functions:change_env_variables:320] Wait for all the services to come up after change_env_vars update 2021-06-08 06:28:31,081 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from couchbase_utils/cluster_utils/cluster_ready_functions.py -> change_env_variables():L322 2021-06-08 06:28:31,082 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-06-08 06:28:41,082 | test | INFO | MainThread | [basetestcase:setUp:342] Couchbase Cluster: C1, Nodes: 172.23.106.196, 172.23.106.199, 172.23.106.168, 172.23.106.198 initialized 2021-06-08 06:28:41,084 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.196 with username: root, password: couchbase 2021-06-08 06:28:41,161 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 06:28:41,359 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: ls /etc/ 2021-06-08 06:28:41,414 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 06:28:41,414 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: ls /cygdrive/c/Windows 2021-06-08 06:28:41,426 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: uname -m 2021-06-08 06:28:41,437 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: cat /proc/cpuinfo 2021-06-08 06:28:41,450 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: df -Thl 2021-06-08 06:28:41,464 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: cat /proc/meminfo 2021-06-08 06:28:41,476 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: hostname 2021-06-08 06:28:41,487 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: hostname -d 2021-06-08 06:28:41,500 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: echo 'y' | /opt/couchbase/bin/couchbase-cli enable-developer-preview --enable -c 172.23.106.196:8091 -u Administrator -p password 2021-06-08 06:28:41,742 | infra | DEBUG | MainThread | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.196 2021-06-08 06:28:41,743 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.199 with username: root, password: couchbase 2021-06-08 06:28:41,813 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 06:28:42,019 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: ls /etc/ 2021-06-08 06:28:42,072 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 06:28:42,072 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: ls /cygdrive/c/Windows 2021-06-08 06:28:42,084 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: uname -m 2021-06-08 06:28:42,095 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: cat /proc/cpuinfo 2021-06-08 06:28:42,107 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: df -Thl 2021-06-08 06:28:42,121 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: cat /proc/meminfo 2021-06-08 06:28:42,134 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: hostname 2021-06-08 06:28:42,144 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: hostname -d 2021-06-08 06:28:42,157 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: echo 'y' | /opt/couchbase/bin/couchbase-cli enable-developer-preview --enable -c 172.23.106.199:8091 -u Administrator -p password 2021-06-08 06:28:42,357 | infra | DEBUG | MainThread | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.199 2021-06-08 06:28:42,358 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.168 with username: root, password: couchbase 2021-06-08 06:28:42,440 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 06:28:42,644 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: ls /etc/ 2021-06-08 06:28:42,697 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 06:28:42,697 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: ls /cygdrive/c/Windows 2021-06-08 06:28:42,710 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: uname -m 2021-06-08 06:28:42,721 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: cat /proc/cpuinfo 2021-06-08 06:28:42,733 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: df -Thl 2021-06-08 06:28:42,747 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: cat /proc/meminfo 2021-06-08 06:28:42,759 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: hostname 2021-06-08 06:28:42,772 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: hostname -d 2021-06-08 06:28:42,783 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: echo 'y' | /opt/couchbase/bin/couchbase-cli enable-developer-preview --enable -c 172.23.106.168:8091 -u Administrator -p password 2021-06-08 06:28:43,042 | infra | DEBUG | MainThread | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.168 2021-06-08 06:28:43,045 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.198 with username: root, password: couchbase 2021-06-08 06:28:43,121 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 06:28:43,323 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: ls /etc/ 2021-06-08 06:28:43,378 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 06:28:43,378 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: ls /cygdrive/c/Windows 2021-06-08 06:28:43,391 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: uname -m 2021-06-08 06:28:43,404 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: cat /proc/cpuinfo 2021-06-08 06:28:43,417 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: df -Thl 2021-06-08 06:28:43,433 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: cat /proc/meminfo 2021-06-08 06:28:43,444 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: hostname 2021-06-08 06:28:43,457 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: hostname -d 2021-06-08 06:28:43,469 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: echo 'y' | /opt/couchbase/bin/couchbase-cli enable-developer-preview --enable -c 172.23.106.198:8091 -u Administrator -p password 2021-06-08 06:28:43,665 | infra | DEBUG | MainThread | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.198 2021-06-08 06:28:43,676 | test | INFO | MainThread | [basetestcase:log_setup_status:546] ========= BaseTestCase setup finished for test #3 test_disk_full_reduce_replica ========= 2021-06-08 06:28:43,709 | test | DEBUG | MainThread | [bucket_ready_functions:add_rbac_user:3934] **** Add built-in 'cbadminbucket' user to node 172.23.106.196 **** 2021-06-08 06:28:43,786 | test | DEBUG | MainThread | [bucket_ready_functions:add_rbac_user:3938] **** Add 'admin' role to 'cbadminbucket' user **** 2021-06-08 06:28:43,805 | test | DEBUG | MainThread | [rest_client:init_cluster:770] settings/web params on 172.23.106.196:8091:password=password&port=8091&username=Administrator 2021-06-08 06:28:43,878 | infra | INFO | MainThread | [task_manager:add_new_task:23] Added new task: Rebalance_task_IN=[172.23.106.199,172.23.106.168,172.23.106.198]_OUT=[]_1623158923.86 2021-06-08 06:28:43,878 | infra | DEBUG | MainThread | [task_manager:get_task_result:26] Getting task result for Rebalance_task_IN=[172.23.106.199,172.23.106.168,172.23.106.198]_OUT=[]_1623158923.86 2021-06-08 06:28:43,878 | infra | INFO | pool-5-thread-5 | [task:start_task:87] Thread 'Rebalance_task_IN=[172.23.106.199,172.23.106.168,172.23.106.198]_OUT=[]_1623158923.86' started 2021-06-08 06:28:43,880 | test | DEBUG | pool-5-thread-5 | [rest_client:add_node:1089] Adding remote node 172.23.106.199:8091 to cluster 172.23.106.196:8091 2021-06-08 06:28:46,924 | test | DEBUG | pool-5-thread-5 | [rest_client:add_node:1089] Adding remote node 172.23.106.168:8091 to cluster 172.23.106.196:8091 2021-06-08 06:28:50,086 | test | DEBUG | pool-5-thread-5 | [rest_client:add_node:1089] Adding remote node 172.23.106.198:8091 to cluster 172.23.106.196:8091 2021-06-08 06:28:52,992 | test | DEBUG | pool-5-thread-5 | [rest_client:rebalance:1339] Rebalance params: {'password': 'password', 'knownNodes': u'ns_1@172.23.106.168,ns_1@172.23.106.198,ns_1@172.23.106.196,ns_1@172.23.106.199', 'ejectedNodes': '', 'user': 'Administrator'} 2021-06-08 06:28:53,046 | test | DEBUG | pool-5-thread-5 | [rest_client:rebalance:1344] Rebalance operation started 2021-06-08 06:28:53,048 | test | INFO | pool-5-thread-5 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.106.196 | kv | 7.0.0-5270-enterprise | 1.12937633329 | Cluster node | | 172.23.106.199 | ['kv'] | | | <--- IN --- | | 172.23.106.168 | ['kv'] | | | <--- IN --- | | 172.23.106.198 | ['kv'] | | | <--- IN --- | +----------------+----------+-----------------------+---------------+--------------+ 2021-06-08 06:28:53,059 | test | DEBUG | pool-5-thread-5 | [rest_client:_rebalance_status_and_progress:1553] Rebalance percentage: 0.00 % 2021-06-08 06:28:53,059 | test | INFO | pool-5-thread-5 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-06-08 06:28:53,071 | infra | DEBUG | pool-5-thread-5 | [task:check:382] Wait before next rebalance progress check 2021-06-08 06:28:53,072 | infra | DEBUG | pool-5-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L383 2021-06-08 06:28:53,073 | infra | DEBUG | pool-5-thread-5 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:28:58,085 | infra | DEBUG | pool-5-thread-5 | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _rebalance_status_and_progress():L1557 2021-06-08 06:28:58,085 | infra | DEBUG | pool-5-thread-5 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:29:03,094 | test | INFO | pool-5-thread-5 | [task:check:355] Rebalance - status: none, progress: 100 2021-06-08 06:29:03,107 | test | INFO | pool-5-thread-5 | [task:check:414] Rebalance completed with progress: 100% in 10.0600001812 sec 2021-06-08 06:29:03,107 | infra | INFO | pool-5-thread-5 | [task:complete_task:102] Thread 'Rebalance_task_IN=[172.23.106.199,172.23.106.168,172.23.106.198]_OUT=[]_1623158923.86' completed 2021-06-08 06:29:03,125 | test | DEBUG | MainThread | [bucket_ready_functions:async_create_bucket:1462] Creating bucket: default 2021-06-08 06:29:03,125 | infra | INFO | MainThread | [task_manager:add_new_task:23] Added new task: bucket_default_create_task 2021-06-08 06:29:03,125 | infra | DEBUG | MainThread | [task_manager:get_task_result:26] Getting task result for bucket_default_create_task 2021-06-08 06:29:03,177 | infra | INFO | pool-5-thread-6 | [BucketOperations_Rest:create_bucket:342] Creating 'couchbase' bucket default 2021-06-08 06:29:03,177 | infra | DEBUG | pool-5-thread-6 | [BucketOperations_Rest:create_bucket:344] http://172.23.106.196:8091/pools/default/buckets with param: storageBackend=magma&viewFragmentationThreshold%255Bpercentage%255D=50&fragmentationPercentage=50&replicaIndex=1&maxTTL=0&indexCompactionMode=circular&databaseFragmentationThreshold%255Bpercentage%255D=50&flushEnabled=0¶llelDBAndViewCompaction=false&durabilityMinLevel=none&compressionMode=off&purgeInterval=3&bucketType=couchbase&autoCompactionDefined=false&conflictResolutionType=seqno&name=default&replicaNumber=3&ramQuotaMB=8350&threadsNumber=3&evictionPolicy=fullEviction 2021-06-08 06:29:03,210 | infra | DEBUG | pool-5-thread-6 | [BucketOperations_Rest:create_bucket:368] 0.03 seconds to create bucket default 2021-06-08 06:29:03,210 | infra | INFO | pool-5-thread-6 | [task:complete_task:102] Thread 'bucket_default_create_task' completed 2021-06-08 06:29:03,210 | test | DEBUG | MainThread | [bucket_ready_functions:create_bucket:1474] Wait for memcached to accept cbstats or any other bucket request connections 2021-06-08 06:29:03,217 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> create_bucket():L1476 2021-06-08 06:29:03,217 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-06-08 06:29:05,267 | test | DEBUG | MainThread | [cluster_ready_functions:get_nodes_from_services_map:597] All nodes in cluster: [ip:172.23.106.168 port:8091 ssh_username:root, ip:172.23.106.198 port:8091 ssh_username:root, ip:172.23.106.196 port:8091 ssh_username:root, ip:172.23.106.199 port:8091 ssh_username:root] 2021-06-08 06:29:05,267 | test | DEBUG | MainThread | [bucket_ready_functions:_wait_warmup_completed:3884] Waiting for bucket default to complete warm-up 2021-06-08 06:29:05,269 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.196 with username: root, password: couchbase 2021-06-08 06:29:05,354 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 06:29:05,582 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: ls /etc/ 2021-06-08 06:29:05,654 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 06:29:05,654 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: ls /cygdrive/c/Windows 2021-06-08 06:29:05,667 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: uname -m 2021-06-08 06:29:05,684 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: cat /proc/cpuinfo 2021-06-08 06:29:05,698 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: df -Thl 2021-06-08 06:29:05,717 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: cat /proc/meminfo 2021-06-08 06:29:05,732 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: hostname 2021-06-08 06:29:05,746 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: hostname -d 2021-06-08 06:29:05,760 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:29:05,966 | infra | DEBUG | MainThread | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.196 2021-06-08 06:29:05,967 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.199 with username: root, password: couchbase 2021-06-08 06:29:06,053 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 06:29:06,262 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: ls /etc/ 2021-06-08 06:29:06,315 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 06:29:06,315 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: ls /cygdrive/c/Windows 2021-06-08 06:29:06,328 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: uname -m 2021-06-08 06:29:06,339 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: cat /proc/cpuinfo 2021-06-08 06:29:06,351 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: df -Thl 2021-06-08 06:29:06,368 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: cat /proc/meminfo 2021-06-08 06:29:06,380 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: hostname 2021-06-08 06:29:06,391 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: hostname -d 2021-06-08 06:29:06,404 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:29:06,592 | infra | DEBUG | MainThread | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.199 2021-06-08 06:29:06,594 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.168 with username: root, password: couchbase 2021-06-08 06:29:06,663 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 06:29:06,871 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: ls /etc/ 2021-06-08 06:29:06,924 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 06:29:06,926 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: ls /cygdrive/c/Windows 2021-06-08 06:29:06,938 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: uname -m 2021-06-08 06:29:06,950 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: cat /proc/cpuinfo 2021-06-08 06:29:06,961 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: df -Thl 2021-06-08 06:29:06,976 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: cat /proc/meminfo 2021-06-08 06:29:06,989 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: hostname 2021-06-08 06:29:07,000 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: hostname -d 2021-06-08 06:29:07,013 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:29:07,200 | infra | DEBUG | MainThread | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.168 2021-06-08 06:29:07,203 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.198 with username: root, password: couchbase 2021-06-08 06:29:07,276 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 06:29:07,483 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: ls /etc/ 2021-06-08 06:29:07,538 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 06:29:07,538 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: ls /cygdrive/c/Windows 2021-06-08 06:29:07,551 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: uname -m 2021-06-08 06:29:07,562 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: cat /proc/cpuinfo 2021-06-08 06:29:07,572 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: df -Thl 2021-06-08 06:29:07,588 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: cat /proc/meminfo 2021-06-08 06:29:07,599 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: hostname 2021-06-08 06:29:07,611 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: hostname -d 2021-06-08 06:29:07,622 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:29:07,815 | infra | DEBUG | MainThread | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.198 2021-06-08 06:29:07,815 | test | INFO | MainThread | [magma_base:setUp:120] Scopes list is ['_default'] 2021-06-08 06:29:07,815 | test | DEBUG | MainThread | [magma_base:setUp:135] Collections list == ['_default'] 2021-06-08 06:29:07,818 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.196 with username: root, password: couchbase 2021-06-08 06:29:07,892 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 06:29:08,094 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: ls /etc/ 2021-06-08 06:29:08,148 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 06:29:08,150 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: ls /cygdrive/c/Windows 2021-06-08 06:29:08,161 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: uname -m 2021-06-08 06:29:08,174 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: cat /proc/cpuinfo 2021-06-08 06:29:08,184 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: df -Thl 2021-06-08 06:29:08,200 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: cat /proc/meminfo 2021-06-08 06:29:08,211 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: hostname 2021-06-08 06:29:08,223 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: hostname -d 2021-06-08 06:29:08,237 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: du -cm /data/default/magma.*/kv* | tail -1 | awk '{print $1}' 2021-06-08 06:29:08,404 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: du -cm /data/default/magma.*/wal | tail -1 | awk '{print $1}' 2021-06-08 06:29:08,426 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: du -cm /data/default/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2021-06-08 06:29:08,500 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: du -cm /data/default/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2021-06-08 06:29:08,576 | infra | DEBUG | MainThread | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.196 2021-06-08 06:29:08,578 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.199 with username: root, password: couchbase 2021-06-08 06:29:08,647 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 06:29:08,846 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: ls /etc/ 2021-06-08 06:29:08,900 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 06:29:08,900 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: ls /cygdrive/c/Windows 2021-06-08 06:29:08,913 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: uname -m 2021-06-08 06:29:08,924 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: cat /proc/cpuinfo 2021-06-08 06:29:08,936 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: df -Thl 2021-06-08 06:29:08,950 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: cat /proc/meminfo 2021-06-08 06:29:08,963 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: hostname 2021-06-08 06:29:08,974 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: hostname -d 2021-06-08 06:29:08,987 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: du -cm /data/default/magma.*/kv* | tail -1 | awk '{print $1}' 2021-06-08 06:29:09,138 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: du -cm /data/default/magma.*/wal | tail -1 | awk '{print $1}' 2021-06-08 06:29:09,153 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: du -cm /data/default/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2021-06-08 06:29:09,224 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: du -cm /data/default/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2021-06-08 06:29:09,301 | infra | DEBUG | MainThread | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.199 2021-06-08 06:29:09,302 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.168 with username: root, password: couchbase 2021-06-08 06:29:09,371 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 06:29:09,578 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: ls /etc/ 2021-06-08 06:29:09,631 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 06:29:09,631 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: ls /cygdrive/c/Windows 2021-06-08 06:29:09,644 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: uname -m 2021-06-08 06:29:09,654 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: cat /proc/cpuinfo 2021-06-08 06:29:09,667 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: df -Thl 2021-06-08 06:29:09,681 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: cat /proc/meminfo 2021-06-08 06:29:09,694 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: hostname 2021-06-08 06:29:09,704 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: hostname -d 2021-06-08 06:29:09,717 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: du -cm /data/default/magma.*/kv* | tail -1 | awk '{print $1}' 2021-06-08 06:29:09,871 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: du -cm /data/default/magma.*/wal | tail -1 | awk '{print $1}' 2021-06-08 06:29:09,885 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: du -cm /data/default/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2021-06-08 06:29:09,960 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: du -cm /data/default/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2021-06-08 06:29:10,035 | infra | DEBUG | MainThread | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.168 2021-06-08 06:29:10,036 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.198 with username: root, password: couchbase 2021-06-08 06:29:10,111 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 06:29:10,313 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: ls /etc/ 2021-06-08 06:29:10,367 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 06:29:10,367 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: ls /cygdrive/c/Windows 2021-06-08 06:29:10,380 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: uname -m 2021-06-08 06:29:10,391 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: cat /proc/cpuinfo 2021-06-08 06:29:10,404 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: df -Thl 2021-06-08 06:29:10,418 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: cat /proc/meminfo 2021-06-08 06:29:10,430 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: hostname 2021-06-08 06:29:10,441 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: hostname -d 2021-06-08 06:29:10,453 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: du -cm /data/default/magma.*/kv* | tail -1 | awk '{print $1}' 2021-06-08 06:29:10,609 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: du -cm /data/default/magma.*/wal | tail -1 | awk '{print $1}' 2021-06-08 06:29:10,625 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: du -cm /data/default/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2021-06-08 06:29:10,703 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: du -cm /data/default/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2021-06-08 06:29:10,786 | infra | DEBUG | MainThread | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.198 2021-06-08 06:29:10,786 | test | INFO | MainThread | [magma_base:get_disk_usage:815] Disk usage stats for bucekt default is below 2021-06-08 06:29:10,786 | test | INFO | MainThread | [magma_base:get_disk_usage:816] Total Disk usage for kvstore is 48MB 2021-06-08 06:29:10,944 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/magma/magma_base.py -> get_bucket_dgm():L771 2021-06-08 06:29:10,944 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: 'vb_active_resident_items_ratio' 2021-06-08 06:29:16,094 | test | INFO | MainThread | [magma_base:get_bucket_dgm:767] Active Resident Threshold of default is 100 2021-06-08 06:29:16,094 | test | DEBUG | MainThread | [magma_base:get_disk_usage:818] Total Disk usage for wal is 32MB 2021-06-08 06:29:16,095 | test | DEBUG | MainThread | [magma_base:get_disk_usage:819] Total Disk usage for keyTree is 16MB 2021-06-08 06:29:16,095 | test | DEBUG | MainThread | [magma_base:get_disk_usage:820] Total Disk usage for seqTree is 16MB 2021-06-08 06:29:16,095 | test | INFO | MainThread | [magma_base:setUp:178] Empty magma bucket disk usage: 48 2021-06-08 06:29:16,095 | test | INFO | MainThread | [magma_base:setUp:243] Creating SDK clients for client_pool 2021-06-08 06:29:16,095 | test | DEBUG | MainThread | [sdk_client3:_SDKClient__create_conn:237] Creating SDK connection for 'default' 2021-06-08 06:29:16,448 | test | DEBUG | MainThread | [sdk_client3:_SDKClient__create_conn:237] Creating SDK connection for 'default' 2021-06-08 06:29:16,779 | test | DEBUG | MainThread | [sdk_client3:_SDKClient__create_conn:237] Creating SDK connection for 'default' 2021-06-08 06:29:17,118 | test | DEBUG | MainThread | [sdk_client3:_SDKClient__create_conn:237] Creating SDK connection for 'default' 2021-06-08 06:29:17,459 | test | DEBUG | MainThread | [sdk_client3:_SDKClient__create_conn:237] Creating SDK connection for 'default' 2021-06-08 06:29:17,888 | test | DEBUG | MainThread | [sdk_client3:_SDKClient__create_conn:237] Creating SDK connection for 'default' 2021-06-08 06:29:18,259 | test | DEBUG | MainThread | [sdk_client3:_SDKClient__create_conn:237] Creating SDK connection for 'default' 2021-06-08 06:29:18,588 | test | DEBUG | MainThread | [sdk_client3:_SDKClient__create_conn:237] Creating SDK connection for 'default' 2021-06-08 06:29:18,903 | test | DEBUG | MainThread | [sdk_client3:_SDKClient__create_conn:237] Creating SDK connection for 'default' 2021-06-08 06:29:19,259 | test | DEBUG | MainThread | [sdk_client3:_SDKClient__create_conn:237] Creating SDK connection for 'default' 2021-06-08 06:29:19,618 | test | DEBUG | MainThread | [sdk_client3:_SDKClient__create_conn:237] Creating SDK connection for 'default' 2021-06-08 06:29:19,947 | test | DEBUG | MainThread | [sdk_client3:_SDKClient__create_conn:237] Creating SDK connection for 'default' 2021-06-08 06:29:20,267 | test | DEBUG | MainThread | [sdk_client3:_SDKClient__create_conn:237] Creating SDK connection for 'default' 2021-06-08 06:29:20,709 | test | DEBUG | MainThread | [sdk_client3:_SDKClient__create_conn:237] Creating SDK connection for 'default' 2021-06-08 06:29:21,038 | test | DEBUG | MainThread | [sdk_client3:_SDKClient__create_conn:237] Creating SDK connection for 'default' 2021-06-08 06:29:21,469 | test | DEBUG | MainThread | [sdk_client3:_SDKClient__create_conn:237] Creating SDK connection for 'default' 2021-06-08 06:29:21,898 | test | DEBUG | MainThread | [sdk_client3:_SDKClient__create_conn:237] Creating SDK connection for 'default' 2021-06-08 06:29:22,229 | test | DEBUG | MainThread | [sdk_client3:_SDKClient__create_conn:237] Creating SDK connection for 'default' 2021-06-08 06:29:22,737 | test | DEBUG | MainThread | [sdk_client3:_SDKClient__create_conn:237] Creating SDK connection for 'default' 2021-06-08 06:29:23,069 | test | DEBUG | MainThread | [sdk_client3:_SDKClient__create_conn:237] Creating SDK connection for 'default' 2021-06-08 06:29:23,510 | test | DEBUG | MainThread | [magma_base:initial_load:350] initial_items_in_each_collection 5000000 2021-06-08 06:29:23,515 | infra | DEBUG | MainThread | [cluster:async_load_gen_docs_from_spec:105] Initializing mutation task for given spec 2021-06-08 06:29:23,515 | infra | INFO | MainThread | [task_manager:add_new_task:23] Added new task: MutateDocsFromSpecTask_1623158963.52 2021-06-08 06:29:23,515 | infra | DEBUG | MainThread | [task_manager:get_task_result:26] Getting task result for MutateDocsFromSpecTask_1623158963.52 2021-06-08 06:29:23,515 | infra | INFO | pool-5-thread-7 | [task:start_task:87] Thread 'MutateDocsFromSpecTask_1623158963.52' started 2021-06-08 06:29:23,542 | infra | INFO | pool-5-thread-7 | [task_manager:add_new_task:23] Added new task: PrintBucketStats_default_1623158963.53 2021-06-08 06:29:23,542 | infra | INFO | pool-5-thread-8 | [task:start_task:87] Thread 'PrintBucketStats_default_1623158963.53' started 2021-06-08 06:29:23,542 | infra | INFO | pool-5-thread-7 | [task_manager:add_new_task:23] Added new task: LoadDocs_MutateDocsFromSpecTask_1623158963.52_default__default__default_ttl=0_create__0_250000_1623158963.53 2021-06-08 06:29:23,542 | infra | INFO | pool-5-thread-9 | [task:start_task:87] Thread 'LoadDocs_MutateDocsFromSpecTask_1623158963.52_default__default__default_ttl=0_create__0_250000_1623158963.53' started 2021-06-08 06:29:23,542 | infra | INFO | pool-5-thread-7 | [task_manager:add_new_task:23] Added new task: LoadDocs_MutateDocsFromSpecTask_1623158963.52_default__default__default_ttl=0_create__250000_500000_1623158963.53 2021-06-08 06:29:23,542 | infra | INFO | pool-5-thread-10 | [task:start_task:87] Thread 'LoadDocs_MutateDocsFromSpecTask_1623158963.52_default__default__default_ttl=0_create__250000_500000_1623158963.53' started 2021-06-08 06:29:23,542 | infra | INFO | pool-5-thread-7 | [task_manager:add_new_task:23] Added new task: LoadDocs_MutateDocsFromSpecTask_1623158963.52_default__default__default_ttl=0_create__500000_750000_1623158963.53 2021-06-08 06:29:23,543 | infra | INFO | pool-5-thread-7 | [task_manager:add_new_task:23] Added new task: LoadDocs_MutateDocsFromSpecTask_1623158963.52_default__default__default_ttl=0_create__750000_1000000_1623158963.53 2021-06-08 06:29:23,543 | infra | INFO | pool-5-thread-7 | [task_manager:add_new_task:23] Added new task: LoadDocs_MutateDocsFromSpecTask_1623158963.52_default__default__default_ttl=0_create__1000000_1250000_1623158963.53 2021-06-08 06:29:23,543 | infra | INFO | pool-5-thread-12 | [task:start_task:87] Thread 'LoadDocs_MutateDocsFromSpecTask_1623158963.52_default__default__default_ttl=0_create__750000_1000000_1623158963.53' started 2021-06-08 06:29:23,545 | infra | INFO | pool-5-thread-13 | [task:start_task:87] Thread 'LoadDocs_MutateDocsFromSpecTask_1623158963.52_default__default__default_ttl=0_create__1000000_1250000_1623158963.53' started 2021-06-08 06:29:23,545 | infra | INFO | pool-5-thread-7 | [task_manager:add_new_task:23] Added new task: LoadDocs_MutateDocsFromSpecTask_1623158963.52_default__default__default_ttl=0_create__1250000_1500000_1623158963.53 2021-06-08 06:29:23,546 | infra | INFO | pool-5-thread-14 | [task:start_task:87] Thread 'LoadDocs_MutateDocsFromSpecTask_1623158963.52_default__default__default_ttl=0_create__1250000_1500000_1623158963.53' started 2021-06-08 06:29:23,543 | infra | INFO | pool-5-thread-11 | [task:start_task:87] Thread 'LoadDocs_MutateDocsFromSpecTask_1623158963.52_default__default__default_ttl=0_create__500000_750000_1623158963.53' started 2021-06-08 06:29:23,546 | infra | INFO | pool-5-thread-7 | [task_manager:add_new_task:23] Added new task: LoadDocs_MutateDocsFromSpecTask_1623158963.52_default__default__default_ttl=0_create__1500000_1750000_1623158963.53 2021-06-08 06:29:23,546 | infra | INFO | pool-5-thread-15 | [task:start_task:87] Thread 'LoadDocs_MutateDocsFromSpecTask_1623158963.52_default__default__default_ttl=0_create__1500000_1750000_1623158963.53' started 2021-06-08 06:29:23,548 | infra | INFO | pool-5-thread-7 | [task_manager:add_new_task:23] Added new task: LoadDocs_MutateDocsFromSpecTask_1623158963.52_default__default__default_ttl=0_create__1750000_2000000_1623158963.53 2021-06-08 06:29:23,549 | infra | INFO | pool-5-thread-16 | [task:start_task:87] Thread 'LoadDocs_MutateDocsFromSpecTask_1623158963.52_default__default__default_ttl=0_create__1750000_2000000_1623158963.53' started 2021-06-08 06:29:23,549 | infra | INFO | pool-5-thread-17 | [task:start_task:87] Thread 'LoadDocs_MutateDocsFromSpecTask_1623158963.52_default__default__default_ttl=0_create__2000000_2250000_1623158963.53' started 2021-06-08 06:29:23,549 | infra | INFO | pool-5-thread-7 | [task_manager:add_new_task:23] Added new task: LoadDocs_MutateDocsFromSpecTask_1623158963.52_default__default__default_ttl=0_create__2000000_2250000_1623158963.53 2021-06-08 06:29:23,551 | infra | INFO | pool-5-thread-7 | [task_manager:add_new_task:23] Added new task: LoadDocs_MutateDocsFromSpecTask_1623158963.52_default__default__default_ttl=0_create__2250000_2500000_1623158963.53 2021-06-08 06:29:23,551 | infra | INFO | pool-5-thread-18 | [task:start_task:87] Thread 'LoadDocs_MutateDocsFromSpecTask_1623158963.52_default__default__default_ttl=0_create__2250000_2500000_1623158963.53' started 2021-06-08 06:29:23,552 | infra | INFO | pool-5-thread-7 | [task_manager:add_new_task:23] Added new task: LoadDocs_MutateDocsFromSpecTask_1623158963.52_default__default__default_ttl=0_create__2500000_2750000_1623158963.53 2021-06-08 06:29:23,553 | infra | INFO | pool-5-thread-7 | [task_manager:add_new_task:23] Added new task: LoadDocs_MutateDocsFromSpecTask_1623158963.52_default__default__default_ttl=0_create__2750000_3000000_1623158963.53 2021-06-08 06:29:23,553 | infra | INFO | pool-5-thread-20 | [task:start_task:87] Thread 'LoadDocs_MutateDocsFromSpecTask_1623158963.52_default__default__default_ttl=0_create__2750000_3000000_1623158963.53' started 2021-06-08 06:29:23,553 | infra | INFO | pool-5-thread-21 | [task:start_task:87] Thread 'LoadDocs_MutateDocsFromSpecTask_1623158963.52_default__default__default_ttl=0_create__3000000_3250000_1623158963.53' started 2021-06-08 06:29:23,552 | infra | INFO | pool-5-thread-19 | [task:start_task:87] Thread 'LoadDocs_MutateDocsFromSpecTask_1623158963.52_default__default__default_ttl=0_create__2500000_2750000_1623158963.53' started 2021-06-08 06:29:23,553 | infra | INFO | pool-5-thread-7 | [task_manager:add_new_task:23] Added new task: LoadDocs_MutateDocsFromSpecTask_1623158963.52_default__default__default_ttl=0_create__3000000_3250000_1623158963.53 2021-06-08 06:29:23,556 | infra | INFO | pool-5-thread-22 | [task:start_task:87] Thread 'LoadDocs_MutateDocsFromSpecTask_1623158963.52_default__default__default_ttl=0_create__3250000_3500000_1623158963.53' started 2021-06-08 06:29:23,556 | infra | INFO | pool-5-thread-7 | [task_manager:add_new_task:23] Added new task: LoadDocs_MutateDocsFromSpecTask_1623158963.52_default__default__default_ttl=0_create__3250000_3500000_1623158963.53 2021-06-08 06:29:23,563 | infra | INFO | pool-5-thread-7 | [task_manager:add_new_task:23] Added new task: LoadDocs_MutateDocsFromSpecTask_1623158963.52_default__default__default_ttl=0_create__3500000_3750000_1623158963.53 2021-06-08 06:29:23,563 | infra | INFO | pool-5-thread-23 | [task:start_task:87] Thread 'LoadDocs_MutateDocsFromSpecTask_1623158963.52_default__default__default_ttl=0_create__3500000_3750000_1623158963.53' started 2021-06-08 06:29:23,565 | infra | INFO | pool-5-thread-7 | [task_manager:add_new_task:23] Added new task: LoadDocs_MutateDocsFromSpecTask_1623158963.52_default__default__default_ttl=0_create__3750000_4000000_1623158963.53 2021-06-08 06:29:23,565 | infra | INFO | pool-5-thread-24 | [task:start_task:87] Thread 'LoadDocs_MutateDocsFromSpecTask_1623158963.52_default__default__default_ttl=0_create__3750000_4000000_1623158963.53' started 2021-06-08 06:29:23,565 | infra | INFO | pool-5-thread-7 | [task_manager:add_new_task:23] Added new task: LoadDocs_MutateDocsFromSpecTask_1623158963.52_default__default__default_ttl=0_create__4000000_4250000_1623158963.53 2021-06-08 06:29:23,565 | infra | INFO | pool-5-thread-7 | [task_manager:add_new_task:23] Added new task: LoadDocs_MutateDocsFromSpecTask_1623158963.52_default__default__default_ttl=0_create__4250000_4500000_1623158963.53 2021-06-08 06:29:23,566 | infra | INFO | pool-5-thread-26 | [task:start_task:87] Thread 'LoadDocs_MutateDocsFromSpecTask_1623158963.52_default__default__default_ttl=0_create__4250000_4500000_1623158963.53' started 2021-06-08 06:29:23,566 | infra | INFO | pool-5-thread-7 | [task_manager:add_new_task:23] Added new task: LoadDocs_MutateDocsFromSpecTask_1623158963.52_default__default__default_ttl=0_create__4500000_4750000_1623158963.53 2021-06-08 06:29:23,566 | infra | INFO | pool-5-thread-27 | [task:start_task:87] Thread 'LoadDocs_MutateDocsFromSpecTask_1623158963.52_default__default__default_ttl=0_create__4500000_4750000_1623158963.53' started 2021-06-08 06:29:23,566 | infra | INFO | pool-5-thread-7 | [task_manager:add_new_task:23] Added new task: LoadDocs_MutateDocsFromSpecTask_1623158963.52_default__default__default_ttl=0_create__4750000_5000000_1623158963.53 2021-06-08 06:29:23,568 | infra | INFO | pool-5-thread-28 | [task:start_task:87] Thread 'LoadDocs_MutateDocsFromSpecTask_1623158963.52_default__default__default_ttl=0_create__4750000_5000000_1623158963.53' started 2021-06-08 06:29:23,568 | infra | DEBUG | pool-5-thread-7 | [task_manager:get_task_result:26] Getting task result for LoadDocs_MutateDocsFromSpecTask_1623158963.52_default__default__default_ttl=0_create__0_250000_1623158963.53 2021-06-08 06:29:23,569 | infra | INFO | pool-5-thread-25 | [task:start_task:87] Thread 'LoadDocs_MutateDocsFromSpecTask_1623158963.52_default__default__default_ttl=0_create__4000000_4250000_1623158963.53' started 2021-06-08 06:29:23,782 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 06:29:23,783 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 06:29:23,783 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 06:29:25,085 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 06:29:25,086 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 06:29:25,086 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 06:29:26,420 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 06:29:26,421 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 06:29:26,423 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 06:29:27,782 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 06:29:27,785 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 06:29:27,785 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 06:29:29,132 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 06:29:29,135 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 06:29:29,135 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 06:29:30,453 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 06:29:30,456 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 06:29:30,457 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 06:29:31,808 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 92445.000000 2021-06-08 06:29:31,811 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 06:29:31,812 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 06:29:33,134 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 92445.000000 2021-06-08 06:29:33,135 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 06:29:33,135 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 06:29:34,421 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 92445.000000 2021-06-08 06:29:34,423 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 06:29:34,423 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 06:29:35,756 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 92445.000000 2021-06-08 06:29:35,759 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 06:29:35,759 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 06:29:37,171 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 92445.000000 2021-06-08 06:29:37,174 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 06:29:37,174 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 06:29:38,582 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 92445.000000 2021-06-08 06:29:38,585 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 06:29:38,585 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 06:29:39,970 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 92445.000000 2021-06-08 06:29:39,971 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 06:29:39,973 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 06:29:41,372 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 92445.000000 2021-06-08 06:29:41,378 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 06:29:41,380 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 06:29:42,726 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 135341.900000 2021-06-08 06:29:42,727 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 06:29:42,727 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 06:29:44,046 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 135341.900000 2021-06-08 06:29:44,049 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 06:29:44,049 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 06:29:45,428 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 135341.900000 2021-06-08 06:29:45,431 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 06:29:45,433 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 06:29:46,864 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 135341.900000 2021-06-08 06:29:46,867 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 06:29:46,867 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 06:29:48,216 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 135341.900000 2021-06-08 06:29:48,219 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 06:29:48,219 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 06:29:49,608 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 135341.900000 2021-06-08 06:29:49,611 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 06:29:49,611 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 06:29:51,019 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 135341.900000 2021-06-08 06:29:51,023 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 06:29:51,023 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 06:29:52,384 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 121996.300000 2021-06-08 06:29:52,388 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 06:29:52,388 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 06:29:53,799 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 121996.300000 2021-06-08 06:29:53,808 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 06:29:53,808 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 06:29:55,206 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 121996.300000 2021-06-08 06:29:55,210 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 06:29:55,216 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 06:29:56,571 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 121996.300000 2021-06-08 06:29:56,573 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 06:29:56,573 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 06:29:57,986 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 121996.300000 2021-06-08 06:29:57,990 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 06:29:57,990 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 06:29:59,368 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 121996.300000 2021-06-08 06:29:59,371 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 06:29:59,371 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 06:30:00,740 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 121996.300000 2021-06-08 06:30:00,742 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 06:30:00,742 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 06:30:01,311 | infra | INFO | pool-5-thread-9 | [task:complete_task:102] Thread 'LoadDocs_MutateDocsFromSpecTask_1623158963.52_default__default__default_ttl=0_create__0_250000_1623158963.53' completed 2021-06-08 06:30:01,312 | infra | DEBUG | pool-5-thread-7 | [task:call:3816] Items loaded in task LoadDocs_MutateDocsFromSpecTask_1623158963.52_default__default__default_ttl=0_create__0_250000_1623158963.53 are 250000 2021-06-08 06:30:01,312 | test | DEBUG | pool-5-thread-7 | [task:call:3845] Failed to load 0 docs from 0 to 250000 2021-06-08 06:30:01,312 | infra | DEBUG | pool-5-thread-7 | [task_manager:get_task_result:26] Getting task result for LoadDocs_MutateDocsFromSpecTask_1623158963.52_default__default__default_ttl=0_create__250000_500000_1623158963.53 2021-06-08 06:30:01,318 | infra | INFO | pool-5-thread-11 | [task:complete_task:102] Thread 'LoadDocs_MutateDocsFromSpecTask_1623158963.52_default__default__default_ttl=0_create__500000_750000_1623158963.53' completed 2021-06-08 06:30:01,809 | infra | INFO | pool-5-thread-18 | [task:complete_task:102] Thread 'LoadDocs_MutateDocsFromSpecTask_1623158963.52_default__default__default_ttl=0_create__2250000_2500000_1623158963.53' completed 2021-06-08 06:30:01,825 | infra | INFO | pool-5-thread-21 | [task:complete_task:102] Thread 'LoadDocs_MutateDocsFromSpecTask_1623158963.52_default__default__default_ttl=0_create__3000000_3250000_1623158963.53' completed 2021-06-08 06:30:01,969 | infra | INFO | pool-5-thread-15 | [task:complete_task:102] Thread 'LoadDocs_MutateDocsFromSpecTask_1623158963.52_default__default__default_ttl=0_create__1500000_1750000_1623158963.53' completed 2021-06-08 06:30:01,982 | infra | INFO | pool-5-thread-27 | [task:complete_task:102] Thread 'LoadDocs_MutateDocsFromSpecTask_1623158963.52_default__default__default_ttl=0_create__4500000_4750000_1623158963.53' completed 2021-06-08 06:30:02,026 | infra | INFO | pool-5-thread-10 | [task:complete_task:102] Thread 'LoadDocs_MutateDocsFromSpecTask_1623158963.52_default__default__default_ttl=0_create__250000_500000_1623158963.53' completed 2021-06-08 06:30:02,026 | infra | DEBUG | pool-5-thread-7 | [task:call:3816] Items loaded in task LoadDocs_MutateDocsFromSpecTask_1623158963.52_default__default__default_ttl=0_create__250000_500000_1623158963.53 are 250000 2021-06-08 06:30:02,026 | test | DEBUG | pool-5-thread-7 | [task:call:3845] Failed to load 0 docs from 250000 to 500000 2021-06-08 06:30:02,026 | infra | DEBUG | pool-5-thread-7 | [task_manager:get_task_result:26] Getting task result for LoadDocs_MutateDocsFromSpecTask_1623158963.52_default__default__default_ttl=0_create__500000_750000_1623158963.53 2021-06-08 06:30:02,026 | infra | DEBUG | pool-5-thread-7 | [task:call:3816] Items loaded in task LoadDocs_MutateDocsFromSpecTask_1623158963.52_default__default__default_ttl=0_create__500000_750000_1623158963.53 are 250000 2021-06-08 06:30:02,026 | test | DEBUG | pool-5-thread-7 | [task:call:3845] Failed to load 0 docs from 500000 to 750000 2021-06-08 06:30:02,026 | infra | DEBUG | pool-5-thread-7 | [task_manager:get_task_result:26] Getting task result for LoadDocs_MutateDocsFromSpecTask_1623158963.52_default__default__default_ttl=0_create__750000_1000000_1623158963.53 2021-06-08 06:30:02,111 | infra | INFO | pool-5-thread-13 | [task:complete_task:102] Thread 'LoadDocs_MutateDocsFromSpecTask_1623158963.52_default__default__default_ttl=0_create__1000000_1250000_1623158963.53' completed 2021-06-08 06:30:02,111 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 133016.435359 2021-06-08 06:30:02,114 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 06:30:02,114 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 06:30:02,161 | infra | INFO | pool-5-thread-26 | [task:complete_task:102] Thread 'LoadDocs_MutateDocsFromSpecTask_1623158963.52_default__default__default_ttl=0_create__4250000_4500000_1623158963.53' completed 2021-06-08 06:30:02,188 | infra | INFO | pool-5-thread-12 | [task:complete_task:102] Thread 'LoadDocs_MutateDocsFromSpecTask_1623158963.52_default__default__default_ttl=0_create__750000_1000000_1623158963.53' completed 2021-06-08 06:30:02,190 | infra | DEBUG | pool-5-thread-7 | [task:call:3816] Items loaded in task LoadDocs_MutateDocsFromSpecTask_1623158963.52_default__default__default_ttl=0_create__750000_1000000_1623158963.53 are 250000 2021-06-08 06:30:02,190 | test | DEBUG | pool-5-thread-7 | [task:call:3845] Failed to load 0 docs from 750000 to 1000000 2021-06-08 06:30:02,190 | infra | DEBUG | pool-5-thread-7 | [task_manager:get_task_result:26] Getting task result for LoadDocs_MutateDocsFromSpecTask_1623158963.52_default__default__default_ttl=0_create__1000000_1250000_1623158963.53 2021-06-08 06:30:02,190 | infra | DEBUG | pool-5-thread-7 | [task:call:3816] Items loaded in task LoadDocs_MutateDocsFromSpecTask_1623158963.52_default__default__default_ttl=0_create__1000000_1250000_1623158963.53 are 250000 2021-06-08 06:30:02,190 | test | DEBUG | pool-5-thread-7 | [task:call:3845] Failed to load 0 docs from 1000000 to 1250000 2021-06-08 06:30:02,190 | infra | DEBUG | pool-5-thread-7 | [task_manager:get_task_result:26] Getting task result for LoadDocs_MutateDocsFromSpecTask_1623158963.52_default__default__default_ttl=0_create__1250000_1500000_1623158963.53 2021-06-08 06:30:02,197 | infra | INFO | pool-5-thread-14 | [task:complete_task:102] Thread 'LoadDocs_MutateDocsFromSpecTask_1623158963.52_default__default__default_ttl=0_create__1250000_1500000_1623158963.53' completed 2021-06-08 06:30:02,197 | infra | DEBUG | pool-5-thread-7 | [task:call:3816] Items loaded in task LoadDocs_MutateDocsFromSpecTask_1623158963.52_default__default__default_ttl=0_create__1250000_1500000_1623158963.53 are 250000 2021-06-08 06:30:02,197 | test | DEBUG | pool-5-thread-7 | [task:call:3845] Failed to load 0 docs from 1250000 to 1500000 2021-06-08 06:30:02,197 | infra | DEBUG | pool-5-thread-7 | [task_manager:get_task_result:26] Getting task result for LoadDocs_MutateDocsFromSpecTask_1623158963.52_default__default__default_ttl=0_create__1500000_1750000_1623158963.53 2021-06-08 06:30:02,197 | infra | DEBUG | pool-5-thread-7 | [task:call:3816] Items loaded in task LoadDocs_MutateDocsFromSpecTask_1623158963.52_default__default__default_ttl=0_create__1500000_1750000_1623158963.53 are 250000 2021-06-08 06:30:02,197 | test | DEBUG | pool-5-thread-7 | [task:call:3845] Failed to load 0 docs from 1500000 to 1750000 2021-06-08 06:30:02,197 | infra | DEBUG | pool-5-thread-7 | [task_manager:get_task_result:26] Getting task result for LoadDocs_MutateDocsFromSpecTask_1623158963.52_default__default__default_ttl=0_create__1750000_2000000_1623158963.53 2021-06-08 06:30:02,209 | infra | INFO | pool-5-thread-16 | [task:complete_task:102] Thread 'LoadDocs_MutateDocsFromSpecTask_1623158963.52_default__default__default_ttl=0_create__1750000_2000000_1623158963.53' completed 2021-06-08 06:30:02,209 | infra | DEBUG | pool-5-thread-7 | [task:call:3816] Items loaded in task LoadDocs_MutateDocsFromSpecTask_1623158963.52_default__default__default_ttl=0_create__1750000_2000000_1623158963.53 are 250000 2021-06-08 06:30:02,209 | test | DEBUG | pool-5-thread-7 | [task:call:3845] Failed to load 0 docs from 1750000 to 2000000 2021-06-08 06:30:02,210 | infra | DEBUG | pool-5-thread-7 | [task_manager:get_task_result:26] Getting task result for LoadDocs_MutateDocsFromSpecTask_1623158963.52_default__default__default_ttl=0_create__2000000_2250000_1623158963.53 2021-06-08 06:30:02,283 | infra | INFO | pool-5-thread-19 | [task:complete_task:102] Thread 'LoadDocs_MutateDocsFromSpecTask_1623158963.52_default__default__default_ttl=0_create__2500000_2750000_1623158963.53' completed 2021-06-08 06:30:02,283 | infra | INFO | pool-5-thread-28 | [task:complete_task:102] Thread 'LoadDocs_MutateDocsFromSpecTask_1623158963.52_default__default__default_ttl=0_create__4750000_5000000_1623158963.53' completed 2021-06-08 06:30:02,325 | infra | INFO | pool-5-thread-23 | [task:complete_task:102] Thread 'LoadDocs_MutateDocsFromSpecTask_1623158963.52_default__default__default_ttl=0_create__3500000_3750000_1623158963.53' completed 2021-06-08 06:30:02,349 | infra | INFO | pool-5-thread-17 | [task:complete_task:102] Thread 'LoadDocs_MutateDocsFromSpecTask_1623158963.52_default__default__default_ttl=0_create__2000000_2250000_1623158963.53' completed 2021-06-08 06:30:02,349 | infra | DEBUG | pool-5-thread-7 | [task:call:3816] Items loaded in task LoadDocs_MutateDocsFromSpecTask_1623158963.52_default__default__default_ttl=0_create__2000000_2250000_1623158963.53 are 250000 2021-06-08 06:30:02,349 | test | DEBUG | pool-5-thread-7 | [task:call:3845] Failed to load 0 docs from 2000000 to 2250000 2021-06-08 06:30:02,349 | infra | DEBUG | pool-5-thread-7 | [task_manager:get_task_result:26] Getting task result for LoadDocs_MutateDocsFromSpecTask_1623158963.52_default__default__default_ttl=0_create__2250000_2500000_1623158963.53 2021-06-08 06:30:02,351 | infra | DEBUG | pool-5-thread-7 | [task:call:3816] Items loaded in task LoadDocs_MutateDocsFromSpecTask_1623158963.52_default__default__default_ttl=0_create__2250000_2500000_1623158963.53 are 250000 2021-06-08 06:30:02,351 | test | DEBUG | pool-5-thread-7 | [task:call:3845] Failed to load 0 docs from 2250000 to 2500000 2021-06-08 06:30:02,351 | infra | DEBUG | pool-5-thread-7 | [task_manager:get_task_result:26] Getting task result for LoadDocs_MutateDocsFromSpecTask_1623158963.52_default__default__default_ttl=0_create__2500000_2750000_1623158963.53 2021-06-08 06:30:02,351 | infra | DEBUG | pool-5-thread-7 | [task:call:3816] Items loaded in task LoadDocs_MutateDocsFromSpecTask_1623158963.52_default__default__default_ttl=0_create__2500000_2750000_1623158963.53 are 250000 2021-06-08 06:30:02,351 | test | DEBUG | pool-5-thread-7 | [task:call:3845] Failed to load 0 docs from 2500000 to 2750000 2021-06-08 06:30:02,351 | infra | DEBUG | pool-5-thread-7 | [task_manager:get_task_result:26] Getting task result for LoadDocs_MutateDocsFromSpecTask_1623158963.52_default__default__default_ttl=0_create__2750000_3000000_1623158963.53 2021-06-08 06:30:02,357 | infra | INFO | pool-5-thread-20 | [task:complete_task:102] Thread 'LoadDocs_MutateDocsFromSpecTask_1623158963.52_default__default__default_ttl=0_create__2750000_3000000_1623158963.53' completed 2021-06-08 06:30:02,357 | infra | DEBUG | pool-5-thread-7 | [task:call:3816] Items loaded in task LoadDocs_MutateDocsFromSpecTask_1623158963.52_default__default__default_ttl=0_create__2750000_3000000_1623158963.53 are 250000 2021-06-08 06:30:02,358 | test | DEBUG | pool-5-thread-7 | [task:call:3845] Failed to load 0 docs from 2750000 to 3000000 2021-06-08 06:30:02,358 | infra | DEBUG | pool-5-thread-7 | [task_manager:get_task_result:26] Getting task result for LoadDocs_MutateDocsFromSpecTask_1623158963.52_default__default__default_ttl=0_create__3000000_3250000_1623158963.53 2021-06-08 06:30:02,358 | infra | DEBUG | pool-5-thread-7 | [task:call:3816] Items loaded in task LoadDocs_MutateDocsFromSpecTask_1623158963.52_default__default__default_ttl=0_create__3000000_3250000_1623158963.53 are 250000 2021-06-08 06:30:02,358 | test | DEBUG | pool-5-thread-7 | [task:call:3845] Failed to load 0 docs from 3000000 to 3250000 2021-06-08 06:30:02,358 | infra | DEBUG | pool-5-thread-7 | [task_manager:get_task_result:26] Getting task result for LoadDocs_MutateDocsFromSpecTask_1623158963.52_default__default__default_ttl=0_create__3250000_3500000_1623158963.53 2021-06-08 06:30:02,371 | infra | INFO | pool-5-thread-22 | [task:complete_task:102] Thread 'LoadDocs_MutateDocsFromSpecTask_1623158963.52_default__default__default_ttl=0_create__3250000_3500000_1623158963.53' completed 2021-06-08 06:30:02,371 | infra | DEBUG | pool-5-thread-7 | [task:call:3816] Items loaded in task LoadDocs_MutateDocsFromSpecTask_1623158963.52_default__default__default_ttl=0_create__3250000_3500000_1623158963.53 are 250000 2021-06-08 06:30:02,371 | test | DEBUG | pool-5-thread-7 | [task:call:3845] Failed to load 0 docs from 3250000 to 3500000 2021-06-08 06:30:02,371 | infra | DEBUG | pool-5-thread-7 | [task_manager:get_task_result:26] Getting task result for LoadDocs_MutateDocsFromSpecTask_1623158963.52_default__default__default_ttl=0_create__3500000_3750000_1623158963.53 2021-06-08 06:30:02,371 | infra | DEBUG | pool-5-thread-7 | [task:call:3816] Items loaded in task LoadDocs_MutateDocsFromSpecTask_1623158963.52_default__default__default_ttl=0_create__3500000_3750000_1623158963.53 are 250000 2021-06-08 06:30:02,371 | test | DEBUG | pool-5-thread-7 | [task:call:3845] Failed to load 0 docs from 3500000 to 3750000 2021-06-08 06:30:02,371 | infra | DEBUG | pool-5-thread-7 | [task_manager:get_task_result:26] Getting task result for LoadDocs_MutateDocsFromSpecTask_1623158963.52_default__default__default_ttl=0_create__3750000_4000000_1623158963.53 2021-06-08 06:30:02,423 | infra | INFO | pool-5-thread-24 | [task:complete_task:102] Thread 'LoadDocs_MutateDocsFromSpecTask_1623158963.52_default__default__default_ttl=0_create__3750000_4000000_1623158963.53' completed 2021-06-08 06:30:02,423 | infra | DEBUG | pool-5-thread-7 | [task:call:3816] Items loaded in task LoadDocs_MutateDocsFromSpecTask_1623158963.52_default__default__default_ttl=0_create__3750000_4000000_1623158963.53 are 250000 2021-06-08 06:30:02,424 | test | DEBUG | pool-5-thread-7 | [task:call:3845] Failed to load 0 docs from 3750000 to 4000000 2021-06-08 06:30:02,424 | infra | DEBUG | pool-5-thread-7 | [task_manager:get_task_result:26] Getting task result for LoadDocs_MutateDocsFromSpecTask_1623158963.52_default__default__default_ttl=0_create__4000000_4250000_1623158963.53 2021-06-08 06:30:02,461 | infra | INFO | pool-5-thread-25 | [task:complete_task:102] Thread 'LoadDocs_MutateDocsFromSpecTask_1623158963.52_default__default__default_ttl=0_create__4000000_4250000_1623158963.53' completed 2021-06-08 06:30:02,463 | infra | DEBUG | pool-5-thread-7 | [task:call:3816] Items loaded in task LoadDocs_MutateDocsFromSpecTask_1623158963.52_default__default__default_ttl=0_create__4000000_4250000_1623158963.53 are 250000 2021-06-08 06:30:02,463 | test | DEBUG | pool-5-thread-7 | [task:call:3845] Failed to load 0 docs from 4000000 to 4250000 2021-06-08 06:30:02,463 | infra | DEBUG | pool-5-thread-7 | [task_manager:get_task_result:26] Getting task result for LoadDocs_MutateDocsFromSpecTask_1623158963.52_default__default__default_ttl=0_create__4250000_4500000_1623158963.53 2021-06-08 06:30:02,463 | infra | DEBUG | pool-5-thread-7 | [task:call:3816] Items loaded in task LoadDocs_MutateDocsFromSpecTask_1623158963.52_default__default__default_ttl=0_create__4250000_4500000_1623158963.53 are 250000 2021-06-08 06:30:02,463 | test | DEBUG | pool-5-thread-7 | [task:call:3845] Failed to load 0 docs from 4250000 to 4500000 2021-06-08 06:30:02,463 | infra | DEBUG | pool-5-thread-7 | [task_manager:get_task_result:26] Getting task result for LoadDocs_MutateDocsFromSpecTask_1623158963.52_default__default__default_ttl=0_create__4500000_4750000_1623158963.53 2021-06-08 06:30:02,463 | infra | DEBUG | pool-5-thread-7 | [task:call:3816] Items loaded in task LoadDocs_MutateDocsFromSpecTask_1623158963.52_default__default__default_ttl=0_create__4500000_4750000_1623158963.53 are 250000 2021-06-08 06:30:02,463 | test | DEBUG | pool-5-thread-7 | [task:call:3845] Failed to load 0 docs from 4500000 to 4750000 2021-06-08 06:30:02,463 | infra | DEBUG | pool-5-thread-7 | [task_manager:get_task_result:26] Getting task result for LoadDocs_MutateDocsFromSpecTask_1623158963.52_default__default__default_ttl=0_create__4750000_5000000_1623158963.53 2021-06-08 06:30:02,463 | infra | DEBUG | pool-5-thread-7 | [task:call:3816] Items loaded in task LoadDocs_MutateDocsFromSpecTask_1623158963.52_default__default__default_ttl=0_create__4750000_5000000_1623158963.53 are 250000 2021-06-08 06:30:02,463 | test | DEBUG | pool-5-thread-7 | [task:call:3845] Failed to load 0 docs from 4750000 to 5000000 2021-06-08 06:30:02,463 | infra | DEBUG | pool-5-thread-7 | [task_manager:get_task_result:26] Getting task result for PrintBucketStats_default_1623158963.53 2021-06-08 06:30:03,115 | test | INFO | pool-5-thread-8 | [table_view:display:72] Ops trend for bucket 'default' +------------+---------------------------------------------------------------------------------------------------------------------------+------------+ | Min | Trend | Max | +------------+---------------------------------------------------------------------------------------------------------------------------+------------+ | 0.000 | ******....................................................................********................................******X | 135341.900 | | 121996.300 | .........................................................................................*******........X | 133016.435 | +------------+---------------------------------------------------------------------------------------------------------------------------+------------+ 2021-06-08 06:30:03,115 | infra | INFO | pool-5-thread-8 | [task:complete_task:102] Thread 'PrintBucketStats_default_1623158963.53' completed 2021-06-08 06:30:03,115 | infra | DEBUG | pool-5-thread-7 | [task:call:3861] ========= Tasks in loadgen pool======= 2021-06-08 06:30:03,115 | infra | WARNING | pool-5-thread-7 | [task_manager:print_tasks_in_pool:83] Task 'MutateDocsFromSpecTask_1623158963.52' not completed 2021-06-08 06:30:03,115 | infra | DEBUG | pool-5-thread-7 | [task:call:3863] ====================================== 2021-06-08 06:30:03,115 | infra | DEBUG | pool-5-thread-7 | [task:call:3866] Task 'LoadDocs_MutateDocsFromSpecTask_1623158963.52_default__default__default_ttl=0_create__0_250000_1623158963.53' complete. Loaded 250000 items 2021-06-08 06:30:03,115 | infra | DEBUG | pool-5-thread-7 | [task:call:3866] Task 'LoadDocs_MutateDocsFromSpecTask_1623158963.52_default__default__default_ttl=0_create__250000_500000_1623158963.53' complete. Loaded 250000 items 2021-06-08 06:30:03,115 | infra | DEBUG | pool-5-thread-7 | [task:call:3866] Task 'LoadDocs_MutateDocsFromSpecTask_1623158963.52_default__default__default_ttl=0_create__500000_750000_1623158963.53' complete. Loaded 250000 items 2021-06-08 06:30:03,115 | infra | DEBUG | pool-5-thread-7 | [task:call:3866] Task 'LoadDocs_MutateDocsFromSpecTask_1623158963.52_default__default__default_ttl=0_create__750000_1000000_1623158963.53' complete. Loaded 250000 items 2021-06-08 06:30:03,115 | infra | DEBUG | pool-5-thread-7 | [task:call:3866] Task 'LoadDocs_MutateDocsFromSpecTask_1623158963.52_default__default__default_ttl=0_create__1000000_1250000_1623158963.53' complete. Loaded 250000 items 2021-06-08 06:30:03,115 | infra | DEBUG | pool-5-thread-7 | [task:call:3866] Task 'LoadDocs_MutateDocsFromSpecTask_1623158963.52_default__default__default_ttl=0_create__1250000_1500000_1623158963.53' complete. Loaded 250000 items 2021-06-08 06:30:03,117 | infra | DEBUG | pool-5-thread-7 | [task:call:3866] Task 'LoadDocs_MutateDocsFromSpecTask_1623158963.52_default__default__default_ttl=0_create__1500000_1750000_1623158963.53' complete. Loaded 250000 items 2021-06-08 06:30:03,117 | infra | DEBUG | pool-5-thread-7 | [task:call:3866] Task 'LoadDocs_MutateDocsFromSpecTask_1623158963.52_default__default__default_ttl=0_create__1750000_2000000_1623158963.53' complete. Loaded 250000 items 2021-06-08 06:30:03,117 | infra | DEBUG | pool-5-thread-7 | [task:call:3866] Task 'LoadDocs_MutateDocsFromSpecTask_1623158963.52_default__default__default_ttl=0_create__2000000_2250000_1623158963.53' complete. Loaded 250000 items 2021-06-08 06:30:03,117 | infra | DEBUG | pool-5-thread-7 | [task:call:3866] Task 'LoadDocs_MutateDocsFromSpecTask_1623158963.52_default__default__default_ttl=0_create__2250000_2500000_1623158963.53' complete. Loaded 250000 items 2021-06-08 06:30:03,117 | infra | DEBUG | pool-5-thread-7 | [task:call:3866] Task 'LoadDocs_MutateDocsFromSpecTask_1623158963.52_default__default__default_ttl=0_create__2500000_2750000_1623158963.53' complete. Loaded 250000 items 2021-06-08 06:30:03,117 | infra | DEBUG | pool-5-thread-7 | [task:call:3866] Task 'LoadDocs_MutateDocsFromSpecTask_1623158963.52_default__default__default_ttl=0_create__2750000_3000000_1623158963.53' complete. Loaded 250000 items 2021-06-08 06:30:03,117 | infra | DEBUG | pool-5-thread-7 | [task:call:3866] Task 'LoadDocs_MutateDocsFromSpecTask_1623158963.52_default__default__default_ttl=0_create__3000000_3250000_1623158963.53' complete. Loaded 250000 items 2021-06-08 06:30:03,117 | infra | DEBUG | pool-5-thread-7 | [task:call:3866] Task 'LoadDocs_MutateDocsFromSpecTask_1623158963.52_default__default__default_ttl=0_create__3250000_3500000_1623158963.53' complete. Loaded 250000 items 2021-06-08 06:30:03,117 | infra | DEBUG | pool-5-thread-7 | [task:call:3866] Task 'LoadDocs_MutateDocsFromSpecTask_1623158963.52_default__default__default_ttl=0_create__3500000_3750000_1623158963.53' complete. Loaded 250000 items 2021-06-08 06:30:03,118 | infra | DEBUG | pool-5-thread-7 | [task:call:3866] Task 'LoadDocs_MutateDocsFromSpecTask_1623158963.52_default__default__default_ttl=0_create__3750000_4000000_1623158963.53' complete. Loaded 250000 items 2021-06-08 06:30:03,118 | infra | DEBUG | pool-5-thread-7 | [task:call:3866] Task 'LoadDocs_MutateDocsFromSpecTask_1623158963.52_default__default__default_ttl=0_create__4000000_4250000_1623158963.53' complete. Loaded 250000 items 2021-06-08 06:30:03,118 | infra | DEBUG | pool-5-thread-7 | [task:call:3866] Task 'LoadDocs_MutateDocsFromSpecTask_1623158963.52_default__default__default_ttl=0_create__4250000_4500000_1623158963.53' complete. Loaded 250000 items 2021-06-08 06:30:03,118 | infra | DEBUG | pool-5-thread-7 | [task:call:3866] Task 'LoadDocs_MutateDocsFromSpecTask_1623158963.52_default__default__default_ttl=0_create__4500000_4750000_1623158963.53' complete. Loaded 250000 items 2021-06-08 06:30:03,118 | infra | DEBUG | pool-5-thread-7 | [task:call:3866] Task 'LoadDocs_MutateDocsFromSpecTask_1623158963.52_default__default__default_ttl=0_create__4750000_5000000_1623158963.53' complete. Loaded 250000 items 2021-06-08 06:30:03,118 | infra | INFO | pool-5-thread-7 | [task:complete_task:102] Thread 'MutateDocsFromSpecTask_1623158963.52' completed 2021-06-08 06:30:03,178 | test | DEBUG | MainThread | [cluster_ready_functions:get_nodes_from_services_map:597] All nodes in cluster: [ip:172.23.106.168 port:8091 ssh_username:root, ip:172.23.106.198 port:8091 ssh_username:root, ip:172.23.106.196 port:8091 ssh_username:root, ip:172.23.106.199 port:8091 ssh_username:root] 2021-06-08 06:30:03,180 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.196 with username: root, password: couchbase 2021-06-08 06:30:03,289 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 06:30:03,562 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: ls /etc/ 2021-06-08 06:30:03,640 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 06:30:03,640 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: ls /cygdrive/c/Windows 2021-06-08 06:30:03,654 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: uname -m 2021-06-08 06:30:03,670 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: cat /proc/cpuinfo 2021-06-08 06:30:03,684 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: df -Thl 2021-06-08 06:30:03,707 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: cat /proc/meminfo 2021-06-08 06:30:03,726 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: hostname 2021-06-08 06:30:03,743 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: hostname -d 2021-06-08 06:30:03,762 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:559] Starting StatsWaitTask for num_items_for_persistence on bucket default 2021-06-08 06:30:03,762 | infra | INFO | MainThread | [task_manager:add_new_task:23] Added new task: StatsWaitTask_default_num_items_for_persistence_1623159003.76 2021-06-08 06:30:03,762 | infra | INFO | pool-5-thread-29 | [task:start_task:87] Thread 'StatsWaitTask_default_num_items_for_persistence_1623159003.76' started 2021-06-08 06:30:03,763 | infra | DEBUG | pool-5-thread-29 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:30:03,765 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.199 with username: root, password: couchbase 2021-06-08 06:30:03,858 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 06:30:04,111 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: ls /etc/ 2021-06-08 06:30:04,203 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 06:30:04,203 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: ls /cygdrive/c/Windows 2021-06-08 06:30:04,223 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: uname -m 2021-06-08 06:30:04,243 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: cat /proc/cpuinfo 2021-06-08 06:30:04,259 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: df -Thl 2021-06-08 06:30:04,282 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: cat /proc/meminfo 2021-06-08 06:30:04,305 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: hostname 2021-06-08 06:30:04,322 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: hostname -d 2021-06-08 06:30:04,339 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:559] Starting StatsWaitTask for num_items_for_persistence on bucket default 2021-06-08 06:30:04,339 | infra | INFO | MainThread | [task_manager:add_new_task:23] Added new task: StatsWaitTask_default_num_items_for_persistence_1623159004.34 2021-06-08 06:30:04,341 | infra | INFO | pool-5-thread-30 | [task:start_task:87] Thread 'StatsWaitTask_default_num_items_for_persistence_1623159004.34' started 2021-06-08 06:30:04,341 | infra | DEBUG | pool-5-thread-30 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:30:04,342 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.168 with username: root, password: couchbase 2021-06-08 06:30:04,443 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 06:30:04,723 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: ls /etc/ 2021-06-08 06:30:04,822 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 06:30:04,822 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: ls /cygdrive/c/Windows 2021-06-08 06:30:04,845 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: uname -m 2021-06-08 06:30:04,865 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: cat /proc/cpuinfo 2021-06-08 06:30:04,888 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: df -Thl 2021-06-08 06:30:04,917 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: cat /proc/meminfo 2021-06-08 06:30:04,934 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: hostname 2021-06-08 06:30:04,960 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: hostname -d 2021-06-08 06:30:04,980 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:559] Starting StatsWaitTask for num_items_for_persistence on bucket default 2021-06-08 06:30:04,980 | infra | INFO | MainThread | [task_manager:add_new_task:23] Added new task: StatsWaitTask_default_num_items_for_persistence_1623159004.98 2021-06-08 06:30:04,980 | infra | INFO | pool-5-thread-1 | [task:start_task:87] Thread 'StatsWaitTask_default_num_items_for_persistence_1623159004.98' started 2021-06-08 06:30:04,980 | infra | DEBUG | pool-5-thread-1 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:30:04,983 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.198 with username: root, password: couchbase 2021-06-08 06:30:05,069 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 06:30:05,260 | test | DEBUG | pool-5-thread-29 | [task:_compare:2649] Comparing 2162227 == 0 2021-06-08 06:30:05,260 | test | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 2162227 == 0. Received: {'172.23.106.196': 2162227} for bucket 'default' 2021-06-08 06:30:05,260 | infra | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:30:05,263 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:30:05,263 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:30:05,338 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: ls /etc/ 2021-06-08 06:30:05,427 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 06:30:05,427 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: ls /cygdrive/c/Windows 2021-06-08 06:30:05,447 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: uname -m 2021-06-08 06:30:05,473 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: cat /proc/cpuinfo 2021-06-08 06:30:05,494 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: df -Thl 2021-06-08 06:30:05,517 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: cat /proc/meminfo 2021-06-08 06:30:05,539 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: hostname 2021-06-08 06:30:05,553 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: hostname -d 2021-06-08 06:30:05,582 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:559] Starting StatsWaitTask for num_items_for_persistence on bucket default 2021-06-08 06:30:05,582 | infra | INFO | MainThread | [task_manager:add_new_task:23] Added new task: StatsWaitTask_default_num_items_for_persistence_1623159005.58 2021-06-08 06:30:05,582 | infra | INFO | pool-5-thread-2 | [task:start_task:87] Thread 'StatsWaitTask_default_num_items_for_persistence_1623159005.58' started 2021-06-08 06:30:05,582 | infra | DEBUG | MainThread | [task_manager:get_task_result:26] Getting task result for StatsWaitTask_default_num_items_for_persistence_1623159003.76 2021-06-08 06:30:05,582 | infra | DEBUG | pool-5-thread-2 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:30:05,878 | test | DEBUG | pool-5-thread-30 | [task:_compare:2649] Comparing 2144428 == 0 2021-06-08 06:30:05,878 | test | DEBUG | pool-5-thread-30 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 2144428 == 0. Received: {'172.23.106.199': 2144428} for bucket 'default' 2021-06-08 06:30:05,878 | infra | DEBUG | pool-5-thread-30 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:30:05,880 | infra | DEBUG | pool-5-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:30:05,881 | infra | DEBUG | pool-5-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:30:06,845 | test | DEBUG | pool-5-thread-1 | [task:_compare:2649] Comparing 2273664 == 0 2021-06-08 06:30:06,845 | test | DEBUG | pool-5-thread-1 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 2273664 == 0. Received: {'172.23.106.168': 2273664} for bucket 'default' 2021-06-08 06:30:06,845 | infra | DEBUG | pool-5-thread-1 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:30:06,848 | infra | DEBUG | pool-5-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:30:06,848 | infra | DEBUG | pool-5-thread-1 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:30:07,135 | test | DEBUG | pool-5-thread-2 | [task:_compare:2649] Comparing 2281433 == 0 2021-06-08 06:30:07,135 | test | DEBUG | pool-5-thread-2 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 2281433 == 0. Received: {'172.23.106.198': 2281433} for bucket 'default' 2021-06-08 06:30:07,135 | infra | DEBUG | pool-5-thread-2 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:30:07,138 | infra | DEBUG | pool-5-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:30:07,138 | infra | DEBUG | pool-5-thread-2 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:30:10,263 | infra | DEBUG | pool-5-thread-29 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:30:10,881 | infra | DEBUG | pool-5-thread-30 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:30:11,753 | test | DEBUG | pool-5-thread-29 | [task:_compare:2649] Comparing 2539551 == 0 2021-06-08 06:30:11,753 | test | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 2539551 == 0. Received: {'172.23.106.196': 2539551} for bucket 'default' 2021-06-08 06:30:11,753 | infra | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:30:11,755 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:30:11,755 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:30:11,848 | infra | DEBUG | pool-5-thread-1 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:30:12,138 | infra | DEBUG | pool-5-thread-2 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:30:12,398 | test | DEBUG | pool-5-thread-30 | [task:_compare:2649] Comparing 2518397 == 0 2021-06-08 06:30:12,398 | test | DEBUG | pool-5-thread-30 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 2518397 == 0. Received: {'172.23.106.199': 2518397} for bucket 'default' 2021-06-08 06:30:12,398 | infra | DEBUG | pool-5-thread-30 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:30:12,401 | infra | DEBUG | pool-5-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:30:12,401 | infra | DEBUG | pool-5-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:30:13,450 | test | DEBUG | pool-5-thread-1 | [task:_compare:2649] Comparing 2679113 == 0 2021-06-08 06:30:13,450 | test | DEBUG | pool-5-thread-1 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 2679113 == 0. Received: {'172.23.106.168': 2679113} for bucket 'default' 2021-06-08 06:30:13,450 | infra | DEBUG | pool-5-thread-1 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:30:13,451 | infra | DEBUG | pool-5-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:30:13,451 | infra | DEBUG | pool-5-thread-1 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:30:13,697 | test | DEBUG | pool-5-thread-2 | [task:_compare:2649] Comparing 2622152 == 0 2021-06-08 06:30:13,697 | test | DEBUG | pool-5-thread-2 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 2622152 == 0. Received: {'172.23.106.198': 2622152} for bucket 'default' 2021-06-08 06:30:13,697 | infra | DEBUG | pool-5-thread-2 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:30:13,698 | infra | DEBUG | pool-5-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:30:13,698 | infra | DEBUG | pool-5-thread-2 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:30:16,756 | infra | DEBUG | pool-5-thread-29 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:30:17,401 | infra | DEBUG | pool-5-thread-30 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:30:18,298 | test | DEBUG | pool-5-thread-29 | [task:_compare:2649] Comparing 2878383 == 0 2021-06-08 06:30:18,298 | test | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 2878383 == 0. Received: {'172.23.106.196': 2878383} for bucket 'default' 2021-06-08 06:30:18,298 | infra | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:30:18,299 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:30:18,299 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:30:18,453 | infra | DEBUG | pool-5-thread-1 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:30:18,700 | infra | DEBUG | pool-5-thread-2 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:30:18,845 | test | DEBUG | pool-5-thread-30 | [task:_compare:2649] Comparing 2799786 == 0 2021-06-08 06:30:18,845 | test | DEBUG | pool-5-thread-30 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 2799786 == 0. Received: {'172.23.106.199': 2799786} for bucket 'default' 2021-06-08 06:30:18,845 | infra | DEBUG | pool-5-thread-30 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:30:18,848 | infra | DEBUG | pool-5-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:30:18,848 | infra | DEBUG | pool-5-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:30:19,891 | test | DEBUG | pool-5-thread-1 | [task:_compare:2649] Comparing 2827361 == 0 2021-06-08 06:30:19,891 | test | DEBUG | pool-5-thread-1 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 2827361 == 0. Received: {'172.23.106.168': 2827361} for bucket 'default' 2021-06-08 06:30:19,892 | infra | DEBUG | pool-5-thread-1 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:30:19,894 | infra | DEBUG | pool-5-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:30:19,894 | infra | DEBUG | pool-5-thread-1 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:30:20,145 | test | DEBUG | pool-5-thread-2 | [task:_compare:2649] Comparing 2820772 == 0 2021-06-08 06:30:20,145 | test | DEBUG | pool-5-thread-2 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 2820772 == 0. Received: {'172.23.106.198': 2820772} for bucket 'default' 2021-06-08 06:30:20,147 | infra | DEBUG | pool-5-thread-2 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:30:20,148 | infra | DEBUG | pool-5-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:30:20,148 | infra | DEBUG | pool-5-thread-2 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:30:23,299 | infra | DEBUG | pool-5-thread-29 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:30:23,848 | infra | DEBUG | pool-5-thread-30 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:30:24,684 | test | DEBUG | pool-5-thread-29 | [task:_compare:2649] Comparing 2705555 == 0 2021-06-08 06:30:24,684 | test | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 2705555 == 0. Received: {'172.23.106.196': 2705555} for bucket 'default' 2021-06-08 06:30:24,684 | infra | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:30:24,686 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:30:24,686 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:30:24,894 | infra | DEBUG | pool-5-thread-1 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:30:25,148 | infra | DEBUG | pool-5-thread-2 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:30:25,311 | test | DEBUG | pool-5-thread-30 | [task:_compare:2649] Comparing 2607741 == 0 2021-06-08 06:30:25,311 | test | DEBUG | pool-5-thread-30 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 2607741 == 0. Received: {'172.23.106.199': 2607741} for bucket 'default' 2021-06-08 06:30:25,311 | infra | DEBUG | pool-5-thread-30 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:30:25,312 | infra | DEBUG | pool-5-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:30:25,312 | infra | DEBUG | pool-5-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:30:26,306 | test | DEBUG | pool-5-thread-1 | [task:_compare:2649] Comparing 2701294 == 0 2021-06-08 06:30:26,308 | test | DEBUG | pool-5-thread-1 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 2701294 == 0. Received: {'172.23.106.168': 2701294} for bucket 'default' 2021-06-08 06:30:26,308 | infra | DEBUG | pool-5-thread-1 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:30:26,309 | infra | DEBUG | pool-5-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:30:26,309 | infra | DEBUG | pool-5-thread-1 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:30:26,552 | test | DEBUG | pool-5-thread-2 | [task:_compare:2649] Comparing 2649159 == 0 2021-06-08 06:30:26,552 | test | DEBUG | pool-5-thread-2 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 2649159 == 0. Received: {'172.23.106.198': 2649159} for bucket 'default' 2021-06-08 06:30:26,552 | infra | DEBUG | pool-5-thread-2 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:30:26,555 | infra | DEBUG | pool-5-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:30:26,555 | infra | DEBUG | pool-5-thread-2 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:30:29,686 | infra | DEBUG | pool-5-thread-29 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:30:30,312 | infra | DEBUG | pool-5-thread-30 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:30:31,055 | test | DEBUG | pool-5-thread-29 | [task:_compare:2649] Comparing 2603588 == 0 2021-06-08 06:30:31,055 | test | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 2603588 == 0. Received: {'172.23.106.196': 2603588} for bucket 'default' 2021-06-08 06:30:31,055 | infra | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:30:31,056 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:30:31,058 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:30:31,309 | infra | DEBUG | pool-5-thread-1 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:30:31,555 | infra | DEBUG | pool-5-thread-2 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:30:31,707 | test | DEBUG | pool-5-thread-30 | [task:_compare:2649] Comparing 2471054 == 0 2021-06-08 06:30:31,707 | test | DEBUG | pool-5-thread-30 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 2471054 == 0. Received: {'172.23.106.199': 2471054} for bucket 'default' 2021-06-08 06:30:31,707 | infra | DEBUG | pool-5-thread-30 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:30:31,710 | infra | DEBUG | pool-5-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:30:31,710 | infra | DEBUG | pool-5-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:30:32,709 | test | DEBUG | pool-5-thread-1 | [task:_compare:2649] Comparing 2585192 == 0 2021-06-08 06:30:32,709 | test | DEBUG | pool-5-thread-1 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 2585192 == 0. Received: {'172.23.106.168': 2585192} for bucket 'default' 2021-06-08 06:30:32,709 | infra | DEBUG | pool-5-thread-1 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:30:32,710 | infra | DEBUG | pool-5-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:30:32,711 | infra | DEBUG | pool-5-thread-1 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:30:32,944 | test | DEBUG | pool-5-thread-2 | [task:_compare:2649] Comparing 2554752 == 0 2021-06-08 06:30:32,944 | test | DEBUG | pool-5-thread-2 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 2554752 == 0. Received: {'172.23.106.198': 2554752} for bucket 'default' 2021-06-08 06:30:32,944 | infra | DEBUG | pool-5-thread-2 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:30:32,946 | infra | DEBUG | pool-5-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:30:32,946 | infra | DEBUG | pool-5-thread-2 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:30:36,058 | infra | DEBUG | pool-5-thread-29 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:30:36,710 | infra | DEBUG | pool-5-thread-30 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:30:37,397 | test | DEBUG | pool-5-thread-29 | [task:_compare:2649] Comparing 2515247 == 0 2021-06-08 06:30:37,397 | test | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 2515247 == 0. Received: {'172.23.106.196': 2515247} for bucket 'default' 2021-06-08 06:30:37,397 | infra | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:30:37,398 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:30:37,398 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:30:37,711 | infra | DEBUG | pool-5-thread-1 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:30:37,946 | infra | DEBUG | pool-5-thread-2 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:30:38,098 | test | DEBUG | pool-5-thread-30 | [task:_compare:2649] Comparing 2390446 == 0 2021-06-08 06:30:38,099 | test | DEBUG | pool-5-thread-30 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 2390446 == 0. Received: {'172.23.106.199': 2390446} for bucket 'default' 2021-06-08 06:30:38,099 | infra | DEBUG | pool-5-thread-30 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:30:38,101 | infra | DEBUG | pool-5-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:30:38,101 | infra | DEBUG | pool-5-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:30:39,101 | test | DEBUG | pool-5-thread-1 | [task:_compare:2649] Comparing 2508276 == 0 2021-06-08 06:30:39,101 | test | DEBUG | pool-5-thread-1 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 2508276 == 0. Received: {'172.23.106.168': 2508276} for bucket 'default' 2021-06-08 06:30:39,101 | infra | DEBUG | pool-5-thread-1 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:30:39,104 | infra | DEBUG | pool-5-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:30:39,104 | infra | DEBUG | pool-5-thread-1 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:30:39,305 | test | DEBUG | pool-5-thread-2 | [task:_compare:2649] Comparing 2435110 == 0 2021-06-08 06:30:39,305 | test | DEBUG | pool-5-thread-2 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 2435110 == 0. Received: {'172.23.106.198': 2435110} for bucket 'default' 2021-06-08 06:30:39,305 | infra | DEBUG | pool-5-thread-2 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:30:39,306 | infra | DEBUG | pool-5-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:30:39,306 | infra | DEBUG | pool-5-thread-2 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:30:42,400 | infra | DEBUG | pool-5-thread-29 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:30:43,101 | infra | DEBUG | pool-5-thread-30 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:30:43,766 | test | DEBUG | pool-5-thread-29 | [task:_compare:2649] Comparing 2437874 == 0 2021-06-08 06:30:43,766 | test | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 2437874 == 0. Received: {'172.23.106.196': 2437874} for bucket 'default' 2021-06-08 06:30:43,766 | infra | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:30:43,767 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:30:43,767 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:30:44,105 | infra | DEBUG | pool-5-thread-1 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:30:44,306 | infra | DEBUG | pool-5-thread-2 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:30:44,492 | test | DEBUG | pool-5-thread-30 | [task:_compare:2649] Comparing 2354174 == 0 2021-06-08 06:30:44,493 | test | DEBUG | pool-5-thread-30 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 2354174 == 0. Received: {'172.23.106.199': 2354174} for bucket 'default' 2021-06-08 06:30:44,493 | infra | DEBUG | pool-5-thread-30 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:30:44,494 | infra | DEBUG | pool-5-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:30:44,494 | infra | DEBUG | pool-5-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:30:45,477 | test | DEBUG | pool-5-thread-1 | [task:_compare:2649] Comparing 2424564 == 0 2021-06-08 06:30:45,477 | test | DEBUG | pool-5-thread-1 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 2424564 == 0. Received: {'172.23.106.168': 2424564} for bucket 'default' 2021-06-08 06:30:45,477 | infra | DEBUG | pool-5-thread-1 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:30:45,480 | infra | DEBUG | pool-5-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:30:45,480 | infra | DEBUG | pool-5-thread-1 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:30:45,678 | test | DEBUG | pool-5-thread-2 | [task:_compare:2649] Comparing 2350386 == 0 2021-06-08 06:30:45,678 | test | DEBUG | pool-5-thread-2 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 2350386 == 0. Received: {'172.23.106.198': 2350386} for bucket 'default' 2021-06-08 06:30:45,678 | infra | DEBUG | pool-5-thread-2 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:30:45,680 | infra | DEBUG | pool-5-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:30:45,680 | infra | DEBUG | pool-5-thread-2 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:30:48,769 | infra | DEBUG | pool-5-thread-29 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:30:49,494 | infra | DEBUG | pool-5-thread-30 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:30:50,151 | test | DEBUG | pool-5-thread-29 | [task:_compare:2649] Comparing 2396655 == 0 2021-06-08 06:30:50,153 | test | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 2396655 == 0. Received: {'172.23.106.196': 2396655} for bucket 'default' 2021-06-08 06:30:50,153 | infra | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:30:50,154 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:30:50,154 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:30:50,480 | infra | DEBUG | pool-5-thread-1 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:30:50,680 | infra | DEBUG | pool-5-thread-2 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:30:50,930 | test | DEBUG | pool-5-thread-30 | [task:_compare:2649] Comparing 2301034 == 0 2021-06-08 06:30:50,930 | test | DEBUG | pool-5-thread-30 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 2301034 == 0. Received: {'172.23.106.199': 2301034} for bucket 'default' 2021-06-08 06:30:50,930 | infra | DEBUG | pool-5-thread-30 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:30:50,933 | infra | DEBUG | pool-5-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:30:50,933 | infra | DEBUG | pool-5-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:30:51,861 | test | DEBUG | pool-5-thread-1 | [task:_compare:2649] Comparing 2383692 == 0 2021-06-08 06:30:51,861 | test | DEBUG | pool-5-thread-1 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 2383692 == 0. Received: {'172.23.106.168': 2383692} for bucket 'default' 2021-06-08 06:30:51,861 | infra | DEBUG | pool-5-thread-1 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:30:51,862 | infra | DEBUG | pool-5-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:30:51,862 | infra | DEBUG | pool-5-thread-1 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:30:52,045 | test | DEBUG | pool-5-thread-2 | [task:_compare:2649] Comparing 2327203 == 0 2021-06-08 06:30:52,046 | test | DEBUG | pool-5-thread-2 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 2327203 == 0. Received: {'172.23.106.198': 2327203} for bucket 'default' 2021-06-08 06:30:52,046 | infra | DEBUG | pool-5-thread-2 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:30:52,048 | infra | DEBUG | pool-5-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:30:52,048 | infra | DEBUG | pool-5-thread-2 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:30:55,154 | infra | DEBUG | pool-5-thread-29 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:30:55,933 | infra | DEBUG | pool-5-thread-30 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:30:56,499 | test | DEBUG | pool-5-thread-29 | [task:_compare:2649] Comparing 2333996 == 0 2021-06-08 06:30:56,500 | test | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 2333996 == 0. Received: {'172.23.106.196': 2333996} for bucket 'default' 2021-06-08 06:30:56,500 | infra | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:30:56,500 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:30:56,502 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:30:56,862 | infra | DEBUG | pool-5-thread-1 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:30:57,048 | infra | DEBUG | pool-5-thread-2 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:30:57,326 | test | DEBUG | pool-5-thread-30 | [task:_compare:2649] Comparing 2282280 == 0 2021-06-08 06:30:57,328 | test | DEBUG | pool-5-thread-30 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 2282280 == 0. Received: {'172.23.106.199': 2282280} for bucket 'default' 2021-06-08 06:30:57,328 | infra | DEBUG | pool-5-thread-30 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:30:57,328 | infra | DEBUG | pool-5-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:30:57,329 | infra | DEBUG | pool-5-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:30:58,293 | test | DEBUG | pool-5-thread-1 | [task:_compare:2649] Comparing 2285099 == 0 2021-06-08 06:30:58,293 | test | DEBUG | pool-5-thread-1 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 2285099 == 0. Received: {'172.23.106.168': 2285099} for bucket 'default' 2021-06-08 06:30:58,293 | infra | DEBUG | pool-5-thread-1 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:30:58,296 | infra | DEBUG | pool-5-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:30:58,296 | infra | DEBUG | pool-5-thread-1 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:30:58,434 | test | DEBUG | pool-5-thread-2 | [task:_compare:2649] Comparing 2273235 == 0 2021-06-08 06:30:58,436 | test | DEBUG | pool-5-thread-2 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 2273235 == 0. Received: {'172.23.106.198': 2273235} for bucket 'default' 2021-06-08 06:30:58,436 | infra | DEBUG | pool-5-thread-2 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:30:58,437 | infra | DEBUG | pool-5-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:30:58,437 | infra | DEBUG | pool-5-thread-2 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:31:01,502 | infra | DEBUG | pool-5-thread-29 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:31:02,329 | infra | DEBUG | pool-5-thread-30 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:31:02,871 | test | DEBUG | pool-5-thread-29 | [task:_compare:2649] Comparing 2265961 == 0 2021-06-08 06:31:02,871 | test | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 2265961 == 0. Received: {'172.23.106.196': 2265961} for bucket 'default' 2021-06-08 06:31:02,872 | infra | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:31:02,875 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:31:02,875 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:31:03,296 | infra | DEBUG | pool-5-thread-1 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:31:03,437 | infra | DEBUG | pool-5-thread-2 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:31:03,727 | test | DEBUG | pool-5-thread-30 | [task:_compare:2649] Comparing 2279853 == 0 2021-06-08 06:31:03,727 | test | DEBUG | pool-5-thread-30 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 2279853 == 0. Received: {'172.23.106.199': 2279853} for bucket 'default' 2021-06-08 06:31:03,727 | infra | DEBUG | pool-5-thread-30 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:31:03,730 | infra | DEBUG | pool-5-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:31:03,730 | infra | DEBUG | pool-5-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:31:04,711 | test | DEBUG | pool-5-thread-1 | [task:_compare:2649] Comparing 2248705 == 0 2021-06-08 06:31:04,711 | test | DEBUG | pool-5-thread-1 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 2248705 == 0. Received: {'172.23.106.168': 2248705} for bucket 'default' 2021-06-08 06:31:04,711 | infra | DEBUG | pool-5-thread-1 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:31:04,713 | infra | DEBUG | pool-5-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:31:04,713 | infra | DEBUG | pool-5-thread-1 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:31:04,825 | test | DEBUG | pool-5-thread-2 | [task:_compare:2649] Comparing 2252930 == 0 2021-06-08 06:31:04,825 | test | DEBUG | pool-5-thread-2 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 2252930 == 0. Received: {'172.23.106.198': 2252930} for bucket 'default' 2021-06-08 06:31:04,825 | infra | DEBUG | pool-5-thread-2 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:31:04,826 | infra | DEBUG | pool-5-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:31:04,826 | infra | DEBUG | pool-5-thread-2 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:31:07,875 | infra | DEBUG | pool-5-thread-29 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:31:08,730 | infra | DEBUG | pool-5-thread-30 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:31:09,206 | test | DEBUG | pool-5-thread-29 | [task:_compare:2649] Comparing 2250939 == 0 2021-06-08 06:31:09,206 | test | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 2250939 == 0. Received: {'172.23.106.196': 2250939} for bucket 'default' 2021-06-08 06:31:09,206 | infra | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:31:09,207 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:31:09,207 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:31:09,714 | infra | DEBUG | pool-5-thread-1 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:31:09,826 | infra | DEBUG | pool-5-thread-2 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:31:10,065 | test | DEBUG | pool-5-thread-30 | [task:_compare:2649] Comparing 2243011 == 0 2021-06-08 06:31:10,065 | test | DEBUG | pool-5-thread-30 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 2243011 == 0. Received: {'172.23.106.199': 2243011} for bucket 'default' 2021-06-08 06:31:10,065 | infra | DEBUG | pool-5-thread-30 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:31:10,068 | infra | DEBUG | pool-5-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:31:10,068 | infra | DEBUG | pool-5-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:31:11,134 | test | DEBUG | pool-5-thread-1 | [task:_compare:2649] Comparing 2246076 == 0 2021-06-08 06:31:11,134 | test | DEBUG | pool-5-thread-1 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 2246076 == 0. Received: {'172.23.106.168': 2246076} for bucket 'default' 2021-06-08 06:31:11,134 | infra | DEBUG | pool-5-thread-1 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:31:11,137 | infra | DEBUG | pool-5-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:31:11,137 | infra | DEBUG | pool-5-thread-1 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:31:11,213 | test | DEBUG | pool-5-thread-2 | [task:_compare:2649] Comparing 2235074 == 0 2021-06-08 06:31:11,213 | test | DEBUG | pool-5-thread-2 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 2235074 == 0. Received: {'172.23.106.198': 2235074} for bucket 'default' 2021-06-08 06:31:11,213 | infra | DEBUG | pool-5-thread-2 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:31:11,214 | infra | DEBUG | pool-5-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:31:11,214 | infra | DEBUG | pool-5-thread-2 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:31:14,207 | infra | DEBUG | pool-5-thread-29 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:31:15,068 | infra | DEBUG | pool-5-thread-30 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:31:15,563 | test | DEBUG | pool-5-thread-29 | [task:_compare:2649] Comparing 2229755 == 0 2021-06-08 06:31:15,565 | test | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 2229755 == 0. Received: {'172.23.106.196': 2229755} for bucket 'default' 2021-06-08 06:31:15,565 | infra | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:31:15,565 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:31:15,566 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:31:16,137 | infra | DEBUG | pool-5-thread-1 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:31:16,216 | infra | DEBUG | pool-5-thread-2 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:31:16,411 | test | DEBUG | pool-5-thread-30 | [task:_compare:2649] Comparing 2251413 == 0 2021-06-08 06:31:16,411 | test | DEBUG | pool-5-thread-30 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 2251413 == 0. Received: {'172.23.106.199': 2251413} for bucket 'default' 2021-06-08 06:31:16,411 | infra | DEBUG | pool-5-thread-30 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:31:16,414 | infra | DEBUG | pool-5-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:31:16,414 | infra | DEBUG | pool-5-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:31:17,575 | test | DEBUG | pool-5-thread-1 | [task:_compare:2649] Comparing 2194760 == 0 2021-06-08 06:31:17,575 | test | DEBUG | pool-5-thread-1 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 2194760 == 0. Received: {'172.23.106.168': 2194760} for bucket 'default' 2021-06-08 06:31:17,575 | infra | DEBUG | pool-5-thread-1 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:31:17,578 | infra | DEBUG | pool-5-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:31:17,578 | infra | DEBUG | pool-5-thread-1 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:31:17,635 | test | DEBUG | pool-5-thread-2 | [task:_compare:2649] Comparing 2243005 == 0 2021-06-08 06:31:17,637 | test | DEBUG | pool-5-thread-2 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 2243005 == 0. Received: {'172.23.106.198': 2243005} for bucket 'default' 2021-06-08 06:31:17,637 | infra | DEBUG | pool-5-thread-2 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:31:17,638 | infra | DEBUG | pool-5-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:31:17,638 | infra | DEBUG | pool-5-thread-2 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:31:20,566 | infra | DEBUG | pool-5-thread-29 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:31:21,414 | infra | DEBUG | pool-5-thread-30 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:31:21,885 | test | DEBUG | pool-5-thread-29 | [task:_compare:2649] Comparing 2224970 == 0 2021-06-08 06:31:21,885 | test | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 2224970 == 0. Received: {'172.23.106.196': 2224970} for bucket 'default' 2021-06-08 06:31:21,885 | infra | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:31:21,888 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:31:21,888 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:31:22,578 | infra | DEBUG | pool-5-thread-1 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:31:22,638 | infra | DEBUG | pool-5-thread-2 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:31:22,779 | test | DEBUG | pool-5-thread-30 | [task:_compare:2649] Comparing 2247075 == 0 2021-06-08 06:31:22,780 | test | DEBUG | pool-5-thread-30 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 2247075 == 0. Received: {'172.23.106.199': 2247075} for bucket 'default' 2021-06-08 06:31:22,780 | infra | DEBUG | pool-5-thread-30 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:31:22,782 | infra | DEBUG | pool-5-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:31:22,782 | infra | DEBUG | pool-5-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:31:24,038 | test | DEBUG | pool-5-thread-1 | [task:_compare:2649] Comparing 2178006 == 0 2021-06-08 06:31:24,038 | test | DEBUG | pool-5-thread-1 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 2178006 == 0. Received: {'172.23.106.168': 2178006} for bucket 'default' 2021-06-08 06:31:24,039 | infra | DEBUG | pool-5-thread-1 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:31:24,040 | infra | DEBUG | pool-5-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:31:24,040 | infra | DEBUG | pool-5-thread-1 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:31:24,072 | test | DEBUG | pool-5-thread-2 | [task:_compare:2649] Comparing 2216565 == 0 2021-06-08 06:31:24,072 | test | DEBUG | pool-5-thread-2 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 2216565 == 0. Received: {'172.23.106.198': 2216565} for bucket 'default' 2021-06-08 06:31:24,072 | infra | DEBUG | pool-5-thread-2 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:31:24,075 | infra | DEBUG | pool-5-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:31:24,075 | infra | DEBUG | pool-5-thread-2 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:31:26,888 | infra | DEBUG | pool-5-thread-29 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:31:27,782 | infra | DEBUG | pool-5-thread-30 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:31:28,174 | test | DEBUG | pool-5-thread-29 | [task:_compare:2649] Comparing 2183159 == 0 2021-06-08 06:31:28,174 | test | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 2183159 == 0. Received: {'172.23.106.196': 2183159} for bucket 'default' 2021-06-08 06:31:28,174 | infra | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:31:28,177 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:31:28,177 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:31:29,040 | infra | DEBUG | pool-5-thread-1 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:31:29,075 | infra | DEBUG | pool-5-thread-2 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:31:29,292 | test | DEBUG | pool-5-thread-30 | [task:_compare:2649] Comparing 2242722 == 0 2021-06-08 06:31:29,292 | test | DEBUG | pool-5-thread-30 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 2242722 == 0. Received: {'172.23.106.199': 2242722} for bucket 'default' 2021-06-08 06:31:29,292 | infra | DEBUG | pool-5-thread-30 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:31:29,293 | infra | DEBUG | pool-5-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:31:29,293 | infra | DEBUG | pool-5-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:31:30,490 | test | DEBUG | pool-5-thread-1 | [task:_compare:2649] Comparing 2198341 == 0 2021-06-08 06:31:30,490 | test | DEBUG | pool-5-thread-1 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 2198341 == 0. Received: {'172.23.106.168': 2198341} for bucket 'default' 2021-06-08 06:31:30,490 | infra | DEBUG | pool-5-thread-1 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:31:30,493 | infra | DEBUG | pool-5-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:31:30,493 | infra | DEBUG | pool-5-thread-1 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:31:30,507 | test | DEBUG | pool-5-thread-2 | [task:_compare:2649] Comparing 2192430 == 0 2021-06-08 06:31:30,507 | test | DEBUG | pool-5-thread-2 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 2192430 == 0. Received: {'172.23.106.198': 2192430} for bucket 'default' 2021-06-08 06:31:30,507 | infra | DEBUG | pool-5-thread-2 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:31:30,509 | infra | DEBUG | pool-5-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:31:30,509 | infra | DEBUG | pool-5-thread-2 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:31:33,177 | infra | DEBUG | pool-5-thread-29 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:31:34,295 | infra | DEBUG | pool-5-thread-30 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:31:34,493 | test | DEBUG | pool-5-thread-29 | [task:_compare:2649] Comparing 2065161 == 0 2021-06-08 06:31:34,493 | test | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 2065161 == 0. Received: {'172.23.106.196': 2065161} for bucket 'default' 2021-06-08 06:31:34,493 | infra | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:31:34,494 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:31:34,494 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:31:35,493 | infra | DEBUG | pool-5-thread-1 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:31:35,509 | infra | DEBUG | pool-5-thread-2 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:31:35,648 | test | DEBUG | pool-5-thread-30 | [task:_compare:2649] Comparing 2131627 == 0 2021-06-08 06:31:35,648 | test | DEBUG | pool-5-thread-30 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 2131627 == 0. Received: {'172.23.106.199': 2131627} for bucket 'default' 2021-06-08 06:31:35,650 | infra | DEBUG | pool-5-thread-30 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:31:35,651 | infra | DEBUG | pool-5-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:31:35,651 | infra | DEBUG | pool-5-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:31:36,884 | test | DEBUG | pool-5-thread-2 | [task:_compare:2649] Comparing 2073912 == 0 2021-06-08 06:31:36,884 | test | DEBUG | pool-5-thread-2 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 2073912 == 0. Received: {'172.23.106.198': 2073912} for bucket 'default' 2021-06-08 06:31:36,884 | infra | DEBUG | pool-5-thread-2 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:31:36,885 | infra | DEBUG | pool-5-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:31:36,885 | infra | DEBUG | pool-5-thread-2 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:31:36,898 | test | DEBUG | pool-5-thread-1 | [task:_compare:2649] Comparing 2117402 == 0 2021-06-08 06:31:36,898 | test | DEBUG | pool-5-thread-1 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 2117402 == 0. Received: {'172.23.106.168': 2117402} for bucket 'default' 2021-06-08 06:31:36,898 | infra | DEBUG | pool-5-thread-1 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:31:36,900 | infra | DEBUG | pool-5-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:31:36,900 | infra | DEBUG | pool-5-thread-1 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:31:39,496 | infra | DEBUG | pool-5-thread-29 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:31:40,651 | infra | DEBUG | pool-5-thread-30 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:31:40,836 | test | DEBUG | pool-5-thread-29 | [task:_compare:2649] Comparing 1947861 == 0 2021-06-08 06:31:40,836 | test | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 1947861 == 0. Received: {'172.23.106.196': 1947861} for bucket 'default' 2021-06-08 06:31:40,836 | infra | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:31:40,838 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:31:40,838 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:31:41,887 | infra | DEBUG | pool-5-thread-2 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:31:41,901 | infra | DEBUG | pool-5-thread-1 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:31:41,973 | test | DEBUG | pool-5-thread-30 | [task:_compare:2649] Comparing 2015091 == 0 2021-06-08 06:31:41,973 | test | DEBUG | pool-5-thread-30 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 2015091 == 0. Received: {'172.23.106.199': 2015091} for bucket 'default' 2021-06-08 06:31:41,973 | infra | DEBUG | pool-5-thread-30 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:31:41,976 | infra | DEBUG | pool-5-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:31:41,976 | infra | DEBUG | pool-5-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:31:43,286 | test | DEBUG | pool-5-thread-2 | [task:_compare:2649] Comparing 1969791 == 0 2021-06-08 06:31:43,286 | test | DEBUG | pool-5-thread-2 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 1969791 == 0. Received: {'172.23.106.198': 1969791} for bucket 'default' 2021-06-08 06:31:43,286 | infra | DEBUG | pool-5-thread-2 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:31:43,289 | infra | DEBUG | pool-5-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:31:43,289 | infra | DEBUG | pool-5-thread-2 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:31:43,293 | test | DEBUG | pool-5-thread-1 | [task:_compare:2649] Comparing 2027161 == 0 2021-06-08 06:31:43,293 | test | DEBUG | pool-5-thread-1 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 2027161 == 0. Received: {'172.23.106.168': 2027161} for bucket 'default' 2021-06-08 06:31:43,293 | infra | DEBUG | pool-5-thread-1 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:31:43,296 | infra | DEBUG | pool-5-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:31:43,296 | infra | DEBUG | pool-5-thread-1 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:31:45,838 | infra | DEBUG | pool-5-thread-29 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:31:46,976 | infra | DEBUG | pool-5-thread-30 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:31:47,109 | test | DEBUG | pool-5-thread-29 | [task:_compare:2649] Comparing 1846878 == 0 2021-06-08 06:31:47,109 | test | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 1846878 == 0. Received: {'172.23.106.196': 1846878} for bucket 'default' 2021-06-08 06:31:47,109 | infra | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:31:47,111 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:31:47,111 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:31:48,279 | test | DEBUG | pool-5-thread-30 | [task:_compare:2649] Comparing 1909097 == 0 2021-06-08 06:31:48,279 | test | DEBUG | pool-5-thread-30 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 1909097 == 0. Received: {'172.23.106.199': 1909097} for bucket 'default' 2021-06-08 06:31:48,279 | infra | DEBUG | pool-5-thread-30 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:31:48,282 | infra | DEBUG | pool-5-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:31:48,282 | infra | DEBUG | pool-5-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:31:48,289 | infra | DEBUG | pool-5-thread-2 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:31:48,296 | infra | DEBUG | pool-5-thread-1 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:31:49,668 | test | DEBUG | pool-5-thread-1 | [task:_compare:2649] Comparing 1909471 == 0 2021-06-08 06:31:49,668 | test | DEBUG | pool-5-thread-1 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 1909471 == 0. Received: {'172.23.106.168': 1909471} for bucket 'default' 2021-06-08 06:31:49,670 | infra | DEBUG | pool-5-thread-1 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:31:49,671 | infra | DEBUG | pool-5-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:31:49,671 | infra | DEBUG | pool-5-thread-1 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:31:49,687 | test | DEBUG | pool-5-thread-2 | [task:_compare:2649] Comparing 1877250 == 0 2021-06-08 06:31:49,687 | test | DEBUG | pool-5-thread-2 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 1877250 == 0. Received: {'172.23.106.198': 1877250} for bucket 'default' 2021-06-08 06:31:49,688 | infra | DEBUG | pool-5-thread-2 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:31:49,690 | infra | DEBUG | pool-5-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:31:49,690 | infra | DEBUG | pool-5-thread-2 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:31:52,112 | infra | DEBUG | pool-5-thread-29 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:31:53,282 | infra | DEBUG | pool-5-thread-30 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:31:53,382 | test | DEBUG | pool-5-thread-29 | [task:_compare:2649] Comparing 1738482 == 0 2021-06-08 06:31:53,384 | test | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 1738482 == 0. Received: {'172.23.106.196': 1738482} for bucket 'default' 2021-06-08 06:31:53,384 | infra | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:31:53,384 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:31:53,385 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:31:54,654 | test | DEBUG | pool-5-thread-30 | [task:_compare:2649] Comparing 1801449 == 0 2021-06-08 06:31:54,654 | test | DEBUG | pool-5-thread-30 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 1801449 == 0. Received: {'172.23.106.199': 1801449} for bucket 'default' 2021-06-08 06:31:54,654 | infra | DEBUG | pool-5-thread-30 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:31:54,657 | infra | DEBUG | pool-5-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:31:54,657 | infra | DEBUG | pool-5-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:31:54,671 | infra | DEBUG | pool-5-thread-1 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:31:54,690 | infra | DEBUG | pool-5-thread-2 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:31:56,023 | test | DEBUG | pool-5-thread-2 | [task:_compare:2649] Comparing 1784386 == 0 2021-06-08 06:31:56,023 | test | DEBUG | pool-5-thread-2 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 1784386 == 0. Received: {'172.23.106.198': 1784386} for bucket 'default' 2021-06-08 06:31:56,023 | infra | DEBUG | pool-5-thread-2 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:31:56,026 | infra | DEBUG | pool-5-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:31:56,026 | infra | DEBUG | pool-5-thread-2 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:31:56,069 | test | DEBUG | pool-5-thread-1 | [task:_compare:2649] Comparing 1827887 == 0 2021-06-08 06:31:56,071 | test | DEBUG | pool-5-thread-1 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 1827887 == 0. Received: {'172.23.106.168': 1827887} for bucket 'default' 2021-06-08 06:31:56,071 | infra | DEBUG | pool-5-thread-1 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:31:56,072 | infra | DEBUG | pool-5-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:31:56,072 | infra | DEBUG | pool-5-thread-1 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:31:58,385 | infra | DEBUG | pool-5-thread-29 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:31:59,657 | test | DEBUG | pool-5-thread-29 | [task:_compare:2649] Comparing 1625843 == 0 2021-06-08 06:31:59,657 | test | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 1625843 == 0. Received: {'172.23.106.196': 1625843} for bucket 'default' 2021-06-08 06:31:59,657 | infra | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:31:59,657 | infra | DEBUG | pool-5-thread-30 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:31:59,660 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:31:59,660 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:32:00,937 | test | DEBUG | pool-5-thread-30 | [task:_compare:2649] Comparing 1713746 == 0 2021-06-08 06:32:00,938 | test | DEBUG | pool-5-thread-30 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 1713746 == 0. Received: {'172.23.106.199': 1713746} for bucket 'default' 2021-06-08 06:32:00,938 | infra | DEBUG | pool-5-thread-30 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:32:00,940 | infra | DEBUG | pool-5-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:32:00,940 | infra | DEBUG | pool-5-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:32:01,026 | infra | DEBUG | pool-5-thread-2 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:32:01,072 | infra | DEBUG | pool-5-thread-1 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:32:02,262 | test | DEBUG | pool-5-thread-2 | [task:_compare:2649] Comparing 1679882 == 0 2021-06-08 06:32:02,263 | test | DEBUG | pool-5-thread-2 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 1679882 == 0. Received: {'172.23.106.198': 1679882} for bucket 'default' 2021-06-08 06:32:02,263 | infra | DEBUG | pool-5-thread-2 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:32:02,265 | infra | DEBUG | pool-5-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:32:02,265 | infra | DEBUG | pool-5-thread-2 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:32:02,700 | test | DEBUG | pool-5-thread-1 | [task:_compare:2649] Comparing 1735881 == 0 2021-06-08 06:32:02,701 | test | DEBUG | pool-5-thread-1 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 1735881 == 0. Received: {'172.23.106.168': 1735881} for bucket 'default' 2021-06-08 06:32:02,701 | infra | DEBUG | pool-5-thread-1 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:32:02,703 | infra | DEBUG | pool-5-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:32:02,703 | infra | DEBUG | pool-5-thread-1 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:32:04,660 | infra | DEBUG | pool-5-thread-29 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:32:05,917 | test | DEBUG | pool-5-thread-29 | [task:_compare:2649] Comparing 1533666 == 0 2021-06-08 06:32:05,917 | test | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 1533666 == 0. Received: {'172.23.106.196': 1533666} for bucket 'default' 2021-06-08 06:32:05,918 | infra | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:32:05,920 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:32:05,920 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:32:05,940 | infra | DEBUG | pool-5-thread-30 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:32:07,213 | test | DEBUG | pool-5-thread-30 | [task:_compare:2649] Comparing 1598909 == 0 2021-06-08 06:32:07,213 | test | DEBUG | pool-5-thread-30 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 1598909 == 0. Received: {'172.23.106.199': 1598909} for bucket 'default' 2021-06-08 06:32:07,213 | infra | DEBUG | pool-5-thread-30 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:32:07,216 | infra | DEBUG | pool-5-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:32:07,216 | infra | DEBUG | pool-5-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:32:07,265 | infra | DEBUG | pool-5-thread-2 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:32:07,703 | infra | DEBUG | pool-5-thread-1 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:32:08,499 | test | DEBUG | pool-5-thread-2 | [task:_compare:2649] Comparing 1594875 == 0 2021-06-08 06:32:08,500 | test | DEBUG | pool-5-thread-2 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 1594875 == 0. Received: {'172.23.106.198': 1594875} for bucket 'default' 2021-06-08 06:32:08,500 | infra | DEBUG | pool-5-thread-2 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:32:08,502 | infra | DEBUG | pool-5-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:32:08,502 | infra | DEBUG | pool-5-thread-2 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:32:08,953 | test | DEBUG | pool-5-thread-1 | [task:_compare:2649] Comparing 1619193 == 0 2021-06-08 06:32:08,954 | test | DEBUG | pool-5-thread-1 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 1619193 == 0. Received: {'172.23.106.168': 1619193} for bucket 'default' 2021-06-08 06:32:08,954 | infra | DEBUG | pool-5-thread-1 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:32:08,956 | infra | DEBUG | pool-5-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:32:08,957 | infra | DEBUG | pool-5-thread-1 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:32:10,921 | infra | DEBUG | pool-5-thread-29 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:32:12,180 | test | DEBUG | pool-5-thread-29 | [task:_compare:2649] Comparing 1454871 == 0 2021-06-08 06:32:12,180 | test | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 1454871 == 0. Received: {'172.23.106.196': 1454871} for bucket 'default' 2021-06-08 06:32:12,180 | infra | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:32:12,183 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:32:12,183 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:32:12,216 | infra | DEBUG | pool-5-thread-30 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:32:13,466 | test | DEBUG | pool-5-thread-30 | [task:_compare:2649] Comparing 1487016 == 0 2021-06-08 06:32:13,466 | test | DEBUG | pool-5-thread-30 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 1487016 == 0. Received: {'172.23.106.199': 1487016} for bucket 'default' 2021-06-08 06:32:13,466 | infra | DEBUG | pool-5-thread-30 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:32:13,469 | infra | DEBUG | pool-5-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:32:13,469 | infra | DEBUG | pool-5-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:32:13,502 | infra | DEBUG | pool-5-thread-2 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:32:13,957 | infra | DEBUG | pool-5-thread-1 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:32:14,770 | test | DEBUG | pool-5-thread-2 | [task:_compare:2649] Comparing 1497847 == 0 2021-06-08 06:32:14,770 | test | DEBUG | pool-5-thread-2 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 1497847 == 0. Received: {'172.23.106.198': 1497847} for bucket 'default' 2021-06-08 06:32:14,770 | infra | DEBUG | pool-5-thread-2 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:32:14,773 | infra | DEBUG | pool-5-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:32:14,773 | infra | DEBUG | pool-5-thread-2 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:32:15,227 | test | DEBUG | pool-5-thread-1 | [task:_compare:2649] Comparing 1528268 == 0 2021-06-08 06:32:15,227 | test | DEBUG | pool-5-thread-1 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 1528268 == 0. Received: {'172.23.106.168': 1528268} for bucket 'default' 2021-06-08 06:32:15,229 | infra | DEBUG | pool-5-thread-1 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:32:15,230 | infra | DEBUG | pool-5-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:32:15,230 | infra | DEBUG | pool-5-thread-1 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:32:17,183 | infra | DEBUG | pool-5-thread-29 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:32:18,433 | test | DEBUG | pool-5-thread-29 | [task:_compare:2649] Comparing 1354210 == 0 2021-06-08 06:32:18,433 | test | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 1354210 == 0. Received: {'172.23.106.196': 1354210} for bucket 'default' 2021-06-08 06:32:18,433 | infra | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:32:18,434 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:32:18,434 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:32:18,469 | infra | DEBUG | pool-5-thread-30 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:32:19,726 | test | DEBUG | pool-5-thread-30 | [task:_compare:2649] Comparing 1389564 == 0 2021-06-08 06:32:19,726 | test | DEBUG | pool-5-thread-30 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 1389564 == 0. Received: {'172.23.106.199': 1389564} for bucket 'default' 2021-06-08 06:32:19,726 | infra | DEBUG | pool-5-thread-30 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:32:19,729 | infra | DEBUG | pool-5-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:32:19,730 | infra | DEBUG | pool-5-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:32:19,773 | infra | DEBUG | pool-5-thread-2 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:32:20,230 | infra | DEBUG | pool-5-thread-1 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:32:20,984 | test | DEBUG | pool-5-thread-2 | [task:_compare:2649] Comparing 1417868 == 0 2021-06-08 06:32:20,984 | test | DEBUG | pool-5-thread-2 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 1417868 == 0. Received: {'172.23.106.198': 1417868} for bucket 'default' 2021-06-08 06:32:20,984 | infra | DEBUG | pool-5-thread-2 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:32:20,986 | infra | DEBUG | pool-5-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:32:20,986 | infra | DEBUG | pool-5-thread-2 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:32:21,467 | test | DEBUG | pool-5-thread-1 | [task:_compare:2649] Comparing 1430627 == 0 2021-06-08 06:32:21,467 | test | DEBUG | pool-5-thread-1 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 1430627 == 0. Received: {'172.23.106.168': 1430627} for bucket 'default' 2021-06-08 06:32:21,467 | infra | DEBUG | pool-5-thread-1 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:32:21,470 | infra | DEBUG | pool-5-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:32:21,470 | infra | DEBUG | pool-5-thread-1 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:32:23,436 | infra | DEBUG | pool-5-thread-29 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:32:24,681 | test | DEBUG | pool-5-thread-29 | [task:_compare:2649] Comparing 1261738 == 0 2021-06-08 06:32:24,681 | test | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 1261738 == 0. Received: {'172.23.106.196': 1261738} for bucket 'default' 2021-06-08 06:32:24,681 | infra | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:32:24,684 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:32:24,684 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:32:24,730 | infra | DEBUG | pool-5-thread-30 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:32:25,987 | infra | DEBUG | pool-5-thread-2 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:32:25,993 | test | DEBUG | pool-5-thread-30 | [task:_compare:2649] Comparing 1300305 == 0 2021-06-08 06:32:25,994 | test | DEBUG | pool-5-thread-30 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 1300305 == 0. Received: {'172.23.106.199': 1300305} for bucket 'default' 2021-06-08 06:32:25,994 | infra | DEBUG | pool-5-thread-30 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:32:25,996 | infra | DEBUG | pool-5-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:32:25,996 | infra | DEBUG | pool-5-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:32:26,470 | infra | DEBUG | pool-5-thread-1 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:32:27,187 | test | DEBUG | pool-5-thread-2 | [task:_compare:2649] Comparing 1317906 == 0 2021-06-08 06:32:27,187 | test | DEBUG | pool-5-thread-2 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 1317906 == 0. Received: {'172.23.106.198': 1317906} for bucket 'default' 2021-06-08 06:32:27,187 | infra | DEBUG | pool-5-thread-2 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:32:27,190 | infra | DEBUG | pool-5-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:32:27,190 | infra | DEBUG | pool-5-thread-2 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:32:27,730 | test | DEBUG | pool-5-thread-1 | [task:_compare:2649] Comparing 1318885 == 0 2021-06-08 06:32:27,730 | test | DEBUG | pool-5-thread-1 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 1318885 == 0. Received: {'172.23.106.168': 1318885} for bucket 'default' 2021-06-08 06:32:27,730 | infra | DEBUG | pool-5-thread-1 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:32:27,733 | infra | DEBUG | pool-5-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:32:27,733 | infra | DEBUG | pool-5-thread-1 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:32:29,684 | infra | DEBUG | pool-5-thread-29 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:32:30,924 | test | DEBUG | pool-5-thread-29 | [task:_compare:2649] Comparing 1175672 == 0 2021-06-08 06:32:30,924 | test | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 1175672 == 0. Received: {'172.23.106.196': 1175672} for bucket 'default' 2021-06-08 06:32:30,924 | infra | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:32:30,926 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:32:30,926 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:32:30,996 | infra | DEBUG | pool-5-thread-30 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:32:32,190 | infra | DEBUG | pool-5-thread-2 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:32:32,733 | infra | DEBUG | pool-5-thread-1 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:32:33,407 | test | DEBUG | pool-5-thread-30 | [task:_compare:2649] Comparing 1178872 == 0 2021-06-08 06:32:33,407 | test | DEBUG | pool-5-thread-30 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 1178872 == 0. Received: {'172.23.106.199': 1178872} for bucket 'default' 2021-06-08 06:32:33,407 | infra | DEBUG | pool-5-thread-30 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:32:33,410 | infra | DEBUG | pool-5-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:32:33,410 | infra | DEBUG | pool-5-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:32:33,461 | test | DEBUG | pool-5-thread-2 | [task:_compare:2649] Comparing 1242624 == 0 2021-06-08 06:32:33,461 | test | DEBUG | pool-5-thread-2 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 1242624 == 0. Received: {'172.23.106.198': 1242624} for bucket 'default' 2021-06-08 06:32:33,461 | infra | DEBUG | pool-5-thread-2 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:32:33,463 | infra | DEBUG | pool-5-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:32:33,463 | infra | DEBUG | pool-5-thread-2 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:32:33,956 | test | DEBUG | pool-5-thread-1 | [task:_compare:2649] Comparing 1216351 == 0 2021-06-08 06:32:33,956 | test | DEBUG | pool-5-thread-1 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 1216351 == 0. Received: {'172.23.106.168': 1216351} for bucket 'default' 2021-06-08 06:32:33,956 | infra | DEBUG | pool-5-thread-1 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:32:33,957 | infra | DEBUG | pool-5-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:32:33,957 | infra | DEBUG | pool-5-thread-1 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:32:35,927 | infra | DEBUG | pool-5-thread-29 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:32:37,946 | test | DEBUG | pool-5-thread-29 | [task:_compare:2649] Comparing 1084620 == 0 2021-06-08 06:32:37,947 | test | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 1084620 == 0. Received: {'172.23.106.196': 1084620} for bucket 'default' 2021-06-08 06:32:37,947 | infra | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:32:37,947 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:32:37,948 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:32:38,411 | infra | DEBUG | pool-5-thread-30 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:32:38,463 | infra | DEBUG | pool-5-thread-2 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:32:38,959 | infra | DEBUG | pool-5-thread-1 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:32:39,696 | test | DEBUG | pool-5-thread-2 | [task:_compare:2649] Comparing 1157004 == 0 2021-06-08 06:32:39,697 | test | DEBUG | pool-5-thread-2 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 1157004 == 0. Received: {'172.23.106.198': 1157004} for bucket 'default' 2021-06-08 06:32:39,697 | infra | DEBUG | pool-5-thread-2 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:32:39,698 | infra | DEBUG | pool-5-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:32:39,698 | infra | DEBUG | pool-5-thread-2 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:32:39,766 | test | DEBUG | pool-5-thread-30 | [task:_compare:2649] Comparing 1082488 == 0 2021-06-08 06:32:39,766 | test | DEBUG | pool-5-thread-30 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 1082488 == 0. Received: {'172.23.106.199': 1082488} for bucket 'default' 2021-06-08 06:32:39,766 | infra | DEBUG | pool-5-thread-30 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:32:39,767 | infra | DEBUG | pool-5-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:32:39,767 | infra | DEBUG | pool-5-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:32:40,167 | test | DEBUG | pool-5-thread-1 | [task:_compare:2649] Comparing 1110030 == 0 2021-06-08 06:32:40,167 | test | DEBUG | pool-5-thread-1 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 1110030 == 0. Received: {'172.23.106.168': 1110030} for bucket 'default' 2021-06-08 06:32:40,167 | infra | DEBUG | pool-5-thread-1 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:32:40,170 | infra | DEBUG | pool-5-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:32:40,170 | infra | DEBUG | pool-5-thread-1 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:32:42,948 | infra | DEBUG | pool-5-thread-29 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:32:44,323 | test | DEBUG | pool-5-thread-29 | [task:_compare:2649] Comparing 990526 == 0 2021-06-08 06:32:44,325 | test | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 990526 == 0. Received: {'172.23.106.196': 990526} for bucket 'default' 2021-06-08 06:32:44,325 | infra | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:32:44,325 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:32:44,326 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:32:44,698 | infra | DEBUG | pool-5-thread-2 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:32:44,769 | infra | DEBUG | pool-5-thread-30 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:32:45,170 | infra | DEBUG | pool-5-thread-1 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:32:45,963 | test | DEBUG | pool-5-thread-2 | [task:_compare:2649] Comparing 1071697 == 0 2021-06-08 06:32:45,963 | test | DEBUG | pool-5-thread-2 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 1071697 == 0. Received: {'172.23.106.198': 1071697} for bucket 'default' 2021-06-08 06:32:45,963 | infra | DEBUG | pool-5-thread-2 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:32:45,966 | infra | DEBUG | pool-5-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:32:45,967 | infra | DEBUG | pool-5-thread-2 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:32:46,033 | test | DEBUG | pool-5-thread-30 | [task:_compare:2649] Comparing 992309 == 0 2021-06-08 06:32:46,033 | test | DEBUG | pool-5-thread-30 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 992309 == 0. Received: {'172.23.106.199': 992309} for bucket 'default' 2021-06-08 06:32:46,033 | infra | DEBUG | pool-5-thread-30 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:32:46,036 | infra | DEBUG | pool-5-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:32:46,036 | infra | DEBUG | pool-5-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:32:46,407 | test | DEBUG | pool-5-thread-1 | [task:_compare:2649] Comparing 1003543 == 0 2021-06-08 06:32:46,407 | test | DEBUG | pool-5-thread-1 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 1003543 == 0. Received: {'172.23.106.168': 1003543} for bucket 'default' 2021-06-08 06:32:46,407 | infra | DEBUG | pool-5-thread-1 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:32:46,410 | infra | DEBUG | pool-5-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:32:46,410 | infra | DEBUG | pool-5-thread-1 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:32:49,326 | infra | DEBUG | pool-5-thread-29 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:32:50,543 | test | DEBUG | pool-5-thread-29 | [task:_compare:2649] Comparing 899112 == 0 2021-06-08 06:32:50,543 | test | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 899112 == 0. Received: {'172.23.106.196': 899112} for bucket 'default' 2021-06-08 06:32:50,543 | infra | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:32:50,546 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:32:50,546 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:32:50,967 | infra | DEBUG | pool-5-thread-2 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:32:51,036 | infra | DEBUG | pool-5-thread-30 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:32:51,410 | infra | DEBUG | pool-5-thread-1 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:32:52,197 | test | DEBUG | pool-5-thread-2 | [task:_compare:2649] Comparing 972770 == 0 2021-06-08 06:32:52,197 | test | DEBUG | pool-5-thread-2 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 972770 == 0. Received: {'172.23.106.198': 972770} for bucket 'default' 2021-06-08 06:32:52,197 | infra | DEBUG | pool-5-thread-2 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:32:52,200 | infra | DEBUG | pool-5-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:32:52,200 | infra | DEBUG | pool-5-thread-2 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:32:52,299 | test | DEBUG | pool-5-thread-30 | [task:_compare:2649] Comparing 888741 == 0 2021-06-08 06:32:52,301 | test | DEBUG | pool-5-thread-30 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 888741 == 0. Received: {'172.23.106.199': 888741} for bucket 'default' 2021-06-08 06:32:52,301 | infra | DEBUG | pool-5-thread-30 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:32:52,302 | infra | DEBUG | pool-5-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:32:52,302 | infra | DEBUG | pool-5-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:32:53,358 | test | DEBUG | pool-5-thread-1 | [task:_compare:2649] Comparing 890171 == 0 2021-06-08 06:32:53,358 | test | DEBUG | pool-5-thread-1 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 890171 == 0. Received: {'172.23.106.168': 890171} for bucket 'default' 2021-06-08 06:32:53,358 | infra | DEBUG | pool-5-thread-1 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:32:53,361 | infra | DEBUG | pool-5-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:32:53,361 | infra | DEBUG | pool-5-thread-1 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:32:55,546 | infra | DEBUG | pool-5-thread-29 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:32:56,750 | test | DEBUG | pool-5-thread-29 | [task:_compare:2649] Comparing 807931 == 0 2021-06-08 06:32:56,752 | test | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 807931 == 0. Received: {'172.23.106.196': 807931} for bucket 'default' 2021-06-08 06:32:56,752 | infra | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:32:56,753 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:32:56,753 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:32:57,200 | infra | DEBUG | pool-5-thread-2 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:32:57,302 | infra | DEBUG | pool-5-thread-30 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:32:58,361 | infra | DEBUG | pool-5-thread-1 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:32:58,424 | test | DEBUG | pool-5-thread-2 | [task:_compare:2649] Comparing 901612 == 0 2021-06-08 06:32:58,424 | test | DEBUG | pool-5-thread-2 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 901612 == 0. Received: {'172.23.106.198': 901612} for bucket 'default' 2021-06-08 06:32:58,424 | infra | DEBUG | pool-5-thread-2 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:32:58,427 | infra | DEBUG | pool-5-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:32:58,427 | infra | DEBUG | pool-5-thread-2 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:32:58,546 | test | DEBUG | pool-5-thread-30 | [task:_compare:2649] Comparing 803914 == 0 2021-06-08 06:32:58,546 | test | DEBUG | pool-5-thread-30 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 803914 == 0. Received: {'172.23.106.199': 803914} for bucket 'default' 2021-06-08 06:32:58,546 | infra | DEBUG | pool-5-thread-30 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:32:58,549 | infra | DEBUG | pool-5-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:32:58,549 | infra | DEBUG | pool-5-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:32:59,563 | test | DEBUG | pool-5-thread-1 | [task:_compare:2649] Comparing 803295 == 0 2021-06-08 06:32:59,565 | test | DEBUG | pool-5-thread-1 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 803295 == 0. Received: {'172.23.106.168': 803295} for bucket 'default' 2021-06-08 06:32:59,565 | infra | DEBUG | pool-5-thread-1 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:32:59,565 | infra | DEBUG | pool-5-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:32:59,566 | infra | DEBUG | pool-5-thread-1 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:33:01,753 | infra | DEBUG | pool-5-thread-29 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:33:02,954 | test | DEBUG | pool-5-thread-29 | [task:_compare:2649] Comparing 723300 == 0 2021-06-08 06:33:02,954 | test | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 723300 == 0. Received: {'172.23.106.196': 723300} for bucket 'default' 2021-06-08 06:33:02,954 | infra | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:33:02,957 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:33:02,957 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:33:03,427 | infra | DEBUG | pool-5-thread-2 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:33:03,549 | infra | DEBUG | pool-5-thread-30 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:33:04,566 | infra | DEBUG | pool-5-thread-1 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:33:04,638 | test | DEBUG | pool-5-thread-2 | [task:_compare:2649] Comparing 817631 == 0 2021-06-08 06:33:04,638 | test | DEBUG | pool-5-thread-2 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 817631 == 0. Received: {'172.23.106.198': 817631} for bucket 'default' 2021-06-08 06:33:04,640 | infra | DEBUG | pool-5-thread-2 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:33:04,641 | infra | DEBUG | pool-5-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:33:04,641 | infra | DEBUG | pool-5-thread-2 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:33:04,769 | test | DEBUG | pool-5-thread-30 | [task:_compare:2649] Comparing 700505 == 0 2021-06-08 06:33:04,769 | test | DEBUG | pool-5-thread-30 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 700505 == 0. Received: {'172.23.106.199': 700505} for bucket 'default' 2021-06-08 06:33:04,769 | infra | DEBUG | pool-5-thread-30 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:33:04,772 | infra | DEBUG | pool-5-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:33:04,772 | infra | DEBUG | pool-5-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:33:05,753 | test | DEBUG | pool-5-thread-1 | [task:_compare:2649] Comparing 708659 == 0 2021-06-08 06:33:05,755 | test | DEBUG | pool-5-thread-1 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 708659 == 0. Received: {'172.23.106.168': 708659} for bucket 'default' 2021-06-08 06:33:05,755 | infra | DEBUG | pool-5-thread-1 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:33:05,756 | infra | DEBUG | pool-5-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:33:05,756 | infra | DEBUG | pool-5-thread-1 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:33:07,957 | infra | DEBUG | pool-5-thread-29 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:33:09,157 | test | DEBUG | pool-5-thread-29 | [task:_compare:2649] Comparing 626221 == 0 2021-06-08 06:33:09,157 | test | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 626221 == 0. Received: {'172.23.106.196': 626221} for bucket 'default' 2021-06-08 06:33:09,157 | infra | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:33:09,158 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:33:09,158 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:33:09,641 | infra | DEBUG | pool-5-thread-2 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:33:09,772 | infra | DEBUG | pool-5-thread-30 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:33:10,756 | infra | DEBUG | pool-5-thread-1 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:33:10,848 | test | DEBUG | pool-5-thread-2 | [task:_compare:2649] Comparing 724730 == 0 2021-06-08 06:33:10,848 | test | DEBUG | pool-5-thread-2 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 724730 == 0. Received: {'172.23.106.198': 724730} for bucket 'default' 2021-06-08 06:33:10,848 | infra | DEBUG | pool-5-thread-2 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:33:10,851 | infra | DEBUG | pool-5-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:33:10,851 | infra | DEBUG | pool-5-thread-2 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:33:10,990 | test | DEBUG | pool-5-thread-30 | [task:_compare:2649] Comparing 632403 == 0 2021-06-08 06:33:10,990 | test | DEBUG | pool-5-thread-30 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 632403 == 0. Received: {'172.23.106.199': 632403} for bucket 'default' 2021-06-08 06:33:10,990 | infra | DEBUG | pool-5-thread-30 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:33:10,992 | infra | DEBUG | pool-5-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:33:10,992 | infra | DEBUG | pool-5-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:33:13,444 | test | DEBUG | pool-5-thread-1 | [task:_compare:2649] Comparing 598283 == 0 2021-06-08 06:33:13,444 | test | DEBUG | pool-5-thread-1 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 598283 == 0. Received: {'172.23.106.168': 598283} for bucket 'default' 2021-06-08 06:33:13,444 | infra | DEBUG | pool-5-thread-1 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:33:13,447 | infra | DEBUG | pool-5-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:33:13,447 | infra | DEBUG | pool-5-thread-1 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:33:14,160 | infra | DEBUG | pool-5-thread-29 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:33:15,351 | test | DEBUG | pool-5-thread-29 | [task:_compare:2649] Comparing 544011 == 0 2021-06-08 06:33:15,351 | test | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 544011 == 0. Received: {'172.23.106.196': 544011} for bucket 'default' 2021-06-08 06:33:15,351 | infra | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:33:15,352 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:33:15,352 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:33:15,851 | infra | DEBUG | pool-5-thread-2 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:33:15,992 | infra | DEBUG | pool-5-thread-30 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:33:17,062 | test | DEBUG | pool-5-thread-2 | [task:_compare:2649] Comparing 644500 == 0 2021-06-08 06:33:17,062 | test | DEBUG | pool-5-thread-2 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 644500 == 0. Received: {'172.23.106.198': 644500} for bucket 'default' 2021-06-08 06:33:17,062 | infra | DEBUG | pool-5-thread-2 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:33:17,065 | infra | DEBUG | pool-5-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:33:17,065 | infra | DEBUG | pool-5-thread-2 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:33:17,197 | test | DEBUG | pool-5-thread-30 | [task:_compare:2649] Comparing 543426 == 0 2021-06-08 06:33:17,197 | test | DEBUG | pool-5-thread-30 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 543426 == 0. Received: {'172.23.106.199': 543426} for bucket 'default' 2021-06-08 06:33:17,197 | infra | DEBUG | pool-5-thread-30 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:33:17,200 | infra | DEBUG | pool-5-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:33:17,200 | infra | DEBUG | pool-5-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:33:18,447 | infra | DEBUG | pool-5-thread-1 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:33:19,628 | test | DEBUG | pool-5-thread-1 | [task:_compare:2649] Comparing 518195 == 0 2021-06-08 06:33:19,628 | test | DEBUG | pool-5-thread-1 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 518195 == 0. Received: {'172.23.106.168': 518195} for bucket 'default' 2021-06-08 06:33:19,630 | infra | DEBUG | pool-5-thread-1 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:33:19,631 | infra | DEBUG | pool-5-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:33:19,631 | infra | DEBUG | pool-5-thread-1 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:33:20,354 | infra | DEBUG | pool-5-thread-29 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:33:22,065 | infra | DEBUG | pool-5-thread-2 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:33:22,200 | infra | DEBUG | pool-5-thread-30 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:33:22,427 | test | DEBUG | pool-5-thread-29 | [task:_compare:2649] Comparing 453602 == 0 2021-06-08 06:33:22,428 | test | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 453602 == 0. Received: {'172.23.106.196': 453602} for bucket 'default' 2021-06-08 06:33:22,428 | infra | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:33:22,430 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:33:22,430 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:33:23,262 | test | DEBUG | pool-5-thread-2 | [task:_compare:2649] Comparing 549311 == 0 2021-06-08 06:33:23,262 | test | DEBUG | pool-5-thread-2 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 549311 == 0. Received: {'172.23.106.198': 549311} for bucket 'default' 2021-06-08 06:33:23,262 | infra | DEBUG | pool-5-thread-2 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:33:23,263 | infra | DEBUG | pool-5-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:33:23,263 | infra | DEBUG | pool-5-thread-2 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:33:23,388 | test | DEBUG | pool-5-thread-30 | [task:_compare:2649] Comparing 468753 == 0 2021-06-08 06:33:23,388 | test | DEBUG | pool-5-thread-30 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 468753 == 0. Received: {'172.23.106.199': 468753} for bucket 'default' 2021-06-08 06:33:23,388 | infra | DEBUG | pool-5-thread-30 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:33:23,390 | infra | DEBUG | pool-5-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:33:23,390 | infra | DEBUG | pool-5-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:33:24,631 | infra | DEBUG | pool-5-thread-1 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:33:25,826 | test | DEBUG | pool-5-thread-1 | [task:_compare:2649] Comparing 441636 == 0 2021-06-08 06:33:25,828 | test | DEBUG | pool-5-thread-1 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 441636 == 0. Received: {'172.23.106.168': 441636} for bucket 'default' 2021-06-08 06:33:25,828 | infra | DEBUG | pool-5-thread-1 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:33:25,829 | infra | DEBUG | pool-5-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:33:25,829 | infra | DEBUG | pool-5-thread-1 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:33:27,430 | infra | DEBUG | pool-5-thread-29 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:33:28,263 | infra | DEBUG | pool-5-thread-2 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:33:28,390 | infra | DEBUG | pool-5-thread-30 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:33:28,637 | test | DEBUG | pool-5-thread-29 | [task:_compare:2649] Comparing 384595 == 0 2021-06-08 06:33:28,637 | test | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 384595 == 0. Received: {'172.23.106.196': 384595} for bucket 'default' 2021-06-08 06:33:28,637 | infra | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:33:28,638 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:33:28,638 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:33:29,489 | test | DEBUG | pool-5-thread-2 | [task:_compare:2649] Comparing 465767 == 0 2021-06-08 06:33:29,489 | test | DEBUG | pool-5-thread-2 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 465767 == 0. Received: {'172.23.106.198': 465767} for bucket 'default' 2021-06-08 06:33:29,489 | infra | DEBUG | pool-5-thread-2 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:33:29,490 | infra | DEBUG | pool-5-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:33:29,492 | infra | DEBUG | pool-5-thread-2 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:33:29,608 | test | DEBUG | pool-5-thread-30 | [task:_compare:2649] Comparing 377651 == 0 2021-06-08 06:33:29,608 | test | DEBUG | pool-5-thread-30 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 377651 == 0. Received: {'172.23.106.199': 377651} for bucket 'default' 2021-06-08 06:33:29,608 | infra | DEBUG | pool-5-thread-30 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:33:29,611 | infra | DEBUG | pool-5-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:33:29,611 | infra | DEBUG | pool-5-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:33:30,829 | infra | DEBUG | pool-5-thread-1 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:33:32,851 | test | DEBUG | pool-5-thread-1 | [task:_compare:2649] Comparing 352808 == 0 2021-06-08 06:33:32,852 | test | DEBUG | pool-5-thread-1 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 352808 == 0. Received: {'172.23.106.168': 352808} for bucket 'default' 2021-06-08 06:33:32,852 | infra | DEBUG | pool-5-thread-1 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:33:32,855 | infra | DEBUG | pool-5-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:33:32,855 | infra | DEBUG | pool-5-thread-1 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:33:33,640 | infra | DEBUG | pool-5-thread-29 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:33:34,492 | infra | DEBUG | pool-5-thread-2 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:33:34,611 | infra | DEBUG | pool-5-thread-30 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:33:34,806 | test | DEBUG | pool-5-thread-29 | [task:_compare:2649] Comparing 289189 == 0 2021-06-08 06:33:34,806 | test | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 289189 == 0. Received: {'172.23.106.196': 289189} for bucket 'default' 2021-06-08 06:33:34,806 | infra | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:33:34,809 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:33:34,809 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:33:35,694 | test | DEBUG | pool-5-thread-2 | [task:_compare:2649] Comparing 384591 == 0 2021-06-08 06:33:35,694 | test | DEBUG | pool-5-thread-2 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 384591 == 0. Received: {'172.23.106.198': 384591} for bucket 'default' 2021-06-08 06:33:35,694 | infra | DEBUG | pool-5-thread-2 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:33:35,696 | infra | DEBUG | pool-5-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:33:35,696 | infra | DEBUG | pool-5-thread-2 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:33:35,796 | test | DEBUG | pool-5-thread-30 | [task:_compare:2649] Comparing 307615 == 0 2021-06-08 06:33:35,796 | test | DEBUG | pool-5-thread-30 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 307615 == 0. Received: {'172.23.106.199': 307615} for bucket 'default' 2021-06-08 06:33:35,796 | infra | DEBUG | pool-5-thread-30 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:33:35,798 | infra | DEBUG | pool-5-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:33:35,799 | infra | DEBUG | pool-5-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:33:37,855 | infra | DEBUG | pool-5-thread-1 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:33:39,013 | test | DEBUG | pool-5-thread-1 | [task:_compare:2649] Comparing 268874 == 0 2021-06-08 06:33:39,015 | test | DEBUG | pool-5-thread-1 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 268874 == 0. Received: {'172.23.106.168': 268874} for bucket 'default' 2021-06-08 06:33:39,015 | infra | DEBUG | pool-5-thread-1 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:33:39,016 | infra | DEBUG | pool-5-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:33:39,016 | infra | DEBUG | pool-5-thread-1 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:33:39,809 | infra | DEBUG | pool-5-thread-29 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:33:40,697 | infra | DEBUG | pool-5-thread-2 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:33:40,799 | infra | DEBUG | pool-5-thread-30 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:33:40,970 | test | DEBUG | pool-5-thread-29 | [task:_compare:2649] Comparing 218458 == 0 2021-06-08 06:33:40,970 | test | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 218458 == 0. Received: {'172.23.106.196': 218458} for bucket 'default' 2021-06-08 06:33:40,970 | infra | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:33:40,971 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:33:40,971 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:33:41,898 | test | DEBUG | pool-5-thread-2 | [task:_compare:2649] Comparing 321685 == 0 2021-06-08 06:33:41,898 | test | DEBUG | pool-5-thread-2 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 321685 == 0. Received: {'172.23.106.198': 321685} for bucket 'default' 2021-06-08 06:33:41,898 | infra | DEBUG | pool-5-thread-2 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:33:41,900 | infra | DEBUG | pool-5-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:33:41,901 | infra | DEBUG | pool-5-thread-2 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:33:41,977 | test | DEBUG | pool-5-thread-30 | [task:_compare:2649] Comparing 243156 == 0 2021-06-08 06:33:41,977 | test | DEBUG | pool-5-thread-30 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 243156 == 0. Received: {'172.23.106.199': 243156} for bucket 'default' 2021-06-08 06:33:41,977 | infra | DEBUG | pool-5-thread-30 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:33:41,980 | infra | DEBUG | pool-5-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:33:41,980 | infra | DEBUG | pool-5-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:33:44,016 | infra | DEBUG | pool-5-thread-1 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:33:45,160 | test | DEBUG | pool-5-thread-1 | [task:_compare:2649] Comparing 206678 == 0 2021-06-08 06:33:45,160 | test | DEBUG | pool-5-thread-1 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 206678 == 0. Received: {'172.23.106.168': 206678} for bucket 'default' 2021-06-08 06:33:45,160 | infra | DEBUG | pool-5-thread-1 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:33:45,161 | infra | DEBUG | pool-5-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:33:45,161 | infra | DEBUG | pool-5-thread-1 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:33:45,971 | infra | DEBUG | pool-5-thread-29 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:33:46,901 | infra | DEBUG | pool-5-thread-2 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:33:46,980 | infra | DEBUG | pool-5-thread-30 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:33:47,127 | test | DEBUG | pool-5-thread-29 | [task:_compare:2649] Comparing 151047 == 0 2021-06-08 06:33:47,127 | test | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 151047 == 0. Received: {'172.23.106.196': 151047} for bucket 'default' 2021-06-08 06:33:47,127 | infra | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:33:47,128 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:33:47,128 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:33:48,085 | test | DEBUG | pool-5-thread-2 | [task:_compare:2649] Comparing 245294 == 0 2021-06-08 06:33:48,085 | test | DEBUG | pool-5-thread-2 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 245294 == 0. Received: {'172.23.106.198': 245294} for bucket 'default' 2021-06-08 06:33:48,085 | infra | DEBUG | pool-5-thread-2 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:33:48,088 | infra | DEBUG | pool-5-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:33:48,088 | infra | DEBUG | pool-5-thread-2 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:33:48,153 | test | DEBUG | pool-5-thread-30 | [task:_compare:2649] Comparing 168691 == 0 2021-06-08 06:33:48,153 | test | DEBUG | pool-5-thread-30 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 168691 == 0. Received: {'172.23.106.199': 168691} for bucket 'default' 2021-06-08 06:33:48,153 | infra | DEBUG | pool-5-thread-30 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:33:48,154 | infra | DEBUG | pool-5-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:33:48,154 | infra | DEBUG | pool-5-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:33:50,161 | infra | DEBUG | pool-5-thread-1 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:33:51,308 | test | DEBUG | pool-5-thread-1 | [task:_compare:2649] Comparing 144829 == 0 2021-06-08 06:33:51,308 | test | DEBUG | pool-5-thread-1 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 144829 == 0. Received: {'172.23.106.168': 144829} for bucket 'default' 2021-06-08 06:33:51,308 | infra | DEBUG | pool-5-thread-1 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:33:51,309 | infra | DEBUG | pool-5-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:33:51,309 | infra | DEBUG | pool-5-thread-1 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:33:52,128 | infra | DEBUG | pool-5-thread-29 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:33:53,088 | infra | DEBUG | pool-5-thread-2 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:33:53,154 | infra | DEBUG | pool-5-thread-30 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:33:53,263 | test | DEBUG | pool-5-thread-29 | [task:_compare:2649] Comparing 99210 == 0 2021-06-08 06:33:53,265 | test | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 99210 == 0. Received: {'172.23.106.196': 99210} for bucket 'default' 2021-06-08 06:33:53,265 | infra | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:33:53,266 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:33:53,266 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:33:54,263 | test | DEBUG | pool-5-thread-2 | [task:_compare:2649] Comparing 171903 == 0 2021-06-08 06:33:54,263 | test | DEBUG | pool-5-thread-2 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 171903 == 0. Received: {'172.23.106.198': 171903} for bucket 'default' 2021-06-08 06:33:54,263 | infra | DEBUG | pool-5-thread-2 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:33:54,266 | infra | DEBUG | pool-5-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:33:54,266 | infra | DEBUG | pool-5-thread-2 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:33:54,328 | test | DEBUG | pool-5-thread-30 | [task:_compare:2649] Comparing 102524 == 0 2021-06-08 06:33:54,328 | test | DEBUG | pool-5-thread-30 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 102524 == 0. Received: {'172.23.106.199': 102524} for bucket 'default' 2021-06-08 06:33:54,328 | infra | DEBUG | pool-5-thread-30 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:33:54,329 | infra | DEBUG | pool-5-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:33:54,329 | infra | DEBUG | pool-5-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:33:56,311 | infra | DEBUG | pool-5-thread-1 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:33:57,457 | test | DEBUG | pool-5-thread-1 | [task:_compare:2649] Comparing 85789 == 0 2021-06-08 06:33:57,457 | test | DEBUG | pool-5-thread-1 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 85789 == 0. Received: {'172.23.106.168': 85789} for bucket 'default' 2021-06-08 06:33:57,457 | infra | DEBUG | pool-5-thread-1 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:33:57,460 | infra | DEBUG | pool-5-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:33:57,460 | infra | DEBUG | pool-5-thread-1 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:33:58,266 | infra | DEBUG | pool-5-thread-29 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:33:59,266 | infra | DEBUG | pool-5-thread-2 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:33:59,329 | infra | DEBUG | pool-5-thread-30 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:33:59,392 | test | DEBUG | pool-5-thread-29 | [task:_compare:2649] Comparing 30434 == 0 2021-06-08 06:33:59,394 | test | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 30434 == 0. Received: {'172.23.106.196': 30434} for bucket 'default' 2021-06-08 06:33:59,394 | infra | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:33:59,394 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:33:59,395 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:34:00,424 | test | DEBUG | pool-5-thread-2 | [task:_compare:2649] Comparing 92912 == 0 2021-06-08 06:34:00,424 | test | DEBUG | pool-5-thread-2 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 92912 == 0. Received: {'172.23.106.198': 92912} for bucket 'default' 2021-06-08 06:34:00,424 | infra | DEBUG | pool-5-thread-2 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:34:00,427 | infra | DEBUG | pool-5-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:34:00,427 | infra | DEBUG | pool-5-thread-2 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:34:00,500 | test | DEBUG | pool-5-thread-30 | [task:_compare:2649] Comparing 40924 == 0 2021-06-08 06:34:00,500 | test | DEBUG | pool-5-thread-30 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 40924 == 0. Received: {'172.23.106.199': 40924} for bucket 'default' 2021-06-08 06:34:00,500 | infra | DEBUG | pool-5-thread-30 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:34:00,502 | infra | DEBUG | pool-5-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:34:00,502 | infra | DEBUG | pool-5-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:34:02,460 | infra | DEBUG | pool-5-thread-1 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:34:03,608 | test | DEBUG | pool-5-thread-1 | [task:_compare:2649] Comparing 9430 == 0 2021-06-08 06:34:03,608 | test | DEBUG | pool-5-thread-1 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 9430 == 0. Received: {'172.23.106.168': 9430} for bucket 'default' 2021-06-08 06:34:03,608 | infra | DEBUG | pool-5-thread-1 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:34:03,611 | infra | DEBUG | pool-5-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:34:03,611 | infra | DEBUG | pool-5-thread-1 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:34:04,395 | infra | DEBUG | pool-5-thread-29 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:34:05,427 | infra | DEBUG | pool-5-thread-2 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:34:05,503 | infra | DEBUG | pool-5-thread-30 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:34:05,540 | test | DEBUG | pool-5-thread-29 | [task:_compare:2649] Comparing 1094 == 0 2021-06-08 06:34:05,540 | test | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 1094 == 0. Received: {'172.23.106.196': 1094} for bucket 'default' 2021-06-08 06:34:05,540 | infra | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:34:05,542 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:34:05,542 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:34:06,628 | test | DEBUG | pool-5-thread-2 | [task:_compare:2649] Comparing 12683 == 0 2021-06-08 06:34:06,628 | test | DEBUG | pool-5-thread-2 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 12683 == 0. Received: {'172.23.106.198': 12683} for bucket 'default' 2021-06-08 06:34:06,628 | infra | DEBUG | pool-5-thread-2 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:34:06,630 | infra | DEBUG | pool-5-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:34:06,630 | infra | DEBUG | pool-5-thread-2 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:34:06,670 | test | DEBUG | pool-5-thread-30 | [task:_compare:2649] Comparing 0 == 0 2021-06-08 06:34:06,670 | test | DEBUG | pool-5-thread-30 | [task:_get_checkpoint_stats_and_compare:2638] Ready: num_items_for_persistence 0 == 0. Received: {'172.23.106.199': 0} for bucket 'default' 2021-06-08 06:34:06,670 | infra | INFO | pool-5-thread-30 | [task:complete_task:102] Thread 'StatsWaitTask_default_num_items_for_persistence_1623159004.34' completed 2021-06-08 06:34:08,611 | infra | DEBUG | pool-5-thread-1 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:34:09,743 | test | DEBUG | pool-5-thread-1 | [task:_compare:2649] Comparing 0 == 0 2021-06-08 06:34:09,744 | test | DEBUG | pool-5-thread-1 | [task:_get_checkpoint_stats_and_compare:2638] Ready: num_items_for_persistence 0 == 0. Received: {'172.23.106.168': 0} for bucket 'default' 2021-06-08 06:34:09,744 | infra | INFO | pool-5-thread-1 | [task:complete_task:102] Thread 'StatsWaitTask_default_num_items_for_persistence_1623159004.98' completed 2021-06-08 06:34:10,542 | infra | DEBUG | pool-5-thread-29 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:34:11,631 | infra | DEBUG | pool-5-thread-2 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:34:11,671 | test | DEBUG | pool-5-thread-29 | [task:_compare:2649] Comparing 0 == 0 2021-06-08 06:34:11,671 | test | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2638] Ready: num_items_for_persistence 0 == 0. Received: {'172.23.106.196': 0} for bucket 'default' 2021-06-08 06:34:11,671 | infra | INFO | pool-5-thread-29 | [task:complete_task:102] Thread 'StatsWaitTask_default_num_items_for_persistence_1623159003.76' completed 2021-06-08 06:34:11,671 | infra | DEBUG | MainThread | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.196 2021-06-08 06:34:11,673 | infra | DEBUG | MainThread | [task_manager:get_task_result:26] Getting task result for StatsWaitTask_default_num_items_for_persistence_1623159004.34 2021-06-08 06:34:11,673 | infra | DEBUG | MainThread | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.199 2021-06-08 06:34:11,673 | infra | DEBUG | MainThread | [task_manager:get_task_result:26] Getting task result for StatsWaitTask_default_num_items_for_persistence_1623159004.98 2021-06-08 06:34:11,674 | infra | DEBUG | MainThread | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.168 2021-06-08 06:34:11,674 | infra | DEBUG | MainThread | [task_manager:get_task_result:26] Getting task result for StatsWaitTask_default_num_items_for_persistence_1623159005.58 2021-06-08 06:34:12,796 | test | DEBUG | pool-5-thread-2 | [task:_compare:2649] Comparing 1074 == 0 2021-06-08 06:34:12,796 | test | DEBUG | pool-5-thread-2 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 1074 == 0. Received: {'172.23.106.198': 1074} for bucket 'default' 2021-06-08 06:34:12,796 | infra | DEBUG | pool-5-thread-2 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 06:34:12,799 | infra | DEBUG | pool-5-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 06:34:12,799 | infra | DEBUG | pool-5-thread-2 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:34:17,799 | infra | DEBUG | pool-5-thread-2 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 06:34:18,921 | test | DEBUG | pool-5-thread-2 | [task:_compare:2649] Comparing 0 == 0 2021-06-08 06:34:18,921 | test | DEBUG | pool-5-thread-2 | [task:_get_checkpoint_stats_and_compare:2638] Ready: num_items_for_persistence 0 == 0. Received: {'172.23.106.198': 0} for bucket 'default' 2021-06-08 06:34:18,921 | infra | INFO | pool-5-thread-2 | [task:complete_task:102] Thread 'StatsWaitTask_default_num_items_for_persistence_1623159005.58' completed 2021-06-08 06:34:18,921 | infra | DEBUG | MainThread | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.198 2021-06-08 06:34:18,966 | test | DEBUG | MainThread | [bucket_ready_functions:parse_get_bucket_json:3739] Stats: {u'opsPerSec': 0, u'diskFetches': 0, u'dataUsed': 30506438656L, u'itemCount': 5000000, u'quotaPercentUsed': 80.24867438127895, u'storageTotals': {u'hdd': {u'usedByData': 61491454980L, u'used': 61816661604L, u'quotaTotal': 386354135040L, u'total': 386354135040L, u'free': 324537473436L}, u'ram': {u'usedByData': 28105042552L, u'used': 42840420352L, u'quotaTotal': 35026632704L, u'total': 49176707072L, u'quotaUsed': 35022438400L, u'quotaUsedPerNode': 8755609600L, u'quotaTotalPerNode': 8756658176L}}, u'diskUsed': 61491454980L, u'vbActiveNumNonResident': 0, u'memUsed': 28105042552L} 2021-06-08 06:34:18,967 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.196 with username: root, password: couchbase 2021-06-08 06:34:19,045 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 06:34:19,253 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: ls /etc/ 2021-06-08 06:34:19,306 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 06:34:19,308 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: ls /cygdrive/c/Windows 2021-06-08 06:34:19,319 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: uname -m 2021-06-08 06:34:19,331 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: cat /proc/cpuinfo 2021-06-08 06:34:19,342 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: df -Thl 2021-06-08 06:34:19,357 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: cat /proc/meminfo 2021-06-08 06:34:19,368 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: hostname 2021-06-08 06:34:19,380 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: hostname -d 2021-06-08 06:34:19,391 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: du -cm /data/default/magma.*/kv* | tail -1 | awk '{print $1}' 2021-06-08 06:34:19,599 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: du -cm /data/default/magma.*/wal | tail -1 | awk '{print $1}' 2021-06-08 06:34:19,615 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: du -cm /data/default/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2021-06-08 06:34:19,707 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: du -cm /data/default/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2021-06-08 06:34:19,796 | infra | DEBUG | MainThread | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.196 2021-06-08 06:34:19,799 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.199 with username: root, password: couchbase 2021-06-08 06:34:19,865 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 06:34:20,075 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: ls /etc/ 2021-06-08 06:34:20,127 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 06:34:20,128 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: ls /cygdrive/c/Windows 2021-06-08 06:34:20,140 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: uname -m 2021-06-08 06:34:20,151 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: cat /proc/cpuinfo 2021-06-08 06:34:20,163 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: df -Thl 2021-06-08 06:34:20,177 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: cat /proc/meminfo 2021-06-08 06:34:20,188 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: hostname 2021-06-08 06:34:20,200 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: hostname -d 2021-06-08 06:34:20,213 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: du -cm /data/default/magma.*/kv* | tail -1 | awk '{print $1}' 2021-06-08 06:34:20,424 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: du -cm /data/default/magma.*/wal | tail -1 | awk '{print $1}' 2021-06-08 06:34:20,440 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: du -cm /data/default/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2021-06-08 06:34:20,539 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: du -cm /data/default/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2021-06-08 06:34:20,632 | infra | DEBUG | MainThread | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.199 2021-06-08 06:34:20,634 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.168 with username: root, password: couchbase 2021-06-08 06:34:20,723 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 06:34:20,934 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: ls /etc/ 2021-06-08 06:34:20,986 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 06:34:20,986 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: ls /cygdrive/c/Windows 2021-06-08 06:34:21,000 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: uname -m 2021-06-08 06:34:21,010 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: cat /proc/cpuinfo 2021-06-08 06:34:21,023 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: df -Thl 2021-06-08 06:34:21,038 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: cat /proc/meminfo 2021-06-08 06:34:21,049 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: hostname 2021-06-08 06:34:21,062 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: hostname -d 2021-06-08 06:34:21,073 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: du -cm /data/default/magma.*/kv* | tail -1 | awk '{print $1}' 2021-06-08 06:34:21,290 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: du -cm /data/default/magma.*/wal | tail -1 | awk '{print $1}' 2021-06-08 06:34:21,305 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: du -cm /data/default/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2021-06-08 06:34:21,400 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: du -cm /data/default/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2021-06-08 06:34:21,490 | infra | DEBUG | MainThread | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.168 2021-06-08 06:34:21,493 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.198 with username: root, password: couchbase 2021-06-08 06:34:21,565 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 06:34:21,772 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: ls /etc/ 2021-06-08 06:34:21,823 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 06:34:21,823 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: ls /cygdrive/c/Windows 2021-06-08 06:34:21,835 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: uname -m 2021-06-08 06:34:21,846 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: cat /proc/cpuinfo 2021-06-08 06:34:21,858 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: df -Thl 2021-06-08 06:34:21,875 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: cat /proc/meminfo 2021-06-08 06:34:21,885 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: hostname 2021-06-08 06:34:21,897 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: hostname -d 2021-06-08 06:34:21,910 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: du -cm /data/default/magma.*/kv* | tail -1 | awk '{print $1}' 2021-06-08 06:34:22,122 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: du -cm /data/default/magma.*/wal | tail -1 | awk '{print $1}' 2021-06-08 06:34:22,138 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: du -cm /data/default/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2021-06-08 06:34:22,233 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: du -cm /data/default/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2021-06-08 06:34:22,329 | infra | DEBUG | MainThread | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.198 2021-06-08 06:34:22,329 | test | INFO | MainThread | [magma_base:get_disk_usage:815] Disk usage stats for bucekt default is below 2021-06-08 06:34:22,329 | test | INFO | MainThread | [magma_base:get_disk_usage:816] Total Disk usage for kvstore is 29901MB 2021-06-08 06:34:22,578 | test | INFO | MainThread | [magma_base:get_bucket_dgm:767] Active Resident Threshold of default is 100 2021-06-08 06:34:22,578 | test | DEBUG | MainThread | [magma_base:get_disk_usage:818] Total Disk usage for wal is 30042MB 2021-06-08 06:34:22,578 | test | DEBUG | MainThread | [magma_base:get_disk_usage:819] Total Disk usage for keyTree is 721MB 2021-06-08 06:34:22,578 | test | DEBUG | MainThread | [magma_base:get_disk_usage:820] Total Disk usage for seqTree is 28996MB 2021-06-08 06:34:22,578 | test | INFO | MainThread | [magma_base:initial_load:359] For bucket default disk usage after initial creation is 29901MB 2021-06-08 06:34:22,578 | test | INFO | MainThread | [magma_base:setUp:261] ==========Finished magma base setup======== 2021-06-08 06:34:22,581 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.196 with username: root, password: couchbase 2021-06-08 06:34:22,661 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 06:34:22,861 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: ls /etc/ 2021-06-08 06:34:22,914 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 06:34:22,914 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: ls /cygdrive/c/Windows 2021-06-08 06:34:22,927 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: uname -m 2021-06-08 06:34:22,938 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: cat /proc/cpuinfo 2021-06-08 06:34:22,950 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: df -Thl 2021-06-08 06:34:22,966 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: cat /proc/meminfo 2021-06-08 06:34:22,980 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: hostname 2021-06-08 06:34:22,993 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: hostname -d 2021-06-08 06:34:23,005 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: rm -rf /data/full_disk_* 2021-06-08 06:34:23,016 | infra | DEBUG | MainThread | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.196 2021-06-08 06:34:23,022 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/magma/magma_disk_full.py -> free_disk():L107 2021-06-08 06:34:23,022 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: Wait for files to clean up from the disk 2021-06-08 06:34:33,022 | test | DEBUG | MainThread | [rest_client:update_autofailover_settings:1942] settings/autoFailover params: failoverServerGroup=false&failoverOnDataDiskIssues%5Benabled%5D=false&maxCount=1&timeout=600&enabled=false&canAbortRebalance=false 2021-06-08 06:34:33,039 | infra | DEBUG | Thread-19 | [remote_util:connect:206] Connecting to 172.23.106.168 with username: root, password: couchbase 2021-06-08 06:34:33,039 | infra | DEBUG | Thread-17 | [remote_util:connect:206] Connecting to 172.23.106.196 with username: root, password: couchbase 2021-06-08 06:34:33,039 | infra | DEBUG | Thread-18 | [remote_util:connect:206] Connecting to 172.23.106.199 with username: root, password: couchbase 2021-06-08 06:34:33,039 | infra | DEBUG | Thread-20 | [remote_util:connect:206] Connecting to 172.23.106.198 with username: root, password: couchbase 2021-06-08 06:34:33,108 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 06:34:33,115 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 06:34:33,121 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 06:34:33,124 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 06:34:33,319 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: ls /etc/ 2021-06-08 06:34:33,328 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: ls /etc/ 2021-06-08 06:34:33,332 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: ls /etc/ 2021-06-08 06:34:33,332 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: ls /etc/ 2021-06-08 06:34:33,381 | infra | DEBUG | Thread-17 | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 06:34:33,382 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: ls /cygdrive/c/Windows 2021-06-08 06:34:33,388 | infra | DEBUG | Thread-20 | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 06:34:33,388 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: ls /cygdrive/c/Windows 2021-06-08 06:34:33,395 | infra | DEBUG | Thread-18 | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 06:34:33,395 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: ls /cygdrive/c/Windows 2021-06-08 06:34:33,397 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: uname -m 2021-06-08 06:34:33,398 | infra | DEBUG | Thread-19 | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 06:34:33,398 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: ls /cygdrive/c/Windows 2021-06-08 06:34:33,401 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: uname -m 2021-06-08 06:34:33,408 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: cat /proc/cpuinfo 2021-06-08 06:34:33,410 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: uname -m 2021-06-08 06:34:33,414 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: cat /proc/cpuinfo 2021-06-08 06:34:33,414 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: uname -m 2021-06-08 06:34:33,423 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: df -Thl 2021-06-08 06:34:33,424 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: cat /proc/cpuinfo 2021-06-08 06:34:33,426 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: df -Thl 2021-06-08 06:34:33,427 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: cat /proc/cpuinfo 2021-06-08 06:34:33,437 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: df -Thl 2021-06-08 06:34:33,438 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: cat /proc/meminfo 2021-06-08 06:34:33,440 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: df -Thl 2021-06-08 06:34:33,443 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: cat /proc/meminfo 2021-06-08 06:34:33,453 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: hostname 2021-06-08 06:34:33,453 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: cat /proc/meminfo 2021-06-08 06:34:33,456 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: hostname 2021-06-08 06:34:33,456 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: cat /proc/meminfo 2021-06-08 06:34:33,466 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: hostname -d 2021-06-08 06:34:33,467 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: hostname 2021-06-08 06:34:33,469 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: hostname 2021-06-08 06:34:33,470 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: hostname -d 2021-06-08 06:34:33,480 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: df -lBM /data 2021-06-08 06:34:33,480 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: hostname -d 2021-06-08 06:34:33,483 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: hostname -d 2021-06-08 06:34:33,484 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: df -lBM /data 2021-06-08 06:34:33,493 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_77431MB_1623159273.49 bs=1024M count=1 2021-06-08 06:34:33,493 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: dd if=/dev/zero of=/data/full_disk_77431MB_1623159273.49 bs=1024M count=1 2021-06-08 06:34:33,494 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: df -lBM /data 2021-06-08 06:34:33,496 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: df -lBM /data 2021-06-08 06:34:33,499 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_76985MB_1623159273.5 bs=1024M count=1 2021-06-08 06:34:33,499 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: dd if=/dev/zero of=/data/full_disk_76985MB_1623159273.5 bs=1024M count=1 2021-06-08 06:34:33,509 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_77161MB_1623159273.51 bs=1024M count=1 2021-06-08 06:34:33,509 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: dd if=/dev/zero of=/data/full_disk_77161MB_1623159273.51 bs=1024M count=1 2021-06-08 06:34:33,509 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_76762MB_1623159273.51 bs=1024M count=1 2021-06-08 06:34:33,509 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: dd if=/dev/zero of=/data/full_disk_76762MB_1623159273.51 bs=1024M count=1 2021-06-08 06:34:35,039 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 1.45609 s, 737 MB/s 2021-06-08 06:34:35,039 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_76407MB_1623159275.04 bs=1024M count=1 2021-06-08 06:34:35,039 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: dd if=/dev/zero of=/data/full_disk_76407MB_1623159275.04 bs=1024M count=1 2021-06-08 06:34:35,042 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 1.4443 s, 743 MB/s 2021-06-08 06:34:35,042 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_76137MB_1623159275.04 bs=1024M count=1 2021-06-08 06:34:35,042 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: dd if=/dev/zero of=/data/full_disk_76137MB_1623159275.04 bs=1024M count=1 2021-06-08 06:34:36,457 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 1.32303 s, 812 MB/s 2021-06-08 06:34:36,457 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_75383MB_1623159276.46 bs=1024M count=1 2021-06-08 06:34:36,457 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: dd if=/dev/zero of=/data/full_disk_75383MB_1623159276.46 bs=1024M count=1 2021-06-08 06:34:36,917 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 1.78239 s, 602 MB/s 2021-06-08 06:34:36,917 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_75113MB_1623159276.92 bs=1024M count=1 2021-06-08 06:34:36,917 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: dd if=/dev/zero of=/data/full_disk_75113MB_1623159276.92 bs=1024M count=1 2021-06-08 06:34:49,062 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 12.5116 s, 85.8 MB/s 2021-06-08 06:34:49,062 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_74359MB_1623159289.06 bs=1024M count=1 2021-06-08 06:34:49,063 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: dd if=/dev/zero of=/data/full_disk_74359MB_1623159289.06 bs=1024M count=1 2021-06-08 06:34:49,536 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 12.5182 s, 85.8 MB/s 2021-06-08 06:34:49,536 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_74089MB_1623159289.54 bs=1024M count=1 2021-06-08 06:34:49,536 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: dd if=/dev/zero of=/data/full_disk_74089MB_1623159289.54 bs=1024M count=1 2021-06-08 06:35:00,308 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 10.6729 s, 101 MB/s 2021-06-08 06:35:00,308 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_73065MB_1623159300.31 bs=1024M count=1 2021-06-08 06:35:00,308 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: dd if=/dev/zero of=/data/full_disk_73065MB_1623159300.31 bs=1024M count=1 2021-06-08 06:35:10,348 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 36.7619 s, 29.2 MB/s 2021-06-08 06:35:10,348 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_75961MB_1623159310.35 bs=1024M count=1 2021-06-08 06:35:10,348 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: dd if=/dev/zero of=/data/full_disk_75961MB_1623159310.35 bs=1024M count=1 2021-06-08 06:35:11,311 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 22.1498 s, 48.5 MB/s 2021-06-08 06:35:11,312 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_73335MB_1623159311.31 bs=1024M count=1 2021-06-08 06:35:11,312 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: dd if=/dev/zero of=/data/full_disk_73335MB_1623159311.31 bs=1024M count=1 2021-06-08 06:35:11,799 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 11.3953 s, 94.2 MB/s 2021-06-08 06:35:11,799 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_72041MB_1623159311.8 bs=1024M count=1 2021-06-08 06:35:11,799 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: dd if=/dev/zero of=/data/full_disk_72041MB_1623159311.8 bs=1024M count=1 2021-06-08 06:35:14,368 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 2.46947 s, 435 MB/s 2021-06-08 06:35:14,368 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_71017MB_1623159314.37 bs=1024M count=1 2021-06-08 06:35:14,368 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: dd if=/dev/zero of=/data/full_disk_71017MB_1623159314.37 bs=1024M count=1 2021-06-08 06:35:14,628 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 41.0324 s, 26.2 MB/s 2021-06-08 06:35:14,628 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_75738MB_1623159314.63 bs=1024M count=1 2021-06-08 06:35:14,628 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: dd if=/dev/zero of=/data/full_disk_75738MB_1623159314.63 bs=1024M count=1 2021-06-08 06:35:29,767 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 15.3002 s, 70.2 MB/s 2021-06-08 06:35:29,767 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_69993MB_1623159329.77 bs=1024M count=1 2021-06-08 06:35:29,769 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: dd if=/dev/zero of=/data/full_disk_69993MB_1623159329.77 bs=1024M count=1 2021-06-08 06:35:33,805 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 3.93839 s, 273 MB/s 2021-06-08 06:35:33,805 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_68969MB_1623159333.81 bs=1024M count=1 2021-06-08 06:35:33,805 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: dd if=/dev/zero of=/data/full_disk_68969MB_1623159333.81 bs=1024M count=1 2021-06-08 06:35:38,052 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 26.6476 s, 40.3 MB/s 2021-06-08 06:35:38,052 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_72311MB_1623159338.05 bs=1024M count=1 2021-06-08 06:35:38,052 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: dd if=/dev/zero of=/data/full_disk_72311MB_1623159338.05 bs=1024M count=1 2021-06-08 06:35:41,203 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 7.22983 s, 149 MB/s 2021-06-08 06:35:41,203 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_67945MB_1623159341.2 bs=1024M count=1 2021-06-08 06:35:41,203 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: dd if=/dev/zero of=/data/full_disk_67945MB_1623159341.2 bs=1024M count=1 2021-06-08 06:35:54,729 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 16.5798 s, 64.8 MB/s 2021-06-08 06:35:54,729 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_71287MB_1623159354.73 bs=1024M count=1 2021-06-08 06:35:54,729 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: dd if=/dev/zero of=/data/full_disk_71287MB_1623159354.73 bs=1024M count=1 2021-06-08 06:35:54,861 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 13.5534 s, 79.2 MB/s 2021-06-08 06:35:54,861 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_66921MB_1623159354.86 bs=1024M count=1 2021-06-08 06:35:54,861 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: dd if=/dev/zero of=/data/full_disk_66921MB_1623159354.86 bs=1024M count=1 2021-06-08 06:35:57,858 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 2.89797 s, 371 MB/s 2021-06-08 06:35:57,858 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_65897MB_1623159357.86 bs=1024M count=1 2021-06-08 06:35:57,858 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: dd if=/dev/zero of=/data/full_disk_65897MB_1623159357.86 bs=1024M count=1 2021-06-08 06:35:58,309 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 3.48787 s, 308 MB/s 2021-06-08 06:35:58,311 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_70263MB_1623159358.31 bs=1024M count=1 2021-06-08 06:35:58,311 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: dd if=/dev/zero of=/data/full_disk_70263MB_1623159358.31 bs=1024M count=1 2021-06-08 06:35:59,989 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 45.2632 s, 23.7 MB/s 2021-06-08 06:35:59,990 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_74714MB_1623159359.99 bs=1024M count=1 2021-06-08 06:35:59,990 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: dd if=/dev/zero of=/data/full_disk_74714MB_1623159359.99 bs=1024M count=1 2021-06-08 06:36:00,124 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 49.6742 s, 21.6 MB/s 2021-06-08 06:36:00,125 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_74937MB_1623159360.12 bs=1024M count=1 2021-06-08 06:36:00,125 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: dd if=/dev/zero of=/data/full_disk_74937MB_1623159360.12 bs=1024M count=1 2021-06-08 06:36:11,601 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 13.6425 s, 78.7 MB/s 2021-06-08 06:36:11,601 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_64873MB_1623159371.6 bs=1024M count=1 2021-06-08 06:36:11,601 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: dd if=/dev/zero of=/data/full_disk_64873MB_1623159371.6 bs=1024M count=1 2021-06-08 06:36:14,325 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 15.9202 s, 67.4 MB/s 2021-06-08 06:36:14,325 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_69239MB_1623159374.33 bs=1024M count=1 2021-06-08 06:36:14,326 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: dd if=/dev/zero of=/data/full_disk_69239MB_1623159374.33 bs=1024M count=1 2021-06-08 06:36:17,697 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 3.27859 s, 328 MB/s 2021-06-08 06:36:17,697 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_68215MB_1623159377.7 bs=1024M count=1 2021-06-08 06:36:17,697 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: dd if=/dev/zero of=/data/full_disk_68215MB_1623159377.7 bs=1024M count=1 2021-06-08 06:36:18,358 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 6.65919 s, 161 MB/s 2021-06-08 06:36:18,358 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_63849MB_1623159378.36 bs=1024M count=1 2021-06-08 06:36:18,358 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: dd if=/dev/zero of=/data/full_disk_63849MB_1623159378.36 bs=1024M count=1 2021-06-08 06:36:26,005 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 8.21114 s, 131 MB/s 2021-06-08 06:36:26,006 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_67191MB_1623159386.01 bs=1024M count=1 2021-06-08 06:36:26,006 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: dd if=/dev/zero of=/data/full_disk_67191MB_1623159386.01 bs=1024M count=1 2021-06-08 06:36:26,729 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 8.26546 s, 130 MB/s 2021-06-08 06:36:26,729 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_62825MB_1623159386.73 bs=1024M count=1 2021-06-08 06:36:26,729 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: dd if=/dev/zero of=/data/full_disk_62825MB_1623159386.73 bs=1024M count=1 2021-06-08 06:36:30,476 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 30.393 s, 35.3 MB/s 2021-06-08 06:36:30,476 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_73690MB_1623159390.48 bs=1024M count=1 2021-06-08 06:36:30,476 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: dd if=/dev/zero of=/data/full_disk_73690MB_1623159390.48 bs=1024M count=1 2021-06-08 06:36:36,815 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 36.5921 s, 29.3 MB/s 2021-06-08 06:36:36,815 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_73913MB_1623159396.82 bs=1024M count=1 2021-06-08 06:36:36,815 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: dd if=/dev/zero of=/data/full_disk_73913MB_1623159396.82 bs=1024M count=1 2021-06-08 06:36:38,897 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 12.4423 s, 86.3 MB/s 2021-06-08 06:36:38,898 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_66167MB_1623159398.9 bs=1024M count=1 2021-06-08 06:36:38,898 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: dd if=/dev/zero of=/data/full_disk_66167MB_1623159398.9 bs=1024M count=1 2021-06-08 06:36:39,094 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 12.2591 s, 87.6 MB/s 2021-06-08 06:36:39,094 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_61801MB_1623159399.09 bs=1024M count=1 2021-06-08 06:36:39,094 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: dd if=/dev/zero of=/data/full_disk_61801MB_1623159399.09 bs=1024M count=1 2021-06-08 06:36:39,546 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 8.97104 s, 120 MB/s 2021-06-08 06:36:39,546 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_72666MB_1623159399.55 bs=1024M count=1 2021-06-08 06:36:39,546 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: dd if=/dev/zero of=/data/full_disk_72666MB_1623159399.55 bs=1024M count=1 2021-06-08 06:36:45,967 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 6.97218 s, 154 MB/s 2021-06-08 06:36:45,967 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_65143MB_1623159405.97 bs=1024M count=1 2021-06-08 06:36:45,967 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: dd if=/dev/zero of=/data/full_disk_65143MB_1623159405.97 bs=1024M count=1 2021-06-08 06:36:47,365 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 8.16949 s, 131 MB/s 2021-06-08 06:36:47,365 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_60777MB_1623159407.37 bs=1024M count=1 2021-06-08 06:36:47,365 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: dd if=/dev/zero of=/data/full_disk_60777MB_1623159407.37 bs=1024M count=1 2021-06-08 06:36:48,779 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 9.1391 s, 117 MB/s 2021-06-08 06:36:48,779 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_71642MB_1623159408.78 bs=1024M count=1 2021-06-08 06:36:48,779 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: dd if=/dev/zero of=/data/full_disk_71642MB_1623159408.78 bs=1024M count=1 2021-06-08 06:36:58,345 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 12.2784 s, 87.4 MB/s 2021-06-08 06:36:58,345 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_64119MB_1623159418.35 bs=1024M count=1 2021-06-08 06:36:58,345 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: dd if=/dev/zero of=/data/full_disk_64119MB_1623159418.35 bs=1024M count=1 2021-06-08 06:37:01,183 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 12.3029 s, 87.3 MB/s 2021-06-08 06:37:01,184 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_70618MB_1623159421.18 bs=1024M count=1 2021-06-08 06:37:01,184 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: dd if=/dev/zero of=/data/full_disk_70618MB_1623159421.18 bs=1024M count=1 2021-06-08 06:37:02,710 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 2.10774 s, 509 MB/s 2021-06-08 06:37:02,710 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_63095MB_1623159422.71 bs=1024M count=1 2021-06-08 06:37:02,710 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: dd if=/dev/zero of=/data/full_disk_63095MB_1623159422.71 bs=1024M count=1 2021-06-08 06:37:02,803 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 1.52118 s, 706 MB/s 2021-06-08 06:37:02,803 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_69594MB_1623159422.8 bs=1024M count=1 2021-06-08 06:37:02,803 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: dd if=/dev/zero of=/data/full_disk_69594MB_1623159422.8 bs=1024M count=1 2021-06-08 06:37:03,005 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 26.0916 s, 41.2 MB/s 2021-06-08 06:37:03,005 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_72889MB_1623159423.01 bs=1024M count=1 2021-06-08 06:37:03,006 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: dd if=/dev/zero of=/data/full_disk_72889MB_1623159423.01 bs=1024M count=1 2021-06-08 06:37:03,832 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 15.9146 s, 67.5 MB/s 2021-06-08 06:37:03,832 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_59753MB_1623159423.83 bs=1024M count=1 2021-06-08 06:37:03,832 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: dd if=/dev/zero of=/data/full_disk_59753MB_1623159423.83 bs=1024M count=1 2021-06-08 06:37:08,844 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 6.03403 s, 178 MB/s 2021-06-08 06:37:08,844 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_62071MB_1623159428.84 bs=1024M count=1 2021-06-08 06:37:08,844 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: dd if=/dev/zero of=/data/full_disk_62071MB_1623159428.84 bs=1024M count=1 2021-06-08 06:37:10,259 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 7.15371 s, 150 MB/s 2021-06-08 06:37:10,259 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_71865MB_1623159430.26 bs=1024M count=1 2021-06-08 06:37:10,259 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: dd if=/dev/zero of=/data/full_disk_71865MB_1623159430.26 bs=1024M count=1 2021-06-08 06:37:11,595 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 8.69247 s, 124 MB/s 2021-06-08 06:37:11,595 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_68570MB_1623159431.6 bs=1024M count=1 2021-06-08 06:37:11,595 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: dd if=/dev/zero of=/data/full_disk_68570MB_1623159431.6 bs=1024M count=1 2021-06-08 06:37:22,799 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 13.8509 s, 77.5 MB/s 2021-06-08 06:37:22,799 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_61047MB_1623159442.8 bs=1024M count=1 2021-06-08 06:37:22,799 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: dd if=/dev/zero of=/data/full_disk_61047MB_1623159442.8 bs=1024M count=1 2021-06-08 06:37:23,118 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 12.7585 s, 84.2 MB/s 2021-06-08 06:37:23,118 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_70841MB_1623159443.12 bs=1024M count=1 2021-06-08 06:37:23,118 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: dd if=/dev/zero of=/data/full_disk_70841MB_1623159443.12 bs=1024M count=1 2021-06-08 06:37:23,207 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 11.5081 s, 93.3 MB/s 2021-06-08 06:37:23,207 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_67546MB_1623159443.21 bs=1024M count=1 2021-06-08 06:37:23,207 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: dd if=/dev/zero of=/data/full_disk_67546MB_1623159443.21 bs=1024M count=1 2021-06-08 06:37:24,305 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 1.40989 s, 762 MB/s 2021-06-08 06:37:24,305 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_60023MB_1623159444.31 bs=1024M count=1 2021-06-08 06:37:24,305 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: dd if=/dev/zero of=/data/full_disk_60023MB_1623159444.31 bs=1024M count=1 2021-06-08 06:37:27,174 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 3.96307 s, 271 MB/s 2021-06-08 06:37:27,174 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_69817MB_1623159447.17 bs=1024M count=1 2021-06-08 06:37:27,174 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: dd if=/dev/zero of=/data/full_disk_69817MB_1623159447.17 bs=1024M count=1 2021-06-08 06:37:27,776 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 4.4692 s, 240 MB/s 2021-06-08 06:37:27,776 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_66522MB_1623159447.78 bs=1024M count=1 2021-06-08 06:37:27,776 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: dd if=/dev/zero of=/data/full_disk_66522MB_1623159447.78 bs=1024M count=1 2021-06-08 06:37:29,403 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 25.4658 s, 42.2 MB/s 2021-06-08 06:37:29,403 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_58729MB_1623159449.4 bs=1024M count=1 2021-06-08 06:37:29,403 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: dd if=/dev/zero of=/data/full_disk_58729MB_1623159449.4 bs=1024M count=1 2021-06-08 06:37:34,927 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 7.65406 s, 140 MB/s 2021-06-08 06:37:34,928 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_68793MB_1623159454.93 bs=1024M count=1 2021-06-08 06:37:34,928 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: dd if=/dev/zero of=/data/full_disk_68793MB_1623159454.93 bs=1024M count=1 2021-06-08 06:37:36,322 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 8.44452 s, 127 MB/s 2021-06-08 06:37:36,323 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_65498MB_1623159456.32 bs=1024M count=1 2021-06-08 06:37:36,323 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: dd if=/dev/zero of=/data/full_disk_65498MB_1623159456.32 bs=1024M count=1 2021-06-08 06:37:39,733 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 15.3373 s, 70.0 MB/s 2021-06-08 06:37:39,734 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_58999MB_1623159459.73 bs=1024M count=1 2021-06-08 06:37:39,734 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: dd if=/dev/zero of=/data/full_disk_58999MB_1623159459.73 bs=1024M count=1 2021-06-08 06:37:45,253 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 10.2264 s, 105 MB/s 2021-06-08 06:37:45,253 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_67769MB_1623159465.25 bs=1024M count=1 2021-06-08 06:37:45,253 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: dd if=/dev/zero of=/data/full_disk_67769MB_1623159465.25 bs=1024M count=1 2021-06-08 06:37:45,437 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 9.00893 s, 119 MB/s 2021-06-08 06:37:45,437 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_64474MB_1623159465.44 bs=1024M count=1 2021-06-08 06:37:45,437 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: dd if=/dev/zero of=/data/full_disk_64474MB_1623159465.44 bs=1024M count=1 2021-06-08 06:37:46,868 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 1.50798 s, 712 MB/s 2021-06-08 06:37:46,868 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_66745MB_1623159466.87 bs=1024M count=1 2021-06-08 06:37:46,868 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: dd if=/dev/zero of=/data/full_disk_66745MB_1623159466.87 bs=1024M count=1 2021-06-08 06:37:47,380 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 1.82538 s, 588 MB/s 2021-06-08 06:37:47,380 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_63450MB_1623159467.38 bs=1024M count=1 2021-06-08 06:37:47,381 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: dd if=/dev/zero of=/data/full_disk_63450MB_1623159467.38 bs=1024M count=1 2021-06-08 06:37:52,838 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 5.87427 s, 183 MB/s 2021-06-08 06:37:52,839 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_65721MB_1623159472.84 bs=1024M count=1 2021-06-08 06:37:52,839 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: dd if=/dev/zero of=/data/full_disk_65721MB_1623159472.84 bs=1024M count=1 2021-06-08 06:37:53,329 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 23.8212 s, 45.1 MB/s 2021-06-08 06:37:53,329 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_57705MB_1623159473.33 bs=1024M count=1 2021-06-08 06:37:53,329 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: dd if=/dev/zero of=/data/full_disk_57705MB_1623159473.33 bs=1024M count=1 2021-06-08 06:37:54,161 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 6.67756 s, 161 MB/s 2021-06-08 06:37:54,161 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_62426MB_1623159474.16 bs=1024M count=1 2021-06-08 06:37:54,161 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: dd if=/dev/zero of=/data/full_disk_62426MB_1623159474.16 bs=1024M count=1 2021-06-08 06:38:06,811 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 13.8671 s, 77.4 MB/s 2021-06-08 06:38:06,812 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_64697MB_1623159486.81 bs=1024M count=1 2021-06-08 06:38:06,812 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: dd if=/dev/zero of=/data/full_disk_64697MB_1623159486.81 bs=1024M count=1 2021-06-08 06:38:06,967 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 27.135 s, 39.6 MB/s 2021-06-08 06:38:06,967 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_57975MB_1623159486.97 bs=1024M count=1 2021-06-08 06:38:06,967 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: dd if=/dev/zero of=/data/full_disk_57975MB_1623159486.97 bs=1024M count=1 2021-06-08 06:38:07,092 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 12.8246 s, 83.7 MB/s 2021-06-08 06:38:07,094 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_61402MB_1623159487.09 bs=1024M count=1 2021-06-08 06:38:07,094 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: dd if=/dev/zero of=/data/full_disk_61402MB_1623159487.09 bs=1024M count=1 2021-06-08 06:38:07,154 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 13.7154 s, 78.3 MB/s 2021-06-08 06:38:07,154 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_56681MB_1623159487.15 bs=1024M count=1 2021-06-08 06:38:07,154 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: dd if=/dev/zero of=/data/full_disk_56681MB_1623159487.15 bs=1024M count=1 2021-06-08 06:38:08,319 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 1.39928 s, 767 MB/s 2021-06-08 06:38:08,319 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_63673MB_1623159488.32 bs=1024M count=1 2021-06-08 06:38:08,319 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: dd if=/dev/zero of=/data/full_disk_63673MB_1623159488.32 bs=1024M count=1 2021-06-08 06:38:09,015 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 1.75052 s, 613 MB/s 2021-06-08 06:38:09,015 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_55657MB_1623159489.02 bs=1024M count=1 2021-06-08 06:38:09,015 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: dd if=/dev/zero of=/data/full_disk_55657MB_1623159489.02 bs=1024M count=1 2021-06-08 06:38:12,033 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 4.83279 s, 222 MB/s 2021-06-08 06:38:12,033 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_60378MB_1623159492.03 bs=1024M count=1 2021-06-08 06:38:12,033 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: dd if=/dev/zero of=/data/full_disk_60378MB_1623159492.03 bs=1024M count=1 2021-06-08 06:38:13,719 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 5.29686 s, 203 MB/s 2021-06-08 06:38:13,720 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_62649MB_1623159493.72 bs=1024M count=1 2021-06-08 06:38:13,720 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: dd if=/dev/zero of=/data/full_disk_62649MB_1623159493.72 bs=1024M count=1 2021-06-08 06:38:19,461 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 7.32546 s, 147 MB/s 2021-06-08 06:38:19,463 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_59354MB_1623159499.46 bs=1024M count=1 2021-06-08 06:38:19,463 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: dd if=/dev/zero of=/data/full_disk_59354MB_1623159499.46 bs=1024M count=1 2021-06-08 06:38:23,099 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 16.0365 s, 67.0 MB/s 2021-06-08 06:38:23,099 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_56951MB_1623159503.1 bs=1024M count=1 2021-06-08 06:38:23,099 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: dd if=/dev/zero of=/data/full_disk_56951MB_1623159503.1 bs=1024M count=1 2021-06-08 06:38:28,752 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 19.626 s, 54.7 MB/s 2021-06-08 06:38:28,753 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_54633MB_1623159508.75 bs=1024M count=1 2021-06-08 06:38:28,753 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: dd if=/dev/zero of=/data/full_disk_54633MB_1623159508.75 bs=1024M count=1 2021-06-08 06:38:29,354 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 6.15537 s, 174 MB/s 2021-06-08 06:38:29,354 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_55927MB_1623159509.35 bs=1024M count=1 2021-06-08 06:38:29,354 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: dd if=/dev/zero of=/data/full_disk_55927MB_1623159509.35 bs=1024M count=1 2021-06-08 06:38:29,367 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 9.80089 s, 110 MB/s 2021-06-08 06:38:29,367 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_58330MB_1623159509.37 bs=1024M count=1 2021-06-08 06:38:29,367 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: dd if=/dev/zero of=/data/full_disk_58330MB_1623159509.37 bs=1024M count=1 2021-06-08 06:38:30,098 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 16.2654 s, 66.0 MB/s 2021-06-08 06:38:30,098 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_61625MB_1623159510.1 bs=1024M count=1 2021-06-08 06:38:30,098 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: dd if=/dev/zero of=/data/full_disk_61625MB_1623159510.1 bs=1024M count=1 2021-06-08 06:38:30,467 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 1.6172 s, 664 MB/s 2021-06-08 06:38:30,467 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_53609MB_1623159510.47 bs=1024M count=1 2021-06-08 06:38:30,467 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: dd if=/dev/zero of=/data/full_disk_53609MB_1623159510.47 bs=1024M count=1 2021-06-08 06:38:37,930 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 8.47566 s, 127 MB/s 2021-06-08 06:38:37,930 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_54903MB_1623159517.93 bs=1024M count=1 2021-06-08 06:38:37,931 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: dd if=/dev/zero of=/data/full_disk_54903MB_1623159517.93 bs=1024M count=1 2021-06-08 06:38:38,551 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 9.08376 s, 118 MB/s 2021-06-08 06:38:38,552 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_57306MB_1623159518.55 bs=1024M count=1 2021-06-08 06:38:38,552 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: dd if=/dev/zero of=/data/full_disk_57306MB_1623159518.55 bs=1024M count=1 2021-06-08 06:38:40,214 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 9.65068 s, 111 MB/s 2021-06-08 06:38:40,216 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_52585MB_1623159520.22 bs=1024M count=1 2021-06-08 06:38:40,216 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: dd if=/dev/zero of=/data/full_disk_52585MB_1623159520.22 bs=1024M count=1 2021-06-08 06:38:40,352 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 10.1517 s, 106 MB/s 2021-06-08 06:38:40,352 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_60601MB_1623159520.35 bs=1024M count=1 2021-06-08 06:38:40,352 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: dd if=/dev/zero of=/data/full_disk_60601MB_1623159520.35 bs=1024M count=1 2021-06-08 06:38:51,020 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 12.9875 s, 82.7 MB/s 2021-06-08 06:38:51,020 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_53879MB_1623159531.02 bs=1024M count=1 2021-06-08 06:38:51,022 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: dd if=/dev/zero of=/data/full_disk_53879MB_1623159531.02 bs=1024M count=1 2021-06-08 06:38:51,191 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 12.5375 s, 85.6 MB/s 2021-06-08 06:38:51,191 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_56282MB_1623159531.19 bs=1024M count=1 2021-06-08 06:38:51,191 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: dd if=/dev/zero of=/data/full_disk_56282MB_1623159531.19 bs=1024M count=1 2021-06-08 06:38:51,322 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 10.8653 s, 98.8 MB/s 2021-06-08 06:38:51,322 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_59577MB_1623159531.32 bs=1024M count=1 2021-06-08 06:38:51,322 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: dd if=/dev/zero of=/data/full_disk_59577MB_1623159531.32 bs=1024M count=1 2021-06-08 06:38:51,982 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 11.6512 s, 92.2 MB/s 2021-06-08 06:38:51,982 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_51561MB_1623159531.98 bs=1024M count=1 2021-06-08 06:38:51,982 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: dd if=/dev/zero of=/data/full_disk_51561MB_1623159531.98 bs=1024M count=1 2021-06-08 06:38:56,073 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 4.94973 s, 217 MB/s 2021-06-08 06:38:56,075 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_52855MB_1623159536.08 bs=1024M count=1 2021-06-08 06:38:56,075 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: dd if=/dev/zero of=/data/full_disk_52855MB_1623159536.08 bs=1024M count=1 2021-06-08 06:38:58,780 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 7.47587 s, 144 MB/s 2021-06-08 06:38:58,782 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_55258MB_1623159538.78 bs=1024M count=1 2021-06-08 06:38:58,782 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: dd if=/dev/zero of=/data/full_disk_55258MB_1623159538.78 bs=1024M count=1 2021-06-08 06:39:00,367 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 8.27802 s, 130 MB/s 2021-06-08 06:39:00,367 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_50537MB_1623159540.37 bs=1024M count=1 2021-06-08 06:39:00,368 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: dd if=/dev/zero of=/data/full_disk_50537MB_1623159540.37 bs=1024M count=1 2021-06-08 06:39:02,415 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 10.9888 s, 97.7 MB/s 2021-06-08 06:39:02,415 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_58553MB_1623159542.42 bs=1024M count=1 2021-06-08 06:39:02,417 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: dd if=/dev/zero of=/data/full_disk_58553MB_1623159542.42 bs=1024M count=1 2021-06-08 06:39:12,732 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 16.5498 s, 64.9 MB/s 2021-06-08 06:39:12,733 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_51831MB_1623159552.73 bs=1024M count=1 2021-06-08 06:39:12,733 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: dd if=/dev/zero of=/data/full_disk_51831MB_1623159552.73 bs=1024M count=1 2021-06-08 06:39:12,977 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 14.083 s, 76.2 MB/s 2021-06-08 06:39:12,979 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_54234MB_1623159552.98 bs=1024M count=1 2021-06-08 06:39:12,979 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: dd if=/dev/zero of=/data/full_disk_54234MB_1623159552.98 bs=1024M count=1 2021-06-08 06:39:13,015 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 12.5281 s, 85.7 MB/s 2021-06-08 06:39:13,015 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_49513MB_1623159553.02 bs=1024M count=1 2021-06-08 06:39:13,015 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: dd if=/dev/zero of=/data/full_disk_49513MB_1623159553.02 bs=1024M count=1 2021-06-08 06:39:13,259 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 10.7331 s, 100 MB/s 2021-06-08 06:39:13,259 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_57529MB_1623159553.26 bs=1024M count=1 2021-06-08 06:39:13,259 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: dd if=/dev/zero of=/data/full_disk_57529MB_1623159553.26 bs=1024M count=1 2021-06-08 06:39:15,719 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 2.88796 s, 372 MB/s 2021-06-08 06:39:15,719 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_50807MB_1623159555.72 bs=1024M count=1 2021-06-08 06:39:15,719 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: dd if=/dev/zero of=/data/full_disk_50807MB_1623159555.72 bs=1024M count=1 2021-06-08 06:39:19,467 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 6.37913 s, 168 MB/s 2021-06-08 06:39:19,467 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_53210MB_1623159559.47 bs=1024M count=1 2021-06-08 06:39:19,467 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: dd if=/dev/zero of=/data/full_disk_53210MB_1623159559.47 bs=1024M count=1 2021-06-08 06:39:19,694 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 6.57396 s, 163 MB/s 2021-06-08 06:39:19,694 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_48489MB_1623159559.69 bs=1024M count=1 2021-06-08 06:39:19,694 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: dd if=/dev/zero of=/data/full_disk_48489MB_1623159559.69 bs=1024M count=1 2021-06-08 06:39:22,171 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 8.80185 s, 122 MB/s 2021-06-08 06:39:22,171 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_56505MB_1623159562.17 bs=1024M count=1 2021-06-08 06:39:22,171 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: dd if=/dev/zero of=/data/full_disk_56505MB_1623159562.17 bs=1024M count=1 2021-06-08 06:39:24,661 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 8.83847 s, 121 MB/s 2021-06-08 06:39:24,661 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_49783MB_1623159564.66 bs=1024M count=1 2021-06-08 06:39:24,661 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: dd if=/dev/zero of=/data/full_disk_49783MB_1623159564.66 bs=1024M count=1 2021-06-08 06:39:34,789 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 15.2122 s, 70.6 MB/s 2021-06-08 06:39:34,789 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_52186MB_1623159574.79 bs=1024M count=1 2021-06-08 06:39:34,789 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: dd if=/dev/zero of=/data/full_disk_52186MB_1623159574.79 bs=1024M count=1 2021-06-08 06:39:35,009 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 15.2093 s, 70.6 MB/s 2021-06-08 06:39:35,009 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_47465MB_1623159575.01 bs=1024M count=1 2021-06-08 06:39:35,009 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: dd if=/dev/zero of=/data/full_disk_47465MB_1623159575.01 bs=1024M count=1 2021-06-08 06:39:35,092 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 12.8153 s, 83.8 MB/s 2021-06-08 06:39:35,092 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_55481MB_1623159575.09 bs=1024M count=1 2021-06-08 06:39:35,092 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: dd if=/dev/zero of=/data/full_disk_55481MB_1623159575.09 bs=1024M count=1 2021-06-08 06:39:35,588 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 10.8175 s, 99.3 MB/s 2021-06-08 06:39:35,588 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_48759MB_1623159575.59 bs=1024M count=1 2021-06-08 06:39:35,588 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: dd if=/dev/zero of=/data/full_disk_48759MB_1623159575.59 bs=1024M count=1 2021-06-08 06:39:39,642 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 4.75287 s, 226 MB/s 2021-06-08 06:39:39,644 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_51162MB_1623159579.64 bs=1024M count=1 2021-06-08 06:39:39,644 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: dd if=/dev/zero of=/data/full_disk_51162MB_1623159579.64 bs=1024M count=1 2021-06-08 06:39:40,490 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 5.36798 s, 200 MB/s 2021-06-08 06:39:40,490 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_46441MB_1623159580.49 bs=1024M count=1 2021-06-08 06:39:40,490 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: dd if=/dev/zero of=/data/full_disk_46441MB_1623159580.49 bs=1024M count=1 2021-06-08 06:39:41,540 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 6.34726 s, 169 MB/s 2021-06-08 06:39:41,540 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_54457MB_1623159581.54 bs=1024M count=1 2021-06-08 06:39:41,540 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: dd if=/dev/zero of=/data/full_disk_54457MB_1623159581.54 bs=1024M count=1 2021-06-08 06:39:44,234 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 8.54669 s, 126 MB/s 2021-06-08 06:39:44,236 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_47735MB_1623159584.24 bs=1024M count=1 2021-06-08 06:39:44,236 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: dd if=/dev/zero of=/data/full_disk_47735MB_1623159584.24 bs=1024M count=1 2021-06-08 06:39:55,473 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 14.8757 s, 72.2 MB/s 2021-06-08 06:39:55,473 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_45417MB_1623159595.47 bs=1024M count=1 2021-06-08 06:39:55,473 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: dd if=/dev/zero of=/data/full_disk_45417MB_1623159595.47 bs=1024M count=1 2021-06-08 06:39:55,990 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 16.2401 s, 66.1 MB/s 2021-06-08 06:39:55,992 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_50138MB_1623159595.99 bs=1024M count=1 2021-06-08 06:39:55,992 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: dd if=/dev/zero of=/data/full_disk_50138MB_1623159595.99 bs=1024M count=1 2021-06-08 06:39:57,519 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 13.1761 s, 81.5 MB/s 2021-06-08 06:39:57,520 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_46711MB_1623159597.52 bs=1024M count=1 2021-06-08 06:39:57,520 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: dd if=/dev/zero of=/data/full_disk_46711MB_1623159597.52 bs=1024M count=1 2021-06-08 06:39:57,551 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 1.45338 s, 739 MB/s 2021-06-08 06:39:57,551 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_49114MB_1623159597.55 bs=1024M count=1 2021-06-08 06:39:57,551 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: dd if=/dev/zero of=/data/full_disk_49114MB_1623159597.55 bs=1024M count=1 2021-06-08 06:39:57,582 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 2.00105 s, 537 MB/s 2021-06-08 06:39:57,582 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_44393MB_1623159597.58 bs=1024M count=1 2021-06-08 06:39:57,582 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: dd if=/dev/zero of=/data/full_disk_44393MB_1623159597.58 bs=1024M count=1 2021-06-08 06:39:58,049 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 16.3972 s, 65.5 MB/s 2021-06-08 06:39:58,049 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_53433MB_1623159598.05 bs=1024M count=1 2021-06-08 06:39:58,049 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: dd if=/dev/zero of=/data/full_disk_53433MB_1623159598.05 bs=1024M count=1 2021-06-08 06:40:04,098 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 6.46992 s, 166 MB/s 2021-06-08 06:40:04,098 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_45687MB_1623159604.1 bs=1024M count=1 2021-06-08 06:40:04,099 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: dd if=/dev/zero of=/data/full_disk_45687MB_1623159604.1 bs=1024M count=1 2021-06-08 06:40:04,362 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 6.66787 s, 161 MB/s 2021-06-08 06:40:04,364 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_43369MB_1623159604.36 bs=1024M count=1 2021-06-08 06:40:04,364 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: dd if=/dev/zero of=/data/full_disk_43369MB_1623159604.36 bs=1024M count=1 2021-06-08 06:40:06,184 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 8.02546 s, 134 MB/s 2021-06-08 06:40:06,184 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_52409MB_1623159606.18 bs=1024M count=1 2021-06-08 06:40:06,184 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: dd if=/dev/zero of=/data/full_disk_52409MB_1623159606.18 bs=1024M count=1 2021-06-08 06:40:15,595 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 11.1267 s, 96.5 MB/s 2021-06-08 06:40:15,595 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_42345MB_1623159615.6 bs=1024M count=1 2021-06-08 06:40:15,595 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: dd if=/dev/zero of=/data/full_disk_42345MB_1623159615.6 bs=1024M count=1 2021-06-08 06:40:18,226 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 14.0223 s, 76.6 MB/s 2021-06-08 06:40:18,227 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_44663MB_1623159618.23 bs=1024M count=1 2021-06-08 06:40:18,227 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: dd if=/dev/zero of=/data/full_disk_44663MB_1623159618.23 bs=1024M count=1 2021-06-08 06:40:18,789 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 21.1237 s, 50.8 MB/s 2021-06-08 06:40:18,789 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_48090MB_1623159618.79 bs=1024M count=1 2021-06-08 06:40:18,789 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: dd if=/dev/zero of=/data/full_disk_48090MB_1623159618.79 bs=1024M count=1 2021-06-08 06:40:19,125 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 12.8306 s, 83.7 MB/s 2021-06-08 06:40:19,125 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_51385MB_1623159619.12 bs=1024M count=1 2021-06-08 06:40:19,125 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: dd if=/dev/zero of=/data/full_disk_51385MB_1623159619.12 bs=1024M count=1 2021-06-08 06:40:21,115 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 2.78948 s, 385 MB/s 2021-06-08 06:40:21,115 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_43639MB_1623159621.12 bs=1024M count=1 2021-06-08 06:40:21,115 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: dd if=/dev/zero of=/data/full_disk_43639MB_1623159621.12 bs=1024M count=1 2021-06-08 06:40:22,262 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 3.36251 s, 319 MB/s 2021-06-08 06:40:22,262 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_47066MB_1623159622.26 bs=1024M count=1 2021-06-08 06:40:22,262 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: dd if=/dev/zero of=/data/full_disk_47066MB_1623159622.26 bs=1024M count=1 2021-06-08 06:40:23,947 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 4.71336 s, 228 MB/s 2021-06-08 06:40:23,948 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_50361MB_1623159623.95 bs=1024M count=1 2021-06-08 06:40:23,948 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: dd if=/dev/zero of=/data/full_disk_50361MB_1623159623.95 bs=1024M count=1 2021-06-08 06:40:26,308 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 10.5997 s, 101 MB/s 2021-06-08 06:40:26,309 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_41321MB_1623159626.31 bs=1024M count=1 2021-06-08 06:40:26,309 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: dd if=/dev/zero of=/data/full_disk_41321MB_1623159626.31 bs=1024M count=1 2021-06-08 06:40:29,862 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 8.64004 s, 124 MB/s 2021-06-08 06:40:29,864 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_42615MB_1623159629.86 bs=1024M count=1 2021-06-08 06:40:29,864 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: dd if=/dev/zero of=/data/full_disk_42615MB_1623159629.86 bs=1024M count=1 2021-06-08 06:40:39,473 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 13.0574 s, 82.2 MB/s 2021-06-08 06:40:39,473 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_40297MB_1623159639.47 bs=1024M count=1 2021-06-08 06:40:39,473 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: dd if=/dev/zero of=/data/full_disk_40297MB_1623159639.47 bs=1024M count=1 2021-06-08 06:40:40,789 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 18.4189 s, 58.3 MB/s 2021-06-08 06:40:40,789 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_46042MB_1623159640.79 bs=1024M count=1 2021-06-08 06:40:40,789 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: dd if=/dev/zero of=/data/full_disk_46042MB_1623159640.79 bs=1024M count=1 2021-06-08 06:40:40,865 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 16.8083 s, 63.9 MB/s 2021-06-08 06:40:40,865 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_49337MB_1623159640.87 bs=1024M count=1 2021-06-08 06:40:40,865 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: dd if=/dev/zero of=/data/full_disk_49337MB_1623159640.87 bs=1024M count=1 2021-06-08 06:40:41,142 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 8.99202 s, 119 MB/s 2021-06-08 06:40:41,142 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_41591MB_1623159641.14 bs=1024M count=1 2021-06-08 06:40:41,142 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: dd if=/dev/zero of=/data/full_disk_41591MB_1623159641.14 bs=1024M count=1 2021-06-08 06:40:43,983 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 4.40337 s, 244 MB/s 2021-06-08 06:40:43,984 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_39273MB_1623159643.98 bs=1024M count=1 2021-06-08 06:40:43,984 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: dd if=/dev/zero of=/data/full_disk_39273MB_1623159643.98 bs=1024M count=1 2021-06-08 06:40:45,773 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 4.88335 s, 220 MB/s 2021-06-08 06:40:45,773 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_45018MB_1623159645.77 bs=1024M count=1 2021-06-08 06:40:45,773 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: dd if=/dev/zero of=/data/full_disk_45018MB_1623159645.77 bs=1024M count=1 2021-06-08 06:40:47,437 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 6.45535 s, 166 MB/s 2021-06-08 06:40:47,437 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_48313MB_1623159647.44 bs=1024M count=1 2021-06-08 06:40:47,437 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: dd if=/dev/zero of=/data/full_disk_48313MB_1623159647.44 bs=1024M count=1 2021-06-08 06:40:48,183 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 6.93743 s, 155 MB/s 2021-06-08 06:40:48,183 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_40567MB_1623159648.18 bs=1024M count=1 2021-06-08 06:40:48,183 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: dd if=/dev/zero of=/data/full_disk_40567MB_1623159648.18 bs=1024M count=1 2021-06-08 06:40:56,273 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 12.1861 s, 88.1 MB/s 2021-06-08 06:40:56,273 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_38249MB_1623159656.27 bs=1024M count=1 2021-06-08 06:40:56,273 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: dd if=/dev/zero of=/data/full_disk_38249MB_1623159656.27 bs=1024M count=1 2021-06-08 06:41:02,881 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 17.0042 s, 63.1 MB/s 2021-06-08 06:41:02,881 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_43994MB_1623159662.88 bs=1024M count=1 2021-06-08 06:41:02,881 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: dd if=/dev/zero of=/data/full_disk_43994MB_1623159662.88 bs=1024M count=1 2021-06-08 06:41:03,006 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 15.4537 s, 69.5 MB/s 2021-06-08 06:41:03,006 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_47289MB_1623159663.01 bs=1024M count=1 2021-06-08 06:41:03,006 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: dd if=/dev/zero of=/data/full_disk_47289MB_1623159663.01 bs=1024M count=1 2021-06-08 06:41:03,015 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 14.7278 s, 72.9 MB/s 2021-06-08 06:41:03,016 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_39543MB_1623159663.02 bs=1024M count=1 2021-06-08 06:41:03,016 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: dd if=/dev/zero of=/data/full_disk_39543MB_1623159663.02 bs=1024M count=1 2021-06-08 06:41:03,509 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 7.12919 s, 151 MB/s 2021-06-08 06:41:03,509 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_37225MB_1623159663.51 bs=1024M count=1 2021-06-08 06:41:03,509 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: dd if=/dev/zero of=/data/full_disk_37225MB_1623159663.51 bs=1024M count=1 2021-06-08 06:41:07,688 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 4.57098 s, 235 MB/s 2021-06-08 06:41:07,688 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_46265MB_1623159667.69 bs=1024M count=1 2021-06-08 06:41:07,690 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: dd if=/dev/zero of=/data/full_disk_46265MB_1623159667.69 bs=1024M count=1 2021-06-08 06:41:07,773 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 4.79261 s, 224 MB/s 2021-06-08 06:41:07,773 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_42970MB_1623159667.77 bs=1024M count=1 2021-06-08 06:41:07,773 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: dd if=/dev/zero of=/data/full_disk_42970MB_1623159667.77 bs=1024M count=1 2021-06-08 06:41:09,140 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 6.01038 s, 179 MB/s 2021-06-08 06:41:09,141 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_38519MB_1623159669.14 bs=1024M count=1 2021-06-08 06:41:09,141 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: dd if=/dev/zero of=/data/full_disk_38519MB_1623159669.14 bs=1024M count=1 2021-06-08 06:41:12,312 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 8.69916 s, 123 MB/s 2021-06-08 06:41:12,312 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_36201MB_1623159672.31 bs=1024M count=1 2021-06-08 06:41:12,312 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: dd if=/dev/zero of=/data/full_disk_36201MB_1623159672.31 bs=1024M count=1 2021-06-08 06:41:21,237 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 13.4393 s, 79.9 MB/s 2021-06-08 06:41:21,239 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_45241MB_1623159681.24 bs=1024M count=1 2021-06-08 06:41:21,239 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: dd if=/dev/zero of=/data/full_disk_45241MB_1623159681.24 bs=1024M count=1 2021-06-08 06:41:24,752 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 16.8698 s, 63.6 MB/s 2021-06-08 06:41:24,753 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_41946MB_1623159684.75 bs=1024M count=1 2021-06-08 06:41:24,753 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: dd if=/dev/zero of=/data/full_disk_41946MB_1623159684.75 bs=1024M count=1 2021-06-08 06:41:24,792 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 15.5419 s, 69.1 MB/s 2021-06-08 06:41:24,793 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_37495MB_1623159684.79 bs=1024M count=1 2021-06-08 06:41:24,793 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: dd if=/dev/zero of=/data/full_disk_37495MB_1623159684.79 bs=1024M count=1 2021-06-08 06:41:25,219 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 12.7975 s, 83.9 MB/s 2021-06-08 06:41:25,219 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_35177MB_1623159685.22 bs=1024M count=1 2021-06-08 06:41:25,219 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: dd if=/dev/zero of=/data/full_disk_35177MB_1623159685.22 bs=1024M count=1 2021-06-08 06:41:25,640 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 4.27984 s, 251 MB/s 2021-06-08 06:41:25,640 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_44217MB_1623159685.64 bs=1024M count=1 2021-06-08 06:41:25,640 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: dd if=/dev/zero of=/data/full_disk_44217MB_1623159685.64 bs=1024M count=1 2021-06-08 06:41:29,457 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 4.59169 s, 234 MB/s 2021-06-08 06:41:29,457 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_40922MB_1623159689.46 bs=1024M count=1 2021-06-08 06:41:29,457 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: dd if=/dev/zero of=/data/full_disk_40922MB_1623159689.46 bs=1024M count=1 2021-06-08 06:41:29,640 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 4.73527 s, 227 MB/s 2021-06-08 06:41:29,640 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_36471MB_1623159689.64 bs=1024M count=1 2021-06-08 06:41:29,640 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: dd if=/dev/zero of=/data/full_disk_36471MB_1623159689.64 bs=1024M count=1 2021-06-08 06:41:33,950 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 8.61552 s, 125 MB/s 2021-06-08 06:41:33,950 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_34153MB_1623159693.95 bs=1024M count=1 2021-06-08 06:41:33,950 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: dd if=/dev/zero of=/data/full_disk_34153MB_1623159693.95 bs=1024M count=1 2021-06-08 06:41:35,063 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 9.30765 s, 115 MB/s 2021-06-08 06:41:35,063 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_43193MB_1623159695.06 bs=1024M count=1 2021-06-08 06:41:35,063 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: dd if=/dev/zero of=/data/full_disk_43193MB_1623159695.06 bs=1024M count=1 2021-06-08 06:41:44,767 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 14.7318 s, 72.9 MB/s 2021-06-08 06:41:44,769 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_35447MB_1623159704.77 bs=1024M count=1 2021-06-08 06:41:44,769 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: dd if=/dev/zero of=/data/full_disk_35447MB_1623159704.77 bs=1024M count=1 2021-06-08 06:41:46,517 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 16.9552 s, 63.3 MB/s 2021-06-08 06:41:46,519 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_39898MB_1623159706.52 bs=1024M count=1 2021-06-08 06:41:46,519 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: dd if=/dev/zero of=/data/full_disk_39898MB_1623159706.52 bs=1024M count=1 2021-06-08 06:41:47,023 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 11.8415 s, 90.7 MB/s 2021-06-08 06:41:47,023 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_42169MB_1623159707.02 bs=1024M count=1 2021-06-08 06:41:47,025 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: dd if=/dev/zero of=/data/full_disk_42169MB_1623159707.02 bs=1024M count=1 2021-06-08 06:41:47,316 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 13.2485 s, 81.0 MB/s 2021-06-08 06:41:47,316 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_33129MB_1623159707.32 bs=1024M count=1 2021-06-08 06:41:47,316 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: dd if=/dev/zero of=/data/full_disk_33129MB_1623159707.32 bs=1024M count=1 2021-06-08 06:41:47,694 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 2.80644 s, 383 MB/s 2021-06-08 06:41:47,694 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_34423MB_1623159707.69 bs=1024M count=1 2021-06-08 06:41:47,694 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: dd if=/dev/zero of=/data/full_disk_34423MB_1623159707.69 bs=1024M count=1 2021-06-08 06:41:50,323 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 3.6919 s, 291 MB/s 2021-06-08 06:41:50,323 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_38874MB_1623159710.32 bs=1024M count=1 2021-06-08 06:41:50,323 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: dd if=/dev/zero of=/data/full_disk_38874MB_1623159710.32 bs=1024M count=1 2021-06-08 06:41:54,680 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 7.53386 s, 143 MB/s 2021-06-08 06:41:54,680 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_41145MB_1623159714.68 bs=1024M count=1 2021-06-08 06:41:54,680 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: dd if=/dev/zero of=/data/full_disk_41145MB_1623159714.68 bs=1024M count=1 2021-06-08 06:41:55,418 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 7.60827 s, 141 MB/s 2021-06-08 06:41:55,418 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_33399MB_1623159715.42 bs=1024M count=1 2021-06-08 06:41:55,418 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: dd if=/dev/zero of=/data/full_disk_33399MB_1623159715.42 bs=1024M count=1 2021-06-08 06:41:57,016 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 9.58779 s, 112 MB/s 2021-06-08 06:41:57,016 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_32105MB_1623159717.02 bs=1024M count=1 2021-06-08 06:41:57,016 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: dd if=/dev/zero of=/data/full_disk_32105MB_1623159717.02 bs=1024M count=1 2021-06-08 06:42:09,101 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 14.3042 s, 75.1 MB/s 2021-06-08 06:42:09,101 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_40121MB_1623159729.1 bs=1024M count=1 2021-06-08 06:42:09,101 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: dd if=/dev/zero of=/data/full_disk_40121MB_1623159729.1 bs=1024M count=1 2021-06-08 06:42:09,270 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 13.734 s, 78.2 MB/s 2021-06-08 06:42:09,270 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_32375MB_1623159729.27 bs=1024M count=1 2021-06-08 06:42:09,270 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: dd if=/dev/zero of=/data/full_disk_32375MB_1623159729.27 bs=1024M count=1 2021-06-08 06:42:09,348 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 12.2232 s, 87.8 MB/s 2021-06-08 06:42:09,348 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_31081MB_1623159729.35 bs=1024M count=1 2021-06-08 06:42:09,348 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: dd if=/dev/zero of=/data/full_disk_31081MB_1623159729.35 bs=1024M count=1 2021-06-08 06:42:09,644 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 19.2061 s, 55.9 MB/s 2021-06-08 06:42:09,644 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_37850MB_1623159729.64 bs=1024M count=1 2021-06-08 06:42:09,644 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: dd if=/dev/zero of=/data/full_disk_37850MB_1623159729.64 bs=1024M count=1 2021-06-08 06:42:12,111 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 2.73535 s, 393 MB/s 2021-06-08 06:42:12,111 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_31351MB_1623159732.11 bs=1024M count=1 2021-06-08 06:42:12,111 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: dd if=/dev/zero of=/data/full_disk_31351MB_1623159732.11 bs=1024M count=1 2021-06-08 06:42:12,717 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 3.25784 s, 330 MB/s 2021-06-08 06:42:12,717 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_30057MB_1623159732.72 bs=1024M count=1 2021-06-08 06:42:12,717 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: dd if=/dev/zero of=/data/full_disk_30057MB_1623159732.72 bs=1024M count=1 2021-06-08 06:42:13,066 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 3.85282 s, 279 MB/s 2021-06-08 06:42:13,068 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_39097MB_1623159733.07 bs=1024M count=1 2021-06-08 06:42:13,068 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: dd if=/dev/zero of=/data/full_disk_39097MB_1623159733.07 bs=1024M count=1 2021-06-08 06:42:18,811 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 6.59887 s, 163 MB/s 2021-06-08 06:42:18,811 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_30327MB_1623159738.81 bs=1024M count=1 2021-06-08 06:42:18,811 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: dd if=/dev/zero of=/data/full_disk_30327MB_1623159738.81 bs=1024M count=1 2021-06-08 06:42:21,045 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 8.22396 s, 131 MB/s 2021-06-08 06:42:21,046 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_29033MB_1623159741.05 bs=1024M count=1 2021-06-08 06:42:21,046 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: dd if=/dev/zero of=/data/full_disk_29033MB_1623159741.05 bs=1024M count=1 2021-06-08 06:42:31,214 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 12.3024 s, 87.3 MB/s 2021-06-08 06:42:31,214 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_29303MB_1623159751.21 bs=1024M count=1 2021-06-08 06:42:31,214 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: dd if=/dev/zero of=/data/full_disk_29303MB_1623159751.21 bs=1024M count=1 2021-06-08 06:42:31,246 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 7.30837 s, 147 MB/s 2021-06-08 06:42:31,246 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_28009MB_1623159751.25 bs=1024M count=1 2021-06-08 06:42:31,246 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: dd if=/dev/zero of=/data/full_disk_28009MB_1623159751.25 bs=1024M count=1 2021-06-08 06:42:32,921 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 1.55777 s, 689 MB/s 2021-06-08 06:42:32,921 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_26985MB_1623159752.92 bs=1024M count=1 2021-06-08 06:42:32,923 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: dd if=/dev/zero of=/data/full_disk_26985MB_1623159752.92 bs=1024M count=1 2021-06-08 06:42:33,072 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 1.75216 s, 613 MB/s 2021-06-08 06:42:33,072 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_28279MB_1623159753.07 bs=1024M count=1 2021-06-08 06:42:33,072 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: dd if=/dev/zero of=/data/full_disk_28279MB_1623159753.07 bs=1024M count=1 2021-06-08 06:42:35,326 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 22.1541 s, 48.5 MB/s 2021-06-08 06:42:35,328 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_38073MB_1623159755.33 bs=1024M count=1 2021-06-08 06:42:35,328 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: dd if=/dev/zero of=/data/full_disk_38073MB_1623159755.33 bs=1024M count=1 2021-06-08 06:42:35,578 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 25.8255 s, 41.6 MB/s 2021-06-08 06:42:35,578 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_36826MB_1623159755.58 bs=1024M count=1 2021-06-08 06:42:35,578 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: dd if=/dev/zero of=/data/full_disk_36826MB_1623159755.58 bs=1024M count=1 2021-06-08 06:42:40,555 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 7.52246 s, 143 MB/s 2021-06-08 06:42:40,555 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_25961MB_1623159760.56 bs=1024M count=1 2021-06-08 06:42:40,555 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: dd if=/dev/zero of=/data/full_disk_25961MB_1623159760.56 bs=1024M count=1 2021-06-08 06:42:40,605 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 7.4302 s, 145 MB/s 2021-06-08 06:42:40,605 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_27255MB_1623159760.61 bs=1024M count=1 2021-06-08 06:42:40,605 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: dd if=/dev/zero of=/data/full_disk_27255MB_1623159760.61 bs=1024M count=1 2021-06-08 06:42:53,358 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 12.6024 s, 85.2 MB/s 2021-06-08 06:42:53,359 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_26231MB_1623159773.36 bs=1024M count=1 2021-06-08 06:42:53,359 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: dd if=/dev/zero of=/data/full_disk_26231MB_1623159773.36 bs=1024M count=1 2021-06-08 06:42:54,207 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 13.5348 s, 79.3 MB/s 2021-06-08 06:42:54,207 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_24937MB_1623159774.21 bs=1024M count=1 2021-06-08 06:42:54,207 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: dd if=/dev/zero of=/data/full_disk_24937MB_1623159774.21 bs=1024M count=1 2021-06-08 06:42:55,055 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 19.6078 s, 54.8 MB/s 2021-06-08 06:42:55,055 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_37049MB_1623159775.06 bs=1024M count=1 2021-06-08 06:42:55,055 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: dd if=/dev/zero of=/data/full_disk_37049MB_1623159775.06 bs=1024M count=1 2021-06-08 06:42:55,426 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 1.9559 s, 549 MB/s 2021-06-08 06:42:55,426 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_25207MB_1623159775.43 bs=1024M count=1 2021-06-08 06:42:55,426 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: dd if=/dev/zero of=/data/full_disk_25207MB_1623159775.43 bs=1024M count=1 2021-06-08 06:42:58,336 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 4.01347 s, 268 MB/s 2021-06-08 06:42:58,336 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_23913MB_1623159778.34 bs=1024M count=1 2021-06-08 06:42:58,336 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: dd if=/dev/zero of=/data/full_disk_23913MB_1623159778.34 bs=1024M count=1 2021-06-08 06:43:04,140 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 8.96737 s, 120 MB/s 2021-06-08 06:43:04,140 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_36025MB_1623159784.14 bs=1024M count=1 2021-06-08 06:43:04,141 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: dd if=/dev/zero of=/data/full_disk_36025MB_1623159784.14 bs=1024M count=1 2021-06-08 06:43:04,384 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 28.7005 s, 37.4 MB/s 2021-06-08 06:43:04,384 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_35802MB_1623159784.38 bs=1024M count=1 2021-06-08 06:43:04,384 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: dd if=/dev/zero of=/data/full_disk_35802MB_1623159784.38 bs=1024M count=1 2021-06-08 06:43:14,773 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 16.3064 s, 65.8 MB/s 2021-06-08 06:43:14,775 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_22889MB_1623159794.78 bs=1024M count=1 2021-06-08 06:43:14,775 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: dd if=/dev/zero of=/data/full_disk_22889MB_1623159794.78 bs=1024M count=1 2021-06-08 06:43:15,062 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 19.5262 s, 55.0 MB/s 2021-06-08 06:43:15,062 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_24183MB_1623159795.06 bs=1024M count=1 2021-06-08 06:43:15,062 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: dd if=/dev/zero of=/data/full_disk_24183MB_1623159795.06 bs=1024M count=1 2021-06-08 06:43:16,687 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 12.2002 s, 88.0 MB/s 2021-06-08 06:43:16,687 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_34778MB_1623159796.69 bs=1024M count=1 2021-06-08 06:43:16,687 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: dd if=/dev/zero of=/data/full_disk_34778MB_1623159796.69 bs=1024M count=1 2021-06-08 06:43:17,948 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 13.696 s, 78.4 MB/s 2021-06-08 06:43:17,950 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_35001MB_1623159797.95 bs=1024M count=1 2021-06-08 06:43:17,950 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: dd if=/dev/zero of=/data/full_disk_35001MB_1623159797.95 bs=1024M count=1 2021-06-08 06:43:22,881 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 7.71124 s, 139 MB/s 2021-06-08 06:43:22,882 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_23159MB_1623159802.88 bs=1024M count=1 2021-06-08 06:43:22,882 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: dd if=/dev/zero of=/data/full_disk_23159MB_1623159802.88 bs=1024M count=1 2021-06-08 06:43:33,038 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 16.2448 s, 66.1 MB/s 2021-06-08 06:43:33,038 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_33754MB_1623159813.04 bs=1024M count=1 2021-06-08 06:43:33,038 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: dd if=/dev/zero of=/data/full_disk_33754MB_1623159813.04 bs=1024M count=1 2021-06-08 06:43:35,121 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 20.2288 s, 53.1 MB/s 2021-06-08 06:43:35,121 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_21865MB_1623159815.12 bs=1024M count=1 2021-06-08 06:43:35,121 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: dd if=/dev/zero of=/data/full_disk_21865MB_1623159815.12 bs=1024M count=1 2021-06-08 06:43:35,736 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 17.6771 s, 60.7 MB/s 2021-06-08 06:43:35,736 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_33977MB_1623159815.74 bs=1024M count=1 2021-06-08 06:43:35,736 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: dd if=/dev/zero of=/data/full_disk_33977MB_1623159815.74 bs=1024M count=1 2021-06-08 06:43:37,335 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 14.3359 s, 74.9 MB/s 2021-06-08 06:43:37,335 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_22135MB_1623159817.34 bs=1024M count=1 2021-06-08 06:43:37,335 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: dd if=/dev/zero of=/data/full_disk_22135MB_1623159817.34 bs=1024M count=1 2021-06-08 06:43:38,947 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 5.79536 s, 185 MB/s 2021-06-08 06:43:38,947 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_32730MB_1623159818.95 bs=1024M count=1 2021-06-08 06:43:38,947 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: dd if=/dev/zero of=/data/full_disk_32730MB_1623159818.95 bs=1024M count=1 2021-06-08 06:43:40,661 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 4.81614 s, 223 MB/s 2021-06-08 06:43:40,661 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_32953MB_1623159820.66 bs=1024M count=1 2021-06-08 06:43:40,661 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: dd if=/dev/zero of=/data/full_disk_32953MB_1623159820.66 bs=1024M count=1 2021-06-08 06:43:43,323 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 4.26212 s, 252 MB/s 2021-06-08 06:43:43,323 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_31706MB_1623159823.32 bs=1024M count=1 2021-06-08 06:43:43,323 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: dd if=/dev/zero of=/data/full_disk_31706MB_1623159823.32 bs=1024M count=1 2021-06-08 06:43:46,559 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 5.77936 s, 186 MB/s 2021-06-08 06:43:46,561 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_31929MB_1623159826.56 bs=1024M count=1 2021-06-08 06:43:46,561 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: dd if=/dev/zero of=/data/full_disk_31929MB_1623159826.56 bs=1024M count=1 2021-06-08 06:43:46,989 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 9.54471 s, 112 MB/s 2021-06-08 06:43:46,990 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_21111MB_1623159826.99 bs=1024M count=1 2021-06-08 06:43:46,990 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: dd if=/dev/zero of=/data/full_disk_21111MB_1623159826.99 bs=1024M count=1 2021-06-08 06:43:57,065 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 13.6263 s, 78.8 MB/s 2021-06-08 06:43:57,065 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_30682MB_1623159837.07 bs=1024M count=1 2021-06-08 06:43:57,065 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: dd if=/dev/zero of=/data/full_disk_30682MB_1623159837.07 bs=1024M count=1 2021-06-08 06:44:00,594 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 3.34378 s, 321 MB/s 2021-06-08 06:44:00,595 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_29658MB_1623159840.59 bs=1024M count=1 2021-06-08 06:44:00,595 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: dd if=/dev/zero of=/data/full_disk_29658MB_1623159840.59 bs=1024M count=1 2021-06-08 06:44:01,325 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 14.6469 s, 73.3 MB/s 2021-06-08 06:44:01,325 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_30905MB_1623159841.33 bs=1024M count=1 2021-06-08 06:44:01,325 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: dd if=/dev/zero of=/data/full_disk_30905MB_1623159841.33 bs=1024M count=1 2021-06-08 06:44:02,555 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 27.3139 s, 39.3 MB/s 2021-06-08 06:44:02,555 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_20841MB_1623159842.56 bs=1024M count=1 2021-06-08 06:44:02,555 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: dd if=/dev/zero of=/data/full_disk_20841MB_1623159842.56 bs=1024M count=1 2021-06-08 06:44:03,176 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 2.46466 s, 436 MB/s 2021-06-08 06:44:03,177 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_28634MB_1623159843.18 bs=1024M count=1 2021-06-08 06:44:03,177 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: dd if=/dev/zero of=/data/full_disk_28634MB_1623159843.18 bs=1024M count=1 2021-06-08 06:44:03,948 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 2.51526 s, 427 MB/s 2021-06-08 06:44:03,950 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_29881MB_1623159843.95 bs=1024M count=1 2021-06-08 06:44:03,950 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: dd if=/dev/zero of=/data/full_disk_29881MB_1623159843.95 bs=1024M count=1 2021-06-08 06:44:07,634 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 4.34532 s, 247 MB/s 2021-06-08 06:44:07,635 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_27610MB_1623159847.64 bs=1024M count=1 2021-06-08 06:44:07,635 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: dd if=/dev/zero of=/data/full_disk_27610MB_1623159847.64 bs=1024M count=1 2021-06-08 06:44:08,766 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 4.69602 s, 229 MB/s 2021-06-08 06:44:08,767 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_28857MB_1623159848.77 bs=1024M count=1 2021-06-08 06:44:08,767 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: dd if=/dev/zero of=/data/full_disk_28857MB_1623159848.77 bs=1024M count=1 2021-06-08 06:44:19,187 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 32.0832 s, 33.5 MB/s 2021-06-08 06:44:19,187 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_20087MB_1623159859.19 bs=1024M count=1 2021-06-08 06:44:19,187 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: dd if=/dev/zero of=/data/full_disk_20087MB_1623159859.19 bs=1024M count=1 2021-06-08 06:44:21,585 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 12.6992 s, 84.6 MB/s 2021-06-08 06:44:21,586 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_27833MB_1623159861.59 bs=1024M count=1 2021-06-08 06:44:21,586 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: dd if=/dev/zero of=/data/full_disk_27833MB_1623159861.59 bs=1024M count=1 2021-06-08 06:44:22,474 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 14.7307 s, 72.9 MB/s 2021-06-08 06:44:22,474 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_26586MB_1623159862.47 bs=1024M count=1 2021-06-08 06:44:22,474 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: dd if=/dev/zero of=/data/full_disk_26586MB_1623159862.47 bs=1024M count=1 2021-06-08 06:44:24,174 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 2.46938 s, 435 MB/s 2021-06-08 06:44:24,174 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_26809MB_1623159864.17 bs=1024M count=1 2021-06-08 06:44:24,174 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: dd if=/dev/zero of=/data/full_disk_26809MB_1623159864.17 bs=1024M count=1 2021-06-08 06:44:25,661 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 3.06305 s, 351 MB/s 2021-06-08 06:44:25,661 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_25562MB_1623159865.66 bs=1024M count=1 2021-06-08 06:44:25,661 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: dd if=/dev/zero of=/data/full_disk_25562MB_1623159865.66 bs=1024M count=1 2021-06-08 06:44:25,897 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 23.2209 s, 46.2 MB/s 2021-06-08 06:44:25,897 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_19817MB_1623159865.9 bs=1024M count=1 2021-06-08 06:44:25,897 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: dd if=/dev/zero of=/data/full_disk_19817MB_1623159865.9 bs=1024M count=1 2021-06-08 06:44:27,753 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 1.97274 s, 544 MB/s 2021-06-08 06:44:27,753 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_24538MB_1623159867.75 bs=1024M count=1 2021-06-08 06:44:27,753 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: dd if=/dev/zero of=/data/full_disk_24538MB_1623159867.75 bs=1024M count=1 2021-06-08 06:44:27,937 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 3.6382 s, 295 MB/s 2021-06-08 06:44:27,937 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_25785MB_1623159867.94 bs=1024M count=1 2021-06-08 06:44:27,937 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: dd if=/dev/zero of=/data/full_disk_25785MB_1623159867.94 bs=1024M count=1 2021-06-08 06:44:31,538 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 3.67098 s, 292 MB/s 2021-06-08 06:44:31,538 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_23514MB_1623159871.54 bs=1024M count=1 2021-06-08 06:44:31,538 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: dd if=/dev/zero of=/data/full_disk_23514MB_1623159871.54 bs=1024M count=1 2021-06-08 06:44:40,637 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 14.6192 s, 73.4 MB/s 2021-06-08 06:44:40,638 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_18793MB_1623159880.64 bs=1024M count=1 2021-06-08 06:44:40,638 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: dd if=/dev/zero of=/data/full_disk_18793MB_1623159880.64 bs=1024M count=1 2021-06-08 06:44:43,420 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 24.1091 s, 44.5 MB/s 2021-06-08 06:44:43,421 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_19063MB_1623159883.42 bs=1024M count=1 2021-06-08 06:44:43,421 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: dd if=/dev/zero of=/data/full_disk_19063MB_1623159883.42 bs=1024M count=1 2021-06-08 06:44:43,839 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 15.7774 s, 68.1 MB/s 2021-06-08 06:44:43,841 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_24761MB_1623159883.84 bs=1024M count=1 2021-06-08 06:44:43,841 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: dd if=/dev/zero of=/data/full_disk_24761MB_1623159883.84 bs=1024M count=1 2021-06-08 06:44:45,819 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 2.26872 s, 473 MB/s 2021-06-08 06:44:45,819 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_18039MB_1623159885.82 bs=1024M count=1 2021-06-08 06:44:45,819 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: dd if=/dev/zero of=/data/full_disk_18039MB_1623159885.82 bs=1024M count=1 2021-06-08 06:44:45,838 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 5.08301 s, 211 MB/s 2021-06-08 06:44:45,838 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_17769MB_1623159885.84 bs=1024M count=1 2021-06-08 06:44:45,838 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: dd if=/dev/zero of=/data/full_disk_17769MB_1623159885.84 bs=1024M count=1 2021-06-08 06:44:45,957 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 2.00258 s, 536 MB/s 2021-06-08 06:44:45,957 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_23737MB_1623159885.96 bs=1024M count=1 2021-06-08 06:44:45,957 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: dd if=/dev/zero of=/data/full_disk_23737MB_1623159885.96 bs=1024M count=1 2021-06-08 06:44:46,947 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 15.2879 s, 70.2 MB/s 2021-06-08 06:44:46,947 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_22490MB_1623159886.95 bs=1024M count=1 2021-06-08 06:44:46,947 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: dd if=/dev/zero of=/data/full_disk_22490MB_1623159886.95 bs=1024M count=1 2021-06-08 06:44:51,558 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 5.49132 s, 196 MB/s 2021-06-08 06:44:51,558 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_22713MB_1623159891.56 bs=1024M count=1 2021-06-08 06:44:51,558 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: dd if=/dev/zero of=/data/full_disk_22713MB_1623159891.56 bs=1024M count=1 2021-06-08 06:44:52,878 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 5.80999 s, 185 MB/s 2021-06-08 06:44:52,878 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_21466MB_1623159892.88 bs=1024M count=1 2021-06-08 06:44:52,878 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: dd if=/dev/zero of=/data/full_disk_21466MB_1623159892.88 bs=1024M count=1 2021-06-08 06:44:54,890 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 8.93603 s, 120 MB/s 2021-06-08 06:44:54,890 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_16745MB_1623159894.89 bs=1024M count=1 2021-06-08 06:44:54,890 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: dd if=/dev/zero of=/data/full_disk_16745MB_1623159894.89 bs=1024M count=1 2021-06-08 06:45:05,851 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 8.32924 s, 129 MB/s 2021-06-08 06:45:05,851 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_15721MB_1623159905.85 bs=1024M count=1 2021-06-08 06:45:05,851 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: dd if=/dev/zero of=/data/full_disk_15721MB_1623159905.85 bs=1024M count=1 2021-06-08 06:45:05,982 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 20.043 s, 53.6 MB/s 2021-06-08 06:45:05,983 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_17015MB_1623159905.98 bs=1024M count=1 2021-06-08 06:45:05,983 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: dd if=/dev/zero of=/data/full_disk_17015MB_1623159905.98 bs=1024M count=1 2021-06-08 06:45:06,240 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 13.244 s, 81.1 MB/s 2021-06-08 06:45:06,242 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_20442MB_1623159906.24 bs=1024M count=1 2021-06-08 06:45:06,242 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: dd if=/dev/zero of=/data/full_disk_20442MB_1623159906.24 bs=1024M count=1 2021-06-08 06:45:06,602 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 14.8311 s, 72.4 MB/s 2021-06-08 06:45:06,602 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_21689MB_1623159906.6 bs=1024M count=1 2021-06-08 06:45:06,602 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: dd if=/dev/zero of=/data/full_disk_21689MB_1623159906.6 bs=1024M count=1 2021-06-08 06:45:10,744 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 4.65616 s, 231 MB/s 2021-06-08 06:45:10,746 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_15991MB_1623159910.75 bs=1024M count=1 2021-06-08 06:45:10,746 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: dd if=/dev/zero of=/data/full_disk_15991MB_1623159910.75 bs=1024M count=1 2021-06-08 06:45:11,282 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 4.55613 s, 236 MB/s 2021-06-08 06:45:11,282 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_20665MB_1623159911.28 bs=1024M count=1 2021-06-08 06:45:11,282 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: dd if=/dev/zero of=/data/full_disk_20665MB_1623159911.28 bs=1024M count=1 2021-06-08 06:45:11,576 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 5.61284 s, 191 MB/s 2021-06-08 06:45:11,576 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_14697MB_1623159911.58 bs=1024M count=1 2021-06-08 06:45:11,576 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: dd if=/dev/zero of=/data/full_disk_14697MB_1623159911.58 bs=1024M count=1 2021-06-08 06:45:11,895 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 5.52639 s, 194 MB/s 2021-06-08 06:45:11,897 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_19418MB_1623159911.9 bs=1024M count=1 2021-06-08 06:45:11,897 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: dd if=/dev/zero of=/data/full_disk_19418MB_1623159911.9 bs=1024M count=1 2021-06-08 06:45:16,994 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 5.598 s, 192 MB/s 2021-06-08 06:45:16,996 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_19641MB_1623159916.99 bs=1024M count=1 2021-06-08 06:45:16,996 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: dd if=/dev/zero of=/data/full_disk_19641MB_1623159916.99 bs=1024M count=1 2021-06-08 06:45:26,918 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 15.2322 s, 70.5 MB/s 2021-06-08 06:45:26,920 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_13673MB_1623159926.92 bs=1024M count=1 2021-06-08 06:45:26,920 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: dd if=/dev/zero of=/data/full_disk_13673MB_1623159926.92 bs=1024M count=1 2021-06-08 06:45:27,930 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 15.9109 s, 67.5 MB/s 2021-06-08 06:45:27,931 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_18394MB_1623159927.93 bs=1024M count=1 2021-06-08 06:45:27,931 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: dd if=/dev/zero of=/data/full_disk_18394MB_1623159927.93 bs=1024M count=1 2021-06-08 06:45:28,619 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 1.58285 s, 678 MB/s 2021-06-08 06:45:28,621 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_12649MB_1623159928.62 bs=1024M count=1 2021-06-08 06:45:28,621 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: dd if=/dev/zero of=/data/full_disk_12649MB_1623159928.62 bs=1024M count=1 2021-06-08 06:45:29,756 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 12.6486 s, 84.9 MB/s 2021-06-08 06:45:29,756 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_18617MB_1623159929.76 bs=1024M count=1 2021-06-08 06:45:29,756 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: dd if=/dev/zero of=/data/full_disk_18617MB_1623159929.76 bs=1024M count=1 2021-06-08 06:45:30,608 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 2.54528 s, 422 MB/s 2021-06-08 06:45:30,608 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_17370MB_1623159930.61 bs=1024M count=1 2021-06-08 06:45:30,608 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: dd if=/dev/zero of=/data/full_disk_17370MB_1623159930.61 bs=1024M count=1 2021-06-08 06:45:30,861 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 19.9979 s, 53.7 MB/s 2021-06-08 06:45:30,861 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_14967MB_1623159930.86 bs=1024M count=1 2021-06-08 06:45:30,862 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: dd if=/dev/zero of=/data/full_disk_14967MB_1623159930.86 bs=1024M count=1 2021-06-08 06:45:34,742 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 6.01032 s, 179 MB/s 2021-06-08 06:45:34,742 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_11625MB_1623159934.74 bs=1024M count=1 2021-06-08 06:45:34,742 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: dd if=/dev/zero of=/data/full_disk_11625MB_1623159934.74 bs=1024M count=1 2021-06-08 06:45:38,328 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 7.59714 s, 141 MB/s 2021-06-08 06:45:38,328 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_16346MB_1623159938.33 bs=1024M count=1 2021-06-08 06:45:38,328 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: dd if=/dev/zero of=/data/full_disk_16346MB_1623159938.33 bs=1024M count=1 2021-06-08 06:45:39,114 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 8.13337 s, 132 MB/s 2021-06-08 06:45:39,114 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_13943MB_1623159939.11 bs=1024M count=1 2021-06-08 06:45:39,114 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: dd if=/dev/zero of=/data/full_disk_13943MB_1623159939.11 bs=1024M count=1 2021-06-08 06:45:49,312 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 19.4367 s, 55.2 MB/s 2021-06-08 06:45:49,313 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_17593MB_1623159949.31 bs=1024M count=1 2021-06-08 06:45:49,313 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: dd if=/dev/zero of=/data/full_disk_17593MB_1623159949.31 bs=1024M count=1 2021-06-08 06:45:50,278 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 15.4251 s, 69.6 MB/s 2021-06-08 06:45:50,278 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_10601MB_1623159950.28 bs=1024M count=1 2021-06-08 06:45:50,278 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: dd if=/dev/zero of=/data/full_disk_10601MB_1623159950.28 bs=1024M count=1 2021-06-08 06:45:50,993 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 12.5393 s, 85.6 MB/s 2021-06-08 06:45:50,993 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_15322MB_1623159950.99 bs=1024M count=1 2021-06-08 06:45:50,993 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: dd if=/dev/zero of=/data/full_disk_15322MB_1623159950.99 bs=1024M count=1 2021-06-08 06:45:53,061 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 13.8368 s, 77.6 MB/s 2021-06-08 06:45:53,062 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_12919MB_1623159953.06 bs=1024M count=1 2021-06-08 06:45:53,062 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: dd if=/dev/zero of=/data/full_disk_12919MB_1623159953.06 bs=1024M count=1 2021-06-08 06:45:53,703 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 3.30499 s, 325 MB/s 2021-06-08 06:45:53,703 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_9577MB_1623159953.7 bs=1024M count=1 2021-06-08 06:45:53,703 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: dd if=/dev/zero of=/data/full_disk_9577MB_1623159953.7 bs=1024M count=1 2021-06-08 06:45:55,908 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 4.79271 s, 224 MB/s 2021-06-08 06:45:55,908 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_14298MB_1623159955.91 bs=1024M count=1 2021-06-08 06:45:55,908 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: dd if=/dev/zero of=/data/full_disk_14298MB_1623159955.91 bs=1024M count=1 2021-06-08 06:45:56,526 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 2.65943 s, 404 MB/s 2021-06-08 06:45:56,526 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_8553MB_1623159956.53 bs=1024M count=1 2021-06-08 06:45:56,526 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: dd if=/dev/zero of=/data/full_disk_8553MB_1623159956.53 bs=1024M count=1 2021-06-08 06:45:57,486 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 4.30783 s, 249 MB/s 2021-06-08 06:45:57,486 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_11895MB_1623159957.49 bs=1024M count=1 2021-06-08 06:45:57,486 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: dd if=/dev/zero of=/data/full_disk_11895MB_1623159957.49 bs=1024M count=1 2021-06-08 06:45:59,312 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 3.28044 s, 327 MB/s 2021-06-08 06:45:59,312 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_13274MB_1623159959.31 bs=1024M count=1 2021-06-08 06:45:59,312 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: dd if=/dev/zero of=/data/full_disk_13274MB_1623159959.31 bs=1024M count=1 2021-06-08 06:46:10,806 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 21.3667 s, 50.3 MB/s 2021-06-08 06:46:10,806 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_16569MB_1623159970.81 bs=1024M count=1 2021-06-08 06:46:10,806 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: dd if=/dev/zero of=/data/full_disk_16569MB_1623159970.81 bs=1024M count=1 2021-06-08 06:46:10,914 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 13.3173 s, 80.6 MB/s 2021-06-08 06:46:10,914 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_10871MB_1623159970.91 bs=1024M count=1 2021-06-08 06:46:10,914 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: dd if=/dev/zero of=/data/full_disk_10871MB_1623159970.91 bs=1024M count=1 2021-06-08 06:46:13,500 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 2.4784 s, 433 MB/s 2021-06-08 06:46:13,500 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_9847MB_1623159973.5 bs=1024M count=1 2021-06-08 06:46:13,500 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: dd if=/dev/zero of=/data/full_disk_9847MB_1623159973.5 bs=1024M count=1 2021-06-08 06:46:13,927 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 2.99027 s, 359 MB/s 2021-06-08 06:46:13,927 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_15545MB_1623159973.93 bs=1024M count=1 2021-06-08 06:46:13,927 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: dd if=/dev/zero of=/data/full_disk_15545MB_1623159973.93 bs=1024M count=1 2021-06-08 06:46:14,157 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 17.513 s, 61.3 MB/s 2021-06-08 06:46:14,157 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_7529MB_1623159974.16 bs=1024M count=1 2021-06-08 06:46:14,157 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: dd if=/dev/zero of=/data/full_disk_7529MB_1623159974.16 bs=1024M count=1 2021-06-08 06:46:15,002 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 15.5709 s, 69.0 MB/s 2021-06-08 06:46:15,002 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_12250MB_1623159975.0 bs=1024M count=1 2021-06-08 06:46:15,003 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: dd if=/dev/zero of=/data/full_disk_12250MB_1623159975.0 bs=1024M count=1 2021-06-08 06:46:18,565 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 4.39849 s, 244 MB/s 2021-06-08 06:46:18,565 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_14521MB_1623159978.57 bs=1024M count=1 2021-06-08 06:46:18,565 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: dd if=/dev/zero of=/data/full_disk_14521MB_1623159978.57 bs=1024M count=1 2021-06-08 06:46:22,937 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 8.66274 s, 124 MB/s 2021-06-08 06:46:22,938 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_6505MB_1623159982.94 bs=1024M count=1 2021-06-08 06:46:22,938 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: dd if=/dev/zero of=/data/full_disk_6505MB_1623159982.94 bs=1024M count=1 2021-06-08 06:46:23,848 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 10.2385 s, 105 MB/s 2021-06-08 06:46:23,848 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_8823MB_1623159983.85 bs=1024M count=1 2021-06-08 06:46:23,848 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: dd if=/dev/zero of=/data/full_disk_8823MB_1623159983.85 bs=1024M count=1 2021-06-08 06:46:28,213 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 13.0992 s, 82.0 MB/s 2021-06-08 06:46:28,213 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_11226MB_1623159988.21 bs=1024M count=1 2021-06-08 06:46:28,213 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: dd if=/dev/zero of=/data/full_disk_11226MB_1623159988.21 bs=1024M count=1 2021-06-08 06:46:33,618 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 14.9346 s, 71.9 MB/s 2021-06-08 06:46:33,618 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_13497MB_1623159993.62 bs=1024M count=1 2021-06-08 06:46:33,618 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: dd if=/dev/zero of=/data/full_disk_13497MB_1623159993.62 bs=1024M count=1 2021-06-08 06:46:34,618 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 11.5655 s, 92.8 MB/s 2021-06-08 06:46:34,619 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_5481MB_1623159994.62 bs=1024M count=1 2021-06-08 06:46:34,619 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: dd if=/dev/zero of=/data/full_disk_5481MB_1623159994.62 bs=1024M count=1 2021-06-08 06:46:35,457 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 1.70698 s, 629 MB/s 2021-06-08 06:46:35,457 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_12473MB_1623159995.46 bs=1024M count=1 2021-06-08 06:46:35,457 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: dd if=/dev/zero of=/data/full_disk_12473MB_1623159995.46 bs=1024M count=1 2021-06-08 06:46:36,079 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 7.73229 s, 139 MB/s 2021-06-08 06:46:36,079 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_10202MB_1623159996.08 bs=1024M count=1 2021-06-08 06:46:36,079 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: dd if=/dev/zero of=/data/full_disk_10202MB_1623159996.08 bs=1024M count=1 2021-06-08 06:46:38,155 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 14.2011 s, 75.6 MB/s 2021-06-08 06:46:38,157 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_7799MB_1623159998.16 bs=1024M count=1 2021-06-08 06:46:38,157 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: dd if=/dev/zero of=/data/full_disk_7799MB_1623159998.16 bs=1024M count=1 2021-06-08 06:46:38,549 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 3.80222 s, 282 MB/s 2021-06-08 06:46:38,551 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_4457MB_1623159998.55 bs=1024M count=1 2021-06-08 06:46:38,551 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: dd if=/dev/zero of=/data/full_disk_4457MB_1623159998.55 bs=1024M count=1 2021-06-08 06:46:42,519 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 6.93773 s, 155 MB/s 2021-06-08 06:46:42,519 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_11449MB_1623160002.52 bs=1024M count=1 2021-06-08 06:46:42,519 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: dd if=/dev/zero of=/data/full_disk_11449MB_1623160002.52 bs=1024M count=1 2021-06-08 06:46:49,167 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 10.8918 s, 98.6 MB/s 2021-06-08 06:46:49,167 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_6775MB_1623160009.17 bs=1024M count=1 2021-06-08 06:46:49,167 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: dd if=/dev/zero of=/data/full_disk_6775MB_1623160009.17 bs=1024M count=1 2021-06-08 06:46:54,538 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 15.8674 s, 67.7 MB/s 2021-06-08 06:46:54,538 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_3433MB_1623160014.54 bs=1024M count=1 2021-06-08 06:46:54,538 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: dd if=/dev/zero of=/data/full_disk_3433MB_1623160014.54 bs=1024M count=1 2021-06-08 06:46:54,884 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 18.6846 s, 57.5 MB/s 2021-06-08 06:46:54,884 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_9178MB_1623160014.88 bs=1024M count=1 2021-06-08 06:46:54,884 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: dd if=/dev/zero of=/data/full_disk_9178MB_1623160014.88 bs=1024M count=1 2021-06-08 06:46:55,361 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 12.7138 s, 84.5 MB/s 2021-06-08 06:46:55,361 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_10425MB_1623160015.36 bs=1024M count=1 2021-06-08 06:46:55,361 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: dd if=/dev/zero of=/data/full_disk_10425MB_1623160015.36 bs=1024M count=1 2021-06-08 06:46:56,341 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 1.60402 s, 669 MB/s 2021-06-08 06:46:56,341 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_2409MB_1623160016.34 bs=1024M count=1 2021-06-08 06:46:56,341 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: dd if=/dev/zero of=/data/full_disk_2409MB_1623160016.34 bs=1024M count=1 2021-06-08 06:46:56,802 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 1.78815 s, 600 MB/s 2021-06-08 06:46:56,802 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_8154MB_1623160016.8 bs=1024M count=1 2021-06-08 06:46:56,802 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: dd if=/dev/zero of=/data/full_disk_8154MB_1623160016.8 bs=1024M count=1 2021-06-08 06:46:57,730 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 8.455 s, 127 MB/s 2021-06-08 06:46:57,730 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_5751MB_1623160017.73 bs=1024M count=1 2021-06-08 06:46:57,730 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: dd if=/dev/zero of=/data/full_disk_5751MB_1623160017.73 bs=1024M count=1 2021-06-08 06:47:01,463 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 5.98443 s, 179 MB/s 2021-06-08 06:47:01,463 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_9401MB_1623160021.46 bs=1024M count=1 2021-06-08 06:47:01,463 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: dd if=/dev/zero of=/data/full_disk_9401MB_1623160021.46 bs=1024M count=1 2021-06-08 06:47:03,690 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 7.22746 s, 149 MB/s 2021-06-08 06:47:03,690 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_1385MB_1623160023.69 bs=1024M count=1 2021-06-08 06:47:03,691 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: dd if=/dev/zero of=/data/full_disk_1385MB_1623160023.69 bs=1024M count=1 2021-06-08 06:47:05,875 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 8.02822 s, 134 MB/s 2021-06-08 06:47:05,877 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_4727MB_1623160025.88 bs=1024M count=1 2021-06-08 06:47:05,877 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: dd if=/dev/zero of=/data/full_disk_4727MB_1623160025.88 bs=1024M count=1 2021-06-08 06:47:17,094 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 20.1754 s, 53.2 MB/s 2021-06-08 06:47:17,095 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_7130MB_1623160037.1 bs=1024M count=1 2021-06-08 06:47:17,095 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: dd if=/dev/zero of=/data/full_disk_7130MB_1623160037.1 bs=1024M count=1 2021-06-08 06:47:17,551 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 15.9622 s, 67.3 MB/s 2021-06-08 06:47:17,551 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_8377MB_1623160037.55 bs=1024M count=1 2021-06-08 06:47:17,551 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: dd if=/dev/zero of=/data/full_disk_8377MB_1623160037.55 bs=1024M count=1 2021-06-08 06:47:19,089 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 15.2804 s, 70.3 MB/s 2021-06-08 06:47:19,091 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_361MB_1623160039.09 bs=361M count=1 2021-06-08 06:47:19,091 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: dd if=/dev/zero of=/data/full_disk_361MB_1623160039.09 bs=361M count=1 2021-06-08 06:47:19,536 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 13.5411 s, 79.3 MB/s 2021-06-08 06:47:19,536 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_3703MB_1623160039.54 bs=1024M count=1 2021-06-08 06:47:19,536 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: dd if=/dev/zero of=/data/full_disk_3703MB_1623160039.54 bs=1024M count=1 2021-06-08 06:47:21,358 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 378535936 bytes (379 MB) copied, 2.2111 s, 171 MB/s 2021-06-08 06:47:21,359 | infra | DEBUG | Thread-19 | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.168 2021-06-08 06:47:21,395 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 1.73698 s, 618 MB/s 2021-06-08 06:47:21,395 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_2679MB_1623160041.4 bs=1024M count=1 2021-06-08 06:47:21,397 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: dd if=/dev/zero of=/data/full_disk_2679MB_1623160041.4 bs=1024M count=1 2021-06-08 06:47:21,565 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 3.89535 s, 276 MB/s 2021-06-08 06:47:21,565 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_7353MB_1623160041.57 bs=1024M count=1 2021-06-08 06:47:21,566 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: dd if=/dev/zero of=/data/full_disk_7353MB_1623160041.57 bs=1024M count=1 2021-06-08 06:47:24,153 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 2.46049 s, 436 MB/s 2021-06-08 06:47:24,153 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_6329MB_1623160044.15 bs=1024M count=1 2021-06-08 06:47:24,153 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: dd if=/dev/zero of=/data/full_disk_6329MB_1623160044.15 bs=1024M count=1 2021-06-08 06:47:25,698 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 4.18307 s, 257 MB/s 2021-06-08 06:47:25,698 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_1655MB_1623160045.7 bs=1024M count=1 2021-06-08 06:47:25,698 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: dd if=/dev/zero of=/data/full_disk_1655MB_1623160045.7 bs=1024M count=1 2021-06-08 06:47:36,622 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 12.3491 s, 86.9 MB/s 2021-06-08 06:47:36,622 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_5305MB_1623160056.62 bs=1024M count=1 2021-06-08 06:47:36,624 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: dd if=/dev/zero of=/data/full_disk_5305MB_1623160056.62 bs=1024M count=1 2021-06-08 06:47:38,831 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 13.0155 s, 82.5 MB/s 2021-06-08 06:47:38,831 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_631MB_1623160058.83 bs=631M count=1 2021-06-08 06:47:38,831 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: dd if=/dev/zero of=/data/full_disk_631MB_1623160058.83 bs=631M count=1 2021-06-08 06:47:38,974 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 21.7637 s, 49.3 MB/s 2021-06-08 06:47:38,974 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_6106MB_1623160058.97 bs=1024M count=1 2021-06-08 06:47:38,974 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: dd if=/dev/zero of=/data/full_disk_6106MB_1623160058.97 bs=1024M count=1 2021-06-08 06:47:40,096 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 661651456 bytes (662 MB) copied, 1.18447 s, 559 MB/s 2021-06-08 06:47:40,098 | infra | DEBUG | Thread-17 | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.196 2021-06-08 06:47:41,239 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 4.48369 s, 239 MB/s 2021-06-08 06:47:41,239 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_4281MB_1623160061.24 bs=1024M count=1 2021-06-08 06:47:41,239 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: dd if=/dev/zero of=/data/full_disk_4281MB_1623160061.24 bs=1024M count=1 2021-06-08 06:47:41,555 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 2.45135 s, 438 MB/s 2021-06-08 06:47:41,555 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_5082MB_1623160061.56 bs=1024M count=1 2021-06-08 06:47:41,555 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: dd if=/dev/zero of=/data/full_disk_5082MB_1623160061.56 bs=1024M count=1 2021-06-08 06:47:44,045 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 2.68815 s, 399 MB/s 2021-06-08 06:47:44,045 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_3257MB_1623160064.05 bs=1024M count=1 2021-06-08 06:47:44,046 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: dd if=/dev/zero of=/data/full_disk_3257MB_1623160064.05 bs=1024M count=1 2021-06-08 06:47:44,618 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 2.94495 s, 365 MB/s 2021-06-08 06:47:44,619 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_4058MB_1623160064.62 bs=1024M count=1 2021-06-08 06:47:44,619 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: dd if=/dev/zero of=/data/full_disk_4058MB_1623160064.62 bs=1024M count=1 2021-06-08 06:47:48,176 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 4.00123 s, 268 MB/s 2021-06-08 06:47:48,177 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_2233MB_1623160068.18 bs=1024M count=1 2021-06-08 06:47:48,177 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: dd if=/dev/zero of=/data/full_disk_2233MB_1623160068.18 bs=1024M count=1 2021-06-08 06:47:48,984 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 4.25104 s, 253 MB/s 2021-06-08 06:47:48,986 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_3034MB_1623160068.99 bs=1024M count=1 2021-06-08 06:47:48,986 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: dd if=/dev/zero of=/data/full_disk_3034MB_1623160068.99 bs=1024M count=1 2021-06-08 06:48:00,934 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 12.6428 s, 84.9 MB/s 2021-06-08 06:48:00,936 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_1209MB_1623160080.93 bs=1024M count=1 2021-06-08 06:48:00,936 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: dd if=/dev/zero of=/data/full_disk_1209MB_1623160080.93 bs=1024M count=1 2021-06-08 06:48:01,496 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 12.3935 s, 86.6 MB/s 2021-06-08 06:48:01,496 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_2010MB_1623160081.5 bs=1024M count=1 2021-06-08 06:48:01,496 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: dd if=/dev/zero of=/data/full_disk_2010MB_1623160081.5 bs=1024M count=1 2021-06-08 06:48:02,858 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 1.79326 s, 599 MB/s 2021-06-08 06:48:02,858 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_185MB_1623160082.86 bs=185M count=1 2021-06-08 06:48:02,858 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: dd if=/dev/zero of=/data/full_disk_185MB_1623160082.86 bs=185M count=1 2021-06-08 06:48:03,134 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 193986560 bytes (194 MB) copied, 0.233638 s, 830 MB/s 2021-06-08 06:48:03,134 | infra | DEBUG | Thread-20 | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.198 2021-06-08 06:48:04,428 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 2.80482 s, 383 MB/s 2021-06-08 06:48:04,430 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:87] dd if=/dev/zero of=/data/full_disk_986MB_1623160084.43 bs=986M count=1 2021-06-08 06:48:04,430 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: dd if=/dev/zero of=/data/full_disk_986MB_1623160084.43 bs=986M count=1 2021-06-08 06:48:08,653 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:91] 1+0 records in 1+0 records out 1033895936 bytes (1.0 GB) copied, 4.0986 s, 252 MB/s 2021-06-08 06:48:08,653 | infra | DEBUG | Thread-18 | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.199 2021-06-08 06:48:08,655 | infra | INFO | MainThread | [task_manager:add_new_task:23] Added new task: LoadDocsGen_default__default__default_create_5000000_6000000_1623160088.66 2021-06-08 06:48:08,657 | infra | DEBUG | MainThread | [task_manager:get_task_result:26] Getting task result for LoadDocsGen_default__default__default_create_5000000_6000000_1623160088.66 2021-06-08 06:48:08,657 | infra | INFO | pool-5-thread-3 | [task:start_task:87] Thread 'LoadDocsGen_default__default__default_create_5000000_6000000_1623160088.66' started 2021-06-08 06:48:08,688 | infra | INFO | pool-5-thread-3 | [task_manager:add_new_task:23] Added new task: PrintBucketStats_default_1623160088.67 2021-06-08 06:48:08,688 | infra | INFO | pool-5-thread-4 | [task:start_task:87] Thread 'PrintBucketStats_default_1623160088.67' started 2021-06-08 06:48:08,688 | infra | INFO | pool-5-thread-3 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623160088.66_create__5000000_5050000_1623160088.67 2021-06-08 06:48:08,688 | infra | INFO | pool-5-thread-5 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623160088.66_create__5000000_5050000_1623160088.67' started 2021-06-08 06:48:08,688 | infra | INFO | pool-5-thread-3 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623160088.66_create__5050000_5100000_1623160088.67 2021-06-08 06:48:08,690 | infra | INFO | pool-5-thread-6 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623160088.66_create__5050000_5100000_1623160088.67' started 2021-06-08 06:48:08,690 | infra | INFO | pool-5-thread-3 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623160088.66_create__5100000_5150000_1623160088.67 2021-06-08 06:48:08,690 | infra | INFO | pool-5-thread-9 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623160088.66_create__5100000_5150000_1623160088.67' started 2021-06-08 06:48:08,693 | infra | INFO | pool-5-thread-3 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623160088.66_create__5150000_5200000_1623160088.67 2021-06-08 06:48:08,693 | infra | INFO | pool-5-thread-11 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623160088.66_create__5150000_5200000_1623160088.67' started 2021-06-08 06:48:08,693 | infra | INFO | pool-5-thread-3 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623160088.66_create__5200000_5250000_1623160088.67 2021-06-08 06:48:08,693 | infra | INFO | pool-5-thread-18 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623160088.66_create__5200000_5250000_1623160088.67' started 2021-06-08 06:48:08,694 | infra | INFO | pool-5-thread-3 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623160088.66_create__5250000_5300000_1623160088.67 2021-06-08 06:48:08,694 | infra | INFO | pool-5-thread-3 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623160088.66_create__5300000_5350000_1623160088.67 2021-06-08 06:48:08,694 | infra | INFO | pool-5-thread-3 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623160088.66_create__5350000_5400000_1623160088.67 2021-06-08 06:48:08,694 | infra | INFO | pool-5-thread-3 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623160088.66_create__5400000_5450000_1623160088.67 2021-06-08 06:48:08,694 | infra | INFO | pool-5-thread-3 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623160088.66_create__5450000_5500000_1623160088.67 2021-06-08 06:48:08,694 | infra | INFO | pool-5-thread-3 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623160088.66_create__5500000_5550000_1623160088.67 2021-06-08 06:48:08,694 | infra | INFO | pool-5-thread-3 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623160088.66_create__5550000_5600000_1623160088.67 2021-06-08 06:48:08,694 | infra | INFO | pool-5-thread-3 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623160088.66_create__5600000_5650000_1623160088.67 2021-06-08 06:48:08,694 | infra | INFO | pool-5-thread-21 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623160088.66_create__5250000_5300000_1623160088.67' started 2021-06-08 06:48:08,694 | infra | INFO | pool-5-thread-3 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623160088.66_create__5650000_5700000_1623160088.67 2021-06-08 06:48:08,694 | infra | INFO | pool-5-thread-27 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623160088.66_create__5350000_5400000_1623160088.67' started 2021-06-08 06:48:08,696 | infra | INFO | pool-5-thread-14 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623160088.66_create__5600000_5650000_1623160088.67' started 2021-06-08 06:48:08,694 | infra | INFO | pool-5-thread-15 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623160088.66_create__5300000_5350000_1623160088.67' started 2021-06-08 06:48:08,694 | infra | INFO | pool-5-thread-10 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623160088.66_create__5400000_5450000_1623160088.67' started 2021-06-08 06:48:08,694 | infra | INFO | pool-5-thread-13 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623160088.66_create__5450000_5500000_1623160088.67' started 2021-06-08 06:48:08,694 | infra | INFO | pool-5-thread-12 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623160088.66_create__5550000_5600000_1623160088.67' started 2021-06-08 06:48:08,694 | infra | INFO | pool-5-thread-26 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623160088.66_create__5500000_5550000_1623160088.67' started 2021-06-08 06:48:08,696 | infra | INFO | pool-5-thread-16 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623160088.66_create__5650000_5700000_1623160088.67' started 2021-06-08 06:48:08,696 | infra | INFO | pool-5-thread-3 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623160088.66_create__5700000_5750000_1623160088.67 2021-06-08 06:48:08,697 | infra | INFO | pool-5-thread-19 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623160088.66_create__5700000_5750000_1623160088.67' started 2021-06-08 06:48:08,701 | infra | INFO | pool-5-thread-3 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623160088.66_create__5750000_5800000_1623160088.67 2021-06-08 06:48:08,701 | infra | INFO | pool-5-thread-28 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623160088.66_create__5750000_5800000_1623160088.67' started 2021-06-08 06:48:08,703 | infra | INFO | pool-5-thread-3 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623160088.66_create__5800000_5850000_1623160088.67 2021-06-08 06:48:08,703 | infra | INFO | pool-5-thread-23 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623160088.66_create__5800000_5850000_1623160088.67' started 2021-06-08 06:48:08,707 | infra | INFO | pool-5-thread-3 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623160088.66_create__5850000_5900000_1623160088.67 2021-06-08 06:48:08,707 | infra | INFO | pool-5-thread-17 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623160088.66_create__5850000_5900000_1623160088.67' started 2021-06-08 06:48:08,707 | infra | INFO | pool-5-thread-3 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623160088.66_create__5900000_5950000_1623160088.67 2021-06-08 06:48:08,709 | infra | INFO | pool-5-thread-20 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623160088.66_create__5900000_5950000_1623160088.67' started 2021-06-08 06:48:08,710 | infra | INFO | pool-5-thread-22 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623160088.66_create__5950000_6000000_1623160088.67' started 2021-06-08 06:48:08,710 | infra | INFO | pool-5-thread-3 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623160088.66_create__5950000_6000000_1623160088.67 2021-06-08 06:48:08,710 | infra | DEBUG | pool-5-thread-3 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623160088.66_create__5000000_5050000_1623160088.67 2021-06-08 06:48:08,858 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 06:48:08,859 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 06:48:10,016 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 06:48:10,016 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 06:48:11,221 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 06:48:11,221 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 06:48:12,453 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 06:48:12,453 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 06:48:13,773 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 06:48:13,775 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 06:48:15,091 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 06:48:15,091 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 06:48:16,437 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 06:48:16,437 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 06:48:17,779 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 06:48:17,779 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 06:48:19,203 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 06:48:19,203 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 06:48:20,006 | infra | INFO | pool-5-thread-22 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623160088.66_create__5950000_6000000_1623160088.67' completed 2021-06-08 06:48:20,068 | infra | INFO | pool-5-thread-26 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623160088.66_create__5500000_5550000_1623160088.67' completed 2021-06-08 06:48:20,091 | infra | INFO | pool-5-thread-23 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623160088.66_create__5800000_5850000_1623160088.67' completed 2021-06-08 06:48:20,092 | infra | INFO | pool-5-thread-12 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623160088.66_create__5550000_5600000_1623160088.67' completed 2021-06-08 06:48:20,134 | infra | INFO | pool-5-thread-20 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623160088.66_create__5900000_5950000_1623160088.67' completed 2021-06-08 06:48:20,148 | infra | INFO | pool-5-thread-28 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623160088.66_create__5750000_5800000_1623160088.67' completed 2021-06-08 06:48:20,151 | infra | INFO | pool-5-thread-11 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623160088.66_create__5150000_5200000_1623160088.67' completed 2021-06-08 06:48:20,164 | infra | INFO | pool-5-thread-9 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623160088.66_create__5100000_5150000_1623160088.67' completed 2021-06-08 06:48:20,174 | infra | INFO | pool-5-thread-27 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623160088.66_create__5350000_5400000_1623160088.67' completed 2021-06-08 06:48:20,188 | infra | INFO | pool-5-thread-21 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623160088.66_create__5250000_5300000_1623160088.67' completed 2021-06-08 06:48:20,236 | infra | INFO | pool-5-thread-19 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623160088.66_create__5700000_5750000_1623160088.67' completed 2021-06-08 06:48:20,266 | infra | INFO | pool-5-thread-6 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623160088.66_create__5050000_5100000_1623160088.67' completed 2021-06-08 06:48:20,288 | infra | INFO | pool-5-thread-17 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623160088.66_create__5850000_5900000_1623160088.67' completed 2021-06-08 06:48:20,292 | infra | INFO | pool-5-thread-16 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623160088.66_create__5650000_5700000_1623160088.67' completed 2021-06-08 06:48:20,306 | infra | INFO | pool-5-thread-15 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623160088.66_create__5300000_5350000_1623160088.67' completed 2021-06-08 06:48:20,338 | infra | INFO | pool-5-thread-18 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623160088.66_create__5200000_5250000_1623160088.67' completed 2021-06-08 06:48:20,341 | infra | INFO | pool-5-thread-5 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623160088.66_create__5000000_5050000_1623160088.67' completed 2021-06-08 06:48:20,344 | infra | DEBUG | pool-5-thread-3 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623160088.66_create__5000000_5050000_1623160088.67 are 50000 2021-06-08 06:48:20,344 | infra | INFO | pool-5-thread-14 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623160088.66_create__5600000_5650000_1623160088.67' completed 2021-06-08 06:48:20,344 | infra | DEBUG | pool-5-thread-3 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623160088.66_create__5050000_5100000_1623160088.67 2021-06-08 06:48:20,345 | infra | DEBUG | pool-5-thread-3 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623160088.66_create__5050000_5100000_1623160088.67 are 50000 2021-06-08 06:48:20,345 | infra | DEBUG | pool-5-thread-3 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623160088.66_create__5100000_5150000_1623160088.67 2021-06-08 06:48:20,345 | infra | DEBUG | pool-5-thread-3 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623160088.66_create__5100000_5150000_1623160088.67 are 50000 2021-06-08 06:48:20,345 | infra | DEBUG | pool-5-thread-3 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623160088.66_create__5150000_5200000_1623160088.67 2021-06-08 06:48:20,345 | infra | INFO | pool-5-thread-10 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623160088.66_create__5400000_5450000_1623160088.67' completed 2021-06-08 06:48:20,345 | infra | DEBUG | pool-5-thread-3 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623160088.66_create__5150000_5200000_1623160088.67 are 50000 2021-06-08 06:48:20,346 | infra | DEBUG | pool-5-thread-3 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623160088.66_create__5200000_5250000_1623160088.67 2021-06-08 06:48:20,346 | infra | DEBUG | pool-5-thread-3 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623160088.66_create__5200000_5250000_1623160088.67 are 50000 2021-06-08 06:48:20,346 | infra | DEBUG | pool-5-thread-3 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623160088.66_create__5250000_5300000_1623160088.67 2021-06-08 06:48:20,346 | infra | DEBUG | pool-5-thread-3 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623160088.66_create__5250000_5300000_1623160088.67 are 50000 2021-06-08 06:48:20,346 | infra | DEBUG | pool-5-thread-3 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623160088.66_create__5300000_5350000_1623160088.67 2021-06-08 06:48:20,346 | infra | DEBUG | pool-5-thread-3 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623160088.66_create__5300000_5350000_1623160088.67 are 50000 2021-06-08 06:48:20,346 | infra | DEBUG | pool-5-thread-3 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623160088.66_create__5350000_5400000_1623160088.67 2021-06-08 06:48:20,346 | infra | DEBUG | pool-5-thread-3 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623160088.66_create__5350000_5400000_1623160088.67 are 50000 2021-06-08 06:48:20,348 | infra | DEBUG | pool-5-thread-3 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623160088.66_create__5400000_5450000_1623160088.67 2021-06-08 06:48:20,348 | infra | DEBUG | pool-5-thread-3 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623160088.66_create__5400000_5450000_1623160088.67 are 50000 2021-06-08 06:48:20,348 | infra | DEBUG | pool-5-thread-3 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623160088.66_create__5450000_5500000_1623160088.67 2021-06-08 06:48:20,482 | infra | INFO | pool-5-thread-13 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623160088.66_create__5450000_5500000_1623160088.67' completed 2021-06-08 06:48:20,482 | infra | DEBUG | pool-5-thread-3 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623160088.66_create__5450000_5500000_1623160088.67 are 50000 2021-06-08 06:48:20,483 | infra | DEBUG | pool-5-thread-3 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623160088.66_create__5500000_5550000_1623160088.67 2021-06-08 06:48:20,483 | infra | DEBUG | pool-5-thread-3 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623160088.66_create__5500000_5550000_1623160088.67 are 50000 2021-06-08 06:48:20,483 | infra | DEBUG | pool-5-thread-3 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623160088.66_create__5550000_5600000_1623160088.67 2021-06-08 06:48:20,483 | infra | DEBUG | pool-5-thread-3 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623160088.66_create__5550000_5600000_1623160088.67 are 50000 2021-06-08 06:48:20,483 | infra | DEBUG | pool-5-thread-3 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623160088.66_create__5600000_5650000_1623160088.67 2021-06-08 06:48:20,483 | infra | DEBUG | pool-5-thread-3 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623160088.66_create__5600000_5650000_1623160088.67 are 50000 2021-06-08 06:48:20,483 | infra | DEBUG | pool-5-thread-3 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623160088.66_create__5650000_5700000_1623160088.67 2021-06-08 06:48:20,483 | infra | DEBUG | pool-5-thread-3 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623160088.66_create__5650000_5700000_1623160088.67 are 50000 2021-06-08 06:48:20,483 | infra | DEBUG | pool-5-thread-3 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623160088.66_create__5700000_5750000_1623160088.67 2021-06-08 06:48:20,483 | infra | DEBUG | pool-5-thread-3 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623160088.66_create__5700000_5750000_1623160088.67 are 50000 2021-06-08 06:48:20,483 | infra | DEBUG | pool-5-thread-3 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623160088.66_create__5750000_5800000_1623160088.67 2021-06-08 06:48:20,483 | infra | DEBUG | pool-5-thread-3 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623160088.66_create__5750000_5800000_1623160088.67 are 50000 2021-06-08 06:48:20,483 | infra | DEBUG | pool-5-thread-3 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623160088.66_create__5800000_5850000_1623160088.67 2021-06-08 06:48:20,483 | infra | DEBUG | pool-5-thread-3 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623160088.66_create__5800000_5850000_1623160088.67 are 50000 2021-06-08 06:48:20,483 | infra | DEBUG | pool-5-thread-3 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623160088.66_create__5850000_5900000_1623160088.67 2021-06-08 06:48:20,483 | infra | DEBUG | pool-5-thread-3 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623160088.66_create__5850000_5900000_1623160088.67 are 50000 2021-06-08 06:48:20,483 | infra | DEBUG | pool-5-thread-3 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623160088.66_create__5900000_5950000_1623160088.67 2021-06-08 06:48:20,483 | infra | DEBUG | pool-5-thread-3 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623160088.66_create__5900000_5950000_1623160088.67 are 50000 2021-06-08 06:48:20,483 | infra | DEBUG | pool-5-thread-3 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623160088.66_create__5950000_6000000_1623160088.67 2021-06-08 06:48:20,483 | infra | DEBUG | pool-5-thread-3 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623160088.66_create__5950000_6000000_1623160088.67 are 50000 2021-06-08 06:48:20,484 | infra | DEBUG | pool-5-thread-3 | [task_manager:get_task_result:26] Getting task result for PrintBucketStats_default_1623160088.67 2021-06-08 06:48:20,486 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 06:48:20,486 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 06:48:21,486 | infra | INFO | pool-5-thread-4 | [task:complete_task:102] Thread 'PrintBucketStats_default_1623160088.67' completed 2021-06-08 06:48:21,486 | infra | DEBUG | pool-5-thread-3 | [task:call:1663] ========= Tasks in loadgen pool======= 2021-06-08 06:48:21,487 | infra | WARNING | pool-5-thread-3 | [task_manager:print_tasks_in_pool:83] Task 'LoadDocsGen_default__default__default_create_5000000_6000000_1623160088.66' not completed 2021-06-08 06:48:21,487 | infra | DEBUG | pool-5-thread-3 | [task:call:1665] ====================================== 2021-06-08 06:48:21,487 | infra | DEBUG | pool-5-thread-3 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623160088.66_create__5000000_5050000_1623160088.67' complete. Loaded 50000 items 2021-06-08 06:48:21,487 | infra | DEBUG | pool-5-thread-3 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623160088.66_create__5050000_5100000_1623160088.67' complete. Loaded 50000 items 2021-06-08 06:48:21,487 | infra | DEBUG | pool-5-thread-3 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623160088.66_create__5100000_5150000_1623160088.67' complete. Loaded 50000 items 2021-06-08 06:48:21,487 | infra | DEBUG | pool-5-thread-3 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623160088.66_create__5150000_5200000_1623160088.67' complete. Loaded 50000 items 2021-06-08 06:48:21,487 | infra | DEBUG | pool-5-thread-3 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623160088.66_create__5200000_5250000_1623160088.67' complete. Loaded 50000 items 2021-06-08 06:48:21,487 | infra | DEBUG | pool-5-thread-3 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623160088.66_create__5250000_5300000_1623160088.67' complete. Loaded 50000 items 2021-06-08 06:48:21,487 | infra | DEBUG | pool-5-thread-3 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623160088.66_create__5300000_5350000_1623160088.67' complete. Loaded 50000 items 2021-06-08 06:48:21,489 | infra | DEBUG | pool-5-thread-3 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623160088.66_create__5350000_5400000_1623160088.67' complete. Loaded 50000 items 2021-06-08 06:48:21,489 | infra | DEBUG | pool-5-thread-3 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623160088.66_create__5400000_5450000_1623160088.67' complete. Loaded 50000 items 2021-06-08 06:48:21,489 | infra | DEBUG | pool-5-thread-3 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623160088.66_create__5450000_5500000_1623160088.67' complete. Loaded 50000 items 2021-06-08 06:48:21,489 | infra | DEBUG | pool-5-thread-3 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623160088.66_create__5500000_5550000_1623160088.67' complete. Loaded 50000 items 2021-06-08 06:48:21,489 | infra | DEBUG | pool-5-thread-3 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623160088.66_create__5550000_5600000_1623160088.67' complete. Loaded 50000 items 2021-06-08 06:48:21,489 | infra | DEBUG | pool-5-thread-3 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623160088.66_create__5600000_5650000_1623160088.67' complete. Loaded 50000 items 2021-06-08 06:48:21,489 | infra | DEBUG | pool-5-thread-3 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623160088.66_create__5650000_5700000_1623160088.67' complete. Loaded 50000 items 2021-06-08 06:48:21,489 | infra | DEBUG | pool-5-thread-3 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623160088.66_create__5700000_5750000_1623160088.67' complete. Loaded 50000 items 2021-06-08 06:48:21,489 | infra | DEBUG | pool-5-thread-3 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623160088.66_create__5750000_5800000_1623160088.67' complete. Loaded 50000 items 2021-06-08 06:48:21,489 | infra | DEBUG | pool-5-thread-3 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623160088.66_create__5800000_5850000_1623160088.67' complete. Loaded 50000 items 2021-06-08 06:48:21,490 | infra | DEBUG | pool-5-thread-3 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623160088.66_create__5850000_5900000_1623160088.67' complete. Loaded 50000 items 2021-06-08 06:48:21,490 | infra | DEBUG | pool-5-thread-3 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623160088.66_create__5900000_5950000_1623160088.67' complete. Loaded 50000 items 2021-06-08 06:48:21,490 | infra | DEBUG | pool-5-thread-3 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623160088.66_create__5950000_6000000_1623160088.67' complete. Loaded 50000 items 2021-06-08 06:48:21,490 | infra | INFO | pool-5-thread-3 | [task:complete_task:102] Thread 'LoadDocsGen_default__default__default_create_5000000_6000000_1623160088.66' completed 2021-06-08 06:48:21,492 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.198 with username: root, password: couchbase 2021-06-08 06:48:21,900 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 06:48:22,286 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: ls /etc/ 2021-06-08 06:48:22,375 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 06:48:22,375 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: ls /cygdrive/c/Windows 2021-06-08 06:48:22,395 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: uname -m 2021-06-08 06:48:22,413 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: cat /proc/cpuinfo 2021-06-08 06:48:22,434 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: df -Thl 2021-06-08 06:48:22,490 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: cat /proc/meminfo 2021-06-08 06:48:22,513 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: hostname 2021-06-08 06:48:22,559 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: hostname -d 2021-06-08 06:48:22,579 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:559] Starting StatsWaitTask for ep_data_write_failed on bucket default 2021-06-08 06:48:22,579 | infra | INFO | MainThread | [task_manager:add_new_task:23] Added new task: StatsWaitTask_default_ep_data_write_failed_1623160102.58 2021-06-08 06:48:22,579 | infra | INFO | pool-5-thread-24 | [task:start_task:87] Thread 'StatsWaitTask_default_ep_data_write_failed_1623160102.58' started 2021-06-08 06:48:22,581 | infra | DEBUG | pool-5-thread-24 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:48:22,582 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.196 with username: root, password: couchbase 2021-06-08 06:48:22,687 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 06:48:22,997 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: ls /etc/ 2021-06-08 06:48:23,082 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 06:48:23,082 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: ls /cygdrive/c/Windows 2021-06-08 06:48:23,101 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: uname -m 2021-06-08 06:48:23,117 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: cat /proc/cpuinfo 2021-06-08 06:48:23,132 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: df -Thl 2021-06-08 06:48:23,157 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: cat /proc/meminfo 2021-06-08 06:48:23,174 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: hostname 2021-06-08 06:48:23,190 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: hostname -d 2021-06-08 06:48:23,207 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:559] Starting StatsWaitTask for ep_data_write_failed on bucket default 2021-06-08 06:48:23,207 | infra | INFO | MainThread | [task_manager:add_new_task:23] Added new task: StatsWaitTask_default_ep_data_write_failed_1623160103.21 2021-06-08 06:48:23,209 | infra | INFO | pool-5-thread-25 | [task:start_task:87] Thread 'StatsWaitTask_default_ep_data_write_failed_1623160103.21' started 2021-06-08 06:48:23,209 | infra | DEBUG | pool-5-thread-25 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:48:23,210 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.199 with username: root, password: couchbase 2021-06-08 06:48:23,467 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 06:48:23,473 | test | DEBUG | pool-5-thread-24 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:48:23,474 | infra | DEBUG | pool-5-thread-24 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.198': '0'} for bucket 'default' 2021-06-08 06:48:23,474 | infra | DEBUG | pool-5-thread-24 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:48:23,476 | infra | DEBUG | pool-5-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:48:23,476 | infra | DEBUG | pool-5-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:48:23,665 | test | DEBUG | pool-5-thread-25 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:48:23,665 | infra | DEBUG | pool-5-thread-25 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.196': '0'} for bucket 'default' 2021-06-08 06:48:23,665 | infra | DEBUG | pool-5-thread-25 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:48:23,667 | infra | DEBUG | pool-5-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:48:23,667 | infra | DEBUG | pool-5-thread-25 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:48:23,841 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: ls /etc/ 2021-06-08 06:48:23,944 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 06:48:23,944 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: ls /cygdrive/c/Windows 2021-06-08 06:48:23,963 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: uname -m 2021-06-08 06:48:23,986 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: cat /proc/cpuinfo 2021-06-08 06:48:24,017 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: df -Thl 2021-06-08 06:48:24,046 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: cat /proc/meminfo 2021-06-08 06:48:24,066 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: hostname 2021-06-08 06:48:24,095 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: hostname -d 2021-06-08 06:48:24,125 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:559] Starting StatsWaitTask for ep_data_write_failed on bucket default 2021-06-08 06:48:24,125 | infra | INFO | MainThread | [task_manager:add_new_task:23] Added new task: StatsWaitTask_default_ep_data_write_failed_1623160104.12 2021-06-08 06:48:24,125 | infra | INFO | pool-5-thread-8 | [task:start_task:87] Thread 'StatsWaitTask_default_ep_data_write_failed_1623160104.12' started 2021-06-08 06:48:24,127 | infra | DEBUG | pool-5-thread-8 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:48:24,128 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.168 with username: root, password: couchbase 2021-06-08 06:48:24,388 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 06:48:24,690 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: ls /etc/ 2021-06-08 06:48:24,690 | test | DEBUG | pool-5-thread-8 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:48:24,690 | infra | DEBUG | pool-5-thread-8 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.199': '0'} for bucket 'default' 2021-06-08 06:48:24,690 | infra | DEBUG | pool-5-thread-8 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:48:24,691 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:48:24,691 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:48:24,786 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 06:48:24,786 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: ls /cygdrive/c/Windows 2021-06-08 06:48:24,815 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: uname -m 2021-06-08 06:48:24,848 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: cat /proc/cpuinfo 2021-06-08 06:48:24,868 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: df -Thl 2021-06-08 06:48:24,894 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: cat /proc/meminfo 2021-06-08 06:48:24,910 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: hostname 2021-06-08 06:48:24,940 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: hostname -d 2021-06-08 06:48:24,973 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:559] Starting StatsWaitTask for ep_data_write_failed on bucket default 2021-06-08 06:48:24,973 | infra | INFO | MainThread | [task_manager:add_new_task:23] Added new task: StatsWaitTask_default_ep_data_write_failed_1623160104.97 2021-06-08 06:48:24,973 | infra | INFO | pool-5-thread-7 | [task:start_task:87] Thread 'StatsWaitTask_default_ep_data_write_failed_1623160104.97' started 2021-06-08 06:48:24,973 | infra | DEBUG | MainThread | [task_manager:get_task_result:26] Getting task result for StatsWaitTask_default_ep_data_write_failed_1623160102.58 2021-06-08 06:48:24,973 | infra | DEBUG | pool-5-thread-7 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:48:25,522 | test | DEBUG | pool-5-thread-7 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:48:25,522 | infra | DEBUG | pool-5-thread-7 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.168': '0'} for bucket 'default' 2021-06-08 06:48:25,522 | infra | DEBUG | pool-5-thread-7 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:48:25,523 | infra | DEBUG | pool-5-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:48:25,523 | infra | DEBUG | pool-5-thread-7 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:48:28,476 | infra | DEBUG | pool-5-thread-24 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:48:28,667 | infra | DEBUG | pool-5-thread-25 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:48:29,461 | test | DEBUG | pool-5-thread-25 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:48:29,461 | infra | DEBUG | pool-5-thread-25 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.196': '0'} for bucket 'default' 2021-06-08 06:48:29,461 | infra | DEBUG | pool-5-thread-25 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:48:29,463 | infra | DEBUG | pool-5-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:48:29,463 | infra | DEBUG | pool-5-thread-25 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:48:29,693 | infra | DEBUG | pool-5-thread-8 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:48:30,259 | test | DEBUG | pool-5-thread-24 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:48:30,259 | infra | DEBUG | pool-5-thread-24 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.198': '0'} for bucket 'default' 2021-06-08 06:48:30,259 | infra | DEBUG | pool-5-thread-24 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:48:30,260 | infra | DEBUG | pool-5-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:48:30,260 | infra | DEBUG | pool-5-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:48:30,525 | infra | DEBUG | pool-5-thread-7 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:48:30,877 | test | DEBUG | pool-5-thread-7 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:48:30,877 | infra | DEBUG | pool-5-thread-7 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.168': '0'} for bucket 'default' 2021-06-08 06:48:30,877 | infra | DEBUG | pool-5-thread-7 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:48:30,878 | infra | DEBUG | pool-5-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:48:30,878 | infra | DEBUG | pool-5-thread-7 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:48:31,785 | test | DEBUG | pool-5-thread-8 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:48:31,785 | infra | DEBUG | pool-5-thread-8 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.199': '0'} for bucket 'default' 2021-06-08 06:48:31,785 | infra | DEBUG | pool-5-thread-8 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:48:31,786 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:48:31,786 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:48:34,463 | infra | DEBUG | pool-5-thread-25 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:48:35,262 | infra | DEBUG | pool-5-thread-24 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:48:35,293 | test | DEBUG | pool-5-thread-25 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:48:35,293 | infra | DEBUG | pool-5-thread-25 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.196': '0'} for bucket 'default' 2021-06-08 06:48:35,295 | infra | DEBUG | pool-5-thread-25 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:48:35,296 | infra | DEBUG | pool-5-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:48:35,296 | infra | DEBUG | pool-5-thread-25 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:48:35,694 | test | DEBUG | pool-5-thread-24 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:48:35,694 | infra | DEBUG | pool-5-thread-24 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.198': '0'} for bucket 'default' 2021-06-08 06:48:35,694 | infra | DEBUG | pool-5-thread-24 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:48:35,696 | infra | DEBUG | pool-5-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:48:35,696 | infra | DEBUG | pool-5-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:48:35,880 | infra | DEBUG | pool-5-thread-7 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:48:36,141 | test | DEBUG | pool-5-thread-7 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:48:36,141 | infra | DEBUG | pool-5-thread-7 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.168': '0'} for bucket 'default' 2021-06-08 06:48:36,141 | infra | DEBUG | pool-5-thread-7 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:48:36,142 | infra | DEBUG | pool-5-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:48:36,144 | infra | DEBUG | pool-5-thread-7 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:48:36,786 | infra | DEBUG | pool-5-thread-8 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:48:38,134 | test | DEBUG | pool-5-thread-8 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:48:38,135 | infra | DEBUG | pool-5-thread-8 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.199': '0'} for bucket 'default' 2021-06-08 06:48:38,135 | infra | DEBUG | pool-5-thread-8 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:48:38,137 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:48:38,138 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:48:40,296 | infra | DEBUG | pool-5-thread-25 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:48:40,697 | infra | DEBUG | pool-5-thread-24 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:48:41,144 | infra | DEBUG | pool-5-thread-7 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:48:41,377 | test | DEBUG | pool-5-thread-7 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:48:41,377 | infra | DEBUG | pool-5-thread-7 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.168': '0'} for bucket 'default' 2021-06-08 06:48:41,377 | infra | DEBUG | pool-5-thread-7 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:48:41,378 | infra | DEBUG | pool-5-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:48:41,378 | infra | DEBUG | pool-5-thread-7 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:48:41,444 | test | DEBUG | pool-5-thread-25 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:48:41,444 | infra | DEBUG | pool-5-thread-25 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.196': '0'} for bucket 'default' 2021-06-08 06:48:41,444 | infra | DEBUG | pool-5-thread-25 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:48:41,446 | infra | DEBUG | pool-5-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:48:41,446 | infra | DEBUG | pool-5-thread-25 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:48:41,967 | test | DEBUG | pool-5-thread-24 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:48:41,967 | infra | DEBUG | pool-5-thread-24 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.198': '0'} for bucket 'default' 2021-06-08 06:48:41,967 | infra | DEBUG | pool-5-thread-24 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:48:41,969 | infra | DEBUG | pool-5-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:48:41,969 | infra | DEBUG | pool-5-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:48:43,138 | infra | DEBUG | pool-5-thread-8 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:48:43,796 | test | DEBUG | pool-5-thread-8 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:48:43,796 | infra | DEBUG | pool-5-thread-8 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.199': '0'} for bucket 'default' 2021-06-08 06:48:43,796 | infra | DEBUG | pool-5-thread-8 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:48:43,799 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:48:43,799 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:48:46,378 | infra | DEBUG | pool-5-thread-7 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:48:46,446 | infra | DEBUG | pool-5-thread-25 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:48:46,969 | infra | DEBUG | pool-5-thread-24 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:48:47,029 | test | DEBUG | pool-5-thread-7 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:48:47,029 | infra | DEBUG | pool-5-thread-7 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.168': '0'} for bucket 'default' 2021-06-08 06:48:47,029 | infra | DEBUG | pool-5-thread-7 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:48:47,030 | infra | DEBUG | pool-5-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:48:47,030 | infra | DEBUG | pool-5-thread-7 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:48:47,394 | test | DEBUG | pool-5-thread-25 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:48:47,394 | infra | DEBUG | pool-5-thread-25 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.196': '0'} for bucket 'default' 2021-06-08 06:48:47,394 | infra | DEBUG | pool-5-thread-25 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:48:47,395 | infra | DEBUG | pool-5-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:48:47,395 | infra | DEBUG | pool-5-thread-25 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:48:47,967 | test | DEBUG | pool-5-thread-24 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:48:47,967 | infra | DEBUG | pool-5-thread-24 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.198': '0'} for bucket 'default' 2021-06-08 06:48:47,967 | infra | DEBUG | pool-5-thread-24 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:48:47,970 | infra | DEBUG | pool-5-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:48:47,970 | infra | DEBUG | pool-5-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:48:48,799 | infra | DEBUG | pool-5-thread-8 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:48:49,506 | test | DEBUG | pool-5-thread-8 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:48:49,506 | infra | DEBUG | pool-5-thread-8 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.199': '0'} for bucket 'default' 2021-06-08 06:48:49,506 | infra | DEBUG | pool-5-thread-8 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:48:49,509 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:48:49,509 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:48:52,030 | infra | DEBUG | pool-5-thread-7 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:48:52,342 | test | DEBUG | pool-5-thread-7 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:48:52,342 | infra | DEBUG | pool-5-thread-7 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.168': '0'} for bucket 'default' 2021-06-08 06:48:52,342 | infra | DEBUG | pool-5-thread-7 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:48:52,344 | infra | DEBUG | pool-5-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:48:52,344 | infra | DEBUG | pool-5-thread-7 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:48:52,397 | infra | DEBUG | pool-5-thread-25 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:48:52,970 | infra | DEBUG | pool-5-thread-24 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:48:52,970 | test | DEBUG | pool-5-thread-25 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:48:52,971 | infra | DEBUG | pool-5-thread-25 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.196': '0'} for bucket 'default' 2021-06-08 06:48:52,971 | infra | DEBUG | pool-5-thread-25 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:48:52,973 | infra | DEBUG | pool-5-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:48:52,973 | infra | DEBUG | pool-5-thread-25 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:48:53,321 | test | DEBUG | pool-5-thread-24 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:48:53,321 | infra | DEBUG | pool-5-thread-24 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.198': '0'} for bucket 'default' 2021-06-08 06:48:53,321 | infra | DEBUG | pool-5-thread-24 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:48:53,322 | infra | DEBUG | pool-5-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:48:53,322 | infra | DEBUG | pool-5-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:48:54,509 | infra | DEBUG | pool-5-thread-8 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:48:55,048 | test | DEBUG | pool-5-thread-8 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:48:55,048 | infra | DEBUG | pool-5-thread-8 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.199': '0'} for bucket 'default' 2021-06-08 06:48:55,049 | infra | DEBUG | pool-5-thread-8 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:48:55,049 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:48:55,049 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:48:57,344 | infra | DEBUG | pool-5-thread-7 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:48:57,619 | test | DEBUG | pool-5-thread-7 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:48:57,621 | infra | DEBUG | pool-5-thread-7 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.168': '0'} for bucket 'default' 2021-06-08 06:48:57,621 | infra | DEBUG | pool-5-thread-7 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:48:57,621 | infra | DEBUG | pool-5-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:48:57,622 | infra | DEBUG | pool-5-thread-7 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:48:57,973 | infra | DEBUG | pool-5-thread-25 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:48:58,322 | infra | DEBUG | pool-5-thread-24 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:48:58,338 | test | DEBUG | pool-5-thread-25 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:48:58,338 | infra | DEBUG | pool-5-thread-25 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.196': '0'} for bucket 'default' 2021-06-08 06:48:58,338 | infra | DEBUG | pool-5-thread-25 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:48:58,341 | infra | DEBUG | pool-5-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:48:58,341 | infra | DEBUG | pool-5-thread-25 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:48:59,085 | test | DEBUG | pool-5-thread-24 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:48:59,085 | infra | DEBUG | pool-5-thread-24 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.198': '0'} for bucket 'default' 2021-06-08 06:48:59,085 | infra | DEBUG | pool-5-thread-24 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:48:59,086 | infra | DEBUG | pool-5-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:48:59,086 | infra | DEBUG | pool-5-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:49:00,051 | infra | DEBUG | pool-5-thread-8 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:49:01,076 | test | DEBUG | pool-5-thread-8 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:49:01,076 | infra | DEBUG | pool-5-thread-8 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.199': '0'} for bucket 'default' 2021-06-08 06:49:01,076 | infra | DEBUG | pool-5-thread-8 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:49:01,078 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:49:01,078 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:49:02,622 | infra | DEBUG | pool-5-thread-7 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:49:03,006 | test | DEBUG | pool-5-thread-7 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:49:03,006 | infra | DEBUG | pool-5-thread-7 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.168': '0'} for bucket 'default' 2021-06-08 06:49:03,006 | infra | DEBUG | pool-5-thread-7 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:49:03,009 | infra | DEBUG | pool-5-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:49:03,009 | infra | DEBUG | pool-5-thread-7 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:49:03,341 | infra | DEBUG | pool-5-thread-25 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:49:03,739 | test | DEBUG | pool-5-thread-25 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:49:03,739 | infra | DEBUG | pool-5-thread-25 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.196': '0'} for bucket 'default' 2021-06-08 06:49:03,739 | infra | DEBUG | pool-5-thread-25 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:49:03,740 | infra | DEBUG | pool-5-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:49:03,740 | infra | DEBUG | pool-5-thread-25 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:49:04,086 | infra | DEBUG | pool-5-thread-24 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:49:04,365 | test | DEBUG | pool-5-thread-24 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:49:04,365 | infra | DEBUG | pool-5-thread-24 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.198': '0'} for bucket 'default' 2021-06-08 06:49:04,365 | infra | DEBUG | pool-5-thread-24 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:49:04,367 | infra | DEBUG | pool-5-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:49:04,367 | infra | DEBUG | pool-5-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:49:06,078 | infra | DEBUG | pool-5-thread-8 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:49:06,292 | test | DEBUG | pool-5-thread-8 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:49:06,293 | infra | DEBUG | pool-5-thread-8 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.199': '0'} for bucket 'default' 2021-06-08 06:49:06,293 | infra | DEBUG | pool-5-thread-8 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:49:06,295 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:49:06,296 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:49:08,009 | infra | DEBUG | pool-5-thread-7 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:49:08,220 | test | DEBUG | pool-5-thread-7 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:49:08,220 | infra | DEBUG | pool-5-thread-7 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.168': '0'} for bucket 'default' 2021-06-08 06:49:08,220 | infra | DEBUG | pool-5-thread-7 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:49:08,221 | infra | DEBUG | pool-5-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:49:08,221 | infra | DEBUG | pool-5-thread-7 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:49:08,742 | infra | DEBUG | pool-5-thread-25 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:49:08,950 | test | DEBUG | pool-5-thread-25 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:49:08,950 | infra | DEBUG | pool-5-thread-25 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.196': '0'} for bucket 'default' 2021-06-08 06:49:08,950 | infra | DEBUG | pool-5-thread-25 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:49:08,951 | infra | DEBUG | pool-5-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:49:08,951 | infra | DEBUG | pool-5-thread-25 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:49:09,367 | infra | DEBUG | pool-5-thread-24 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:49:09,575 | test | DEBUG | pool-5-thread-24 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:49:09,575 | infra | DEBUG | pool-5-thread-24 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.198': '0'} for bucket 'default' 2021-06-08 06:49:09,575 | infra | DEBUG | pool-5-thread-24 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:49:09,576 | infra | DEBUG | pool-5-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:49:09,576 | infra | DEBUG | pool-5-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:49:11,296 | infra | DEBUG | pool-5-thread-8 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:49:11,500 | test | DEBUG | pool-5-thread-8 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:49:11,500 | infra | DEBUG | pool-5-thread-8 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.199': '0'} for bucket 'default' 2021-06-08 06:49:11,500 | infra | DEBUG | pool-5-thread-8 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:49:11,502 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:49:11,502 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:49:13,223 | infra | DEBUG | pool-5-thread-7 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:49:13,441 | test | DEBUG | pool-5-thread-7 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:49:13,441 | infra | DEBUG | pool-5-thread-7 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.168': '0'} for bucket 'default' 2021-06-08 06:49:13,443 | infra | DEBUG | pool-5-thread-7 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:49:13,444 | infra | DEBUG | pool-5-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:49:13,444 | infra | DEBUG | pool-5-thread-7 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:49:13,951 | infra | DEBUG | pool-5-thread-25 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:49:14,154 | test | DEBUG | pool-5-thread-25 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:49:14,155 | infra | DEBUG | pool-5-thread-25 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.196': '0'} for bucket 'default' 2021-06-08 06:49:14,155 | infra | DEBUG | pool-5-thread-25 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:49:14,157 | infra | DEBUG | pool-5-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:49:14,157 | infra | DEBUG | pool-5-thread-25 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:49:14,576 | infra | DEBUG | pool-5-thread-24 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:49:14,782 | test | DEBUG | pool-5-thread-24 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:49:14,782 | infra | DEBUG | pool-5-thread-24 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.198': '0'} for bucket 'default' 2021-06-08 06:49:14,782 | infra | DEBUG | pool-5-thread-24 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:49:14,785 | infra | DEBUG | pool-5-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:49:14,785 | infra | DEBUG | pool-5-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:49:16,503 | infra | DEBUG | pool-5-thread-8 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:49:16,706 | test | DEBUG | pool-5-thread-8 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:49:16,706 | infra | DEBUG | pool-5-thread-8 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.199': '0'} for bucket 'default' 2021-06-08 06:49:16,706 | infra | DEBUG | pool-5-thread-8 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:49:16,707 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:49:16,707 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:49:18,444 | infra | DEBUG | pool-5-thread-7 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:49:18,654 | test | DEBUG | pool-5-thread-7 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:49:18,654 | infra | DEBUG | pool-5-thread-7 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.168': '0'} for bucket 'default' 2021-06-08 06:49:18,654 | infra | DEBUG | pool-5-thread-7 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:49:18,655 | infra | DEBUG | pool-5-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:49:18,655 | infra | DEBUG | pool-5-thread-7 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:49:19,157 | infra | DEBUG | pool-5-thread-25 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:49:19,365 | test | DEBUG | pool-5-thread-25 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:49:19,365 | infra | DEBUG | pool-5-thread-25 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.196': '0'} for bucket 'default' 2021-06-08 06:49:19,365 | infra | DEBUG | pool-5-thread-25 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:49:19,367 | infra | DEBUG | pool-5-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:49:19,367 | infra | DEBUG | pool-5-thread-25 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:49:19,785 | infra | DEBUG | pool-5-thread-24 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:49:19,992 | test | DEBUG | pool-5-thread-24 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:49:19,992 | infra | DEBUG | pool-5-thread-24 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.198': '0'} for bucket 'default' 2021-06-08 06:49:19,992 | infra | DEBUG | pool-5-thread-24 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:49:19,993 | infra | DEBUG | pool-5-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:49:19,993 | infra | DEBUG | pool-5-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:49:21,707 | infra | DEBUG | pool-5-thread-8 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:49:21,917 | test | DEBUG | pool-5-thread-8 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:49:21,917 | infra | DEBUG | pool-5-thread-8 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.199': '0'} for bucket 'default' 2021-06-08 06:49:21,917 | infra | DEBUG | pool-5-thread-8 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:49:21,918 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:49:21,918 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:49:23,657 | infra | DEBUG | pool-5-thread-7 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:49:23,875 | test | DEBUG | pool-5-thread-7 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:49:23,875 | infra | DEBUG | pool-5-thread-7 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.168': '0'} for bucket 'default' 2021-06-08 06:49:23,875 | infra | DEBUG | pool-5-thread-7 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:49:23,878 | infra | DEBUG | pool-5-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:49:23,878 | infra | DEBUG | pool-5-thread-7 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:49:24,368 | infra | DEBUG | pool-5-thread-25 | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.196 2021-06-08 06:49:24,368 | infra | INFO | pool-5-thread-25 | [task:complete_task:102] Thread 'StatsWaitTask_default_ep_data_write_failed_1623160103.21' completed 2021-06-08 06:49:24,993 | infra | DEBUG | pool-5-thread-24 | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.198 2021-06-08 06:49:24,994 | infra | INFO | pool-5-thread-24 | [task:complete_task:102] Thread 'StatsWaitTask_default_ep_data_write_failed_1623160102.58' completed 2021-06-08 06:49:25,000 | infra | INFO | MainThread | [task_manager:add_new_task:23] Added new task: LoadDocsGen_default__default__default_create_6000000_7000000_1623160165.0 2021-06-08 06:49:25,000 | infra | INFO | pool-5-thread-30 | [task:start_task:87] Thread 'LoadDocsGen_default__default__default_create_6000000_7000000_1623160165.0' started 2021-06-08 06:49:25,000 | infra | DEBUG | MainThread | [task_manager:get_task_result:26] Getting task result for LoadDocsGen_default__default__default_create_6000000_7000000_1623160165.0 2021-06-08 06:49:25,023 | infra | INFO | pool-5-thread-30 | [task_manager:add_new_task:23] Added new task: PrintBucketStats_default_1623160165.01 2021-06-08 06:49:25,023 | infra | INFO | pool-5-thread-1 | [task:start_task:87] Thread 'PrintBucketStats_default_1623160165.01' started 2021-06-08 06:49:25,025 | infra | INFO | pool-5-thread-30 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623160165.0_create__6000000_6050000_1623160165.01 2021-06-08 06:49:25,025 | infra | INFO | pool-5-thread-29 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623160165.0_create__6000000_6050000_1623160165.01' started 2021-06-08 06:49:25,025 | infra | INFO | pool-5-thread-30 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623160165.0_create__6050000_6100000_1623160165.01 2021-06-08 06:49:25,025 | infra | INFO | pool-5-thread-2 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623160165.0_create__6050000_6100000_1623160165.01' started 2021-06-08 06:49:25,025 | infra | INFO | pool-5-thread-30 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623160165.0_create__6100000_6150000_1623160165.01 2021-06-08 06:49:25,026 | infra | INFO | pool-5-thread-22 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623160165.0_create__6100000_6150000_1623160165.01' started 2021-06-08 06:49:25,026 | infra | INFO | pool-5-thread-26 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623160165.0_create__6150000_6200000_1623160165.01' started 2021-06-08 06:49:25,026 | infra | INFO | pool-5-thread-30 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623160165.0_create__6150000_6200000_1623160165.01 2021-06-08 06:49:25,028 | infra | INFO | pool-5-thread-30 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623160165.0_create__6200000_6250000_1623160165.01 2021-06-08 06:49:25,028 | infra | INFO | pool-5-thread-23 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623160165.0_create__6200000_6250000_1623160165.01' started 2021-06-08 06:49:25,028 | infra | INFO | pool-5-thread-30 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623160165.0_create__6250000_6300000_1623160165.01 2021-06-08 06:49:25,028 | infra | INFO | pool-5-thread-12 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623160165.0_create__6250000_6300000_1623160165.01' started 2021-06-08 06:49:25,029 | infra | INFO | pool-5-thread-20 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623160165.0_create__6300000_6350000_1623160165.01' started 2021-06-08 06:49:25,029 | infra | INFO | pool-5-thread-30 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623160165.0_create__6300000_6350000_1623160165.01 2021-06-08 06:49:25,030 | infra | INFO | pool-5-thread-30 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623160165.0_create__6350000_6400000_1623160165.01 2021-06-08 06:49:25,032 | infra | INFO | pool-5-thread-28 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623160165.0_create__6350000_6400000_1623160165.01' started 2021-06-08 06:49:25,032 | infra | INFO | pool-5-thread-30 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623160165.0_create__6400000_6450000_1623160165.01 2021-06-08 06:49:25,032 | infra | INFO | pool-5-thread-30 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623160165.0_create__6450000_6500000_1623160165.01 2021-06-08 06:49:25,032 | infra | INFO | pool-5-thread-11 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623160165.0_create__6400000_6450000_1623160165.01' started 2021-06-08 06:49:25,032 | infra | INFO | pool-5-thread-9 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623160165.0_create__6450000_6500000_1623160165.01' started 2021-06-08 06:49:25,033 | infra | INFO | pool-5-thread-27 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623160165.0_create__6500000_6550000_1623160165.01' started 2021-06-08 06:49:25,033 | infra | INFO | pool-5-thread-30 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623160165.0_create__6500000_6550000_1623160165.01 2021-06-08 06:49:25,035 | infra | INFO | pool-5-thread-30 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623160165.0_create__6550000_6600000_1623160165.01 2021-06-08 06:49:25,035 | infra | INFO | pool-5-thread-21 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623160165.0_create__6550000_6600000_1623160165.01' started 2021-06-08 06:49:25,035 | infra | INFO | pool-5-thread-30 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623160165.0_create__6600000_6650000_1623160165.01 2021-06-08 06:49:25,035 | infra | INFO | pool-5-thread-19 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623160165.0_create__6600000_6650000_1623160165.01' started 2021-06-08 06:49:25,039 | infra | INFO | pool-5-thread-6 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623160165.0_create__6650000_6700000_1623160165.01' started 2021-06-08 06:49:25,039 | infra | INFO | pool-5-thread-30 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623160165.0_create__6650000_6700000_1623160165.01 2021-06-08 06:49:25,040 | infra | INFO | pool-5-thread-30 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623160165.0_create__6700000_6750000_1623160165.01 2021-06-08 06:49:25,040 | infra | INFO | pool-5-thread-17 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623160165.0_create__6700000_6750000_1623160165.01' started 2021-06-08 06:49:25,040 | infra | INFO | pool-5-thread-30 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623160165.0_create__6750000_6800000_1623160165.01 2021-06-08 06:49:25,040 | infra | INFO | pool-5-thread-16 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623160165.0_create__6750000_6800000_1623160165.01' started 2021-06-08 06:49:25,042 | infra | INFO | pool-5-thread-15 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623160165.0_create__6800000_6850000_1623160165.01' started 2021-06-08 06:49:25,042 | infra | INFO | pool-5-thread-30 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623160165.0_create__6800000_6850000_1623160165.01 2021-06-08 06:49:25,046 | infra | INFO | pool-5-thread-30 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623160165.0_create__6850000_6900000_1623160165.01 2021-06-08 06:49:25,046 | infra | INFO | pool-5-thread-18 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623160165.0_create__6850000_6900000_1623160165.01' started 2021-06-08 06:49:25,049 | infra | INFO | pool-5-thread-30 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623160165.0_create__6900000_6950000_1623160165.01 2021-06-08 06:49:25,049 | infra | INFO | pool-5-thread-5 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623160165.0_create__6900000_6950000_1623160165.01' started 2021-06-08 06:49:25,055 | infra | INFO | pool-5-thread-30 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623160165.0_create__6950000_7000000_1623160165.01 2021-06-08 06:49:25,055 | infra | INFO | pool-5-thread-14 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623160165.0_create__6950000_7000000_1623160165.01' started 2021-06-08 06:49:25,055 | infra | DEBUG | pool-5-thread-30 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623160165.0_create__6000000_6050000_1623160165.01 2021-06-08 06:49:25,378 | test | DEBUG | pool-5-thread-1 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 06:49:25,378 | test | DEBUG | pool-5-thread-1 | [task:print_ep_queue_size:3508] ep_queue_size for default: 0 2021-06-08 06:49:25,381 | infra | DEBUG | pool-5-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 06:49:25,381 | infra | DEBUG | pool-5-thread-1 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 06:49:26,753 | test | DEBUG | pool-5-thread-1 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 06:49:26,753 | test | DEBUG | pool-5-thread-1 | [task:print_ep_queue_size:3508] ep_queue_size for default: 0 2021-06-08 06:49:26,756 | infra | DEBUG | pool-5-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 06:49:26,756 | infra | DEBUG | pool-5-thread-1 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 06:49:26,918 | infra | DEBUG | pool-5-thread-8 | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.199 2021-06-08 06:49:26,921 | infra | INFO | pool-5-thread-8 | [task:complete_task:102] Thread 'StatsWaitTask_default_ep_data_write_failed_1623160104.12' completed 2021-06-08 06:49:28,164 | test | DEBUG | pool-5-thread-1 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 06:49:28,164 | test | DEBUG | pool-5-thread-1 | [task:print_ep_queue_size:3508] ep_queue_size for default: 0 2021-06-08 06:49:28,167 | infra | DEBUG | pool-5-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 06:49:28,167 | infra | DEBUG | pool-5-thread-1 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 06:49:28,878 | infra | DEBUG | pool-5-thread-7 | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.168 2021-06-08 06:49:28,880 | infra | INFO | pool-5-thread-7 | [task:complete_task:102] Thread 'StatsWaitTask_default_ep_data_write_failed_1623160104.97' completed 2021-06-08 06:49:29,588 | test | DEBUG | pool-5-thread-1 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 06:49:29,588 | test | DEBUG | pool-5-thread-1 | [task:print_ep_queue_size:3508] ep_queue_size for default: 0 2021-06-08 06:49:29,591 | infra | DEBUG | pool-5-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 06:49:29,591 | infra | DEBUG | pool-5-thread-1 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 06:49:31,033 | test | DEBUG | pool-5-thread-1 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 06:49:31,035 | test | DEBUG | pool-5-thread-1 | [task:print_ep_queue_size:3508] ep_queue_size for default: 0 2021-06-08 06:49:31,038 | infra | DEBUG | pool-5-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 06:49:31,039 | infra | DEBUG | pool-5-thread-1 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 06:49:32,473 | test | DEBUG | pool-5-thread-1 | [task:record_bucket_ops:3495] Ops rate for 'default': 66759.580576 2021-06-08 06:49:32,473 | test | DEBUG | pool-5-thread-1 | [task:print_ep_queue_size:3508] ep_queue_size for default: 883152 2021-06-08 06:49:32,476 | infra | DEBUG | pool-5-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 06:49:32,476 | infra | DEBUG | pool-5-thread-1 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 06:49:33,181 | infra | INFO | pool-5-thread-9 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623160165.0_create__6450000_6500000_1623160165.01' completed 2021-06-08 06:49:33,234 | infra | INFO | pool-5-thread-26 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623160165.0_create__6150000_6200000_1623160165.01' completed 2021-06-08 06:49:33,246 | infra | INFO | pool-5-thread-19 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623160165.0_create__6600000_6650000_1623160165.01' completed 2021-06-08 06:49:33,263 | infra | INFO | pool-5-thread-5 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623160165.0_create__6900000_6950000_1623160165.01' completed 2021-06-08 06:49:33,301 | infra | INFO | pool-5-thread-15 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623160165.0_create__6800000_6850000_1623160165.01' completed 2021-06-08 06:49:33,385 | infra | INFO | pool-5-thread-12 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623160165.0_create__6250000_6300000_1623160165.01' completed 2021-06-08 06:49:33,391 | infra | INFO | pool-5-thread-21 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623160165.0_create__6550000_6600000_1623160165.01' completed 2021-06-08 06:49:33,404 | infra | INFO | pool-5-thread-11 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623160165.0_create__6400000_6450000_1623160165.01' completed 2021-06-08 06:49:33,415 | infra | INFO | pool-5-thread-20 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623160165.0_create__6300000_6350000_1623160165.01' completed 2021-06-08 06:49:33,834 | test | DEBUG | pool-5-thread-1 | [task:record_bucket_ops:3495] Ops rate for 'default': 66759.580576 2021-06-08 06:49:33,834 | test | DEBUG | pool-5-thread-1 | [task:print_ep_queue_size:3508] ep_queue_size for default: 883152 2021-06-08 06:49:33,835 | infra | DEBUG | pool-5-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 06:49:33,835 | infra | DEBUG | pool-5-thread-1 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 06:49:34,302 | infra | INFO | pool-5-thread-16 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623160165.0_create__6750000_6800000_1623160165.01' completed 2021-06-08 06:49:34,305 | infra | INFO | pool-5-thread-17 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623160165.0_create__6700000_6750000_1623160165.01' completed 2021-06-08 06:49:34,319 | infra | INFO | pool-5-thread-2 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623160165.0_create__6050000_6100000_1623160165.01' completed 2021-06-08 06:49:34,322 | infra | INFO | pool-5-thread-6 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623160165.0_create__6650000_6700000_1623160165.01' completed 2021-06-08 06:49:34,372 | infra | INFO | pool-5-thread-23 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623160165.0_create__6200000_6250000_1623160165.01' completed 2021-06-08 06:49:34,375 | infra | INFO | pool-5-thread-22 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623160165.0_create__6100000_6150000_1623160165.01' completed 2021-06-08 06:49:34,384 | infra | INFO | pool-5-thread-27 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623160165.0_create__6500000_6550000_1623160165.01' completed 2021-06-08 06:49:34,384 | infra | INFO | pool-5-thread-29 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623160165.0_create__6000000_6050000_1623160165.01' completed 2021-06-08 06:49:34,385 | infra | INFO | pool-5-thread-18 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623160165.0_create__6850000_6900000_1623160165.01' completed 2021-06-08 06:49:34,388 | infra | DEBUG | pool-5-thread-30 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623160165.0_create__6000000_6050000_1623160165.01 are 50000 2021-06-08 06:49:34,388 | infra | DEBUG | pool-5-thread-30 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623160165.0_create__6050000_6100000_1623160165.01 2021-06-08 06:49:34,388 | infra | DEBUG | pool-5-thread-30 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623160165.0_create__6050000_6100000_1623160165.01 are 50000 2021-06-08 06:49:34,388 | infra | DEBUG | pool-5-thread-30 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623160165.0_create__6100000_6150000_1623160165.01 2021-06-08 06:49:34,388 | infra | DEBUG | pool-5-thread-30 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623160165.0_create__6100000_6150000_1623160165.01 are 50000 2021-06-08 06:49:34,388 | infra | DEBUG | pool-5-thread-30 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623160165.0_create__6150000_6200000_1623160165.01 2021-06-08 06:49:34,388 | infra | DEBUG | pool-5-thread-30 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623160165.0_create__6150000_6200000_1623160165.01 are 50000 2021-06-08 06:49:34,388 | infra | DEBUG | pool-5-thread-30 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623160165.0_create__6200000_6250000_1623160165.01 2021-06-08 06:49:34,388 | infra | DEBUG | pool-5-thread-30 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623160165.0_create__6200000_6250000_1623160165.01 are 50000 2021-06-08 06:49:34,388 | infra | DEBUG | pool-5-thread-30 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623160165.0_create__6250000_6300000_1623160165.01 2021-06-08 06:49:34,388 | infra | DEBUG | pool-5-thread-30 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623160165.0_create__6250000_6300000_1623160165.01 are 50000 2021-06-08 06:49:34,388 | infra | DEBUG | pool-5-thread-30 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623160165.0_create__6300000_6350000_1623160165.01 2021-06-08 06:49:34,388 | infra | DEBUG | pool-5-thread-30 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623160165.0_create__6300000_6350000_1623160165.01 are 50000 2021-06-08 06:49:34,388 | infra | DEBUG | pool-5-thread-30 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623160165.0_create__6350000_6400000_1623160165.01 2021-06-08 06:49:34,621 | infra | INFO | pool-5-thread-14 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623160165.0_create__6950000_7000000_1623160165.01' completed 2021-06-08 06:49:34,628 | infra | INFO | pool-5-thread-28 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623160165.0_create__6350000_6400000_1623160165.01' completed 2021-06-08 06:49:34,628 | infra | DEBUG | pool-5-thread-30 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623160165.0_create__6350000_6400000_1623160165.01 are 50000 2021-06-08 06:49:34,628 | infra | DEBUG | pool-5-thread-30 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623160165.0_create__6400000_6450000_1623160165.01 2021-06-08 06:49:34,628 | infra | DEBUG | pool-5-thread-30 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623160165.0_create__6400000_6450000_1623160165.01 are 50000 2021-06-08 06:49:34,628 | infra | DEBUG | pool-5-thread-30 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623160165.0_create__6450000_6500000_1623160165.01 2021-06-08 06:49:34,628 | infra | DEBUG | pool-5-thread-30 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623160165.0_create__6450000_6500000_1623160165.01 are 50000 2021-06-08 06:49:34,628 | infra | DEBUG | pool-5-thread-30 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623160165.0_create__6500000_6550000_1623160165.01 2021-06-08 06:49:34,628 | infra | DEBUG | pool-5-thread-30 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623160165.0_create__6500000_6550000_1623160165.01 are 50000 2021-06-08 06:49:34,628 | infra | DEBUG | pool-5-thread-30 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623160165.0_create__6550000_6600000_1623160165.01 2021-06-08 06:49:34,628 | infra | DEBUG | pool-5-thread-30 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623160165.0_create__6550000_6600000_1623160165.01 are 50000 2021-06-08 06:49:34,628 | infra | DEBUG | pool-5-thread-30 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623160165.0_create__6600000_6650000_1623160165.01 2021-06-08 06:49:34,628 | infra | DEBUG | pool-5-thread-30 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623160165.0_create__6600000_6650000_1623160165.01 are 50000 2021-06-08 06:49:34,628 | infra | DEBUG | pool-5-thread-30 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623160165.0_create__6650000_6700000_1623160165.01 2021-06-08 06:49:34,628 | infra | DEBUG | pool-5-thread-30 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623160165.0_create__6650000_6700000_1623160165.01 are 50000 2021-06-08 06:49:34,628 | infra | DEBUG | pool-5-thread-30 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623160165.0_create__6700000_6750000_1623160165.01 2021-06-08 06:49:34,628 | infra | DEBUG | pool-5-thread-30 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623160165.0_create__6700000_6750000_1623160165.01 are 50000 2021-06-08 06:49:34,630 | infra | DEBUG | pool-5-thread-30 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623160165.0_create__6750000_6800000_1623160165.01 2021-06-08 06:49:34,630 | infra | DEBUG | pool-5-thread-30 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623160165.0_create__6750000_6800000_1623160165.01 are 50000 2021-06-08 06:49:34,630 | infra | DEBUG | pool-5-thread-30 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623160165.0_create__6800000_6850000_1623160165.01 2021-06-08 06:49:34,630 | infra | DEBUG | pool-5-thread-30 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623160165.0_create__6800000_6850000_1623160165.01 are 50000 2021-06-08 06:49:34,630 | infra | DEBUG | pool-5-thread-30 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623160165.0_create__6850000_6900000_1623160165.01 2021-06-08 06:49:34,630 | infra | DEBUG | pool-5-thread-30 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623160165.0_create__6850000_6900000_1623160165.01 are 50000 2021-06-08 06:49:34,630 | infra | DEBUG | pool-5-thread-30 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623160165.0_create__6900000_6950000_1623160165.01 2021-06-08 06:49:34,630 | infra | DEBUG | pool-5-thread-30 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623160165.0_create__6900000_6950000_1623160165.01 are 50000 2021-06-08 06:49:34,630 | infra | DEBUG | pool-5-thread-30 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623160165.0_create__6950000_7000000_1623160165.01 2021-06-08 06:49:34,630 | infra | DEBUG | pool-5-thread-30 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623160165.0_create__6950000_7000000_1623160165.01 are 50000 2021-06-08 06:49:34,630 | infra | DEBUG | pool-5-thread-30 | [task_manager:get_task_result:26] Getting task result for PrintBucketStats_default_1623160165.01 2021-06-08 06:49:34,836 | test | INFO | pool-5-thread-1 | [table_view:display:72] Ops trend for bucket 'default' +-------+-------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-------+-------------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | *****....................................................................................................*X | 66759.581 | +-------+-------------------------------------------------------------------------------------------------------------+-----------+ 2021-06-08 06:49:34,836 | infra | INFO | pool-5-thread-1 | [task:complete_task:102] Thread 'PrintBucketStats_default_1623160165.01' completed 2021-06-08 06:49:34,836 | infra | DEBUG | pool-5-thread-30 | [task:call:1663] ========= Tasks in loadgen pool======= 2021-06-08 06:49:34,836 | infra | WARNING | pool-5-thread-30 | [task_manager:print_tasks_in_pool:83] Task 'LoadDocsGen_default__default__default_create_6000000_7000000_1623160165.0' not completed 2021-06-08 06:49:34,838 | infra | DEBUG | pool-5-thread-30 | [task:call:1665] ====================================== 2021-06-08 06:49:34,838 | infra | DEBUG | pool-5-thread-30 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623160165.0_create__6000000_6050000_1623160165.01' complete. Loaded 50000 items 2021-06-08 06:49:34,838 | infra | DEBUG | pool-5-thread-30 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623160165.0_create__6050000_6100000_1623160165.01' complete. Loaded 50000 items 2021-06-08 06:49:34,838 | infra | DEBUG | pool-5-thread-30 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623160165.0_create__6100000_6150000_1623160165.01' complete. Loaded 50000 items 2021-06-08 06:49:34,838 | infra | DEBUG | pool-5-thread-30 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623160165.0_create__6150000_6200000_1623160165.01' complete. Loaded 50000 items 2021-06-08 06:49:34,838 | infra | DEBUG | pool-5-thread-30 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623160165.0_create__6200000_6250000_1623160165.01' complete. Loaded 50000 items 2021-06-08 06:49:34,838 | infra | DEBUG | pool-5-thread-30 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623160165.0_create__6250000_6300000_1623160165.01' complete. Loaded 50000 items 2021-06-08 06:49:34,838 | infra | DEBUG | pool-5-thread-30 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623160165.0_create__6300000_6350000_1623160165.01' complete. Loaded 50000 items 2021-06-08 06:49:34,838 | infra | DEBUG | pool-5-thread-30 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623160165.0_create__6350000_6400000_1623160165.01' complete. Loaded 50000 items 2021-06-08 06:49:34,838 | infra | DEBUG | pool-5-thread-30 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623160165.0_create__6400000_6450000_1623160165.01' complete. Loaded 50000 items 2021-06-08 06:49:34,838 | infra | DEBUG | pool-5-thread-30 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623160165.0_create__6450000_6500000_1623160165.01' complete. Loaded 50000 items 2021-06-08 06:49:34,838 | infra | DEBUG | pool-5-thread-30 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623160165.0_create__6500000_6550000_1623160165.01' complete. Loaded 50000 items 2021-06-08 06:49:34,838 | infra | DEBUG | pool-5-thread-30 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623160165.0_create__6550000_6600000_1623160165.01' complete. Loaded 50000 items 2021-06-08 06:49:34,838 | infra | DEBUG | pool-5-thread-30 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623160165.0_create__6600000_6650000_1623160165.01' complete. Loaded 50000 items 2021-06-08 06:49:34,838 | infra | DEBUG | pool-5-thread-30 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623160165.0_create__6650000_6700000_1623160165.01' complete. Loaded 50000 items 2021-06-08 06:49:34,838 | infra | DEBUG | pool-5-thread-30 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623160165.0_create__6700000_6750000_1623160165.01' complete. Loaded 50000 items 2021-06-08 06:49:34,838 | infra | DEBUG | pool-5-thread-30 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623160165.0_create__6750000_6800000_1623160165.01' complete. Loaded 50000 items 2021-06-08 06:49:34,838 | infra | DEBUG | pool-5-thread-30 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623160165.0_create__6800000_6850000_1623160165.01' complete. Loaded 50000 items 2021-06-08 06:49:34,838 | infra | DEBUG | pool-5-thread-30 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623160165.0_create__6850000_6900000_1623160165.01' complete. Loaded 50000 items 2021-06-08 06:49:34,838 | infra | DEBUG | pool-5-thread-30 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623160165.0_create__6900000_6950000_1623160165.01' complete. Loaded 50000 items 2021-06-08 06:49:34,839 | infra | DEBUG | pool-5-thread-30 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623160165.0_create__6950000_7000000_1623160165.01' complete. Loaded 50000 items 2021-06-08 06:49:34,839 | infra | INFO | pool-5-thread-30 | [task:complete_task:102] Thread 'LoadDocsGen_default__default__default_create_6000000_7000000_1623160165.0' completed 2021-06-08 06:49:34,842 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.198 with username: root, password: couchbase 2021-06-08 06:49:34,936 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 06:49:35,204 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: ls /etc/ 2021-06-08 06:49:35,283 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 06:49:35,283 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: ls /cygdrive/c/Windows 2021-06-08 06:49:35,305 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: uname -m 2021-06-08 06:49:35,328 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: cat /proc/cpuinfo 2021-06-08 06:49:35,345 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: df -Thl 2021-06-08 06:49:35,365 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: cat /proc/meminfo 2021-06-08 06:49:35,384 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: hostname 2021-06-08 06:49:35,401 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: hostname -d 2021-06-08 06:49:35,418 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:559] Starting StatsWaitTask for ep_data_write_failed on bucket default 2021-06-08 06:49:35,418 | infra | INFO | MainThread | [task_manager:add_new_task:23] Added new task: StatsWaitTask_default_ep_data_write_failed_1623160175.42 2021-06-08 06:49:35,418 | infra | INFO | pool-5-thread-10 | [task:start_task:87] Thread 'StatsWaitTask_default_ep_data_write_failed_1623160175.42' started 2021-06-08 06:49:35,420 | infra | DEBUG | pool-5-thread-10 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:49:35,421 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.196 with username: root, password: couchbase 2021-06-08 06:49:35,507 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 06:49:35,766 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: ls /etc/ 2021-06-08 06:49:35,845 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 06:49:35,845 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: ls /cygdrive/c/Windows 2021-06-08 06:49:35,861 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: uname -m 2021-06-08 06:49:35,878 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: cat /proc/cpuinfo 2021-06-08 06:49:35,895 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: df -Thl 2021-06-08 06:49:35,917 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: cat /proc/meminfo 2021-06-08 06:49:35,934 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: hostname 2021-06-08 06:49:35,950 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: hostname -d 2021-06-08 06:49:35,966 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:559] Starting StatsWaitTask for ep_data_write_failed on bucket default 2021-06-08 06:49:35,966 | infra | INFO | MainThread | [task_manager:add_new_task:23] Added new task: StatsWaitTask_default_ep_data_write_failed_1623160175.97 2021-06-08 06:49:35,966 | infra | INFO | pool-5-thread-13 | [task:start_task:87] Thread 'StatsWaitTask_default_ep_data_write_failed_1623160175.97' started 2021-06-08 06:49:35,967 | infra | DEBUG | pool-5-thread-13 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:49:35,969 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.199 with username: root, password: couchbase 2021-06-08 06:49:36,062 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 06:49:36,233 | test | DEBUG | pool-5-thread-10 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:49:36,233 | infra | DEBUG | pool-5-thread-10 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.198': '0'} for bucket 'default' 2021-06-08 06:49:36,233 | infra | DEBUG | pool-5-thread-10 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:49:36,234 | infra | DEBUG | pool-5-thread-10 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:49:36,234 | infra | DEBUG | pool-5-thread-10 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:49:36,342 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: ls /etc/ 2021-06-08 06:49:36,433 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 06:49:36,433 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: ls /cygdrive/c/Windows 2021-06-08 06:49:36,450 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: uname -m 2021-06-08 06:49:36,483 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: cat /proc/cpuinfo 2021-06-08 06:49:36,503 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: df -Thl 2021-06-08 06:49:36,519 | test | DEBUG | pool-5-thread-13 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:49:36,519 | infra | DEBUG | pool-5-thread-13 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.196': '0'} for bucket 'default' 2021-06-08 06:49:36,519 | infra | DEBUG | pool-5-thread-13 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:49:36,520 | infra | DEBUG | pool-5-thread-13 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:49:36,520 | infra | DEBUG | pool-5-thread-13 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:49:36,529 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: cat /proc/meminfo 2021-06-08 06:49:36,546 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: hostname 2021-06-08 06:49:36,566 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: hostname -d 2021-06-08 06:49:36,585 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:559] Starting StatsWaitTask for ep_data_write_failed on bucket default 2021-06-08 06:49:36,585 | infra | INFO | MainThread | [task_manager:add_new_task:23] Added new task: StatsWaitTask_default_ep_data_write_failed_1623160176.59 2021-06-08 06:49:36,585 | infra | INFO | pool-5-thread-4 | [task:start_task:87] Thread 'StatsWaitTask_default_ep_data_write_failed_1623160176.59' started 2021-06-08 06:49:36,586 | infra | DEBUG | pool-5-thread-4 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:49:36,588 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.168 with username: root, password: couchbase 2021-06-08 06:49:36,686 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 06:49:36,964 | test | DEBUG | pool-5-thread-4 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:49:36,964 | infra | DEBUG | pool-5-thread-4 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.199': '0'} for bucket 'default' 2021-06-08 06:49:36,964 | infra | DEBUG | pool-5-thread-4 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:49:36,967 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:49:36,967 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:49:36,987 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: ls /etc/ 2021-06-08 06:49:37,084 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 06:49:37,085 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: ls /cygdrive/c/Windows 2021-06-08 06:49:37,105 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: uname -m 2021-06-08 06:49:37,124 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: cat /proc/cpuinfo 2021-06-08 06:49:37,142 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: df -Thl 2021-06-08 06:49:37,167 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: cat /proc/meminfo 2021-06-08 06:49:37,187 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: hostname 2021-06-08 06:49:37,204 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: hostname -d 2021-06-08 06:49:37,226 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:559] Starting StatsWaitTask for ep_data_write_failed on bucket default 2021-06-08 06:49:37,226 | infra | INFO | MainThread | [task_manager:add_new_task:23] Added new task: StatsWaitTask_default_ep_data_write_failed_1623160177.23 2021-06-08 06:49:37,226 | infra | INFO | pool-5-thread-3 | [task:start_task:87] Thread 'StatsWaitTask_default_ep_data_write_failed_1623160177.23' started 2021-06-08 06:49:37,226 | infra | DEBUG | MainThread | [task_manager:get_task_result:26] Getting task result for StatsWaitTask_default_ep_data_write_failed_1623160175.42 2021-06-08 06:49:37,226 | infra | DEBUG | pool-5-thread-3 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:49:38,338 | test | DEBUG | pool-5-thread-3 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:49:38,338 | infra | DEBUG | pool-5-thread-3 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.168': '0'} for bucket 'default' 2021-06-08 06:49:38,339 | infra | DEBUG | pool-5-thread-3 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:49:38,341 | infra | DEBUG | pool-5-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:49:38,341 | infra | DEBUG | pool-5-thread-3 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:49:41,236 | infra | DEBUG | pool-5-thread-10 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:49:41,522 | infra | DEBUG | pool-5-thread-13 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:49:41,795 | test | DEBUG | pool-5-thread-10 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:49:41,795 | infra | DEBUG | pool-5-thread-10 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.198': '0'} for bucket 'default' 2021-06-08 06:49:41,795 | infra | DEBUG | pool-5-thread-10 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:49:41,796 | infra | DEBUG | pool-5-thread-10 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:49:41,796 | infra | DEBUG | pool-5-thread-10 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:49:41,967 | infra | DEBUG | pool-5-thread-4 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:49:41,989 | test | DEBUG | pool-5-thread-13 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:49:41,990 | infra | DEBUG | pool-5-thread-13 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.196': '0'} for bucket 'default' 2021-06-08 06:49:41,990 | infra | DEBUG | pool-5-thread-13 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:49:41,990 | infra | DEBUG | pool-5-thread-13 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:49:41,992 | infra | DEBUG | pool-5-thread-13 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:49:42,904 | test | DEBUG | pool-5-thread-4 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:49:42,904 | infra | DEBUG | pool-5-thread-4 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.199': '0'} for bucket 'default' 2021-06-08 06:49:42,904 | infra | DEBUG | pool-5-thread-4 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:49:42,905 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:49:42,905 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:49:43,342 | infra | DEBUG | pool-5-thread-3 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:49:44,020 | test | DEBUG | pool-5-thread-3 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:49:44,020 | infra | DEBUG | pool-5-thread-3 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.168': '0'} for bucket 'default' 2021-06-08 06:49:44,020 | infra | DEBUG | pool-5-thread-3 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:49:44,023 | infra | DEBUG | pool-5-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:49:44,023 | infra | DEBUG | pool-5-thread-3 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:49:46,796 | infra | DEBUG | pool-5-thread-10 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:49:46,992 | infra | DEBUG | pool-5-thread-13 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:49:47,628 | test | DEBUG | pool-5-thread-13 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:49:47,628 | infra | DEBUG | pool-5-thread-13 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.196': '0'} for bucket 'default' 2021-06-08 06:49:47,628 | infra | DEBUG | pool-5-thread-13 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:49:47,630 | infra | DEBUG | pool-5-thread-13 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:49:47,630 | infra | DEBUG | pool-5-thread-13 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:49:47,631 | test | DEBUG | pool-5-thread-10 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:49:47,631 | infra | DEBUG | pool-5-thread-10 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.198': '0'} for bucket 'default' 2021-06-08 06:49:47,631 | infra | DEBUG | pool-5-thread-10 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:49:47,632 | infra | DEBUG | pool-5-thread-10 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:49:47,632 | infra | DEBUG | pool-5-thread-10 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:49:47,907 | infra | DEBUG | pool-5-thread-4 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:49:48,881 | test | DEBUG | pool-5-thread-4 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:49:48,881 | infra | DEBUG | pool-5-thread-4 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.199': '0'} for bucket 'default' 2021-06-08 06:49:48,882 | infra | DEBUG | pool-5-thread-4 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:49:48,884 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:49:48,884 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:49:49,023 | infra | DEBUG | pool-5-thread-3 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:49:49,901 | test | DEBUG | pool-5-thread-3 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:49:49,903 | infra | DEBUG | pool-5-thread-3 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.168': '0'} for bucket 'default' 2021-06-08 06:49:49,903 | infra | DEBUG | pool-5-thread-3 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:49:49,904 | infra | DEBUG | pool-5-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:49:49,904 | infra | DEBUG | pool-5-thread-3 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:49:52,631 | infra | DEBUG | pool-5-thread-13 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:49:52,632 | infra | DEBUG | pool-5-thread-10 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:49:53,884 | infra | DEBUG | pool-5-thread-4 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:49:54,378 | test | DEBUG | pool-5-thread-10 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:49:54,378 | infra | DEBUG | pool-5-thread-10 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.198': '0'} for bucket 'default' 2021-06-08 06:49:54,378 | infra | DEBUG | pool-5-thread-10 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:49:54,381 | infra | DEBUG | pool-5-thread-10 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:49:54,381 | infra | DEBUG | pool-5-thread-10 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:49:54,381 | test | DEBUG | pool-5-thread-13 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:49:54,381 | infra | DEBUG | pool-5-thread-13 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.196': '0'} for bucket 'default' 2021-06-08 06:49:54,381 | infra | DEBUG | pool-5-thread-13 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:49:54,384 | infra | DEBUG | pool-5-thread-13 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:49:54,384 | infra | DEBUG | pool-5-thread-13 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:49:54,418 | test | DEBUG | pool-5-thread-4 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:49:54,418 | infra | DEBUG | pool-5-thread-4 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.199': '0'} for bucket 'default' 2021-06-08 06:49:54,418 | infra | DEBUG | pool-5-thread-4 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:49:54,421 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:49:54,421 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:49:54,904 | infra | DEBUG | pool-5-thread-3 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:49:55,250 | test | DEBUG | pool-5-thread-3 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:49:55,250 | infra | DEBUG | pool-5-thread-3 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.168': '0'} for bucket 'default' 2021-06-08 06:49:55,250 | infra | DEBUG | pool-5-thread-3 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:49:55,253 | infra | DEBUG | pool-5-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:49:55,253 | infra | DEBUG | pool-5-thread-3 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:49:59,381 | infra | DEBUG | pool-5-thread-10 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:49:59,384 | infra | DEBUG | pool-5-thread-13 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:49:59,421 | infra | DEBUG | pool-5-thread-4 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:50:00,035 | test | DEBUG | pool-5-thread-4 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:50:00,036 | infra | DEBUG | pool-5-thread-4 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.199': '0'} for bucket 'default' 2021-06-08 06:50:00,036 | infra | DEBUG | pool-5-thread-4 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:50:00,038 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:50:00,038 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:50:00,253 | infra | DEBUG | pool-5-thread-3 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:50:00,280 | test | DEBUG | pool-5-thread-13 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:50:00,280 | infra | DEBUG | pool-5-thread-13 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.196': '0'} for bucket 'default' 2021-06-08 06:50:00,280 | infra | DEBUG | pool-5-thread-13 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:50:00,282 | infra | DEBUG | pool-5-thread-13 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:50:00,282 | infra | DEBUG | pool-5-thread-13 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:50:00,505 | test | DEBUG | pool-5-thread-10 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:50:00,505 | infra | DEBUG | pool-5-thread-10 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.198': '0'} for bucket 'default' 2021-06-08 06:50:00,505 | infra | DEBUG | pool-5-thread-10 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:50:00,506 | infra | DEBUG | pool-5-thread-10 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:50:00,506 | infra | DEBUG | pool-5-thread-10 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:50:01,444 | test | DEBUG | pool-5-thread-3 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:50:01,444 | infra | DEBUG | pool-5-thread-3 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.168': '0'} for bucket 'default' 2021-06-08 06:50:01,444 | infra | DEBUG | pool-5-thread-3 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:50:01,446 | infra | DEBUG | pool-5-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:50:01,446 | infra | DEBUG | pool-5-thread-3 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:50:05,038 | infra | DEBUG | pool-5-thread-4 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:50:05,282 | infra | DEBUG | pool-5-thread-13 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:50:05,492 | test | DEBUG | pool-5-thread-4 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:50:05,492 | infra | DEBUG | pool-5-thread-4 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.199': '0'} for bucket 'default' 2021-06-08 06:50:05,492 | infra | DEBUG | pool-5-thread-4 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:50:05,493 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:50:05,493 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:50:05,506 | infra | DEBUG | pool-5-thread-10 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:50:05,967 | test | DEBUG | pool-5-thread-13 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:50:05,967 | infra | DEBUG | pool-5-thread-13 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.196': '0'} for bucket 'default' 2021-06-08 06:50:05,967 | infra | DEBUG | pool-5-thread-13 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:50:05,969 | infra | DEBUG | pool-5-thread-13 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:50:05,969 | infra | DEBUG | pool-5-thread-13 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:50:06,223 | test | DEBUG | pool-5-thread-10 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:50:06,223 | infra | DEBUG | pool-5-thread-10 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.198': '0'} for bucket 'default' 2021-06-08 06:50:06,223 | infra | DEBUG | pool-5-thread-10 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:50:06,224 | infra | DEBUG | pool-5-thread-10 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:50:06,224 | infra | DEBUG | pool-5-thread-10 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:50:06,447 | infra | DEBUG | pool-5-thread-3 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:50:07,263 | test | DEBUG | pool-5-thread-3 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:50:07,263 | infra | DEBUG | pool-5-thread-3 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.168': '0'} for bucket 'default' 2021-06-08 06:50:07,263 | infra | DEBUG | pool-5-thread-3 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:50:07,265 | infra | DEBUG | pool-5-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:50:07,265 | infra | DEBUG | pool-5-thread-3 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:50:10,494 | infra | DEBUG | pool-5-thread-4 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:50:10,969 | infra | DEBUG | pool-5-thread-13 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:50:11,191 | test | DEBUG | pool-5-thread-4 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:50:11,191 | infra | DEBUG | pool-5-thread-4 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.199': '0'} for bucket 'default' 2021-06-08 06:50:11,191 | infra | DEBUG | pool-5-thread-4 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:50:11,194 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:50:11,194 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:50:11,224 | infra | DEBUG | pool-5-thread-10 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:50:12,180 | test | DEBUG | pool-5-thread-13 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:50:12,180 | infra | DEBUG | pool-5-thread-13 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.196': '0'} for bucket 'default' 2021-06-08 06:50:12,180 | infra | DEBUG | pool-5-thread-13 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:50:12,183 | infra | DEBUG | pool-5-thread-13 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:50:12,183 | infra | DEBUG | pool-5-thread-13 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:50:12,265 | infra | DEBUG | pool-5-thread-3 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:50:12,344 | test | DEBUG | pool-5-thread-10 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:50:12,345 | infra | DEBUG | pool-5-thread-10 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.198': '0'} for bucket 'default' 2021-06-08 06:50:12,345 | infra | DEBUG | pool-5-thread-10 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:50:12,345 | infra | DEBUG | pool-5-thread-10 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:50:12,345 | infra | DEBUG | pool-5-thread-10 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:50:12,670 | test | DEBUG | pool-5-thread-3 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:50:12,670 | infra | DEBUG | pool-5-thread-3 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.168': '0'} for bucket 'default' 2021-06-08 06:50:12,670 | infra | DEBUG | pool-5-thread-3 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:50:12,671 | infra | DEBUG | pool-5-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:50:12,671 | infra | DEBUG | pool-5-thread-3 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:50:16,194 | infra | DEBUG | pool-5-thread-4 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:50:17,183 | infra | DEBUG | pool-5-thread-13 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:50:17,197 | test | DEBUG | pool-5-thread-4 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:50:17,197 | infra | DEBUG | pool-5-thread-4 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.199': '0'} for bucket 'default' 2021-06-08 06:50:17,197 | infra | DEBUG | pool-5-thread-4 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:50:17,198 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:50:17,198 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:50:17,346 | infra | DEBUG | pool-5-thread-10 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:50:17,657 | test | DEBUG | pool-5-thread-13 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:50:17,657 | infra | DEBUG | pool-5-thread-13 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.196': '0'} for bucket 'default' 2021-06-08 06:50:17,657 | infra | DEBUG | pool-5-thread-13 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:50:17,660 | infra | DEBUG | pool-5-thread-13 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:50:17,660 | infra | DEBUG | pool-5-thread-13 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:50:17,671 | infra | DEBUG | pool-5-thread-3 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:50:17,681 | test | DEBUG | pool-5-thread-10 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:50:17,683 | infra | DEBUG | pool-5-thread-10 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.198': '0'} for bucket 'default' 2021-06-08 06:50:17,683 | infra | DEBUG | pool-5-thread-10 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:50:17,684 | infra | DEBUG | pool-5-thread-10 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:50:17,684 | infra | DEBUG | pool-5-thread-10 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:50:18,096 | test | DEBUG | pool-5-thread-3 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:50:18,096 | infra | DEBUG | pool-5-thread-3 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.168': '0'} for bucket 'default' 2021-06-08 06:50:18,096 | infra | DEBUG | pool-5-thread-3 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:50:18,098 | infra | DEBUG | pool-5-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:50:18,098 | infra | DEBUG | pool-5-thread-3 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:50:22,198 | infra | DEBUG | pool-5-thread-4 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:50:22,414 | test | DEBUG | pool-5-thread-4 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:50:22,415 | infra | DEBUG | pool-5-thread-4 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.199': '0'} for bucket 'default' 2021-06-08 06:50:22,415 | infra | DEBUG | pool-5-thread-4 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:50:22,417 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:50:22,417 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:50:22,661 | infra | DEBUG | pool-5-thread-13 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:50:22,684 | infra | DEBUG | pool-5-thread-10 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:50:22,904 | test | DEBUG | pool-5-thread-10 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:50:22,904 | infra | DEBUG | pool-5-thread-10 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.198': '0'} for bucket 'default' 2021-06-08 06:50:22,904 | infra | DEBUG | pool-5-thread-10 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:50:22,905 | infra | DEBUG | pool-5-thread-10 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:50:22,905 | infra | DEBUG | pool-5-thread-10 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:50:23,098 | infra | DEBUG | pool-5-thread-3 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:50:23,127 | test | DEBUG | pool-5-thread-13 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:50:23,128 | infra | DEBUG | pool-5-thread-13 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.196': '0'} for bucket 'default' 2021-06-08 06:50:23,128 | infra | DEBUG | pool-5-thread-13 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:50:23,128 | infra | DEBUG | pool-5-thread-13 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:50:23,130 | infra | DEBUG | pool-5-thread-13 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:50:23,403 | test | DEBUG | pool-5-thread-3 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:50:23,403 | infra | DEBUG | pool-5-thread-3 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.168': '0'} for bucket 'default' 2021-06-08 06:50:23,403 | infra | DEBUG | pool-5-thread-3 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:50:23,404 | infra | DEBUG | pool-5-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:50:23,404 | infra | DEBUG | pool-5-thread-3 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:50:27,417 | infra | DEBUG | pool-5-thread-4 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:50:27,829 | test | DEBUG | pool-5-thread-4 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:50:27,829 | infra | DEBUG | pool-5-thread-4 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.199': '0'} for bucket 'default' 2021-06-08 06:50:27,829 | infra | DEBUG | pool-5-thread-4 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:50:27,832 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:50:27,832 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:50:27,905 | infra | DEBUG | pool-5-thread-10 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:50:28,115 | test | DEBUG | pool-5-thread-10 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:50:28,115 | infra | DEBUG | pool-5-thread-10 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.198': '0'} for bucket 'default' 2021-06-08 06:50:28,115 | infra | DEBUG | pool-5-thread-10 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:50:28,118 | infra | DEBUG | pool-5-thread-10 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:50:28,118 | infra | DEBUG | pool-5-thread-10 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:50:28,130 | infra | DEBUG | pool-5-thread-13 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:50:28,348 | test | DEBUG | pool-5-thread-13 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:50:28,348 | infra | DEBUG | pool-5-thread-13 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.196': '0'} for bucket 'default' 2021-06-08 06:50:28,348 | infra | DEBUG | pool-5-thread-13 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:50:28,349 | infra | DEBUG | pool-5-thread-13 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:50:28,349 | infra | DEBUG | pool-5-thread-13 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:50:28,405 | infra | DEBUG | pool-5-thread-3 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:50:28,617 | test | DEBUG | pool-5-thread-3 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:50:28,617 | infra | DEBUG | pool-5-thread-3 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.168': '0'} for bucket 'default' 2021-06-08 06:50:28,618 | infra | DEBUG | pool-5-thread-3 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:50:28,618 | infra | DEBUG | pool-5-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:50:28,618 | infra | DEBUG | pool-5-thread-3 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:50:32,832 | infra | DEBUG | pool-5-thread-4 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:50:33,040 | test | DEBUG | pool-5-thread-4 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:50:33,040 | infra | DEBUG | pool-5-thread-4 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.199': '0'} for bucket 'default' 2021-06-08 06:50:33,042 | infra | DEBUG | pool-5-thread-4 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:50:33,042 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:50:33,042 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:50:33,118 | infra | DEBUG | pool-5-thread-10 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:50:33,335 | test | DEBUG | pool-5-thread-10 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:50:33,335 | infra | DEBUG | pool-5-thread-10 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.198': '0'} for bucket 'default' 2021-06-08 06:50:33,336 | infra | DEBUG | pool-5-thread-10 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:50:33,338 | infra | DEBUG | pool-5-thread-10 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:50:33,338 | infra | DEBUG | pool-5-thread-10 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:50:33,349 | infra | DEBUG | pool-5-thread-13 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:50:33,578 | test | DEBUG | pool-5-thread-13 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:50:33,578 | infra | DEBUG | pool-5-thread-13 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.196': '0'} for bucket 'default' 2021-06-08 06:50:33,578 | infra | DEBUG | pool-5-thread-13 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:50:33,579 | infra | DEBUG | pool-5-thread-13 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:50:33,579 | infra | DEBUG | pool-5-thread-13 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:50:33,619 | infra | DEBUG | pool-5-thread-3 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:50:33,835 | test | DEBUG | pool-5-thread-3 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:50:33,835 | infra | DEBUG | pool-5-thread-3 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.168': '0'} for bucket 'default' 2021-06-08 06:50:33,835 | infra | DEBUG | pool-5-thread-3 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:50:33,836 | infra | DEBUG | pool-5-thread-3 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:50:33,836 | infra | DEBUG | pool-5-thread-3 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:50:38,043 | infra | DEBUG | pool-5-thread-4 | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.199 2021-06-08 06:50:38,045 | infra | INFO | pool-5-thread-4 | [task:complete_task:102] Thread 'StatsWaitTask_default_ep_data_write_failed_1623160176.59' completed 2021-06-08 06:50:38,338 | infra | DEBUG | pool-5-thread-10 | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.198 2021-06-08 06:50:38,338 | infra | INFO | pool-5-thread-10 | [task:complete_task:102] Thread 'StatsWaitTask_default_ep_data_write_failed_1623160175.42' completed 2021-06-08 06:50:38,344 | infra | INFO | MainThread | [task_manager:add_new_task:23] Added new task: LoadDocsGen_default__default__default_create_7000000_8000000_1623160238.34 2021-06-08 06:50:38,344 | infra | INFO | pool-5-thread-25 | [task:start_task:87] Thread 'LoadDocsGen_default__default__default_create_7000000_8000000_1623160238.34' started 2021-06-08 06:50:38,344 | infra | DEBUG | MainThread | [task_manager:get_task_result:26] Getting task result for LoadDocsGen_default__default__default_create_7000000_8000000_1623160238.34 2021-06-08 06:50:38,371 | infra | INFO | pool-5-thread-25 | [task_manager:add_new_task:23] Added new task: PrintBucketStats_default_1623160238.36 2021-06-08 06:50:38,371 | infra | INFO | pool-5-thread-24 | [task:start_task:87] Thread 'PrintBucketStats_default_1623160238.36' started 2021-06-08 06:50:38,371 | infra | INFO | pool-5-thread-25 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623160238.34_create__7000000_7050000_1623160238.36 2021-06-08 06:50:38,371 | infra | INFO | pool-5-thread-8 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623160238.34_create__7000000_7050000_1623160238.36' started 2021-06-08 06:50:38,371 | infra | INFO | pool-5-thread-25 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623160238.34_create__7050000_7100000_1623160238.36 2021-06-08 06:50:38,371 | infra | INFO | pool-5-thread-7 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623160238.34_create__7050000_7100000_1623160238.36' started 2021-06-08 06:50:38,371 | infra | INFO | pool-5-thread-25 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623160238.34_create__7100000_7150000_1623160238.36 2021-06-08 06:50:38,371 | infra | INFO | pool-5-thread-9 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623160238.34_create__7100000_7150000_1623160238.36' started 2021-06-08 06:50:38,371 | infra | INFO | pool-5-thread-26 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623160238.34_create__7150000_7200000_1623160238.36' started 2021-06-08 06:50:38,371 | infra | INFO | pool-5-thread-25 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623160238.34_create__7150000_7200000_1623160238.36 2021-06-08 06:50:38,375 | infra | INFO | pool-5-thread-25 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623160238.34_create__7200000_7250000_1623160238.36 2021-06-08 06:50:38,375 | infra | INFO | pool-5-thread-19 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623160238.34_create__7200000_7250000_1623160238.36' started 2021-06-08 06:50:38,375 | infra | INFO | pool-5-thread-25 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623160238.34_create__7250000_7300000_1623160238.36 2021-06-08 06:50:38,375 | infra | INFO | pool-5-thread-5 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623160238.34_create__7250000_7300000_1623160238.36' started 2021-06-08 06:50:38,375 | infra | INFO | pool-5-thread-25 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623160238.34_create__7300000_7350000_1623160238.36 2021-06-08 06:50:38,375 | infra | INFO | pool-5-thread-15 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623160238.34_create__7300000_7350000_1623160238.36' started 2021-06-08 06:50:38,377 | infra | INFO | pool-5-thread-25 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623160238.34_create__7350000_7400000_1623160238.36 2021-06-08 06:50:38,377 | infra | INFO | pool-5-thread-12 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623160238.34_create__7350000_7400000_1623160238.36' started 2021-06-08 06:50:38,378 | infra | INFO | pool-5-thread-25 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623160238.34_create__7400000_7450000_1623160238.36 2021-06-08 06:50:38,378 | infra | INFO | pool-5-thread-21 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623160238.34_create__7400000_7450000_1623160238.36' started 2021-06-08 06:50:38,378 | infra | INFO | pool-5-thread-25 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623160238.34_create__7450000_7500000_1623160238.36 2021-06-08 06:50:38,378 | infra | INFO | pool-5-thread-11 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623160238.34_create__7450000_7500000_1623160238.36' started 2021-06-08 06:50:38,378 | infra | INFO | pool-5-thread-25 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623160238.34_create__7500000_7550000_1623160238.36 2021-06-08 06:50:38,378 | infra | INFO | pool-5-thread-20 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623160238.34_create__7500000_7550000_1623160238.36' started 2021-06-08 06:50:38,381 | infra | INFO | pool-5-thread-16 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623160238.34_create__7550000_7600000_1623160238.36' started 2021-06-08 06:50:38,381 | infra | INFO | pool-5-thread-25 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623160238.34_create__7550000_7600000_1623160238.36 2021-06-08 06:50:38,382 | infra | INFO | pool-5-thread-25 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623160238.34_create__7600000_7650000_1623160238.36 2021-06-08 06:50:38,382 | infra | INFO | pool-5-thread-17 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623160238.34_create__7600000_7650000_1623160238.36' started 2021-06-08 06:50:38,382 | infra | INFO | pool-5-thread-2 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623160238.34_create__7650000_7700000_1623160238.36' started 2021-06-08 06:50:38,382 | infra | INFO | pool-5-thread-25 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623160238.34_create__7650000_7700000_1623160238.36 2021-06-08 06:50:38,384 | infra | INFO | pool-5-thread-6 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623160238.34_create__7700000_7750000_1623160238.36' started 2021-06-08 06:50:38,384 | infra | INFO | pool-5-thread-25 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623160238.34_create__7700000_7750000_1623160238.36 2021-06-08 06:50:38,388 | infra | INFO | pool-5-thread-25 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623160238.34_create__7750000_7800000_1623160238.36 2021-06-08 06:50:38,388 | infra | INFO | pool-5-thread-23 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623160238.34_create__7750000_7800000_1623160238.36' started 2021-06-08 06:50:38,388 | infra | INFO | pool-5-thread-25 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623160238.34_create__7800000_7850000_1623160238.36 2021-06-08 06:50:38,388 | infra | INFO | pool-5-thread-22 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623160238.34_create__7800000_7850000_1623160238.36' started 2021-06-08 06:50:38,390 | infra | INFO | pool-5-thread-27 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623160238.34_create__7850000_7900000_1623160238.36' started 2021-06-08 06:50:38,390 | infra | INFO | pool-5-thread-25 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623160238.34_create__7850000_7900000_1623160238.36 2021-06-08 06:50:38,391 | infra | INFO | pool-5-thread-25 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623160238.34_create__7900000_7950000_1623160238.36 2021-06-08 06:50:38,391 | infra | INFO | pool-5-thread-29 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623160238.34_create__7900000_7950000_1623160238.36' started 2021-06-08 06:50:38,391 | infra | INFO | pool-5-thread-18 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623160238.34_create__7950000_8000000_1623160238.36' started 2021-06-08 06:50:38,391 | infra | INFO | pool-5-thread-25 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623160238.34_create__7950000_8000000_1623160238.36 2021-06-08 06:50:38,391 | infra | DEBUG | pool-5-thread-25 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623160238.34_create__7000000_7050000_1623160238.36 2021-06-08 06:50:38,581 | infra | DEBUG | pool-5-thread-13 | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.196 2021-06-08 06:50:38,582 | infra | INFO | pool-5-thread-13 | [task:complete_task:102] Thread 'StatsWaitTask_default_ep_data_write_failed_1623160175.97' completed 2021-06-08 06:50:38,755 | test | DEBUG | pool-5-thread-24 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 06:50:38,756 | test | DEBUG | pool-5-thread-24 | [task:print_ep_queue_size:3508] ep_queue_size for default: 0 2021-06-08 06:50:38,756 | infra | DEBUG | pool-5-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 06:50:38,756 | infra | DEBUG | pool-5-thread-24 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 06:50:38,836 | infra | DEBUG | pool-5-thread-3 | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.168 2021-06-08 06:50:38,838 | infra | INFO | pool-5-thread-3 | [task:complete_task:102] Thread 'StatsWaitTask_default_ep_data_write_failed_1623160177.23' completed 2021-06-08 06:50:40,233 | test | DEBUG | pool-5-thread-24 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 06:50:40,234 | test | DEBUG | pool-5-thread-24 | [task:print_ep_queue_size:3508] ep_queue_size for default: 0 2021-06-08 06:50:40,236 | infra | DEBUG | pool-5-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 06:50:40,236 | infra | DEBUG | pool-5-thread-24 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 06:50:41,782 | test | DEBUG | pool-5-thread-24 | [task:record_bucket_ops:3495] Ops rate for 'default': 24264.500000 2021-06-08 06:50:41,782 | test | DEBUG | pool-5-thread-24 | [task:print_ep_queue_size:3508] ep_queue_size for default: 135284 2021-06-08 06:50:41,788 | infra | DEBUG | pool-5-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 06:50:41,788 | infra | DEBUG | pool-5-thread-24 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 06:50:43,184 | test | DEBUG | pool-5-thread-24 | [task:record_bucket_ops:3495] Ops rate for 'default': 33782.800000 2021-06-08 06:50:43,184 | test | DEBUG | pool-5-thread-24 | [task:print_ep_queue_size:3508] ep_queue_size for default: 472455 2021-06-08 06:50:43,186 | infra | DEBUG | pool-5-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 06:50:43,186 | infra | DEBUG | pool-5-thread-24 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 06:50:44,589 | test | DEBUG | pool-5-thread-24 | [task:record_bucket_ops:3495] Ops rate for 'default': 33782.800000 2021-06-08 06:50:44,589 | test | DEBUG | pool-5-thread-24 | [task:print_ep_queue_size:3508] ep_queue_size for default: 472455 2021-06-08 06:50:44,592 | infra | DEBUG | pool-5-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 06:50:44,594 | infra | DEBUG | pool-5-thread-24 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 06:50:46,065 | test | DEBUG | pool-5-thread-24 | [task:record_bucket_ops:3495] Ops rate for 'default': 33782.800000 2021-06-08 06:50:46,065 | test | DEBUG | pool-5-thread-24 | [task:print_ep_queue_size:3508] ep_queue_size for default: 472455 2021-06-08 06:50:46,068 | infra | DEBUG | pool-5-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 06:50:46,068 | infra | DEBUG | pool-5-thread-24 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 06:50:46,394 | infra | INFO | pool-5-thread-29 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623160238.34_create__7900000_7950000_1623160238.36' completed 2021-06-08 06:50:46,470 | infra | INFO | pool-5-thread-9 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623160238.34_create__7100000_7150000_1623160238.36' completed 2021-06-08 06:50:46,506 | infra | INFO | pool-5-thread-22 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623160238.34_create__7800000_7850000_1623160238.36' completed 2021-06-08 06:50:46,595 | infra | INFO | pool-5-thread-12 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623160238.34_create__7350000_7400000_1623160238.36' completed 2021-06-08 06:50:46,732 | infra | INFO | pool-5-thread-19 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623160238.34_create__7200000_7250000_1623160238.36' completed 2021-06-08 06:50:46,766 | infra | INFO | pool-5-thread-8 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623160238.34_create__7000000_7050000_1623160238.36' completed 2021-06-08 06:50:46,767 | infra | DEBUG | pool-5-thread-25 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623160238.34_create__7000000_7050000_1623160238.36 are 50000 2021-06-08 06:50:46,767 | infra | DEBUG | pool-5-thread-25 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623160238.34_create__7050000_7100000_1623160238.36 2021-06-08 06:50:46,779 | infra | INFO | pool-5-thread-2 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623160238.34_create__7650000_7700000_1623160238.36' completed 2021-06-08 06:50:46,802 | infra | INFO | pool-5-thread-26 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623160238.34_create__7150000_7200000_1623160238.36' completed 2021-06-08 06:50:46,812 | infra | INFO | pool-5-thread-21 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623160238.34_create__7400000_7450000_1623160238.36' completed 2021-06-08 06:50:46,815 | infra | INFO | pool-5-thread-11 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623160238.34_create__7450000_7500000_1623160238.36' completed 2021-06-08 06:50:46,816 | infra | INFO | pool-5-thread-18 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623160238.34_create__7950000_8000000_1623160238.36' completed 2021-06-08 06:50:46,826 | infra | INFO | pool-5-thread-20 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623160238.34_create__7500000_7550000_1623160238.36' completed 2021-06-08 06:50:46,826 | infra | INFO | pool-5-thread-17 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623160238.34_create__7600000_7650000_1623160238.36' completed 2021-06-08 06:50:46,832 | infra | INFO | pool-5-thread-15 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623160238.34_create__7300000_7350000_1623160238.36' completed 2021-06-08 06:50:46,891 | infra | INFO | pool-5-thread-6 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623160238.34_create__7700000_7750000_1623160238.36' completed 2021-06-08 06:50:46,894 | infra | INFO | pool-5-thread-23 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623160238.34_create__7750000_7800000_1623160238.36' completed 2021-06-08 06:50:46,936 | infra | INFO | pool-5-thread-27 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623160238.34_create__7850000_7900000_1623160238.36' completed 2021-06-08 06:50:46,941 | infra | INFO | pool-5-thread-5 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623160238.34_create__7250000_7300000_1623160238.36' completed 2021-06-08 06:50:47,025 | infra | INFO | pool-5-thread-7 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623160238.34_create__7050000_7100000_1623160238.36' completed 2021-06-08 06:50:47,025 | infra | INFO | pool-5-thread-16 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623160238.34_create__7550000_7600000_1623160238.36' completed 2021-06-08 06:50:47,025 | infra | DEBUG | pool-5-thread-25 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623160238.34_create__7050000_7100000_1623160238.36 are 50000 2021-06-08 06:50:47,025 | infra | DEBUG | pool-5-thread-25 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623160238.34_create__7100000_7150000_1623160238.36 2021-06-08 06:50:47,025 | infra | DEBUG | pool-5-thread-25 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623160238.34_create__7100000_7150000_1623160238.36 are 50000 2021-06-08 06:50:47,026 | infra | DEBUG | pool-5-thread-25 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623160238.34_create__7150000_7200000_1623160238.36 2021-06-08 06:50:47,026 | infra | DEBUG | pool-5-thread-25 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623160238.34_create__7150000_7200000_1623160238.36 are 50000 2021-06-08 06:50:47,026 | infra | DEBUG | pool-5-thread-25 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623160238.34_create__7200000_7250000_1623160238.36 2021-06-08 06:50:47,026 | infra | DEBUG | pool-5-thread-25 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623160238.34_create__7200000_7250000_1623160238.36 are 50000 2021-06-08 06:50:47,026 | infra | DEBUG | pool-5-thread-25 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623160238.34_create__7250000_7300000_1623160238.36 2021-06-08 06:50:47,026 | infra | DEBUG | pool-5-thread-25 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623160238.34_create__7250000_7300000_1623160238.36 are 50000 2021-06-08 06:50:47,026 | infra | DEBUG | pool-5-thread-25 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623160238.34_create__7300000_7350000_1623160238.36 2021-06-08 06:50:47,026 | infra | DEBUG | pool-5-thread-25 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623160238.34_create__7300000_7350000_1623160238.36 are 50000 2021-06-08 06:50:47,026 | infra | DEBUG | pool-5-thread-25 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623160238.34_create__7350000_7400000_1623160238.36 2021-06-08 06:50:47,026 | infra | DEBUG | pool-5-thread-25 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623160238.34_create__7350000_7400000_1623160238.36 are 50000 2021-06-08 06:50:47,026 | infra | DEBUG | pool-5-thread-25 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623160238.34_create__7400000_7450000_1623160238.36 2021-06-08 06:50:47,026 | infra | DEBUG | pool-5-thread-25 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623160238.34_create__7400000_7450000_1623160238.36 are 50000 2021-06-08 06:50:47,026 | infra | DEBUG | pool-5-thread-25 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623160238.34_create__7450000_7500000_1623160238.36 2021-06-08 06:50:47,026 | infra | DEBUG | pool-5-thread-25 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623160238.34_create__7450000_7500000_1623160238.36 are 50000 2021-06-08 06:50:47,026 | infra | DEBUG | pool-5-thread-25 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623160238.34_create__7500000_7550000_1623160238.36 2021-06-08 06:50:47,026 | infra | DEBUG | pool-5-thread-25 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623160238.34_create__7500000_7550000_1623160238.36 are 50000 2021-06-08 06:50:47,026 | infra | DEBUG | pool-5-thread-25 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623160238.34_create__7550000_7600000_1623160238.36 2021-06-08 06:50:47,026 | infra | DEBUG | pool-5-thread-25 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623160238.34_create__7550000_7600000_1623160238.36 are 50000 2021-06-08 06:50:47,026 | infra | DEBUG | pool-5-thread-25 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623160238.34_create__7600000_7650000_1623160238.36 2021-06-08 06:50:47,026 | infra | DEBUG | pool-5-thread-25 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623160238.34_create__7600000_7650000_1623160238.36 are 50000 2021-06-08 06:50:47,026 | infra | DEBUG | pool-5-thread-25 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623160238.34_create__7650000_7700000_1623160238.36 2021-06-08 06:50:47,028 | infra | DEBUG | pool-5-thread-25 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623160238.34_create__7650000_7700000_1623160238.36 are 50000 2021-06-08 06:50:47,028 | infra | DEBUG | pool-5-thread-25 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623160238.34_create__7700000_7750000_1623160238.36 2021-06-08 06:50:47,028 | infra | DEBUG | pool-5-thread-25 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623160238.34_create__7700000_7750000_1623160238.36 are 50000 2021-06-08 06:50:47,028 | infra | DEBUG | pool-5-thread-25 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623160238.34_create__7750000_7800000_1623160238.36 2021-06-08 06:50:47,028 | infra | DEBUG | pool-5-thread-25 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623160238.34_create__7750000_7800000_1623160238.36 are 50000 2021-06-08 06:50:47,028 | infra | DEBUG | pool-5-thread-25 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623160238.34_create__7800000_7850000_1623160238.36 2021-06-08 06:50:47,028 | infra | DEBUG | pool-5-thread-25 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623160238.34_create__7800000_7850000_1623160238.36 are 50000 2021-06-08 06:50:47,028 | infra | DEBUG | pool-5-thread-25 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623160238.34_create__7850000_7900000_1623160238.36 2021-06-08 06:50:47,028 | infra | DEBUG | pool-5-thread-25 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623160238.34_create__7850000_7900000_1623160238.36 are 50000 2021-06-08 06:50:47,029 | infra | DEBUG | pool-5-thread-25 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623160238.34_create__7900000_7950000_1623160238.36 2021-06-08 06:50:47,029 | infra | DEBUG | pool-5-thread-25 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623160238.34_create__7900000_7950000_1623160238.36 are 50000 2021-06-08 06:50:47,029 | infra | DEBUG | pool-5-thread-25 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623160238.34_create__7950000_8000000_1623160238.36 2021-06-08 06:50:47,029 | infra | DEBUG | pool-5-thread-25 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623160238.34_create__7950000_8000000_1623160238.36 are 50000 2021-06-08 06:50:47,029 | infra | DEBUG | pool-5-thread-25 | [task_manager:get_task_result:26] Getting task result for PrintBucketStats_default_1623160238.36 2021-06-08 06:50:47,071 | test | INFO | pool-5-thread-24 | [table_view:display:72] Ops trend for bucket 'default' +-------+------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-------+------------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | **.......................................................................*.............................**X | 33782.800 | +-------+------------------------------------------------------------------------------------------------------------+-----------+ 2021-06-08 06:50:47,071 | infra | INFO | pool-5-thread-24 | [task:complete_task:102] Thread 'PrintBucketStats_default_1623160238.36' completed 2021-06-08 06:50:47,071 | infra | DEBUG | pool-5-thread-25 | [task:call:1663] ========= Tasks in loadgen pool======= 2021-06-08 06:50:47,072 | infra | WARNING | pool-5-thread-25 | [task_manager:print_tasks_in_pool:83] Task 'LoadDocsGen_default__default__default_create_7000000_8000000_1623160238.34' not completed 2021-06-08 06:50:47,072 | infra | DEBUG | pool-5-thread-25 | [task:call:1665] ====================================== 2021-06-08 06:50:47,072 | infra | DEBUG | pool-5-thread-25 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623160238.34_create__7000000_7050000_1623160238.36' complete. Loaded 50000 items 2021-06-08 06:50:47,072 | infra | DEBUG | pool-5-thread-25 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623160238.34_create__7050000_7100000_1623160238.36' complete. Loaded 50000 items 2021-06-08 06:50:47,072 | infra | DEBUG | pool-5-thread-25 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623160238.34_create__7100000_7150000_1623160238.36' complete. Loaded 50000 items 2021-06-08 06:50:47,072 | infra | DEBUG | pool-5-thread-25 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623160238.34_create__7150000_7200000_1623160238.36' complete. Loaded 50000 items 2021-06-08 06:50:47,072 | infra | DEBUG | pool-5-thread-25 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623160238.34_create__7200000_7250000_1623160238.36' complete. Loaded 50000 items 2021-06-08 06:50:47,072 | infra | DEBUG | pool-5-thread-25 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623160238.34_create__7250000_7300000_1623160238.36' complete. Loaded 50000 items 2021-06-08 06:50:47,072 | infra | DEBUG | pool-5-thread-25 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623160238.34_create__7300000_7350000_1623160238.36' complete. Loaded 50000 items 2021-06-08 06:50:47,072 | infra | DEBUG | pool-5-thread-25 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623160238.34_create__7350000_7400000_1623160238.36' complete. Loaded 50000 items 2021-06-08 06:50:47,072 | infra | DEBUG | pool-5-thread-25 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623160238.34_create__7400000_7450000_1623160238.36' complete. Loaded 50000 items 2021-06-08 06:50:47,072 | infra | DEBUG | pool-5-thread-25 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623160238.34_create__7450000_7500000_1623160238.36' complete. Loaded 50000 items 2021-06-08 06:50:47,072 | infra | DEBUG | pool-5-thread-25 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623160238.34_create__7500000_7550000_1623160238.36' complete. Loaded 50000 items 2021-06-08 06:50:47,072 | infra | DEBUG | pool-5-thread-25 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623160238.34_create__7550000_7600000_1623160238.36' complete. Loaded 50000 items 2021-06-08 06:50:47,072 | infra | DEBUG | pool-5-thread-25 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623160238.34_create__7600000_7650000_1623160238.36' complete. Loaded 50000 items 2021-06-08 06:50:47,072 | infra | DEBUG | pool-5-thread-25 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623160238.34_create__7650000_7700000_1623160238.36' complete. Loaded 50000 items 2021-06-08 06:50:47,072 | infra | DEBUG | pool-5-thread-25 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623160238.34_create__7700000_7750000_1623160238.36' complete. Loaded 50000 items 2021-06-08 06:50:47,072 | infra | DEBUG | pool-5-thread-25 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623160238.34_create__7750000_7800000_1623160238.36' complete. Loaded 50000 items 2021-06-08 06:50:47,072 | infra | DEBUG | pool-5-thread-25 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623160238.34_create__7800000_7850000_1623160238.36' complete. Loaded 50000 items 2021-06-08 06:50:47,072 | infra | DEBUG | pool-5-thread-25 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623160238.34_create__7850000_7900000_1623160238.36' complete. Loaded 50000 items 2021-06-08 06:50:47,073 | infra | DEBUG | pool-5-thread-25 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623160238.34_create__7900000_7950000_1623160238.36' complete. Loaded 50000 items 2021-06-08 06:50:47,073 | infra | DEBUG | pool-5-thread-25 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623160238.34_create__7950000_8000000_1623160238.36' complete. Loaded 50000 items 2021-06-08 06:50:47,073 | infra | INFO | pool-5-thread-25 | [task:complete_task:102] Thread 'LoadDocsGen_default__default__default_create_7000000_8000000_1623160238.34' completed 2021-06-08 06:50:47,075 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.198 with username: root, password: couchbase 2021-06-08 06:50:47,171 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 06:50:47,448 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: ls /etc/ 2021-06-08 06:50:47,533 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 06:50:47,535 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: ls /cygdrive/c/Windows 2021-06-08 06:50:47,553 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: uname -m 2021-06-08 06:50:47,568 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: cat /proc/cpuinfo 2021-06-08 06:50:47,585 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: df -Thl 2021-06-08 06:50:47,608 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: cat /proc/meminfo 2021-06-08 06:50:47,630 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: hostname 2021-06-08 06:50:47,657 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: hostname -d 2021-06-08 06:50:47,677 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:559] Starting StatsWaitTask for ep_data_write_failed on bucket default 2021-06-08 06:50:47,677 | infra | INFO | MainThread | [task_manager:add_new_task:23] Added new task: StatsWaitTask_default_ep_data_write_failed_1623160247.68 2021-06-08 06:50:47,677 | infra | INFO | pool-5-thread-14 | [task:start_task:87] Thread 'StatsWaitTask_default_ep_data_write_failed_1623160247.68' started 2021-06-08 06:50:47,677 | infra | DEBUG | pool-5-thread-14 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:50:47,680 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.196 with username: root, password: couchbase 2021-06-08 06:50:47,776 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 06:50:48,039 | test | DEBUG | pool-5-thread-14 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:50:48,039 | infra | DEBUG | pool-5-thread-14 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.198': '0'} for bucket 'default' 2021-06-08 06:50:48,039 | infra | DEBUG | pool-5-thread-14 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:50:48,040 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: ls /etc/ 2021-06-08 06:50:48,042 | infra | DEBUG | pool-5-thread-14 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:50:48,042 | infra | DEBUG | pool-5-thread-14 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:50:48,112 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 06:50:48,112 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: ls /cygdrive/c/Windows 2021-06-08 06:50:48,131 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: uname -m 2021-06-08 06:50:48,148 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: cat /proc/cpuinfo 2021-06-08 06:50:48,164 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: df -Thl 2021-06-08 06:50:48,187 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: cat /proc/meminfo 2021-06-08 06:50:48,201 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: hostname 2021-06-08 06:50:48,217 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: hostname -d 2021-06-08 06:50:48,234 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:559] Starting StatsWaitTask for ep_data_write_failed on bucket default 2021-06-08 06:50:48,234 | infra | INFO | MainThread | [task_manager:add_new_task:23] Added new task: StatsWaitTask_default_ep_data_write_failed_1623160248.23 2021-06-08 06:50:48,234 | infra | INFO | pool-5-thread-28 | [task:start_task:87] Thread 'StatsWaitTask_default_ep_data_write_failed_1623160248.23' started 2021-06-08 06:50:48,236 | infra | DEBUG | pool-5-thread-28 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:50:48,237 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.199 with username: root, password: couchbase 2021-06-08 06:50:48,328 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 06:50:48,604 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: ls /etc/ 2021-06-08 06:50:48,687 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 06:50:48,687 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: ls /cygdrive/c/Windows 2021-06-08 06:50:48,706 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: uname -m 2021-06-08 06:50:48,724 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: cat /proc/cpuinfo 2021-06-08 06:50:48,746 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: df -Thl 2021-06-08 06:50:48,769 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: cat /proc/meminfo 2021-06-08 06:50:48,796 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: hostname 2021-06-08 06:50:48,826 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: hostname -d 2021-06-08 06:50:48,845 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:559] Starting StatsWaitTask for ep_data_write_failed on bucket default 2021-06-08 06:50:48,845 | infra | INFO | MainThread | [task_manager:add_new_task:23] Added new task: StatsWaitTask_default_ep_data_write_failed_1623160248.85 2021-06-08 06:50:48,846 | infra | INFO | pool-5-thread-1 | [task:start_task:87] Thread 'StatsWaitTask_default_ep_data_write_failed_1623160248.85' started 2021-06-08 06:50:48,846 | infra | DEBUG | pool-5-thread-1 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:50:48,848 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.168 with username: root, password: couchbase 2021-06-08 06:50:48,867 | test | DEBUG | pool-5-thread-28 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:50:48,867 | infra | DEBUG | pool-5-thread-28 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.196': '0'} for bucket 'default' 2021-06-08 06:50:48,867 | infra | DEBUG | pool-5-thread-28 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:50:48,868 | infra | DEBUG | pool-5-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:50:48,868 | infra | DEBUG | pool-5-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:50:48,976 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 06:50:49,243 | test | DEBUG | pool-5-thread-1 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:50:49,243 | infra | DEBUG | pool-5-thread-1 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.199': '0'} for bucket 'default' 2021-06-08 06:50:49,243 | infra | DEBUG | pool-5-thread-1 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:50:49,246 | infra | DEBUG | pool-5-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:50:49,246 | infra | DEBUG | pool-5-thread-1 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:50:49,280 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: ls /etc/ 2021-06-08 06:50:49,377 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 06:50:49,377 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: ls /cygdrive/c/Windows 2021-06-08 06:50:49,411 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: uname -m 2021-06-08 06:50:49,433 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: cat /proc/cpuinfo 2021-06-08 06:50:49,451 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: df -Thl 2021-06-08 06:50:49,479 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: cat /proc/meminfo 2021-06-08 06:50:49,494 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: hostname 2021-06-08 06:50:49,517 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: hostname -d 2021-06-08 06:50:49,546 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:559] Starting StatsWaitTask for ep_data_write_failed on bucket default 2021-06-08 06:50:49,546 | infra | INFO | MainThread | [task_manager:add_new_task:23] Added new task: StatsWaitTask_default_ep_data_write_failed_1623160249.55 2021-06-08 06:50:49,546 | infra | INFO | pool-5-thread-30 | [task:start_task:87] Thread 'StatsWaitTask_default_ep_data_write_failed_1623160249.55' started 2021-06-08 06:50:49,546 | infra | DEBUG | MainThread | [task_manager:get_task_result:26] Getting task result for StatsWaitTask_default_ep_data_write_failed_1623160247.68 2021-06-08 06:50:49,546 | infra | DEBUG | pool-5-thread-30 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:50:49,854 | test | DEBUG | pool-5-thread-30 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:50:49,854 | infra | DEBUG | pool-5-thread-30 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.168': '0'} for bucket 'default' 2021-06-08 06:50:49,854 | infra | DEBUG | pool-5-thread-30 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:50:49,855 | infra | DEBUG | pool-5-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:50:49,855 | infra | DEBUG | pool-5-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:50:53,042 | infra | DEBUG | pool-5-thread-14 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:50:53,566 | test | DEBUG | pool-5-thread-14 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:50:53,566 | infra | DEBUG | pool-5-thread-14 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.198': '0'} for bucket 'default' 2021-06-08 06:50:53,566 | infra | DEBUG | pool-5-thread-14 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:50:53,569 | infra | DEBUG | pool-5-thread-14 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:50:53,569 | infra | DEBUG | pool-5-thread-14 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:50:53,869 | infra | DEBUG | pool-5-thread-28 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:50:54,246 | infra | DEBUG | pool-5-thread-1 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:50:54,634 | test | DEBUG | pool-5-thread-1 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:50:54,634 | infra | DEBUG | pool-5-thread-1 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.199': '0'} for bucket 'default' 2021-06-08 06:50:54,634 | infra | DEBUG | pool-5-thread-1 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:50:54,635 | infra | DEBUG | pool-5-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:50:54,635 | infra | DEBUG | pool-5-thread-1 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:50:54,644 | test | DEBUG | pool-5-thread-28 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:50:54,644 | infra | DEBUG | pool-5-thread-28 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.196': '0'} for bucket 'default' 2021-06-08 06:50:54,644 | infra | DEBUG | pool-5-thread-28 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:50:54,647 | infra | DEBUG | pool-5-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:50:54,647 | infra | DEBUG | pool-5-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:50:54,855 | infra | DEBUG | pool-5-thread-30 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:50:55,414 | test | DEBUG | pool-5-thread-30 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:50:55,414 | infra | DEBUG | pool-5-thread-30 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.168': '0'} for bucket 'default' 2021-06-08 06:50:55,414 | infra | DEBUG | pool-5-thread-30 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:50:55,415 | infra | DEBUG | pool-5-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:50:55,415 | infra | DEBUG | pool-5-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:50:58,569 | infra | DEBUG | pool-5-thread-14 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:50:59,627 | test | DEBUG | pool-5-thread-14 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:50:59,627 | infra | DEBUG | pool-5-thread-14 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.198': '0'} for bucket 'default' 2021-06-08 06:50:59,628 | infra | DEBUG | pool-5-thread-14 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:50:59,628 | infra | DEBUG | pool-5-thread-14 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:50:59,628 | infra | DEBUG | pool-5-thread-14 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:50:59,637 | infra | DEBUG | pool-5-thread-1 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:50:59,647 | infra | DEBUG | pool-5-thread-28 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:51:00,015 | test | DEBUG | pool-5-thread-1 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:51:00,015 | infra | DEBUG | pool-5-thread-1 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.199': '0'} for bucket 'default' 2021-06-08 06:51:00,015 | infra | DEBUG | pool-5-thread-1 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:51:00,016 | infra | DEBUG | pool-5-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:51:00,016 | infra | DEBUG | pool-5-thread-1 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:51:00,417 | infra | DEBUG | pool-5-thread-30 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:51:01,552 | test | DEBUG | pool-5-thread-30 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:51:01,552 | infra | DEBUG | pool-5-thread-30 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.168': '0'} for bucket 'default' 2021-06-08 06:51:01,552 | infra | DEBUG | pool-5-thread-30 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:51:01,555 | infra | DEBUG | pool-5-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:51:01,555 | infra | DEBUG | pool-5-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:51:01,585 | test | DEBUG | pool-5-thread-28 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:51:01,585 | infra | DEBUG | pool-5-thread-28 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.196': '0'} for bucket 'default' 2021-06-08 06:51:01,585 | infra | DEBUG | pool-5-thread-28 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:51:01,586 | infra | DEBUG | pool-5-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:51:01,586 | infra | DEBUG | pool-5-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:51:04,630 | infra | DEBUG | pool-5-thread-14 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:51:04,867 | test | DEBUG | pool-5-thread-14 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:51:04,867 | infra | DEBUG | pool-5-thread-14 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.198': '0'} for bucket 'default' 2021-06-08 06:51:04,867 | infra | DEBUG | pool-5-thread-14 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:51:04,868 | infra | DEBUG | pool-5-thread-14 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:51:04,868 | infra | DEBUG | pool-5-thread-14 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:51:05,016 | infra | DEBUG | pool-5-thread-1 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:51:06,009 | test | DEBUG | pool-5-thread-1 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:51:06,009 | infra | DEBUG | pool-5-thread-1 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.199': '0'} for bucket 'default' 2021-06-08 06:51:06,009 | infra | DEBUG | pool-5-thread-1 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:51:06,010 | infra | DEBUG | pool-5-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:51:06,010 | infra | DEBUG | pool-5-thread-1 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:51:06,555 | infra | DEBUG | pool-5-thread-30 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:51:06,586 | infra | DEBUG | pool-5-thread-28 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:51:06,956 | test | DEBUG | pool-5-thread-28 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:51:06,957 | infra | DEBUG | pool-5-thread-28 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.196': '0'} for bucket 'default' 2021-06-08 06:51:06,957 | infra | DEBUG | pool-5-thread-28 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:51:06,957 | infra | DEBUG | pool-5-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:51:06,957 | infra | DEBUG | pool-5-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:51:07,743 | test | DEBUG | pool-5-thread-30 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:51:07,743 | infra | DEBUG | pool-5-thread-30 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.168': '0'} for bucket 'default' 2021-06-08 06:51:07,743 | infra | DEBUG | pool-5-thread-30 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:51:07,744 | infra | DEBUG | pool-5-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:51:07,744 | infra | DEBUG | pool-5-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:51:09,868 | infra | DEBUG | pool-5-thread-14 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:51:11,010 | infra | DEBUG | pool-5-thread-1 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:51:11,043 | test | DEBUG | pool-5-thread-14 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:51:11,043 | infra | DEBUG | pool-5-thread-14 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.198': '0'} for bucket 'default' 2021-06-08 06:51:11,043 | infra | DEBUG | pool-5-thread-14 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:51:11,045 | infra | DEBUG | pool-5-thread-14 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:51:11,046 | infra | DEBUG | pool-5-thread-14 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:51:11,959 | infra | DEBUG | pool-5-thread-28 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:51:12,170 | test | DEBUG | pool-5-thread-28 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:51:12,170 | infra | DEBUG | pool-5-thread-28 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.196': '0'} for bucket 'default' 2021-06-08 06:51:12,170 | infra | DEBUG | pool-5-thread-28 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:51:12,171 | infra | DEBUG | pool-5-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:51:12,171 | infra | DEBUG | pool-5-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:51:12,265 | test | DEBUG | pool-5-thread-1 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:51:12,265 | infra | DEBUG | pool-5-thread-1 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.199': '0'} for bucket 'default' 2021-06-08 06:51:12,266 | infra | DEBUG | pool-5-thread-1 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:51:12,266 | infra | DEBUG | pool-5-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:51:12,266 | infra | DEBUG | pool-5-thread-1 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:51:12,744 | infra | DEBUG | pool-5-thread-30 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:51:13,999 | test | DEBUG | pool-5-thread-30 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:51:13,999 | infra | DEBUG | pool-5-thread-30 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.168': '0'} for bucket 'default' 2021-06-08 06:51:13,999 | infra | DEBUG | pool-5-thread-30 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:51:14,000 | infra | DEBUG | pool-5-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:51:14,000 | infra | DEBUG | pool-5-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:51:16,046 | infra | DEBUG | pool-5-thread-14 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:51:17,171 | infra | DEBUG | pool-5-thread-28 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:51:17,187 | test | DEBUG | pool-5-thread-14 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:51:17,188 | infra | DEBUG | pool-5-thread-14 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.198': '0'} for bucket 'default' 2021-06-08 06:51:17,188 | infra | DEBUG | pool-5-thread-14 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:51:17,190 | infra | DEBUG | pool-5-thread-14 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:51:17,190 | infra | DEBUG | pool-5-thread-14 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:51:17,267 | infra | DEBUG | pool-5-thread-1 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:51:17,740 | test | DEBUG | pool-5-thread-28 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:51:17,740 | infra | DEBUG | pool-5-thread-28 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.196': '0'} for bucket 'default' 2021-06-08 06:51:17,740 | infra | DEBUG | pool-5-thread-28 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:51:17,742 | infra | DEBUG | pool-5-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:51:17,742 | infra | DEBUG | pool-5-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:51:18,763 | test | DEBUG | pool-5-thread-1 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:51:18,765 | infra | DEBUG | pool-5-thread-1 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.199': '0'} for bucket 'default' 2021-06-08 06:51:18,765 | infra | DEBUG | pool-5-thread-1 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:51:18,766 | infra | DEBUG | pool-5-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:51:18,766 | infra | DEBUG | pool-5-thread-1 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:51:19,000 | infra | DEBUG | pool-5-thread-30 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:51:20,427 | test | DEBUG | pool-5-thread-30 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:51:20,427 | infra | DEBUG | pool-5-thread-30 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.168': '0'} for bucket 'default' 2021-06-08 06:51:20,427 | infra | DEBUG | pool-5-thread-30 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:51:20,430 | infra | DEBUG | pool-5-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:51:20,430 | infra | DEBUG | pool-5-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:51:22,190 | infra | DEBUG | pool-5-thread-14 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:51:22,742 | infra | DEBUG | pool-5-thread-28 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:51:23,766 | infra | DEBUG | pool-5-thread-1 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:51:24,145 | test | DEBUG | pool-5-thread-28 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:51:24,145 | infra | DEBUG | pool-5-thread-28 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.196': '0'} for bucket 'default' 2021-06-08 06:51:24,145 | infra | DEBUG | pool-5-thread-28 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:51:24,148 | infra | DEBUG | pool-5-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:51:24,148 | infra | DEBUG | pool-5-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:51:24,621 | test | DEBUG | pool-5-thread-14 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:51:24,621 | infra | DEBUG | pool-5-thread-14 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.198': '0'} for bucket 'default' 2021-06-08 06:51:24,621 | infra | DEBUG | pool-5-thread-14 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:51:24,624 | infra | DEBUG | pool-5-thread-14 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:51:24,624 | infra | DEBUG | pool-5-thread-14 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:51:25,042 | test | DEBUG | pool-5-thread-1 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:51:25,042 | infra | DEBUG | pool-5-thread-1 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.199': '0'} for bucket 'default' 2021-06-08 06:51:25,042 | infra | DEBUG | pool-5-thread-1 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:51:25,042 | infra | DEBUG | pool-5-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:51:25,043 | infra | DEBUG | pool-5-thread-1 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:51:25,430 | infra | DEBUG | pool-5-thread-30 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:51:27,095 | test | DEBUG | pool-5-thread-30 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:51:27,095 | infra | DEBUG | pool-5-thread-30 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.168': '0'} for bucket 'default' 2021-06-08 06:51:27,095 | infra | DEBUG | pool-5-thread-30 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:51:27,096 | infra | DEBUG | pool-5-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:51:27,096 | infra | DEBUG | pool-5-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:51:29,148 | infra | DEBUG | pool-5-thread-28 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:51:29,624 | infra | DEBUG | pool-5-thread-14 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:51:30,043 | infra | DEBUG | pool-5-thread-1 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:51:30,257 | test | DEBUG | pool-5-thread-28 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:51:30,257 | infra | DEBUG | pool-5-thread-28 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.196': '0'} for bucket 'default' 2021-06-08 06:51:30,257 | infra | DEBUG | pool-5-thread-28 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:51:30,259 | infra | DEBUG | pool-5-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:51:30,259 | infra | DEBUG | pool-5-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:51:30,959 | test | DEBUG | pool-5-thread-14 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:51:30,959 | infra | DEBUG | pool-5-thread-14 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.198': '0'} for bucket 'default' 2021-06-08 06:51:30,959 | infra | DEBUG | pool-5-thread-14 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:51:30,960 | infra | DEBUG | pool-5-thread-14 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:51:30,960 | infra | DEBUG | pool-5-thread-14 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:51:31,900 | test | DEBUG | pool-5-thread-1 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:51:31,901 | infra | DEBUG | pool-5-thread-1 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.199': '0'} for bucket 'default' 2021-06-08 06:51:31,901 | infra | DEBUG | pool-5-thread-1 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:51:31,901 | infra | DEBUG | pool-5-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:51:31,901 | infra | DEBUG | pool-5-thread-1 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:51:32,098 | infra | DEBUG | pool-5-thread-30 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:51:32,674 | test | DEBUG | pool-5-thread-30 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:51:32,674 | infra | DEBUG | pool-5-thread-30 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.168': '0'} for bucket 'default' 2021-06-08 06:51:32,674 | infra | DEBUG | pool-5-thread-30 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:51:32,676 | infra | DEBUG | pool-5-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:51:32,676 | infra | DEBUG | pool-5-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:51:35,260 | infra | DEBUG | pool-5-thread-28 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:51:35,960 | infra | DEBUG | pool-5-thread-14 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:51:36,523 | test | DEBUG | pool-5-thread-28 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:51:36,523 | infra | DEBUG | pool-5-thread-28 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.196': '0'} for bucket 'default' 2021-06-08 06:51:36,523 | infra | DEBUG | pool-5-thread-28 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:51:36,526 | infra | DEBUG | pool-5-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:51:36,526 | infra | DEBUG | pool-5-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:51:36,638 | test | DEBUG | pool-5-thread-14 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:51:36,638 | infra | DEBUG | pool-5-thread-14 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.198': '0'} for bucket 'default' 2021-06-08 06:51:36,638 | infra | DEBUG | pool-5-thread-14 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:51:36,640 | infra | DEBUG | pool-5-thread-14 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:51:36,640 | infra | DEBUG | pool-5-thread-14 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:51:36,903 | infra | DEBUG | pool-5-thread-1 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:51:37,664 | test | DEBUG | pool-5-thread-1 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:51:37,664 | infra | DEBUG | pool-5-thread-1 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.199': '0'} for bucket 'default' 2021-06-08 06:51:37,664 | infra | DEBUG | pool-5-thread-1 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:51:37,665 | infra | DEBUG | pool-5-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:51:37,665 | infra | DEBUG | pool-5-thread-1 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:51:37,676 | infra | DEBUG | pool-5-thread-30 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:51:39,256 | test | DEBUG | pool-5-thread-30 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:51:39,256 | infra | DEBUG | pool-5-thread-30 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.168': '0'} for bucket 'default' 2021-06-08 06:51:39,256 | infra | DEBUG | pool-5-thread-30 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:51:39,257 | infra | DEBUG | pool-5-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:51:39,257 | infra | DEBUG | pool-5-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:51:41,526 | infra | DEBUG | pool-5-thread-28 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:51:41,641 | infra | DEBUG | pool-5-thread-14 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:51:42,282 | test | DEBUG | pool-5-thread-14 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:51:42,282 | infra | DEBUG | pool-5-thread-14 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.198': '0'} for bucket 'default' 2021-06-08 06:51:42,282 | infra | DEBUG | pool-5-thread-14 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:51:42,285 | infra | DEBUG | pool-5-thread-14 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:51:42,285 | infra | DEBUG | pool-5-thread-14 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:51:42,391 | test | DEBUG | pool-5-thread-28 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:51:42,391 | infra | DEBUG | pool-5-thread-28 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.196': '0'} for bucket 'default' 2021-06-08 06:51:42,391 | infra | DEBUG | pool-5-thread-28 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:51:42,394 | infra | DEBUG | pool-5-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:51:42,394 | infra | DEBUG | pool-5-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:51:42,665 | infra | DEBUG | pool-5-thread-1 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:51:43,075 | test | DEBUG | pool-5-thread-1 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:51:43,075 | infra | DEBUG | pool-5-thread-1 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.199': '0'} for bucket 'default' 2021-06-08 06:51:43,075 | infra | DEBUG | pool-5-thread-1 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:51:43,078 | infra | DEBUG | pool-5-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:51:43,078 | infra | DEBUG | pool-5-thread-1 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:51:44,259 | infra | DEBUG | pool-5-thread-30 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:51:44,476 | test | DEBUG | pool-5-thread-30 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:51:44,476 | infra | DEBUG | pool-5-thread-30 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.168': '0'} for bucket 'default' 2021-06-08 06:51:44,476 | infra | DEBUG | pool-5-thread-30 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:51:44,477 | infra | DEBUG | pool-5-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:51:44,479 | infra | DEBUG | pool-5-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:51:47,285 | infra | DEBUG | pool-5-thread-14 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:51:47,394 | infra | DEBUG | pool-5-thread-28 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:51:47,490 | test | DEBUG | pool-5-thread-14 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:51:47,490 | infra | DEBUG | pool-5-thread-14 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.198': '0'} for bucket 'default' 2021-06-08 06:51:47,490 | infra | DEBUG | pool-5-thread-14 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:51:47,493 | infra | DEBUG | pool-5-thread-14 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:51:47,493 | infra | DEBUG | pool-5-thread-14 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:51:47,595 | test | DEBUG | pool-5-thread-28 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:51:47,595 | infra | DEBUG | pool-5-thread-28 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.196': '0'} for bucket 'default' 2021-06-08 06:51:47,595 | infra | DEBUG | pool-5-thread-28 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:51:47,596 | infra | DEBUG | pool-5-thread-28 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:51:47,596 | infra | DEBUG | pool-5-thread-28 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:51:48,078 | infra | DEBUG | pool-5-thread-1 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:51:48,278 | test | DEBUG | pool-5-thread-1 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:51:48,278 | infra | DEBUG | pool-5-thread-1 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.199': '0'} for bucket 'default' 2021-06-08 06:51:48,278 | infra | DEBUG | pool-5-thread-1 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:51:48,279 | infra | DEBUG | pool-5-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:51:48,279 | infra | DEBUG | pool-5-thread-1 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:51:49,479 | infra | DEBUG | pool-5-thread-30 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:51:49,678 | test | DEBUG | pool-5-thread-30 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:51:49,678 | infra | DEBUG | pool-5-thread-30 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.168': '0'} for bucket 'default' 2021-06-08 06:51:49,678 | infra | DEBUG | pool-5-thread-30 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:51:49,680 | infra | DEBUG | pool-5-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:51:49,680 | infra | DEBUG | pool-5-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:51:52,493 | infra | DEBUG | pool-5-thread-14 | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.198 2021-06-08 06:51:52,493 | infra | INFO | pool-5-thread-14 | [task:complete_task:102] Thread 'StatsWaitTask_default_ep_data_write_failed_1623160247.68' completed 2021-06-08 06:51:52,499 | infra | INFO | MainThread | [task_manager:add_new_task:23] Added new task: LoadDocsGen_default__default__default_create_8000000_9000000_1623160312.5 2021-06-08 06:51:52,499 | infra | DEBUG | MainThread | [task_manager:get_task_result:26] Getting task result for LoadDocsGen_default__default__default_create_8000000_9000000_1623160312.5 2021-06-08 06:51:52,499 | infra | INFO | pool-5-thread-4 | [task:start_task:87] Thread 'LoadDocsGen_default__default__default_create_8000000_9000000_1623160312.5' started 2021-06-08 06:51:52,525 | infra | INFO | pool-5-thread-4 | [task_manager:add_new_task:23] Added new task: PrintBucketStats_default_1623160312.51 2021-06-08 06:51:52,525 | infra | INFO | pool-5-thread-10 | [task:start_task:87] Thread 'PrintBucketStats_default_1623160312.51' started 2021-06-08 06:51:52,525 | infra | INFO | pool-5-thread-4 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623160312.5_create__8000000_8050000_1623160312.51 2021-06-08 06:51:52,525 | infra | INFO | pool-5-thread-4 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623160312.5_create__8050000_8100000_1623160312.51 2021-06-08 06:51:52,525 | infra | INFO | pool-5-thread-3 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623160312.5_create__8050000_8100000_1623160312.51' started 2021-06-08 06:51:52,525 | infra | INFO | pool-5-thread-13 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623160312.5_create__8000000_8050000_1623160312.51' started 2021-06-08 06:51:52,525 | infra | INFO | pool-5-thread-4 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623160312.5_create__8100000_8150000_1623160312.51 2021-06-08 06:51:52,525 | infra | INFO | pool-5-thread-29 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623160312.5_create__8100000_8150000_1623160312.51' started 2021-06-08 06:51:52,526 | infra | INFO | pool-5-thread-4 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623160312.5_create__8150000_8200000_1623160312.51 2021-06-08 06:51:52,526 | infra | INFO | pool-5-thread-4 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623160312.5_create__8200000_8250000_1623160312.51 2021-06-08 06:51:52,526 | infra | INFO | pool-5-thread-9 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623160312.5_create__8150000_8200000_1623160312.51' started 2021-06-08 06:51:52,526 | infra | INFO | pool-5-thread-4 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623160312.5_create__8250000_8300000_1623160312.51 2021-06-08 06:51:52,526 | infra | INFO | pool-5-thread-22 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623160312.5_create__8200000_8250000_1623160312.51' started 2021-06-08 06:51:52,526 | infra | INFO | pool-5-thread-12 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623160312.5_create__8250000_8300000_1623160312.51' started 2021-06-08 06:51:52,528 | infra | INFO | pool-5-thread-4 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623160312.5_create__8300000_8350000_1623160312.51 2021-06-08 06:51:52,528 | infra | INFO | pool-5-thread-19 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623160312.5_create__8300000_8350000_1623160312.51' started 2021-06-08 06:51:52,529 | infra | INFO | pool-5-thread-4 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623160312.5_create__8350000_8400000_1623160312.51 2021-06-08 06:51:52,529 | infra | INFO | pool-5-thread-8 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623160312.5_create__8350000_8400000_1623160312.51' started 2021-06-08 06:51:52,529 | infra | INFO | pool-5-thread-4 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623160312.5_create__8400000_8450000_1623160312.51 2021-06-08 06:51:52,529 | infra | INFO | pool-5-thread-2 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623160312.5_create__8400000_8450000_1623160312.51' started 2021-06-08 06:51:52,530 | infra | INFO | pool-5-thread-4 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623160312.5_create__8450000_8500000_1623160312.51 2021-06-08 06:51:52,530 | infra | INFO | pool-5-thread-26 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623160312.5_create__8450000_8500000_1623160312.51' started 2021-06-08 06:51:52,532 | infra | INFO | pool-5-thread-21 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623160312.5_create__8500000_8550000_1623160312.51' started 2021-06-08 06:51:52,533 | infra | INFO | pool-5-thread-4 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623160312.5_create__8500000_8550000_1623160312.51 2021-06-08 06:51:52,533 | infra | INFO | pool-5-thread-4 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623160312.5_create__8550000_8600000_1623160312.51 2021-06-08 06:51:52,533 | infra | INFO | pool-5-thread-4 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623160312.5_create__8600000_8650000_1623160312.51 2021-06-08 06:51:52,535 | infra | INFO | pool-5-thread-11 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623160312.5_create__8550000_8600000_1623160312.51' started 2021-06-08 06:51:52,535 | infra | INFO | pool-5-thread-18 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623160312.5_create__8600000_8650000_1623160312.51' started 2021-06-08 06:51:52,535 | infra | INFO | pool-5-thread-20 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623160312.5_create__8650000_8700000_1623160312.51' started 2021-06-08 06:51:52,533 | infra | INFO | pool-5-thread-4 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623160312.5_create__8650000_8700000_1623160312.51 2021-06-08 06:51:52,536 | infra | INFO | pool-5-thread-17 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623160312.5_create__8700000_8750000_1623160312.51' started 2021-06-08 06:51:52,536 | infra | INFO | pool-5-thread-4 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623160312.5_create__8700000_8750000_1623160312.51 2021-06-08 06:51:52,539 | infra | INFO | pool-5-thread-4 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623160312.5_create__8750000_8800000_1623160312.51 2021-06-08 06:51:52,539 | infra | INFO | pool-5-thread-15 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623160312.5_create__8750000_8800000_1623160312.51' started 2021-06-08 06:51:52,539 | infra | INFO | pool-5-thread-4 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623160312.5_create__8800000_8850000_1623160312.51 2021-06-08 06:51:52,539 | infra | INFO | pool-5-thread-6 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623160312.5_create__8800000_8850000_1623160312.51' started 2021-06-08 06:51:52,539 | infra | INFO | pool-5-thread-23 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623160312.5_create__8850000_8900000_1623160312.51' started 2021-06-08 06:51:52,546 | infra | INFO | pool-5-thread-4 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623160312.5_create__8850000_8900000_1623160312.51 2021-06-08 06:51:52,546 | infra | INFO | pool-5-thread-27 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623160312.5_create__8900000_8950000_1623160312.51' started 2021-06-08 06:51:52,546 | infra | INFO | pool-5-thread-4 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623160312.5_create__8900000_8950000_1623160312.51 2021-06-08 06:51:52,551 | infra | INFO | pool-5-thread-4 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623160312.5_create__8950000_9000000_1623160312.51 2021-06-08 06:51:52,551 | infra | INFO | pool-5-thread-5 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623160312.5_create__8950000_9000000_1623160312.51' started 2021-06-08 06:51:52,551 | infra | DEBUG | pool-5-thread-4 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623160312.5_create__8000000_8050000_1623160312.51 2021-06-08 06:51:52,596 | infra | DEBUG | pool-5-thread-28 | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.196 2021-06-08 06:51:52,601 | infra | INFO | pool-5-thread-28 | [task:complete_task:102] Thread 'StatsWaitTask_default_ep_data_write_failed_1623160248.23' completed 2021-06-08 06:51:53,003 | test | DEBUG | pool-5-thread-10 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 06:51:53,003 | test | DEBUG | pool-5-thread-10 | [task:print_ep_queue_size:3508] ep_queue_size for default: 0 2021-06-08 06:51:53,005 | infra | DEBUG | pool-5-thread-10 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 06:51:53,006 | infra | DEBUG | pool-5-thread-10 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 06:51:53,279 | infra | DEBUG | pool-5-thread-1 | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.199 2021-06-08 06:51:53,282 | infra | INFO | pool-5-thread-1 | [task:complete_task:102] Thread 'StatsWaitTask_default_ep_data_write_failed_1623160248.85' completed 2021-06-08 06:51:54,404 | test | DEBUG | pool-5-thread-10 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 06:51:54,404 | test | DEBUG | pool-5-thread-10 | [task:print_ep_queue_size:3508] ep_queue_size for default: 0 2021-06-08 06:51:54,407 | infra | DEBUG | pool-5-thread-10 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 06:51:54,407 | infra | DEBUG | pool-5-thread-10 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 06:51:54,680 | infra | DEBUG | pool-5-thread-30 | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.168 2021-06-08 06:51:54,684 | infra | INFO | pool-5-thread-30 | [task:complete_task:102] Thread 'StatsWaitTask_default_ep_data_write_failed_1623160249.55' completed 2021-06-08 06:51:55,861 | test | DEBUG | pool-5-thread-10 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 06:51:55,862 | test | DEBUG | pool-5-thread-10 | [task:print_ep_queue_size:3508] ep_queue_size for default: 0 2021-06-08 06:51:55,865 | infra | DEBUG | pool-5-thread-10 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 06:51:55,865 | infra | DEBUG | pool-5-thread-10 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 06:51:57,211 | test | DEBUG | pool-5-thread-10 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 06:51:57,213 | test | DEBUG | pool-5-thread-10 | [task:print_ep_queue_size:3508] ep_queue_size for default: 0 2021-06-08 06:51:57,214 | infra | DEBUG | pool-5-thread-10 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 06:51:57,214 | infra | DEBUG | pool-5-thread-10 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 06:51:58,630 | test | DEBUG | pool-5-thread-10 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 06:51:58,631 | test | DEBUG | pool-5-thread-10 | [task:print_ep_queue_size:3508] ep_queue_size for default: 0 2021-06-08 06:51:58,632 | infra | DEBUG | pool-5-thread-10 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 06:51:58,632 | infra | DEBUG | pool-5-thread-10 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 06:52:00,046 | test | DEBUG | pool-5-thread-10 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 06:52:00,046 | test | DEBUG | pool-5-thread-10 | [task:print_ep_queue_size:3508] ep_queue_size for default: 0 2021-06-08 06:52:00,051 | infra | DEBUG | pool-5-thread-10 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 06:52:00,051 | infra | DEBUG | pool-5-thread-10 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 06:52:00,102 | infra | INFO | pool-5-thread-3 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623160312.5_create__8050000_8100000_1623160312.51' completed 2021-06-08 06:52:00,480 | infra | INFO | pool-5-thread-5 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623160312.5_create__8950000_9000000_1623160312.51' completed 2021-06-08 06:52:00,576 | infra | INFO | pool-5-thread-22 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623160312.5_create__8200000_8250000_1623160312.51' completed 2021-06-08 06:52:00,611 | infra | INFO | pool-5-thread-6 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623160312.5_create__8800000_8850000_1623160312.51' completed 2021-06-08 06:52:00,644 | infra | INFO | pool-5-thread-11 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623160312.5_create__8550000_8600000_1623160312.51' completed 2021-06-08 06:52:00,703 | infra | INFO | pool-5-thread-20 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623160312.5_create__8650000_8700000_1623160312.51' completed 2021-06-08 06:52:00,706 | infra | INFO | pool-5-thread-2 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623160312.5_create__8400000_8450000_1623160312.51' completed 2021-06-08 06:52:00,736 | infra | INFO | pool-5-thread-27 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623160312.5_create__8900000_8950000_1623160312.51' completed 2021-06-08 06:52:00,760 | infra | INFO | pool-5-thread-12 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623160312.5_create__8250000_8300000_1623160312.51' completed 2021-06-08 06:52:00,762 | infra | INFO | pool-5-thread-19 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623160312.5_create__8300000_8350000_1623160312.51' completed 2021-06-08 06:52:00,808 | infra | INFO | pool-5-thread-9 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623160312.5_create__8150000_8200000_1623160312.51' completed 2021-06-08 06:52:00,819 | infra | INFO | pool-5-thread-13 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623160312.5_create__8000000_8050000_1623160312.51' completed 2021-06-08 06:52:00,819 | infra | DEBUG | pool-5-thread-4 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623160312.5_create__8000000_8050000_1623160312.51 are 50000 2021-06-08 06:52:00,819 | infra | DEBUG | pool-5-thread-4 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623160312.5_create__8050000_8100000_1623160312.51 2021-06-08 06:52:00,819 | infra | DEBUG | pool-5-thread-4 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623160312.5_create__8050000_8100000_1623160312.51 are 50000 2021-06-08 06:52:00,819 | infra | DEBUG | pool-5-thread-4 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623160312.5_create__8100000_8150000_1623160312.51 2021-06-08 06:52:00,868 | infra | INFO | pool-5-thread-23 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623160312.5_create__8850000_8900000_1623160312.51' completed 2021-06-08 06:52:00,872 | infra | INFO | pool-5-thread-17 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623160312.5_create__8700000_8750000_1623160312.51' completed 2021-06-08 06:52:00,875 | infra | INFO | pool-5-thread-8 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623160312.5_create__8350000_8400000_1623160312.51' completed 2021-06-08 06:52:00,881 | infra | INFO | pool-5-thread-15 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623160312.5_create__8750000_8800000_1623160312.51' completed 2021-06-08 06:52:00,887 | infra | INFO | pool-5-thread-29 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623160312.5_create__8100000_8150000_1623160312.51' completed 2021-06-08 06:52:00,887 | infra | DEBUG | pool-5-thread-4 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623160312.5_create__8100000_8150000_1623160312.51 are 50000 2021-06-08 06:52:00,888 | infra | DEBUG | pool-5-thread-4 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623160312.5_create__8150000_8200000_1623160312.51 2021-06-08 06:52:00,888 | infra | INFO | pool-5-thread-18 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623160312.5_create__8600000_8650000_1623160312.51' completed 2021-06-08 06:52:00,888 | infra | DEBUG | pool-5-thread-4 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623160312.5_create__8150000_8200000_1623160312.51 are 50000 2021-06-08 06:52:00,888 | infra | DEBUG | pool-5-thread-4 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623160312.5_create__8200000_8250000_1623160312.51 2021-06-08 06:52:00,888 | infra | DEBUG | pool-5-thread-4 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623160312.5_create__8200000_8250000_1623160312.51 are 50000 2021-06-08 06:52:00,888 | infra | DEBUG | pool-5-thread-4 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623160312.5_create__8250000_8300000_1623160312.51 2021-06-08 06:52:00,888 | infra | DEBUG | pool-5-thread-4 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623160312.5_create__8250000_8300000_1623160312.51 are 50000 2021-06-08 06:52:00,888 | infra | DEBUG | pool-5-thread-4 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623160312.5_create__8300000_8350000_1623160312.51 2021-06-08 06:52:00,888 | infra | DEBUG | pool-5-thread-4 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623160312.5_create__8300000_8350000_1623160312.51 are 50000 2021-06-08 06:52:00,888 | infra | DEBUG | pool-5-thread-4 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623160312.5_create__8350000_8400000_1623160312.51 2021-06-08 06:52:00,888 | infra | DEBUG | pool-5-thread-4 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623160312.5_create__8350000_8400000_1623160312.51 are 50000 2021-06-08 06:52:00,888 | infra | DEBUG | pool-5-thread-4 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623160312.5_create__8400000_8450000_1623160312.51 2021-06-08 06:52:00,888 | infra | DEBUG | pool-5-thread-4 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623160312.5_create__8400000_8450000_1623160312.51 are 50000 2021-06-08 06:52:00,888 | infra | DEBUG | pool-5-thread-4 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623160312.5_create__8450000_8500000_1623160312.51 2021-06-08 06:52:00,891 | infra | INFO | pool-5-thread-21 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623160312.5_create__8500000_8550000_1623160312.51' completed 2021-06-08 06:52:00,960 | infra | INFO | pool-5-thread-26 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623160312.5_create__8450000_8500000_1623160312.51' completed 2021-06-08 06:52:00,961 | infra | DEBUG | pool-5-thread-4 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623160312.5_create__8450000_8500000_1623160312.51 are 50000 2021-06-08 06:52:00,961 | infra | DEBUG | pool-5-thread-4 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623160312.5_create__8500000_8550000_1623160312.51 2021-06-08 06:52:00,961 | infra | DEBUG | pool-5-thread-4 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623160312.5_create__8500000_8550000_1623160312.51 are 50000 2021-06-08 06:52:00,961 | infra | DEBUG | pool-5-thread-4 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623160312.5_create__8550000_8600000_1623160312.51 2021-06-08 06:52:00,961 | infra | DEBUG | pool-5-thread-4 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623160312.5_create__8550000_8600000_1623160312.51 are 50000 2021-06-08 06:52:00,961 | infra | DEBUG | pool-5-thread-4 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623160312.5_create__8600000_8650000_1623160312.51 2021-06-08 06:52:00,961 | infra | DEBUG | pool-5-thread-4 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623160312.5_create__8600000_8650000_1623160312.51 are 50000 2021-06-08 06:52:00,961 | infra | DEBUG | pool-5-thread-4 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623160312.5_create__8650000_8700000_1623160312.51 2021-06-08 06:52:00,961 | infra | DEBUG | pool-5-thread-4 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623160312.5_create__8650000_8700000_1623160312.51 are 50000 2021-06-08 06:52:00,963 | infra | DEBUG | pool-5-thread-4 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623160312.5_create__8700000_8750000_1623160312.51 2021-06-08 06:52:00,963 | infra | DEBUG | pool-5-thread-4 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623160312.5_create__8700000_8750000_1623160312.51 are 50000 2021-06-08 06:52:00,963 | infra | DEBUG | pool-5-thread-4 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623160312.5_create__8750000_8800000_1623160312.51 2021-06-08 06:52:00,963 | infra | DEBUG | pool-5-thread-4 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623160312.5_create__8750000_8800000_1623160312.51 are 50000 2021-06-08 06:52:00,963 | infra | DEBUG | pool-5-thread-4 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623160312.5_create__8800000_8850000_1623160312.51 2021-06-08 06:52:00,963 | infra | DEBUG | pool-5-thread-4 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623160312.5_create__8800000_8850000_1623160312.51 are 50000 2021-06-08 06:52:00,963 | infra | DEBUG | pool-5-thread-4 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623160312.5_create__8850000_8900000_1623160312.51 2021-06-08 06:52:00,963 | infra | DEBUG | pool-5-thread-4 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623160312.5_create__8850000_8900000_1623160312.51 are 50000 2021-06-08 06:52:00,963 | infra | DEBUG | pool-5-thread-4 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623160312.5_create__8900000_8950000_1623160312.51 2021-06-08 06:52:00,963 | infra | DEBUG | pool-5-thread-4 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623160312.5_create__8900000_8950000_1623160312.51 are 50000 2021-06-08 06:52:00,963 | infra | DEBUG | pool-5-thread-4 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623160312.5_create__8950000_9000000_1623160312.51 2021-06-08 06:52:00,963 | infra | DEBUG | pool-5-thread-4 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623160312.5_create__8950000_9000000_1623160312.51 are 50000 2021-06-08 06:52:00,963 | infra | DEBUG | pool-5-thread-4 | [task_manager:get_task_result:26] Getting task result for PrintBucketStats_default_1623160312.51 2021-06-08 06:52:01,052 | infra | INFO | pool-5-thread-10 | [task:complete_task:102] Thread 'PrintBucketStats_default_1623160312.51' completed 2021-06-08 06:52:01,052 | infra | DEBUG | pool-5-thread-4 | [task:call:1663] ========= Tasks in loadgen pool======= 2021-06-08 06:52:01,052 | infra | WARNING | pool-5-thread-4 | [task_manager:print_tasks_in_pool:83] Task 'LoadDocsGen_default__default__default_create_8000000_9000000_1623160312.5' not completed 2021-06-08 06:52:01,052 | infra | DEBUG | pool-5-thread-4 | [task:call:1665] ====================================== 2021-06-08 06:52:01,052 | infra | DEBUG | pool-5-thread-4 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623160312.5_create__8000000_8050000_1623160312.51' complete. Loaded 50000 items 2021-06-08 06:52:01,052 | infra | DEBUG | pool-5-thread-4 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623160312.5_create__8050000_8100000_1623160312.51' complete. Loaded 50000 items 2021-06-08 06:52:01,052 | infra | DEBUG | pool-5-thread-4 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623160312.5_create__8100000_8150000_1623160312.51' complete. Loaded 50000 items 2021-06-08 06:52:01,052 | infra | DEBUG | pool-5-thread-4 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623160312.5_create__8150000_8200000_1623160312.51' complete. Loaded 50000 items 2021-06-08 06:52:01,052 | infra | DEBUG | pool-5-thread-4 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623160312.5_create__8200000_8250000_1623160312.51' complete. Loaded 50000 items 2021-06-08 06:52:01,052 | infra | DEBUG | pool-5-thread-4 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623160312.5_create__8250000_8300000_1623160312.51' complete. Loaded 50000 items 2021-06-08 06:52:01,052 | infra | DEBUG | pool-5-thread-4 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623160312.5_create__8300000_8350000_1623160312.51' complete. Loaded 50000 items 2021-06-08 06:52:01,052 | infra | DEBUG | pool-5-thread-4 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623160312.5_create__8350000_8400000_1623160312.51' complete. Loaded 50000 items 2021-06-08 06:52:01,052 | infra | DEBUG | pool-5-thread-4 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623160312.5_create__8400000_8450000_1623160312.51' complete. Loaded 50000 items 2021-06-08 06:52:01,052 | infra | DEBUG | pool-5-thread-4 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623160312.5_create__8450000_8500000_1623160312.51' complete. Loaded 50000 items 2021-06-08 06:52:01,053 | infra | DEBUG | pool-5-thread-4 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623160312.5_create__8500000_8550000_1623160312.51' complete. Loaded 50000 items 2021-06-08 06:52:01,053 | infra | DEBUG | pool-5-thread-4 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623160312.5_create__8550000_8600000_1623160312.51' complete. Loaded 50000 items 2021-06-08 06:52:01,053 | infra | DEBUG | pool-5-thread-4 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623160312.5_create__8600000_8650000_1623160312.51' complete. Loaded 50000 items 2021-06-08 06:52:01,053 | infra | DEBUG | pool-5-thread-4 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623160312.5_create__8650000_8700000_1623160312.51' complete. Loaded 50000 items 2021-06-08 06:52:01,053 | infra | DEBUG | pool-5-thread-4 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623160312.5_create__8700000_8750000_1623160312.51' complete. Loaded 50000 items 2021-06-08 06:52:01,053 | infra | DEBUG | pool-5-thread-4 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623160312.5_create__8750000_8800000_1623160312.51' complete. Loaded 50000 items 2021-06-08 06:52:01,053 | infra | DEBUG | pool-5-thread-4 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623160312.5_create__8800000_8850000_1623160312.51' complete. Loaded 50000 items 2021-06-08 06:52:01,053 | infra | DEBUG | pool-5-thread-4 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623160312.5_create__8850000_8900000_1623160312.51' complete. Loaded 50000 items 2021-06-08 06:52:01,053 | infra | DEBUG | pool-5-thread-4 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623160312.5_create__8900000_8950000_1623160312.51' complete. Loaded 50000 items 2021-06-08 06:52:01,053 | infra | DEBUG | pool-5-thread-4 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623160312.5_create__8950000_9000000_1623160312.51' complete. Loaded 50000 items 2021-06-08 06:52:01,055 | infra | INFO | pool-5-thread-4 | [task:complete_task:102] Thread 'LoadDocsGen_default__default__default_create_8000000_9000000_1623160312.5' completed 2021-06-08 06:52:01,056 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.198 with username: root, password: couchbase 2021-06-08 06:52:01,140 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 06:52:01,418 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: ls /etc/ 2021-06-08 06:52:01,496 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 06:52:01,496 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: ls /cygdrive/c/Windows 2021-06-08 06:52:01,513 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: uname -m 2021-06-08 06:52:01,529 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: cat /proc/cpuinfo 2021-06-08 06:52:01,552 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: df -Thl 2021-06-08 06:52:01,571 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: cat /proc/meminfo 2021-06-08 06:52:01,589 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: hostname 2021-06-08 06:52:01,609 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: hostname -d 2021-06-08 06:52:01,628 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:559] Starting StatsWaitTask for ep_data_write_failed on bucket default 2021-06-08 06:52:01,628 | infra | INFO | MainThread | [task_manager:add_new_task:23] Added new task: StatsWaitTask_default_ep_data_write_failed_1623160321.63 2021-06-08 06:52:01,628 | infra | INFO | pool-5-thread-7 | [task:start_task:87] Thread 'StatsWaitTask_default_ep_data_write_failed_1623160321.63' started 2021-06-08 06:52:01,628 | infra | DEBUG | pool-5-thread-7 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:52:01,631 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.196 with username: root, password: couchbase 2021-06-08 06:52:01,736 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 06:52:01,990 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: ls /etc/ 2021-06-08 06:52:02,065 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 06:52:02,065 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: ls /cygdrive/c/Windows 2021-06-08 06:52:02,081 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: uname -m 2021-06-08 06:52:02,095 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: cat /proc/cpuinfo 2021-06-08 06:52:02,109 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: df -Thl 2021-06-08 06:52:02,132 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: cat /proc/meminfo 2021-06-08 06:52:02,148 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: hostname 2021-06-08 06:52:02,164 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: hostname -d 2021-06-08 06:52:02,180 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:559] Starting StatsWaitTask for ep_data_write_failed on bucket default 2021-06-08 06:52:02,180 | infra | INFO | MainThread | [task_manager:add_new_task:23] Added new task: StatsWaitTask_default_ep_data_write_failed_1623160322.18 2021-06-08 06:52:02,180 | infra | INFO | pool-5-thread-16 | [task:start_task:87] Thread 'StatsWaitTask_default_ep_data_write_failed_1623160322.18' started 2021-06-08 06:52:02,181 | infra | DEBUG | pool-5-thread-16 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:52:02,184 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.199 with username: root, password: couchbase 2021-06-08 06:52:02,200 | test | DEBUG | pool-5-thread-7 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:52:02,200 | infra | DEBUG | pool-5-thread-7 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.198': '0'} for bucket 'default' 2021-06-08 06:52:02,200 | infra | DEBUG | pool-5-thread-7 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:52:02,203 | infra | DEBUG | pool-5-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:52:02,203 | infra | DEBUG | pool-5-thread-7 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:52:02,279 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 06:52:02,539 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: ls /etc/ 2021-06-08 06:52:02,618 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 06:52:02,619 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: ls /cygdrive/c/Windows 2021-06-08 06:52:02,635 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: uname -m 2021-06-08 06:52:02,651 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: cat /proc/cpuinfo 2021-06-08 06:52:02,670 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: df -Thl 2021-06-08 06:52:02,691 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: cat /proc/meminfo 2021-06-08 06:52:02,709 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: hostname 2021-06-08 06:52:02,724 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: hostname -d 2021-06-08 06:52:02,747 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:559] Starting StatsWaitTask for ep_data_write_failed on bucket default 2021-06-08 06:52:02,747 | infra | INFO | MainThread | [task_manager:add_new_task:23] Added new task: StatsWaitTask_default_ep_data_write_failed_1623160322.75 2021-06-08 06:52:02,747 | infra | INFO | pool-5-thread-24 | [task:start_task:87] Thread 'StatsWaitTask_default_ep_data_write_failed_1623160322.75' started 2021-06-08 06:52:02,749 | infra | DEBUG | pool-5-thread-24 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:52:02,750 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.168 with username: root, password: couchbase 2021-06-08 06:52:02,826 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 06:52:02,940 | test | DEBUG | pool-5-thread-16 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:52:02,940 | infra | DEBUG | pool-5-thread-16 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.196': '0'} for bucket 'default' 2021-06-08 06:52:02,940 | infra | DEBUG | pool-5-thread-16 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:52:02,941 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:52:02,941 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:52:03,079 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: ls /etc/ 2021-06-08 06:52:03,154 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 06:52:03,154 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: ls /cygdrive/c/Windows 2021-06-08 06:52:03,173 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: uname -m 2021-06-08 06:52:03,194 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: cat /proc/cpuinfo 2021-06-08 06:52:03,226 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: df -Thl 2021-06-08 06:52:03,265 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: cat /proc/meminfo 2021-06-08 06:52:03,286 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: hostname 2021-06-08 06:52:03,305 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: hostname -d 2021-06-08 06:52:03,321 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:559] Starting StatsWaitTask for ep_data_write_failed on bucket default 2021-06-08 06:52:03,321 | infra | INFO | MainThread | [task_manager:add_new_task:23] Added new task: StatsWaitTask_default_ep_data_write_failed_1623160323.32 2021-06-08 06:52:03,321 | infra | INFO | pool-5-thread-25 | [task:start_task:87] Thread 'StatsWaitTask_default_ep_data_write_failed_1623160323.32' started 2021-06-08 06:52:03,321 | infra | DEBUG | MainThread | [task_manager:get_task_result:26] Getting task result for StatsWaitTask_default_ep_data_write_failed_1623160321.63 2021-06-08 06:52:03,322 | infra | DEBUG | pool-5-thread-25 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:52:03,453 | test | DEBUG | pool-5-thread-24 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:52:03,453 | infra | DEBUG | pool-5-thread-24 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.199': '0'} for bucket 'default' 2021-06-08 06:52:03,453 | infra | DEBUG | pool-5-thread-24 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:52:03,454 | infra | DEBUG | pool-5-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:52:03,454 | infra | DEBUG | pool-5-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:52:03,845 | test | DEBUG | pool-5-thread-25 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:52:03,845 | infra | DEBUG | pool-5-thread-25 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.168': '0'} for bucket 'default' 2021-06-08 06:52:03,845 | infra | DEBUG | pool-5-thread-25 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:52:03,846 | infra | DEBUG | pool-5-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:52:03,846 | infra | DEBUG | pool-5-thread-25 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:52:07,203 | infra | DEBUG | pool-5-thread-7 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:52:07,765 | test | DEBUG | pool-5-thread-7 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:52:07,765 | infra | DEBUG | pool-5-thread-7 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.198': '0'} for bucket 'default' 2021-06-08 06:52:07,765 | infra | DEBUG | pool-5-thread-7 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:52:07,766 | infra | DEBUG | pool-5-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:52:07,766 | infra | DEBUG | pool-5-thread-7 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:52:07,941 | infra | DEBUG | pool-5-thread-16 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:52:08,454 | infra | DEBUG | pool-5-thread-24 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:52:08,483 | test | DEBUG | pool-5-thread-16 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:52:08,483 | infra | DEBUG | pool-5-thread-16 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.196': '0'} for bucket 'default' 2021-06-08 06:52:08,483 | infra | DEBUG | pool-5-thread-16 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:52:08,484 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:52:08,484 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:52:08,848 | infra | DEBUG | pool-5-thread-25 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:52:08,967 | test | DEBUG | pool-5-thread-24 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:52:08,967 | infra | DEBUG | pool-5-thread-24 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.199': '0'} for bucket 'default' 2021-06-08 06:52:08,967 | infra | DEBUG | pool-5-thread-24 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:52:08,969 | infra | DEBUG | pool-5-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:52:08,969 | infra | DEBUG | pool-5-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:52:09,996 | test | DEBUG | pool-5-thread-25 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:52:09,996 | infra | DEBUG | pool-5-thread-25 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.168': '0'} for bucket 'default' 2021-06-08 06:52:09,996 | infra | DEBUG | pool-5-thread-25 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:52:09,997 | infra | DEBUG | pool-5-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:52:09,997 | infra | DEBUG | pool-5-thread-25 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:52:12,766 | infra | DEBUG | pool-5-thread-7 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:52:13,484 | infra | DEBUG | pool-5-thread-16 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:52:13,970 | infra | DEBUG | pool-5-thread-24 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:52:14,868 | test | DEBUG | pool-5-thread-16 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:52:14,868 | infra | DEBUG | pool-5-thread-16 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.196': '0'} for bucket 'default' 2021-06-08 06:52:14,868 | infra | DEBUG | pool-5-thread-16 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:52:14,869 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:52:14,871 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:52:14,997 | infra | DEBUG | pool-5-thread-25 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:52:15,148 | test | DEBUG | pool-5-thread-7 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:52:15,148 | infra | DEBUG | pool-5-thread-7 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.198': '0'} for bucket 'default' 2021-06-08 06:52:15,148 | infra | DEBUG | pool-5-thread-7 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:52:15,151 | infra | DEBUG | pool-5-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:52:15,151 | infra | DEBUG | pool-5-thread-7 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:52:15,308 | test | DEBUG | pool-5-thread-24 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:52:15,309 | infra | DEBUG | pool-5-thread-24 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.199': '0'} for bucket 'default' 2021-06-08 06:52:15,309 | infra | DEBUG | pool-5-thread-24 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:52:15,309 | infra | DEBUG | pool-5-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:52:15,309 | infra | DEBUG | pool-5-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:52:15,562 | test | DEBUG | pool-5-thread-25 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:52:15,562 | infra | DEBUG | pool-5-thread-25 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.168': '0'} for bucket 'default' 2021-06-08 06:52:15,562 | infra | DEBUG | pool-5-thread-25 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:52:15,563 | infra | DEBUG | pool-5-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:52:15,563 | infra | DEBUG | pool-5-thread-25 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:52:19,871 | infra | DEBUG | pool-5-thread-16 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:52:20,151 | infra | DEBUG | pool-5-thread-7 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:52:20,311 | infra | DEBUG | pool-5-thread-24 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:52:20,563 | infra | DEBUG | pool-5-thread-25 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:52:20,624 | test | DEBUG | pool-5-thread-16 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:52:20,624 | infra | DEBUG | pool-5-thread-16 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.196': '0'} for bucket 'default' 2021-06-08 06:52:20,624 | infra | DEBUG | pool-5-thread-16 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:52:20,625 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:52:20,625 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:52:21,220 | test | DEBUG | pool-5-thread-25 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:52:21,220 | infra | DEBUG | pool-5-thread-25 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.168': '0'} for bucket 'default' 2021-06-08 06:52:21,220 | infra | DEBUG | pool-5-thread-25 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:52:21,223 | infra | DEBUG | pool-5-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:52:21,223 | infra | DEBUG | pool-5-thread-25 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:52:21,242 | test | DEBUG | pool-5-thread-24 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:52:21,242 | infra | DEBUG | pool-5-thread-24 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.199': '0'} for bucket 'default' 2021-06-08 06:52:21,242 | infra | DEBUG | pool-5-thread-24 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:52:21,243 | infra | DEBUG | pool-5-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:52:21,243 | infra | DEBUG | pool-5-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:52:22,134 | test | DEBUG | pool-5-thread-7 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:52:22,135 | infra | DEBUG | pool-5-thread-7 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.198': '0'} for bucket 'default' 2021-06-08 06:52:22,135 | infra | DEBUG | pool-5-thread-7 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:52:22,137 | infra | DEBUG | pool-5-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:52:22,137 | infra | DEBUG | pool-5-thread-7 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:52:25,627 | infra | DEBUG | pool-5-thread-16 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:52:25,842 | test | DEBUG | pool-5-thread-16 | [task:_compare:2649] Comparing 27 > 0 2021-06-08 06:52:25,842 | test | DEBUG | pool-5-thread-16 | [task:_get_all_stats_and_compare:2601] Ready: ep_data_write_failed 27 > 0. Received: {'172.23.106.196': '27'} for bucket 'default' 2021-06-08 06:52:25,842 | infra | INFO | pool-5-thread-16 | [task:complete_task:102] Thread 'StatsWaitTask_default_ep_data_write_failed_1623160322.18' completed 2021-06-08 06:52:26,223 | infra | DEBUG | pool-5-thread-25 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:52:26,243 | infra | DEBUG | pool-5-thread-24 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:52:26,799 | test | DEBUG | pool-5-thread-25 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:52:26,799 | infra | DEBUG | pool-5-thread-25 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.168': '0'} for bucket 'default' 2021-06-08 06:52:26,799 | infra | DEBUG | pool-5-thread-25 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:52:26,801 | infra | DEBUG | pool-5-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:52:26,801 | infra | DEBUG | pool-5-thread-25 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:52:26,966 | test | DEBUG | pool-5-thread-24 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:52:26,966 | infra | DEBUG | pool-5-thread-24 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.199': '0'} for bucket 'default' 2021-06-08 06:52:26,966 | infra | DEBUG | pool-5-thread-24 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:52:26,967 | infra | DEBUG | pool-5-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:52:26,967 | infra | DEBUG | pool-5-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:52:27,138 | infra | DEBUG | pool-5-thread-7 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:52:28,664 | test | DEBUG | pool-5-thread-7 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:52:28,664 | infra | DEBUG | pool-5-thread-7 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.198': '0'} for bucket 'default' 2021-06-08 06:52:28,664 | infra | DEBUG | pool-5-thread-7 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:52:28,665 | infra | DEBUG | pool-5-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:52:28,665 | infra | DEBUG | pool-5-thread-7 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:52:31,802 | infra | DEBUG | pool-5-thread-25 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:52:31,967 | infra | DEBUG | pool-5-thread-24 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:52:32,019 | test | DEBUG | pool-5-thread-25 | [task:_compare:2649] Comparing 7 > 0 2021-06-08 06:52:32,019 | test | DEBUG | pool-5-thread-25 | [task:_get_all_stats_and_compare:2601] Ready: ep_data_write_failed 7 > 0. Received: {'172.23.106.168': '7'} for bucket 'default' 2021-06-08 06:52:32,019 | infra | INFO | pool-5-thread-25 | [task:complete_task:102] Thread 'StatsWaitTask_default_ep_data_write_failed_1623160323.32' completed 2021-06-08 06:52:32,288 | test | DEBUG | pool-5-thread-24 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:52:32,288 | infra | DEBUG | pool-5-thread-24 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.199': '0'} for bucket 'default' 2021-06-08 06:52:32,288 | infra | DEBUG | pool-5-thread-24 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:52:32,289 | infra | DEBUG | pool-5-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:52:32,289 | infra | DEBUG | pool-5-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:52:33,665 | infra | DEBUG | pool-5-thread-7 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:52:34,107 | test | DEBUG | pool-5-thread-7 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:52:34,107 | infra | DEBUG | pool-5-thread-7 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.198': '0'} for bucket 'default' 2021-06-08 06:52:34,108 | infra | DEBUG | pool-5-thread-7 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:52:34,108 | infra | DEBUG | pool-5-thread-7 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:52:34,108 | infra | DEBUG | pool-5-thread-7 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:52:37,289 | infra | DEBUG | pool-5-thread-24 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:52:37,575 | test | DEBUG | pool-5-thread-24 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 06:52:37,575 | infra | DEBUG | pool-5-thread-24 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.199': '0'} for bucket 'default' 2021-06-08 06:52:37,575 | infra | DEBUG | pool-5-thread-24 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 06:52:37,576 | infra | DEBUG | pool-5-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 06:52:37,578 | infra | DEBUG | pool-5-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:52:39,109 | infra | DEBUG | pool-5-thread-7 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:52:39,331 | test | DEBUG | pool-5-thread-7 | [task:_compare:2649] Comparing 323 > 0 2021-06-08 06:52:39,332 | test | DEBUG | pool-5-thread-7 | [task:_get_all_stats_and_compare:2601] Ready: ep_data_write_failed 323 > 0. Received: {'172.23.106.198': '323'} for bucket 'default' 2021-06-08 06:52:39,332 | infra | INFO | pool-5-thread-7 | [task:complete_task:102] Thread 'StatsWaitTask_default_ep_data_write_failed_1623160321.63' completed 2021-06-08 06:52:39,332 | infra | DEBUG | MainThread | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.198 2021-06-08 06:52:39,332 | infra | DEBUG | MainThread | [task_manager:get_task_result:26] Getting task result for StatsWaitTask_default_ep_data_write_failed_1623160322.18 2021-06-08 06:52:39,332 | infra | DEBUG | MainThread | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.196 2021-06-08 06:52:39,332 | infra | DEBUG | MainThread | [task_manager:get_task_result:26] Getting task result for StatsWaitTask_default_ep_data_write_failed_1623160322.75 2021-06-08 06:52:42,578 | infra | DEBUG | pool-5-thread-24 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 06:52:42,801 | test | DEBUG | pool-5-thread-24 | [task:_compare:2649] Comparing 978 > 0 2021-06-08 06:52:42,801 | test | DEBUG | pool-5-thread-24 | [task:_get_all_stats_and_compare:2601] Ready: ep_data_write_failed 978 > 0. Received: {'172.23.106.199': '978'} for bucket 'default' 2021-06-08 06:52:42,801 | infra | INFO | pool-5-thread-24 | [task:complete_task:102] Thread 'StatsWaitTask_default_ep_data_write_failed_1623160322.75' completed 2021-06-08 06:52:42,802 | infra | DEBUG | MainThread | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.199 2021-06-08 06:52:42,802 | infra | DEBUG | MainThread | [task_manager:get_task_result:26] Getting task result for StatsWaitTask_default_ep_data_write_failed_1623160323.32 2021-06-08 06:52:42,802 | infra | DEBUG | MainThread | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.168 2021-06-08 06:52:42,802 | test | DEBUG | MainThread | [sdk_client3:_SDKClient__create_conn:237] Creating SDK connection for 'default' 2021-06-08 06:52:43,140 | test | DEBUG | MainThread | [sdk_client3:_SDKClient__create_conn:237] Creating SDK connection for 'default' 2021-06-08 06:52:43,453 | test | DEBUG | MainThread | [sdk_client3:_SDKClient__create_conn:237] Creating SDK connection for 'default' 2021-06-08 06:52:43,766 | test | DEBUG | MainThread | [sdk_client3:_SDKClient__create_conn:237] Creating SDK connection for 'default' 2021-06-08 06:52:44,098 | test | DEBUG | MainThread | [sdk_client3:_SDKClient__create_conn:237] Creating SDK connection for 'default' 2021-06-08 06:52:44,428 | test | DEBUG | MainThread | [sdk_client3:_SDKClient__create_conn:237] Creating SDK connection for 'default' 2021-06-08 06:52:44,878 | test | DEBUG | MainThread | [sdk_client3:_SDKClient__create_conn:237] Creating SDK connection for 'default' 2021-06-08 06:52:45,193 | test | DEBUG | MainThread | [sdk_client3:_SDKClient__create_conn:237] Creating SDK connection for 'default' 2021-06-08 06:52:45,517 | test | DEBUG | MainThread | [sdk_client3:_SDKClient__create_conn:237] Creating SDK connection for 'default' 2021-06-08 06:52:45,838 | test | DEBUG | MainThread | [sdk_client3:_SDKClient__create_conn:237] Creating SDK connection for 'default' 2021-06-08 06:52:46,219 | test | DEBUG | MainThread | [sdk_client3:_SDKClient__create_conn:237] Creating SDK connection for 'default' 2021-06-08 06:52:46,549 | test | DEBUG | MainThread | [sdk_client3:_SDKClient__create_conn:237] Creating SDK connection for 'default' 2021-06-08 06:52:46,878 | test | DEBUG | MainThread | [sdk_client3:_SDKClient__create_conn:237] Creating SDK connection for 'default' 2021-06-08 06:52:47,217 | test | DEBUG | MainThread | [sdk_client3:_SDKClient__create_conn:237] Creating SDK connection for 'default' 2021-06-08 06:52:47,588 | test | DEBUG | MainThread | [sdk_client3:_SDKClient__create_conn:237] Creating SDK connection for 'default' 2021-06-08 06:52:47,927 | test | DEBUG | MainThread | [sdk_client3:_SDKClient__create_conn:237] Creating SDK connection for 'default' 2021-06-08 06:52:48,247 | test | DEBUG | MainThread | [sdk_client3:_SDKClient__create_conn:237] Creating SDK connection for 'default' 2021-06-08 06:52:48,608 | test | DEBUG | MainThread | [sdk_client3:_SDKClient__create_conn:237] Creating SDK connection for 'default' 2021-06-08 06:52:48,959 | test | DEBUG | MainThread | [sdk_client3:_SDKClient__create_conn:237] Creating SDK connection for 'default' 2021-06-08 06:52:49,288 | test | DEBUG | MainThread | [sdk_client3:_SDKClient__create_conn:237] Creating SDK connection for 'default' 2021-06-08 06:52:49,737 | infra | INFO | MainThread | [task_manager:add_new_task:23] Added new task: DocumentsValidatorTask_default_create_1623160369.74 2021-06-08 06:52:49,737 | infra | INFO | pool-5-thread-14 | [task:start_task:87] Thread 'DocumentsValidatorTask_default_create_1623160369.74' started 2021-06-08 06:52:49,739 | infra | DEBUG | MainThread | [task_manager:get_task_result:26] Getting task result for DocumentsValidatorTask_default_create_1623160369.74 2021-06-08 06:52:49,750 | infra | INFO | pool-5-thread-14 | [task_manager:add_new_task:23] Added new task: ValidateDocumentsTask-default__default__default_0_250000_create_1623160369.75 2021-06-08 06:52:49,750 | infra | INFO | pool-5-thread-28 | [task:start_task:87] Thread 'ValidateDocumentsTask-default__default__default_0_250000_create_1623160369.75' started 2021-06-08 06:52:49,750 | infra | INFO | pool-5-thread-14 | [task_manager:add_new_task:23] Added new task: ValidateDocumentsTask-default__default__default_250000_500000_create_1623160369.75 2021-06-08 06:52:49,750 | infra | INFO | pool-5-thread-14 | [task_manager:add_new_task:23] Added new task: ValidateDocumentsTask-default__default__default_500000_750000_create_1623160369.75 2021-06-08 06:52:49,752 | infra | INFO | pool-5-thread-14 | [task_manager:add_new_task:23] Added new task: ValidateDocumentsTask-default__default__default_750000_1000000_create_1623160369.75 2021-06-08 06:52:49,752 | infra | INFO | pool-5-thread-1 | [task:start_task:87] Thread 'ValidateDocumentsTask-default__default__default_250000_500000_create_1623160369.75' started 2021-06-08 06:52:49,752 | infra | INFO | pool-5-thread-14 | [task_manager:add_new_task:23] Added new task: ValidateDocumentsTask-default__default__default_1000000_1250000_create_1623160369.75 2021-06-08 06:52:49,752 | infra | INFO | pool-5-thread-30 | [task:start_task:87] Thread 'ValidateDocumentsTask-default__default__default_500000_750000_create_1623160369.75' started 2021-06-08 06:52:49,752 | infra | INFO | pool-5-thread-14 | [task_manager:add_new_task:23] Added new task: ValidateDocumentsTask-default__default__default_1250000_1500000_create_1623160369.75 2021-06-08 06:52:49,752 | infra | INFO | pool-5-thread-3 | [task:start_task:87] Thread 'ValidateDocumentsTask-default__default__default_750000_1000000_create_1623160369.75' started 2021-06-08 06:52:49,752 | infra | INFO | pool-5-thread-5 | [task:start_task:87] Thread 'ValidateDocumentsTask-default__default__default_1000000_1250000_create_1623160369.75' started 2021-06-08 06:52:49,756 | infra | INFO | pool-5-thread-14 | [task_manager:add_new_task:23] Added new task: ValidateDocumentsTask-default__default__default_1500000_1750000_create_1623160369.75 2021-06-08 06:52:49,757 | infra | INFO | pool-5-thread-6 | [task:start_task:87] Thread 'ValidateDocumentsTask-default__default__default_1500000_1750000_create_1623160369.75' started 2021-06-08 06:52:49,757 | infra | INFO | pool-5-thread-14 | [task_manager:add_new_task:23] Added new task: ValidateDocumentsTask-default__default__default_1750000_2000000_create_1623160369.75 2021-06-08 06:52:49,756 | infra | INFO | pool-5-thread-22 | [task:start_task:87] Thread 'ValidateDocumentsTask-default__default__default_1250000_1500000_create_1623160369.75' started 2021-06-08 06:52:49,757 | infra | INFO | pool-5-thread-11 | [task:start_task:87] Thread 'ValidateDocumentsTask-default__default__default_1750000_2000000_create_1623160369.75' started 2021-06-08 06:52:49,759 | infra | INFO | pool-5-thread-14 | [task_manager:add_new_task:23] Added new task: ValidateDocumentsTask-default__default__default_2000000_2250000_create_1623160369.75 2021-06-08 06:52:49,759 | infra | INFO | pool-5-thread-20 | [task:start_task:87] Thread 'ValidateDocumentsTask-default__default__default_2000000_2250000_create_1623160369.75' started 2021-06-08 06:52:49,759 | infra | INFO | pool-5-thread-2 | [task:start_task:87] Thread 'ValidateDocumentsTask-default__default__default_2250000_2500000_create_1623160369.75' started 2021-06-08 06:52:49,759 | infra | INFO | pool-5-thread-14 | [task_manager:add_new_task:23] Added new task: ValidateDocumentsTask-default__default__default_2250000_2500000_create_1623160369.75 2021-06-08 06:52:49,766 | infra | INFO | pool-5-thread-14 | [task_manager:add_new_task:23] Added new task: ValidateDocumentsTask-default__default__default_2500000_2750000_create_1623160369.75 2021-06-08 06:52:49,766 | infra | INFO | pool-5-thread-27 | [task:start_task:87] Thread 'ValidateDocumentsTask-default__default__default_2500000_2750000_create_1623160369.75' started 2021-06-08 06:52:49,767 | infra | INFO | pool-5-thread-12 | [task:start_task:87] Thread 'ValidateDocumentsTask-default__default__default_2750000_3000000_create_1623160369.75' started 2021-06-08 06:52:49,767 | infra | INFO | pool-5-thread-14 | [task_manager:add_new_task:23] Added new task: ValidateDocumentsTask-default__default__default_2750000_3000000_create_1623160369.75 2021-06-08 06:52:49,770 | infra | INFO | pool-5-thread-19 | [task:start_task:87] Thread 'ValidateDocumentsTask-default__default__default_3000000_3250000_create_1623160369.75' started 2021-06-08 06:52:49,769 | infra | INFO | pool-5-thread-14 | [task_manager:add_new_task:23] Added new task: ValidateDocumentsTask-default__default__default_3000000_3250000_create_1623160369.75 2021-06-08 06:52:49,770 | infra | INFO | pool-5-thread-9 | [task:start_task:87] Thread 'ValidateDocumentsTask-default__default__default_3250000_3500000_create_1623160369.75' started 2021-06-08 06:52:49,770 | infra | INFO | pool-5-thread-14 | [task_manager:add_new_task:23] Added new task: ValidateDocumentsTask-default__default__default_3250000_3500000_create_1623160369.75 2021-06-08 06:52:49,773 | infra | INFO | pool-5-thread-14 | [task_manager:add_new_task:23] Added new task: ValidateDocumentsTask-default__default__default_3500000_3750000_create_1623160369.75 2021-06-08 06:52:49,775 | infra | INFO | pool-5-thread-14 | [task_manager:add_new_task:23] Added new task: ValidateDocumentsTask-default__default__default_3750000_4000000_create_1623160369.75 2021-06-08 06:52:49,775 | infra | INFO | pool-5-thread-23 | [task:start_task:87] Thread 'ValidateDocumentsTask-default__default__default_3750000_4000000_create_1623160369.75' started 2021-06-08 06:52:49,775 | infra | INFO | pool-5-thread-14 | [task_manager:add_new_task:23] Added new task: ValidateDocumentsTask-default__default__default_4000000_4250000_create_1623160369.75 2021-06-08 06:52:49,776 | infra | INFO | pool-5-thread-8 | [task:start_task:87] Thread 'ValidateDocumentsTask-default__default__default_4250000_4500000_create_1623160369.75' started 2021-06-08 06:52:49,775 | infra | INFO | pool-5-thread-14 | [task_manager:add_new_task:23] Added new task: ValidateDocumentsTask-default__default__default_4250000_4500000_create_1623160369.75 2021-06-08 06:52:49,776 | infra | INFO | pool-5-thread-17 | [task:start_task:87] Thread 'ValidateDocumentsTask-default__default__default_4000000_4250000_create_1623160369.75' started 2021-06-08 06:52:49,776 | infra | INFO | pool-5-thread-13 | [task:start_task:87] Thread 'ValidateDocumentsTask-default__default__default_3500000_3750000_create_1623160369.75' started 2021-06-08 06:52:49,778 | infra | INFO | pool-5-thread-14 | [task_manager:add_new_task:23] Added new task: ValidateDocumentsTask-default__default__default_4500000_4750000_create_1623160369.75 2021-06-08 06:52:49,778 | infra | INFO | pool-5-thread-15 | [task:start_task:87] Thread 'ValidateDocumentsTask-default__default__default_4500000_4750000_create_1623160369.75' started 2021-06-08 06:52:49,779 | infra | INFO | pool-5-thread-14 | [task_manager:add_new_task:23] Added new task: ValidateDocumentsTask-default__default__default_4750000_5000000_create_1623160369.75 2021-06-08 06:52:49,779 | infra | INFO | pool-5-thread-29 | [task:start_task:87] Thread 'ValidateDocumentsTask-default__default__default_4750000_5000000_create_1623160369.75' started 2021-06-08 06:52:49,779 | infra | DEBUG | pool-5-thread-14 | [task_manager:get_task_result:26] Getting task result for ValidateDocumentsTask-default__default__default_0_250000_create_1623160369.75 2021-06-08 06:54:38,897 | infra | INFO | pool-5-thread-19 | [task:complete_task:102] Thread 'ValidateDocumentsTask-default__default__default_3000000_3250000_create_1623160369.75' completed 2021-06-08 06:54:38,987 | infra | INFO | pool-5-thread-1 | [task:complete_task:102] Thread 'ValidateDocumentsTask-default__default__default_250000_500000_create_1623160369.75' completed 2021-06-08 06:54:39,095 | infra | INFO | pool-5-thread-3 | [task:complete_task:102] Thread 'ValidateDocumentsTask-default__default__default_750000_1000000_create_1623160369.75' completed 2021-06-08 06:54:39,171 | infra | INFO | pool-5-thread-5 | [task:complete_task:102] Thread 'ValidateDocumentsTask-default__default__default_1000000_1250000_create_1623160369.75' completed 2021-06-08 06:54:39,275 | infra | INFO | pool-5-thread-8 | [task:complete_task:102] Thread 'ValidateDocumentsTask-default__default__default_4250000_4500000_create_1623160369.75' completed 2021-06-08 06:54:39,305 | infra | INFO | pool-5-thread-12 | [task:complete_task:102] Thread 'ValidateDocumentsTask-default__default__default_2750000_3000000_create_1623160369.75' completed 2021-06-08 06:54:39,311 | infra | INFO | pool-5-thread-15 | [task:complete_task:102] Thread 'ValidateDocumentsTask-default__default__default_4500000_4750000_create_1623160369.75' completed 2021-06-08 06:54:39,342 | infra | INFO | pool-5-thread-27 | [task:complete_task:102] Thread 'ValidateDocumentsTask-default__default__default_2500000_2750000_create_1623160369.75' completed 2021-06-08 06:54:39,394 | infra | INFO | pool-5-thread-11 | [task:complete_task:102] Thread 'ValidateDocumentsTask-default__default__default_1750000_2000000_create_1623160369.75' completed 2021-06-08 06:54:39,668 | infra | INFO | pool-5-thread-6 | [task:complete_task:102] Thread 'ValidateDocumentsTask-default__default__default_1500000_1750000_create_1623160369.75' completed 2021-06-08 06:54:39,786 | infra | INFO | pool-5-thread-20 | [task:complete_task:102] Thread 'ValidateDocumentsTask-default__default__default_2000000_2250000_create_1623160369.75' completed 2021-06-08 06:54:39,874 | infra | INFO | pool-5-thread-9 | [task:complete_task:102] Thread 'ValidateDocumentsTask-default__default__default_3250000_3500000_create_1623160369.75' completed 2021-06-08 06:54:40,062 | infra | INFO | pool-5-thread-2 | [task:complete_task:102] Thread 'ValidateDocumentsTask-default__default__default_2250000_2500000_create_1623160369.75' completed 2021-06-08 06:54:40,124 | infra | INFO | pool-5-thread-22 | [task:complete_task:102] Thread 'ValidateDocumentsTask-default__default__default_1250000_1500000_create_1623160369.75' completed 2021-06-08 06:54:40,131 | infra | INFO | pool-5-thread-23 | [task:complete_task:102] Thread 'ValidateDocumentsTask-default__default__default_3750000_4000000_create_1623160369.75' completed 2021-06-08 06:54:40,269 | infra | INFO | pool-5-thread-13 | [task:complete_task:102] Thread 'ValidateDocumentsTask-default__default__default_3500000_3750000_create_1623160369.75' completed 2021-06-08 06:54:40,276 | infra | INFO | pool-5-thread-28 | [task:complete_task:102] Thread 'ValidateDocumentsTask-default__default__default_0_250000_create_1623160369.75' completed 2021-06-08 06:54:40,276 | infra | DEBUG | pool-5-thread-14 | [task_manager:get_task_result:26] Getting task result for ValidateDocumentsTask-default__default__default_250000_500000_create_1623160369.75 2021-06-08 06:54:40,278 | infra | DEBUG | pool-5-thread-14 | [task_manager:get_task_result:26] Getting task result for ValidateDocumentsTask-default__default__default_500000_750000_create_1623160369.75 2021-06-08 06:54:40,282 | infra | INFO | pool-5-thread-17 | [task:complete_task:102] Thread 'ValidateDocumentsTask-default__default__default_4000000_4250000_create_1623160369.75' completed 2021-06-08 06:54:40,282 | infra | INFO | pool-5-thread-30 | [task:complete_task:102] Thread 'ValidateDocumentsTask-default__default__default_500000_750000_create_1623160369.75' completed 2021-06-08 06:54:40,282 | infra | DEBUG | pool-5-thread-14 | [task_manager:get_task_result:26] Getting task result for ValidateDocumentsTask-default__default__default_750000_1000000_create_1623160369.75 2021-06-08 06:54:40,282 | infra | DEBUG | pool-5-thread-14 | [task_manager:get_task_result:26] Getting task result for ValidateDocumentsTask-default__default__default_1000000_1250000_create_1623160369.75 2021-06-08 06:54:40,282 | infra | DEBUG | pool-5-thread-14 | [task_manager:get_task_result:26] Getting task result for ValidateDocumentsTask-default__default__default_1250000_1500000_create_1623160369.75 2021-06-08 06:54:40,282 | infra | DEBUG | pool-5-thread-14 | [task_manager:get_task_result:26] Getting task result for ValidateDocumentsTask-default__default__default_1500000_1750000_create_1623160369.75 2021-06-08 06:54:40,282 | infra | DEBUG | pool-5-thread-14 | [task_manager:get_task_result:26] Getting task result for ValidateDocumentsTask-default__default__default_1750000_2000000_create_1623160369.75 2021-06-08 06:54:40,282 | infra | DEBUG | pool-5-thread-14 | [task_manager:get_task_result:26] Getting task result for ValidateDocumentsTask-default__default__default_2000000_2250000_create_1623160369.75 2021-06-08 06:54:40,282 | infra | DEBUG | pool-5-thread-14 | [task_manager:get_task_result:26] Getting task result for ValidateDocumentsTask-default__default__default_2250000_2500000_create_1623160369.75 2021-06-08 06:54:40,282 | infra | DEBUG | pool-5-thread-14 | [task_manager:get_task_result:26] Getting task result for ValidateDocumentsTask-default__default__default_2500000_2750000_create_1623160369.75 2021-06-08 06:54:40,282 | infra | DEBUG | pool-5-thread-14 | [task_manager:get_task_result:26] Getting task result for ValidateDocumentsTask-default__default__default_2750000_3000000_create_1623160369.75 2021-06-08 06:54:40,282 | infra | DEBUG | pool-5-thread-14 | [task_manager:get_task_result:26] Getting task result for ValidateDocumentsTask-default__default__default_3000000_3250000_create_1623160369.75 2021-06-08 06:54:40,282 | infra | DEBUG | pool-5-thread-14 | [task_manager:get_task_result:26] Getting task result for ValidateDocumentsTask-default__default__default_3250000_3500000_create_1623160369.75 2021-06-08 06:54:40,282 | infra | DEBUG | pool-5-thread-14 | [task_manager:get_task_result:26] Getting task result for ValidateDocumentsTask-default__default__default_3500000_3750000_create_1623160369.75 2021-06-08 06:54:40,282 | infra | DEBUG | pool-5-thread-14 | [task_manager:get_task_result:26] Getting task result for ValidateDocumentsTask-default__default__default_3750000_4000000_create_1623160369.75 2021-06-08 06:54:40,282 | infra | DEBUG | pool-5-thread-14 | [task_manager:get_task_result:26] Getting task result for ValidateDocumentsTask-default__default__default_4000000_4250000_create_1623160369.75 2021-06-08 06:54:40,282 | infra | DEBUG | pool-5-thread-14 | [task_manager:get_task_result:26] Getting task result for ValidateDocumentsTask-default__default__default_4250000_4500000_create_1623160369.75 2021-06-08 06:54:40,282 | infra | DEBUG | pool-5-thread-14 | [task_manager:get_task_result:26] Getting task result for ValidateDocumentsTask-default__default__default_4500000_4750000_create_1623160369.75 2021-06-08 06:54:40,283 | infra | DEBUG | pool-5-thread-14 | [task_manager:get_task_result:26] Getting task result for ValidateDocumentsTask-default__default__default_4750000_5000000_create_1623160369.75 2021-06-08 06:54:40,365 | infra | INFO | pool-5-thread-29 | [task:complete_task:102] Thread 'ValidateDocumentsTask-default__default__default_4750000_5000000_create_1623160369.75' completed 2021-06-08 06:54:40,365 | infra | INFO | pool-5-thread-14 | [task:complete_task:102] Thread 'DocumentsValidatorTask_default_create_1623160369.74' completed 2021-06-08 06:54:40,365 | test | DEBUG | pool-5-thread-14 | [sdk_client3:close:307] Closing SDK for bucket 'default' 2021-06-08 06:54:40,381 | test | DEBUG | pool-5-thread-14 | [sdk_client3:close:311] Cluster disconnected and env shutdown 2021-06-08 06:54:40,382 | test | DEBUG | pool-5-thread-14 | [sdk_client3:close:307] Closing SDK for bucket 'default' 2021-06-08 06:54:40,398 | test | DEBUG | pool-5-thread-14 | [sdk_client3:close:311] Cluster disconnected and env shutdown 2021-06-08 06:54:40,398 | test | DEBUG | pool-5-thread-14 | [sdk_client3:close:307] Closing SDK for bucket 'default' 2021-06-08 06:54:40,414 | test | DEBUG | pool-5-thread-14 | [sdk_client3:close:311] Cluster disconnected and env shutdown 2021-06-08 06:54:40,414 | test | DEBUG | pool-5-thread-14 | [sdk_client3:close:307] Closing SDK for bucket 'default' 2021-06-08 06:54:40,430 | test | DEBUG | pool-5-thread-14 | [sdk_client3:close:311] Cluster disconnected and env shutdown 2021-06-08 06:54:40,430 | test | DEBUG | pool-5-thread-14 | [sdk_client3:close:307] Closing SDK for bucket 'default' 2021-06-08 06:54:40,444 | test | DEBUG | pool-5-thread-14 | [sdk_client3:close:311] Cluster disconnected and env shutdown 2021-06-08 06:54:40,444 | test | DEBUG | pool-5-thread-14 | [sdk_client3:close:307] Closing SDK for bucket 'default' 2021-06-08 06:54:40,459 | test | DEBUG | pool-5-thread-14 | [sdk_client3:close:311] Cluster disconnected and env shutdown 2021-06-08 06:54:40,459 | test | DEBUG | pool-5-thread-14 | [sdk_client3:close:307] Closing SDK for bucket 'default' 2021-06-08 06:54:40,473 | test | DEBUG | pool-5-thread-14 | [sdk_client3:close:311] Cluster disconnected and env shutdown 2021-06-08 06:54:40,473 | test | DEBUG | pool-5-thread-14 | [sdk_client3:close:307] Closing SDK for bucket 'default' 2021-06-08 06:54:40,487 | test | DEBUG | pool-5-thread-14 | [sdk_client3:close:311] Cluster disconnected and env shutdown 2021-06-08 06:54:40,487 | test | DEBUG | pool-5-thread-14 | [sdk_client3:close:307] Closing SDK for bucket 'default' 2021-06-08 06:54:40,502 | test | DEBUG | pool-5-thread-14 | [sdk_client3:close:311] Cluster disconnected and env shutdown 2021-06-08 06:54:40,502 | test | DEBUG | pool-5-thread-14 | [sdk_client3:close:307] Closing SDK for bucket 'default' 2021-06-08 06:54:40,516 | test | DEBUG | pool-5-thread-14 | [sdk_client3:close:311] Cluster disconnected and env shutdown 2021-06-08 06:54:40,516 | test | DEBUG | pool-5-thread-14 | [sdk_client3:close:307] Closing SDK for bucket 'default' 2021-06-08 06:54:40,530 | test | DEBUG | pool-5-thread-14 | [sdk_client3:close:311] Cluster disconnected and env shutdown 2021-06-08 06:54:40,530 | test | DEBUG | pool-5-thread-14 | [sdk_client3:close:307] Closing SDK for bucket 'default' 2021-06-08 06:54:40,546 | test | DEBUG | pool-5-thread-14 | [sdk_client3:close:311] Cluster disconnected and env shutdown 2021-06-08 06:54:40,546 | test | DEBUG | pool-5-thread-14 | [sdk_client3:close:307] Closing SDK for bucket 'default' 2021-06-08 06:54:40,559 | test | DEBUG | pool-5-thread-14 | [sdk_client3:close:311] Cluster disconnected and env shutdown 2021-06-08 06:54:40,559 | test | DEBUG | pool-5-thread-14 | [sdk_client3:close:307] Closing SDK for bucket 'default' 2021-06-08 06:54:40,573 | test | DEBUG | pool-5-thread-14 | [sdk_client3:close:311] Cluster disconnected and env shutdown 2021-06-08 06:54:40,573 | test | DEBUG | pool-5-thread-14 | [sdk_client3:close:307] Closing SDK for bucket 'default' 2021-06-08 06:54:40,588 | test | DEBUG | pool-5-thread-14 | [sdk_client3:close:311] Cluster disconnected and env shutdown 2021-06-08 06:54:40,588 | test | DEBUG | pool-5-thread-14 | [sdk_client3:close:307] Closing SDK for bucket 'default' 2021-06-08 06:54:40,602 | test | DEBUG | pool-5-thread-14 | [sdk_client3:close:311] Cluster disconnected and env shutdown 2021-06-08 06:54:40,602 | test | DEBUG | pool-5-thread-14 | [sdk_client3:close:307] Closing SDK for bucket 'default' 2021-06-08 06:54:40,617 | test | DEBUG | pool-5-thread-14 | [sdk_client3:close:311] Cluster disconnected and env shutdown 2021-06-08 06:54:40,618 | test | DEBUG | pool-5-thread-14 | [sdk_client3:close:307] Closing SDK for bucket 'default' 2021-06-08 06:54:40,631 | test | DEBUG | pool-5-thread-14 | [sdk_client3:close:311] Cluster disconnected and env shutdown 2021-06-08 06:54:40,631 | test | DEBUG | pool-5-thread-14 | [sdk_client3:close:307] Closing SDK for bucket 'default' 2021-06-08 06:54:40,645 | test | DEBUG | pool-5-thread-14 | [sdk_client3:close:311] Cluster disconnected and env shutdown 2021-06-08 06:54:40,645 | test | DEBUG | pool-5-thread-14 | [sdk_client3:close:307] Closing SDK for bucket 'default' 2021-06-08 06:54:40,661 | test | DEBUG | pool-5-thread-14 | [sdk_client3:close:311] Cluster disconnected and env shutdown 2021-06-08 06:54:40,661 | infra | INFO | pool-5-thread-14 | [task:complete_task:102] Thread 'DocumentsValidatorTask_default_create_1623160369.74' completed 2021-06-08 06:54:40,661 | test | DEBUG | MainThread | [bucket_ready_functions:update_all_bucket_replicas:2042] Updating replica for bucket default to 1s 2021-06-08 06:54:40,677 | infra | INFO | MainThread | [BucketOperations_Rest:change_bucket_props:422] Updating bucket properties for default 2021-06-08 06:54:40,677 | infra | DEBUG | MainThread | [BucketOperations_Rest:change_bucket_props:423] http://172.23.106.196:8091/pools/default/buckets/default with param: replicaNumber=1 2021-06-08 06:54:40,714 | infra | DEBUG | MainThread | [BucketOperations_Rest:change_bucket_props:432] Bucket default updated 2021-06-08 06:54:40,740 | infra | INFO | MainThread | [task_manager:add_new_task:23] Added new task: Rebalance_task_IN=[]_OUT=[]_1623160480.71 2021-06-08 06:54:40,740 | infra | INFO | pool-5-thread-18 | [task:start_task:87] Thread 'Rebalance_task_IN=[]_OUT=[]_1623160480.71' started 2021-06-08 06:54:40,742 | infra | DEBUG | MainThread | [task_manager:get_task_result:26] Getting task result for Rebalance_task_IN=[]_OUT=[]_1623160480.71 2021-06-08 06:54:40,780 | test | DEBUG | pool-5-thread-18 | [rest_client:rebalance:1339] Rebalance params: {'password': 'password', 'knownNodes': u'ns_1@172.23.106.168,ns_1@172.23.106.198,ns_1@172.23.106.196,ns_1@172.23.106.199', 'ejectedNodes': '', 'user': 'Administrator'} 2021-06-08 06:54:40,842 | test | DEBUG | pool-5-thread-18 | [rest_client:rebalance:1344] Rebalance operation started 2021-06-08 06:54:40,842 | test | INFO | pool-5-thread-18 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.106.168 | kv | 7.0.0-5270-enterprise | 65.5812036274 | Cluster node | | 172.23.106.198 | kv | 7.0.0-5270-enterprise | 56.7836340389 | Cluster node | | 172.23.106.196 | kv | 7.0.0-5270-enterprise | 66.6438356164 | Cluster node | | 172.23.106.199 | kv | 7.0.0-5270-enterprise | 51.9971367215 | Cluster node | +----------------+----------+-----------------------+---------------+--------------+ 2021-06-08 06:54:40,851 | test | DEBUG | pool-5-thread-18 | [rest_client:_rebalance_status_and_progress:1553] Rebalance percentage: 0.00 % 2021-06-08 06:54:40,851 | test | INFO | pool-5-thread-18 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-06-08 06:54:40,865 | infra | DEBUG | pool-5-thread-18 | [task:check:382] Wait before next rebalance progress check 2021-06-08 06:54:40,867 | infra | DEBUG | pool-5-thread-18 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L383 2021-06-08 06:54:40,867 | infra | DEBUG | pool-5-thread-18 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:54:45,881 | test | ERROR | pool-5-thread-18 | [rest_client:_rebalance_status_and_progress:1548] {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'f62b8f34711d33f2a1eb20aa8eba6c7d', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=393e8be0b4665c690b809d2baae32fc7', u'status': u'notRunning'} - rebalance failed 2021-06-08 06:54:45,915 | test | INFO | pool-5-thread-18 | [rest_client:print_UI_logs:2694] Latest logs from UI on 172.23.106.196: 2021-06-08 06:54:45,915 | test | ERROR | pool-5-thread-18 | [rest_client:print_UI_logs:2696] {u'code': 0, u'module': u'ns_log', u'type': u'info', u'node': u'ns_1@172.23.106.168', u'tstamp': 1623160483806L, u'shortText': u'message', u'serverTime': u'2021-06-08T06:54:43.806Z', u'text': u"Service 'memcached' exited with status 134. Restarting. Messages:\n2021-06-08T06:54:42.508438-07:00 CRITICAL #23 /opt/couchbase/bin/memcached() [0x400000+0x299d4d]\n2021-06-08T06:54:42.508465-07:00 CRITICAL #24 /opt/couchbase/bin/../lib/libstdc++.so.6() [0x7fc4d3b0c000+0xcdd40]\n2021-06-08T06:54:42.508471-07:00 CRITICAL #25 /lib64/libpthread.so.0() [0x7fc4d57b6000+0x7e65]\n2021-06-08T06:54:42.508520-07:00 CRITICAL #26 /lib64/libc.so.6(clone+0x6d) [0x7fc4d3224000+0xfe88d]\n2021-06-08T06:54:42.508632-07:00 CRITICAL [(default) magma_6/kvstore-206/rev-000000001/seqIndex]Error occured during memtable flush (IOError: /data/default/magma.6/kvstore-206/rev-000000001/seqIndex/sstable.98.data: Unable to write to the file err=No space left on device offset:507904 len:184320).\n2021-06-08T06:54:42.508644-07:00 CRITICAL [(default) magma_6/kvstore-198/rev-000000001/seqIndex]Failed to write tree state /data/default/magma.6/kvstore-198/rev-000000001/seqIndex/state.44: Unable to open file err=No space left on device\n[*** LOG ERROR ***] [2021-06-08 06:54:42] [spdlog_file_logger] async log: thread pool doesn't exist anymore\npure virtual method called\nterminate called recursively"} 2021-06-08 06:54:45,917 | test | ERROR | pool-5-thread-18 | [rest_client:print_UI_logs:2696] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.106.168', u'tstamp': 1623160483793L, u'shortText': u'message', u'serverTime': u'2021-06-08T06:54:43.793Z', u'text': u"Control connection to memcached on 'ns_1@172.23.106.168' disconnected. Check logs for details."} 2021-06-08 06:54:45,917 | test | ERROR | pool-5-thread-18 | [rest_client:print_UI_logs:2696] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'critical', u'node': u'ns_1@172.23.106.196', u'tstamp': 1623160483763L, u'shortText': u'message', u'serverTime': u'2021-06-08T06:54:43.763Z', u'text': u'Rebalance exited with reason {mover_crashed,\n {unexpected_exit,\n {\'EXIT\',<0.14122.3>,\n {{wait_seqno_persisted_failed,"default",761,\n 8812,\n [{\'ns_1@172.23.106.199\',\n {\'EXIT\',\n {socket_closed,\n {gen_server,call,\n [{\'janitor_agent-default\',\n \'ns_1@172.23.106.199\'},\n {if_rebalance,<0.13558.3>,\n {wait_seqno_persisted,761,8812}},\n infinity]}}}}]},\n [{ns_single_vbucket_mover,\n \'-wait_seqno_persisted_many/5-fun-2-\',5,\n [{file,"src/ns_single_vbucket_mover.erl"},\n {line,482}]},\n {proc_lib,init_p,3,\n [{file,"proc_lib.erl"},{line,234}]}]}}}}.\nRebalance Operation Id = e12f4838d496bb0e6c92f06add901452'} 2021-06-08 06:54:45,917 | test | ERROR | pool-5-thread-18 | [rest_client:print_UI_logs:2696] {u'code': 0, u'module': u'ns_log', u'type': u'info', u'node': u'ns_1@172.23.106.198', u'tstamp': 1623160483762L, u'shortText': u'message', u'serverTime': u'2021-06-08T06:54:43.762Z', u'text': u"Service 'memcached' exited with status 134. Restarting. Messages:\n2021-06-08T06:54:42.674172-07:00 CRITICAL [(default) magma_6/kvstore-366/rev-000000001/keyIndex]Error occured during memtable flush (IOError: /data/default/magma.6/kvstore-366/rev-000000001/keyIndex/sstable.100.data: Unable to open file err=No space left on device).\n2021-06-08T06:54:42.674172-07:00 CRITICAL (default) (default) magma_2 MagmaKVStore::saveDocs vb:714 WriteDocs failed. Status:IOError: /data/default/magma.2/wal/wal.81: Unable to write to the file err=No space left on device offset:0 len:1118208\n2021-06-08T06:54:42.674310-07:00 CRITICAL [(default) magma_6/kvstore-358/rev-000000001/keyIndex]Error occured during memtable flush (IOError: /data/default/magma.6/kvstore-358/rev-000000001/keyIndex/sstable.96.data: Unable to open file err=No space left on device).\n2021-06-08T06:54:42.674344-07:00 CRITICAL [(default) magma_6/kvstore-366/rev-000000001/seqIndex]Error occured during memtable flush (IOError: /data/default/magma.6/kvstore-366/rev-000000001/seqIndex/sstable.62.data: Unable to write to the file err=No space left on device offset:1519616 len:520192)."} 2021-06-08 06:54:45,917 | test | ERROR | pool-5-thread-18 | [rest_client:print_UI_logs:2696] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.106.198', u'tstamp': 1623160483751L, u'shortText': u'message', u'serverTime': u'2021-06-08T06:54:43.751Z', u'text': u"Control connection to memcached on 'ns_1@172.23.106.198' disconnected. Check logs for details."} 2021-06-08 06:54:45,917 | test | ERROR | pool-5-thread-18 | [rest_client:print_UI_logs:2696] {u'code': 0, u'module': u'ns_vbucket_mover', u'type': u'critical', u'node': u'ns_1@172.23.106.196', u'tstamp': 1623160483731L, u'shortText': u'message', u'serverTime': u'2021-06-08T06:54:43.731Z', u'text': u'Worker <0.14054.3> (for action {move,{761,\n [\'ns_1@172.23.106.198\',\n \'ns_1@172.23.106.199\',\n \'ns_1@172.23.106.196\',\n \'ns_1@172.23.106.168\'],\n [\'ns_1@172.23.106.198\',\n \'ns_1@172.23.106.199\'],\n []}}) exited with reason {unexpected_exit,\n {\'EXIT\',\n <0.14122.3>,\n {{wait_seqno_persisted_failed,\n "default",\n 761,8812,\n [{\'ns_1@172.23.106.199\',\n {\'EXIT\',\n {socket_closed,\n {gen_server,\n call,\n [{\'janitor_agent-default\',\n \'ns_1@172.23.106.199\'},\n {if_rebalance,\n <0.13558.3>,\n {wait_seqno_persisted,\n 761,\n 8812}},\n infinity]}}}}]},\n [{ns_single_vbucket_mover,\n \'-wait_seqno_persisted_many/5-fun-2-\',\n 5,\n [{file,\n "src/ns_single_vbucket_mover.erl"},\n {line,\n 482}]},\n {proc_lib,\n init_p,3,\n [{file,\n "proc_lib.erl"},\n {line,\n 234}]}]}}}'} 2021-06-08 06:54:45,917 | test | ERROR | pool-5-thread-18 | [rest_client:print_UI_logs:2696] {u'code': 0, u'module': u'menelaus_web_alerts_srv', u'type': u'info', u'node': u'ns_1@172.23.106.196', u'tstamp': 1623160483385L, u'shortText': u'message', u'serverTime': u'2021-06-08T06:54:43.385Z', u'text': u'Write Commit Failure. Disk write failed for item in Bucket "default" on node 172.23.106.196.'} 2021-06-08 06:54:45,918 | test | ERROR | pool-5-thread-18 | [rest_client:print_UI_logs:2696] {u'code': 0, u'module': u'ns_vbucket_mover', u'type': u'info', u'node': u'ns_1@172.23.106.196', u'tstamp': 1623160481414L, u'shortText': u'message', u'serverTime': u'2021-06-08T06:54:41.414Z', u'text': u'Bucket "default" rebalance does not seem to be swap rebalance'} 2021-06-08 06:54:45,918 | test | ERROR | pool-5-thread-18 | [rest_client:print_UI_logs:2696] {u'code': 0, u'module': u'ns_rebalancer', u'type': u'info', u'node': u'ns_1@172.23.106.196', u'tstamp': 1623160481183L, u'shortText': u'message', u'serverTime': u'2021-06-08T06:54:41.183Z', u'text': u'Started rebalancing bucket default'} 2021-06-08 06:54:45,918 | test | ERROR | pool-5-thread-18 | [rest_client:print_UI_logs:2696] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'info', u'node': u'ns_1@172.23.106.196', u'tstamp': 1623160480786L, u'shortText': u'message', u'serverTime': u'2021-06-08T06:54:40.786Z', u'text': u"Starting rebalance, KeepNodes = ['ns_1@172.23.106.168','ns_1@172.23.106.198',\n 'ns_1@172.23.106.196','ns_1@172.23.106.199'], EjectNodes = [], Failed over and being ejected nodes = []; no delta recovery nodes; Operation Id = e12f4838d496bb0e6c92f06add901452"} 2021-06-08 06:54:45,918 | test | ERROR | pool-5-thread-18 | [task:call:267] Rebalance Failed: {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'f62b8f34711d33f2a1eb20aa8eba6c7d', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=393e8be0b4665c690b809d2baae32fc7', u'status': u'notRunning'} - rebalance failed FAIL 2021-06-08 06:54:45,924 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.196 with username: root, password: couchbase 2021-06-08 06:54:46,013 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 06:54:46,246 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: ls /etc/ 2021-06-08 06:54:46,305 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 06:54:46,306 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: ls /cygdrive/c/Windows 2021-06-08 06:54:46,319 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: uname -m 2021-06-08 06:54:46,332 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: cat /proc/cpuinfo 2021-06-08 06:54:46,345 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: df -Thl 2021-06-08 06:54:46,361 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: cat /proc/meminfo 2021-06-08 06:54:46,374 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: hostname 2021-06-08 06:54:46,388 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: hostname -d 2021-06-08 06:54:46,401 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: rm -rf /data/full_disk_* 2021-06-08 06:54:46,667 | infra | DEBUG | MainThread | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.196 2021-06-08 06:54:46,676 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/magma/magma_disk_full.py -> free_disk():L107 2021-06-08 06:54:46,677 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: Wait for files to clean up from the disk 2021-06-08 06:54:56,678 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.199 with username: root, password: couchbase 2021-06-08 06:54:56,760 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 06:54:56,984 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: ls /etc/ 2021-06-08 06:54:57,045 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 06:54:57,045 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: ls /cygdrive/c/Windows 2021-06-08 06:54:57,058 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: uname -m 2021-06-08 06:54:57,071 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: cat /proc/cpuinfo 2021-06-08 06:54:57,084 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: df -Thl 2021-06-08 06:54:57,102 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: cat /proc/meminfo 2021-06-08 06:54:57,115 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: hostname 2021-06-08 06:54:57,128 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: hostname -d 2021-06-08 06:54:57,141 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: rm -rf /data/full_disk_* 2021-06-08 06:54:57,459 | infra | DEBUG | MainThread | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.199 2021-06-08 06:54:57,467 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/magma/magma_disk_full.py -> free_disk():L107 2021-06-08 06:54:57,467 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: Wait for files to clean up from the disk 2021-06-08 06:55:07,470 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.168 with username: root, password: couchbase 2021-06-08 06:55:07,548 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 06:55:07,769 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: ls /etc/ 2021-06-08 06:55:07,851 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 06:55:07,851 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: ls /cygdrive/c/Windows 2021-06-08 06:55:07,867 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: uname -m 2021-06-08 06:55:07,880 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: cat /proc/cpuinfo 2021-06-08 06:55:07,894 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: df -Thl 2021-06-08 06:55:07,911 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: cat /proc/meminfo 2021-06-08 06:55:07,923 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: hostname 2021-06-08 06:55:07,934 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: hostname -d 2021-06-08 06:55:07,947 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: rm -rf /data/full_disk_* 2021-06-08 06:55:08,216 | infra | DEBUG | MainThread | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.168 2021-06-08 06:55:08,221 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/magma/magma_disk_full.py -> free_disk():L107 2021-06-08 06:55:08,221 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: Wait for files to clean up from the disk 2021-06-08 06:55:18,223 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.198 with username: root, password: couchbase 2021-06-08 06:55:18,309 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 06:55:18,530 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: ls /etc/ 2021-06-08 06:55:18,592 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 06:55:18,592 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: ls /cygdrive/c/Windows 2021-06-08 06:55:18,605 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: uname -m 2021-06-08 06:55:18,618 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: cat /proc/cpuinfo 2021-06-08 06:55:18,631 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: df -Thl 2021-06-08 06:55:18,645 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: cat /proc/meminfo 2021-06-08 06:55:18,658 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: hostname 2021-06-08 06:55:18,671 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: hostname -d 2021-06-08 06:55:18,684 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: rm -rf /data/full_disk_* 2021-06-08 06:55:18,982 | infra | DEBUG | MainThread | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.198 2021-06-08 06:55:18,986 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/magma/magma_disk_full.py -> free_disk():L107 2021-06-08 06:55:18,986 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: Wait for files to clean up from the disk 2021-06-08 06:55:29,019 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+-------------+-------------+----------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+-------------+-------------+----------------+-------------------+-----------------------+ | 172.23.106.168 | kv | 11.095505618 | 12294176768 | 11135389696 | 0 / 3758092288 | 0 / 0 | 7.0.0-5270-enterprise | | 172.23.106.198 | kv | 5.45316839182 | 12294176768 | 11451977728 | 0 / 3758092288 | 0 / 0 | 7.0.0-5270-enterprise | | 172.23.106.196 | kv | 1.78391959799 | 12294176768 | 5811019776 | 0 / 3758092288 | 2188316 / 6731937 | 7.0.0-5270-enterprise | | 172.23.106.199 | kv | 1.83394045974 | 12294176768 | 4129857536 | 0 / 3758092288 | 2285217 / 6687799 | 7.0.0-5270-enterprise | +----------------+----------+-----------------+-------------+-------------+----------------+-------------------+-----------------------+ 2021-06-08 06:55:29,266 | test | INFO | MainThread | [magma_base:tearDown:430] ## Active Resident Threshold of default is 80.623390953 ## 2021-06-08 06:55:29,266 | infra | INFO | MainThread | [task_manager:shutdown:65] Running TaskManager shutdown 2021-06-08 06:55:29,269 | infra | INFO | MainThread | [task_manager:shutdown:65] Running TaskManager shutdown 2021-06-08 06:55:29,269 | infra | WARNING | MainThread | [cluster:shutdown:702] Cluster instance shutdown with force 2021-06-08 06:55:29,269 | infra | INFO | MainThread | [task_manager:abort_all_tasks:87] Stopping task StatsWaitTask_default_ep_data_write_failed_1623160247.68 2021-06-08 06:55:29,269 | infra | DEBUG | MainThread | [task_manager:abort_all_tasks:92] Task StatsWaitTask_default_ep_data_write_failed_1623160247.68 could not be cancelled or already finished. 2021-06-08 06:55:29,269 | infra | INFO | MainThread | [task_manager:abort_all_tasks:87] Stopping task StatsWaitTask_default_ep_data_write_failed_1623160104.97 2021-06-08 06:55:29,269 | infra | DEBUG | MainThread | [task_manager:abort_all_tasks:92] Task StatsWaitTask_default_ep_data_write_failed_1623160104.97 could not be cancelled or already finished. 2021-06-08 06:55:29,269 | infra | INFO | MainThread | [task_manager:abort_all_tasks:87] Stopping task StatsWaitTask_default_ep_data_write_failed_1623160175.42 2021-06-08 06:55:29,269 | infra | DEBUG | MainThread | [task_manager:abort_all_tasks:92] Task StatsWaitTask_default_ep_data_write_failed_1623160175.42 could not be cancelled or already finished. 2021-06-08 06:55:29,269 | infra | INFO | MainThread | [task_manager:abort_all_tasks:87] Stopping task StatsWaitTask_default_ep_data_write_failed_1623160176.59 2021-06-08 06:55:29,269 | infra | DEBUG | MainThread | [task_manager:abort_all_tasks:92] Task StatsWaitTask_default_ep_data_write_failed_1623160176.59 could not be cancelled or already finished. 2021-06-08 06:55:29,269 | infra | INFO | MainThread | [task_manager:abort_all_tasks:87] Stopping task StatsWaitTask_default_ep_data_write_failed_1623160249.55 2021-06-08 06:55:29,269 | infra | DEBUG | MainThread | [task_manager:abort_all_tasks:92] Task StatsWaitTask_default_ep_data_write_failed_1623160249.55 could not be cancelled or already finished. 2021-06-08 06:55:29,270 | infra | INFO | MainThread | [task_manager:abort_all_tasks:87] Stopping task StatsWaitTask_default_ep_data_write_failed_1623160177.23 2021-06-08 06:55:29,270 | infra | DEBUG | MainThread | [task_manager:abort_all_tasks:92] Task StatsWaitTask_default_ep_data_write_failed_1623160177.23 could not be cancelled or already finished. 2021-06-08 06:55:29,270 | infra | INFO | MainThread | [task_manager:abort_all_tasks:87] Stopping task StatsWaitTask_default_ep_data_write_failed_1623160248.85 2021-06-08 06:55:29,270 | infra | DEBUG | MainThread | [task_manager:abort_all_tasks:92] Task StatsWaitTask_default_ep_data_write_failed_1623160248.85 could not be cancelled or already finished. 2021-06-08 06:55:29,270 | infra | INFO | MainThread | [task_manager:abort_all_tasks:87] Stopping task StatsWaitTask_default_ep_data_write_failed_1623160248.23 2021-06-08 06:55:29,270 | infra | DEBUG | MainThread | [task_manager:abort_all_tasks:92] Task StatsWaitTask_default_ep_data_write_failed_1623160248.23 could not be cancelled or already finished. 2021-06-08 06:55:29,270 | infra | INFO | MainThread | [task_manager:abort_all_tasks:87] Stopping task StatsWaitTask_default_ep_data_write_failed_1623160175.97 2021-06-08 06:55:29,270 | infra | DEBUG | MainThread | [task_manager:abort_all_tasks:92] Task StatsWaitTask_default_ep_data_write_failed_1623160175.97 could not be cancelled or already finished. 2021-06-08 06:55:29,270 | infra | INFO | MainThread | [task_manager:abort_all_tasks:87] Stopping task StatsWaitTask_default_ep_data_write_failed_1623160102.58 2021-06-08 06:55:29,270 | infra | DEBUG | MainThread | [task_manager:abort_all_tasks:92] Task StatsWaitTask_default_ep_data_write_failed_1623160102.58 could not be cancelled or already finished. 2021-06-08 06:55:29,270 | infra | INFO | MainThread | [task_manager:abort_all_tasks:87] Stopping task StatsWaitTask_default_ep_data_write_failed_1623160103.21 2021-06-08 06:55:29,272 | infra | DEBUG | MainThread | [task_manager:abort_all_tasks:92] Task StatsWaitTask_default_ep_data_write_failed_1623160103.21 could not be cancelled or already finished. 2021-06-08 06:55:29,272 | infra | INFO | MainThread | [task_manager:abort_all_tasks:87] Stopping task StatsWaitTask_default_ep_data_write_failed_1623160104.12 2021-06-08 06:55:29,272 | infra | DEBUG | MainThread | [task_manager:abort_all_tasks:92] Task StatsWaitTask_default_ep_data_write_failed_1623160104.12 could not be cancelled or already finished. 2021-06-08 06:55:29,272 | infra | DEBUG | MainThread | [sdk_client3:shutdown:76] Closing clients from SDKClientPool 2021-06-08 06:55:29,272 | test | DEBUG | MainThread | [sdk_client3:close:307] Closing SDK for bucket 'default' 2021-06-08 06:55:29,283 | test | DEBUG | MainThread | [sdk_client3:close:311] Cluster disconnected and env shutdown 2021-06-08 06:55:29,285 | test | DEBUG | MainThread | [sdk_client3:close:307] Closing SDK for bucket 'default' 2021-06-08 06:55:29,296 | test | DEBUG | MainThread | [sdk_client3:close:311] Cluster disconnected and env shutdown 2021-06-08 06:55:29,296 | test | DEBUG | MainThread | [sdk_client3:close:307] Closing SDK for bucket 'default' 2021-06-08 06:55:29,309 | test | DEBUG | MainThread | [sdk_client3:close:311] Cluster disconnected and env shutdown 2021-06-08 06:55:29,309 | test | DEBUG | MainThread | [sdk_client3:close:307] Closing SDK for bucket 'default' 2021-06-08 06:55:29,322 | test | DEBUG | MainThread | [sdk_client3:close:311] Cluster disconnected and env shutdown 2021-06-08 06:55:29,322 | test | DEBUG | MainThread | [sdk_client3:close:307] Closing SDK for bucket 'default' 2021-06-08 06:55:29,334 | test | DEBUG | MainThread | [sdk_client3:close:311] Cluster disconnected and env shutdown 2021-06-08 06:55:29,334 | test | DEBUG | MainThread | [sdk_client3:close:307] Closing SDK for bucket 'default' 2021-06-08 06:55:29,345 | test | DEBUG | MainThread | [sdk_client3:close:311] Cluster disconnected and env shutdown 2021-06-08 06:55:29,345 | test | DEBUG | MainThread | [sdk_client3:close:307] Closing SDK for bucket 'default' 2021-06-08 06:55:29,358 | test | DEBUG | MainThread | [sdk_client3:close:311] Cluster disconnected and env shutdown 2021-06-08 06:55:29,358 | test | DEBUG | MainThread | [sdk_client3:close:307] Closing SDK for bucket 'default' 2021-06-08 06:55:29,371 | test | DEBUG | MainThread | [sdk_client3:close:311] Cluster disconnected and env shutdown 2021-06-08 06:55:29,371 | test | DEBUG | MainThread | [sdk_client3:close:307] Closing SDK for bucket 'default' 2021-06-08 06:55:29,382 | test | DEBUG | MainThread | [sdk_client3:close:311] Cluster disconnected and env shutdown 2021-06-08 06:55:29,382 | test | DEBUG | MainThread | [sdk_client3:close:307] Closing SDK for bucket 'default' 2021-06-08 06:55:29,394 | test | DEBUG | MainThread | [sdk_client3:close:311] Cluster disconnected and env shutdown 2021-06-08 06:55:29,395 | test | DEBUG | MainThread | [sdk_client3:close:307] Closing SDK for bucket 'default' 2021-06-08 06:55:29,407 | test | DEBUG | MainThread | [sdk_client3:close:311] Cluster disconnected and env shutdown 2021-06-08 06:55:29,407 | test | DEBUG | MainThread | [sdk_client3:close:307] Closing SDK for bucket 'default' 2021-06-08 06:55:29,420 | test | DEBUG | MainThread | [sdk_client3:close:311] Cluster disconnected and env shutdown 2021-06-08 06:55:29,420 | test | DEBUG | MainThread | [sdk_client3:close:307] Closing SDK for bucket 'default' 2021-06-08 06:55:29,431 | test | DEBUG | MainThread | [sdk_client3:close:311] Cluster disconnected and env shutdown 2021-06-08 06:55:29,431 | test | DEBUG | MainThread | [sdk_client3:close:307] Closing SDK for bucket 'default' 2021-06-08 06:55:29,444 | test | DEBUG | MainThread | [sdk_client3:close:311] Cluster disconnected and env shutdown 2021-06-08 06:55:29,444 | test | DEBUG | MainThread | [sdk_client3:close:307] Closing SDK for bucket 'default' 2021-06-08 06:55:29,457 | test | DEBUG | MainThread | [sdk_client3:close:311] Cluster disconnected and env shutdown 2021-06-08 06:55:29,457 | test | DEBUG | MainThread | [sdk_client3:close:307] Closing SDK for bucket 'default' 2021-06-08 06:55:29,469 | test | DEBUG | MainThread | [sdk_client3:close:311] Cluster disconnected and env shutdown 2021-06-08 06:55:29,469 | test | DEBUG | MainThread | [sdk_client3:close:307] Closing SDK for bucket 'default' 2021-06-08 06:55:29,480 | test | DEBUG | MainThread | [sdk_client3:close:311] Cluster disconnected and env shutdown 2021-06-08 06:55:29,480 | test | DEBUG | MainThread | [sdk_client3:close:307] Closing SDK for bucket 'default' 2021-06-08 06:55:29,565 | test | DEBUG | MainThread | [sdk_client3:close:311] Cluster disconnected and env shutdown 2021-06-08 06:55:29,565 | test | DEBUG | MainThread | [sdk_client3:close:307] Closing SDK for bucket 'default' 2021-06-08 06:55:29,576 | test | DEBUG | MainThread | [sdk_client3:close:311] Cluster disconnected and env shutdown 2021-06-08 06:55:29,576 | test | DEBUG | MainThread | [sdk_client3:close:307] Closing SDK for bucket 'default' 2021-06-08 06:55:29,588 | test | DEBUG | MainThread | [sdk_client3:close:311] Cluster disconnected and env shutdown 2021-06-08 06:55:29,592 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.196 with username: root, password: couchbase 2021-06-08 06:55:29,680 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 06:55:29,887 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: ls /etc/ 2021-06-08 06:55:29,944 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 06:55:29,944 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: ls /cygdrive/c/Windows 2021-06-08 06:55:29,957 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: uname -m 2021-06-08 06:55:29,969 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: cat /proc/cpuinfo 2021-06-08 06:55:29,982 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: df -Thl 2021-06-08 06:55:29,997 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: cat /proc/meminfo 2021-06-08 06:55:30,010 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: hostname 2021-06-08 06:55:30,023 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: hostname -d 2021-06-08 06:55:30,036 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-06-08 06:55:30,051 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:744] 172.23.106.196: No crash files found 2021-06-08 06:55:30,072 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-06-08 06:55:30,086 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000361.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' 2021-06-08 06:55:30,115 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000361.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-06-08 06:55:30,142 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000361.txt 2021-06-08 06:55:30,164 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000361.txt 2021-06-08 06:55:30,181 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000362.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' 2021-06-08 06:55:30,209 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000362.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-06-08 06:55:30,236 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000362.txt 2021-06-08 06:55:30,257 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000362.txt 2021-06-08 06:55:30,278 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000363.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' 2021-06-08 06:55:30,303 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000363.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-06-08 06:55:30,332 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000363.txt 2021-06-08 06:55:30,352 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000363.txt 2021-06-08 06:55:30,371 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000364.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' 2021-06-08 06:55:30,398 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000364.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-06-08 06:55:30,426 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000364.txt 2021-06-08 06:55:30,446 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000364.txt 2021-06-08 06:55:30,464 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000365.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' 2021-06-08 06:55:30,490 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000365.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-06-08 06:55:30,517 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000365.txt 2021-06-08 06:55:30,539 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000365.txt 2021-06-08 06:55:30,558 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000366.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' 2021-06-08 06:55:30,584 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000366.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-06-08 06:55:30,614 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000366.txt 2021-06-08 06:55:30,634 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000366.txt 2021-06-08 06:55:30,654 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000367.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' 2021-06-08 06:55:30,680 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000367.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-06-08 06:55:30,707 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000367.txt 2021-06-08 06:55:30,727 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000367.txt 2021-06-08 06:55:30,746 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000368.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' 2021-06-08 06:55:30,773 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000368.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-06-08 06:55:30,799 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000368.txt 2021-06-08 06:55:30,819 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000368.txt 2021-06-08 06:55:30,838 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000369.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' 2021-06-08 06:55:30,865 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000369.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-06-08 06:55:30,892 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000369.txt 2021-06-08 06:55:30,913 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000369.txt 2021-06-08 06:55:30,931 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000370.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' 2021-06-08 06:55:30,957 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000370.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-06-08 06:55:30,983 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000370.txt 2021-06-08 06:55:31,005 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000370.txt 2021-06-08 06:55:31,023 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000371.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' 2021-06-08 06:55:31,049 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000371.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-06-08 06:55:31,075 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000371.txt 2021-06-08 06:55:31,096 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000371.txt 2021-06-08 06:55:31,115 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000372.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' 2021-06-08 06:55:31,141 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000372.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-06-08 06:55:31,168 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000372.txt 2021-06-08 06:55:31,190 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000372.txt 2021-06-08 06:55:31,209 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000373.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' 2021-06-08 06:55:31,233 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000373.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-06-08 06:55:31,260 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000373.txt 2021-06-08 06:55:31,279 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000373.txt 2021-06-08 06:55:31,298 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000374.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' 2021-06-08 06:55:31,323 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000374.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-06-08 06:55:31,351 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000374.txt 2021-06-08 06:55:31,371 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000374.txt 2021-06-08 06:55:31,390 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000375.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' 2021-06-08 06:55:31,417 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000375.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-06-08 06:55:31,444 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000375.txt 2021-06-08 06:55:31,463 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000375.txt 2021-06-08 06:55:31,482 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000376.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' 2021-06-08 06:55:31,506 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000376.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-06-08 06:55:31,533 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000376.txt 2021-06-08 06:55:31,553 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000376.txt 2021-06-08 06:55:31,572 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000377.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' 2021-06-08 06:55:31,598 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000377.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-06-08 06:55:31,625 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000377.txt 2021-06-08 06:55:31,645 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000377.txt 2021-06-08 06:55:31,664 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000378.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' 2021-06-08 06:55:31,690 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000378.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-06-08 06:55:31,717 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000378.txt 2021-06-08 06:55:31,739 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000378.txt 2021-06-08 06:55:31,759 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000379.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' 2021-06-08 06:55:31,786 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000379.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-06-08 06:55:31,812 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000379.txt 2021-06-08 06:55:31,834 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000379.txt 2021-06-08 06:55:31,852 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000380.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' 2021-06-08 06:55:31,871 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000380.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-06-08 06:55:31,892 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000380.txt 2021-06-08 06:55:31,910 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000380.txt 2021-06-08 06:55:31,926 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-06-08 06:55:31,938 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:763] 172.23.106.196: No 'sanitizers.log.*' files found 2021-06-08 06:55:31,938 | infra | DEBUG | MainThread | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.196 2021-06-08 06:55:31,940 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.199 with username: root, password: couchbase 2021-06-08 06:55:32,006 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 06:55:32,217 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: ls /etc/ 2021-06-08 06:55:32,276 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 06:55:32,276 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: ls /cygdrive/c/Windows 2021-06-08 06:55:32,289 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: uname -m 2021-06-08 06:55:32,299 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: cat /proc/cpuinfo 2021-06-08 06:55:32,312 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: df -Thl 2021-06-08 06:55:32,328 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: cat /proc/meminfo 2021-06-08 06:55:32,339 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: hostname 2021-06-08 06:55:32,351 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: hostname -d 2021-06-08 06:55:32,364 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-06-08 06:55:32,377 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:744] 172.23.106.199: No crash files found 2021-06-08 06:55:32,397 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-06-08 06:55:32,410 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000333.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' 2021-06-08 06:55:32,434 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000333.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-06-08 06:55:32,461 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000333.txt 2021-06-08 06:55:32,480 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000333.txt 2021-06-08 06:55:32,499 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000334.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' 2021-06-08 06:55:32,523 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000334.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-06-08 06:55:32,551 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000334.txt 2021-06-08 06:55:32,571 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000334.txt 2021-06-08 06:55:32,588 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000335.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' 2021-06-08 06:55:32,614 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000335.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-06-08 06:55:32,641 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000335.txt 2021-06-08 06:55:32,661 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000335.txt 2021-06-08 06:55:32,680 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000336.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' 2021-06-08 06:55:32,704 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000336.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-06-08 06:55:32,732 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000336.txt 2021-06-08 06:55:32,753 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000336.txt 2021-06-08 06:55:32,773 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000337.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' 2021-06-08 06:55:32,799 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000337.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-06-08 06:55:32,828 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000337.txt 2021-06-08 06:55:32,848 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000337.txt 2021-06-08 06:55:32,868 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000338.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' 2021-06-08 06:55:32,892 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000338.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-06-08 06:55:32,920 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000338.txt 2021-06-08 06:55:32,940 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000338.txt 2021-06-08 06:55:32,959 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000339.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' 2021-06-08 06:55:32,986 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000339.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-06-08 06:55:33,016 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000339.txt 2021-06-08 06:55:33,036 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000339.txt 2021-06-08 06:55:33,055 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000340.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' 2021-06-08 06:55:33,081 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000340.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-06-08 06:55:33,108 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000340.txt 2021-06-08 06:55:33,128 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000340.txt 2021-06-08 06:55:33,148 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000341.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' 2021-06-08 06:55:33,174 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000341.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-06-08 06:55:33,201 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000341.txt 2021-06-08 06:55:33,223 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000341.txt 2021-06-08 06:55:33,242 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000342.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' 2021-06-08 06:55:33,267 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000342.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-06-08 06:55:33,296 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000342.txt 2021-06-08 06:55:33,316 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000342.txt 2021-06-08 06:55:33,335 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000343.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' 2021-06-08 06:55:33,364 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000343.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-06-08 06:55:33,391 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000343.txt 2021-06-08 06:55:33,414 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000343.txt 2021-06-08 06:55:33,434 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000344.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' 2021-06-08 06:55:33,461 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000344.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-06-08 06:55:33,492 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000344.txt 2021-06-08 06:55:33,516 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000344.txt 2021-06-08 06:55:33,539 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000345.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' 2021-06-08 06:55:33,565 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000345.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-06-08 06:55:33,595 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000345.txt 2021-06-08 06:55:33,617 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000345.txt 2021-06-08 06:55:33,638 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000346.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' 2021-06-08 06:55:33,665 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000346.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-06-08 06:55:33,694 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000346.txt 2021-06-08 06:55:33,714 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000346.txt 2021-06-08 06:55:33,734 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000347.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' 2021-06-08 06:55:33,762 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000347.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-06-08 06:55:33,792 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000347.txt 2021-06-08 06:55:33,812 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000347.txt 2021-06-08 06:55:33,832 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000348.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' 2021-06-08 06:55:33,858 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000348.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-06-08 06:55:33,888 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000348.txt 2021-06-08 06:55:33,907 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000348.txt 2021-06-08 06:55:33,927 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000349.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' 2021-06-08 06:55:33,953 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000349.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-06-08 06:55:33,982 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000349.txt 2021-06-08 06:55:34,002 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000349.txt 2021-06-08 06:55:34,022 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000350.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' 2021-06-08 06:55:34,048 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000350.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-06-08 06:55:34,078 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000350.txt 2021-06-08 06:55:34,101 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000350.txt 2021-06-08 06:55:34,121 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000351.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' 2021-06-08 06:55:34,148 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000351.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-06-08 06:55:34,177 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000351.txt 2021-06-08 06:55:34,198 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000351.txt 2021-06-08 06:55:34,220 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000352.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' 2021-06-08 06:55:34,236 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000352.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-06-08 06:55:34,253 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000352.txt 2021-06-08 06:55:34,269 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000352.txt 2021-06-08 06:55:34,282 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-06-08 06:55:34,296 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:763] 172.23.106.199: No 'sanitizers.log.*' files found 2021-06-08 06:55:34,296 | infra | DEBUG | MainThread | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.199 2021-06-08 06:55:34,298 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.168 with username: root, password: couchbase 2021-06-08 06:55:34,381 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 06:55:34,615 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: ls /etc/ 2021-06-08 06:55:34,677 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 06:55:34,677 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: ls /cygdrive/c/Windows 2021-06-08 06:55:34,694 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: uname -m 2021-06-08 06:55:34,707 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: cat /proc/cpuinfo 2021-06-08 06:55:34,720 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: df -Thl 2021-06-08 06:55:34,740 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: cat /proc/meminfo 2021-06-08 06:55:34,753 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: hostname 2021-06-08 06:55:34,766 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: hostname -d 2021-06-08 06:55:34,782 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-06-08 06:55:34,799 | test | CRITICAL | MainThread | [basetestcase:check_coredump_exist:735] 172.23.106.168: 5 core dump seen 2021-06-08 06:55:34,799 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: rm -rf /opt/couchbase/var/lib/couchbase/crash/9c1677b3-119a-4713-232865bd-6654fec1.core 2021-06-08 06:55:34,812 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: //opt/couchbase/bin/minidump-2-core /opt/couchbase/var/lib/couchbase/crash/9c1677b3-119a-4713-232865bd-6654fec1.dmp > /opt/couchbase/var/lib/couchbase/crash/9c1677b3-119a-4713-232865bd-6654fec1.core 2021-06-08 06:55:34,836 | test | INFO | MainThread | [basetestcase:get_gdb:677] running gdb --batch /opt/couchbase/bin/ memcached -c /opt/couchbase/var/lib/couchbase/crash/9c1677b3-119a-4713-232865bd-6654fec1.core -ex "bt full" -ex quit 2021-06-08 06:55:34,836 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: gdb --batch /opt/couchbase/bin/ memcached -c /opt/couchbase/var/lib/couchbase/crash/9c1677b3-119a-4713-232865bd-6654fec1.core -ex "bt full" -ex quit 2021-06-08 06:55:34,983 | test | CRITICAL | MainThread | [basetestcase:check_coredump_exist:736] 172.23.106.168: Stack Trace of first crash - 9c1677b3-119a-4713-232865bd-6654fec1.dmp Core was generated by `/opt/couchbase/bin/memcached -C /opt/couchbase/var/lib/couchbase/config/memcach'. #0 0x00007fc4d325a337 in ?? () #0 0x00007fc4d325a337 in ?? () No symbol table info available. #1 0x00007fc4d325ba28 in ?? () No symbol table info available. #2 0x0000000000000020 in ?? () No symbol table info available. #3 0x0000000000000000 in ?? () No symbol table info available. 2021-06-08 06:55:34,983 | test | INFO | MainThread | [basetestcase:get_full_thread_dump:686] running gdb -p `(pidof memcached)` -ex "thread apply all bt" -ex detach -ex quit 2021-06-08 06:55:34,983 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: gdb -p `(pidof memcached)` -ex "thread apply all bt" -ex detach -ex quit Detaching from program: /opt/couchbase/bin/memcached, process 90926 2021-06-08 06:55:36,489 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-06-08 06:55:36,503 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000451.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' 2021-06-08 06:55:36,529 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000451.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-06-08 06:55:36,556 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000451.txt 2021-06-08 06:55:36,576 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000451.txt 2021-06-08 06:55:36,595 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-06-08 06:55:36,608 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:763] 172.23.106.168: No 'sanitizers.log.*' files found 2021-06-08 06:55:36,608 | infra | DEBUG | MainThread | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.168 2021-06-08 06:55:36,609 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.198 with username: root, password: couchbase 2021-06-08 06:55:36,687 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 06:55:36,898 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: ls /etc/ 2021-06-08 06:55:36,959 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 06:55:36,959 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: ls /cygdrive/c/Windows 2021-06-08 06:55:36,973 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: uname -m 2021-06-08 06:55:36,984 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: cat /proc/cpuinfo 2021-06-08 06:55:36,997 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: df -Thl 2021-06-08 06:55:37,015 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: cat /proc/meminfo 2021-06-08 06:55:37,026 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: hostname 2021-06-08 06:55:37,039 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: hostname -d 2021-06-08 06:55:37,053 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-06-08 06:55:37,068 | test | CRITICAL | MainThread | [basetestcase:check_coredump_exist:735] 172.23.106.198: 6 core dump seen 2021-06-08 06:55:37,069 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: rm -rf /opt/couchbase/var/lib/couchbase/crash/00c3dbe0-cb0e-4f27-ecc3e08d-1284017f.core 2021-06-08 06:55:37,081 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: //opt/couchbase/bin/minidump-2-core /opt/couchbase/var/lib/couchbase/crash/00c3dbe0-cb0e-4f27-ecc3e08d-1284017f.dmp > /opt/couchbase/var/lib/couchbase/crash/00c3dbe0-cb0e-4f27-ecc3e08d-1284017f.core 2021-06-08 06:55:37,102 | test | INFO | MainThread | [basetestcase:get_gdb:677] running gdb --batch /opt/couchbase/bin/ memcached -c /opt/couchbase/var/lib/couchbase/crash/00c3dbe0-cb0e-4f27-ecc3e08d-1284017f.core -ex "bt full" -ex quit 2021-06-08 06:55:37,102 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: gdb --batch /opt/couchbase/bin/ memcached -c /opt/couchbase/var/lib/couchbase/crash/00c3dbe0-cb0e-4f27-ecc3e08d-1284017f.core -ex "bt full" -ex quit 2021-06-08 06:55:37,236 | test | CRITICAL | MainThread | [basetestcase:check_coredump_exist:736] 172.23.106.198: Stack Trace of first crash - 00c3dbe0-cb0e-4f27-ecc3e08d-1284017f.dmp Core was generated by `/opt/couchbase/bin/memcached -C /opt/couchbase/var/lib/couchbase/config/memcach'. #0 0x00007f3321a2e337 in ?? () #0 0x00007f3321a2e337 in ?? () No symbol table info available. #1 0x00007f3321a2fa28 in ?? () No symbol table info available. #2 0x0000000000000020 in ?? () No symbol table info available. #3 0x0000000000000000 in ?? () No symbol table info available. 2021-06-08 06:55:37,237 | test | INFO | MainThread | [basetestcase:get_full_thread_dump:686] running gdb -p `(pidof memcached)` -ex "thread apply all bt" -ex detach -ex quit 2021-06-08 06:55:37,237 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: gdb -p `(pidof memcached)` -ex "thread apply all bt" -ex detach -ex quit [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Loaded symbols for /lib64/libpthread.so.0 Reading symbols from /lib64/librt.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/librt.so.1 Reading symbols from /opt/couchbase/bin/../lib/libsnappy.so.1...Reading symbols from /usr/lib/debug/opt/couchbase/lib/libsnappy.so.1.2.0.debug...done. done. Loaded symbols for /opt/couchbase/bin/../lib/libsnappy.so.1 Reading symbols from /opt/couchbase/bin/../lib/libjemalloc.so.2...Reading symbols from /usr/lib/debug/opt/couchbase/lib/libjemalloc.so.2.debug...done. done. Loaded symbols for /opt/couchbase/bin/../lib/libjemalloc.so.2 Reading symbols from /opt/couchbase/bin/../lib/libevent_core-2.1.so.7...Reading symbols from /usr/lib/debug/opt/couchbase/lib/libevent_core-2.1.so.7.0.0.debug...done. done. Loaded symbols for /opt/couchbase/bin/../lib/libevent_core-2.1.so.7 Reading symbols from /opt/couchbase/bin/../lib/libevent_extra-2.1.so.7...Reading symbols from /usr/lib/debug/opt/couchbase/lib/libevent_extra-2.1.so.7.0.0.debug...done. done. Loaded symbols for /opt/couchbase/bin/../lib/libevent_extra-2.1.so.7 Reading symbols from /opt/couchbase/bin/../lib/libevent_pthreads-2.1.so.7...Reading symbols from /usr/lib/debug/opt/couchbase/lib/libevent_pthreads-2.1.so.7.0.0.debug...done. done. Loaded symbols for /opt/couchbase/bin/../lib/libevent_pthreads-2.1.so.7 Reading symbols from /opt/couchbase/bin/../lib/libevent_openssl-2.1.so.7...Reading symbols from /usr/lib/debug/opt/couchbase/lib/libevent_openssl-2.1.so.7.0.0.debug...done. done. Loaded symbols for /opt/couchbase/bin/../lib/libevent_openssl-2.1.so.7 Reading symbols from /opt/couchbase/bin/../lib/libssl.so.1.1...Reading symbols from /usr/lib/debug/opt/couchbase/lib/libssl.so.1.1.debug...(no debugging symbols found)...done. (no debugging symbols found)...done. Loaded symbols for /opt/couchbase/bin/../lib/libssl.so.1.1 Reading symbols from /opt/couchbase/bin/../lib/libcrypto.so.1.1...Reading symbols from /usr/lib/debug/opt/couchbase/lib/libcrypto.so.1.1.debug...(no debugging symbols found)...done. (no debugging symbols found)...done. Loaded symbols for /opt/couchbase/bin/../lib/libcrypto.so.1.1 Reading symbols from /opt/couchbase/bin/../lib/libstdc++.so.6...done. Loaded symbols for /opt/couchbase/bin/../lib/libstdc++.so.6 Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done. Loaded symbols for /lib64/libm.so.6 Reading symbols from /opt/couchbase/bin/../lib/libgcc_s.so.1...done. Loaded symbols for /opt/couchbase/bin/../lib/libgcc_s.so.1 Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done. Loaded symbols for /lib64/libc.so.6 Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/ld-linux-x86-64.so.2 0x00007fa0e0602e63 in epoll_wait () from /lib64/libc.so.6 To enable execution of this file add add-auto-load-safe-path /opt/couchbase/lib/libstdc++.so.6.0.28-gdb.py line to your configuration file "/root/.gdbinit". To completely disable this security protection add set auto-load safe-path / line to your configuration file "/root/.gdbinit". For more information about this security protection see the "Auto-loading safe path" section in the GDB manual. E.g., run from the shell: info "(gdb)Auto-loading safe path" Thread 61 (Thread 0x7fa0dfbff700 (LWP 124899)): #0 0x00007fa0e2aa1da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fa0e21de5a7 in background_thread_sleep (tsdn=, interval=10000000000, info=) at src/background_thread.c:255 #2 background_work_sleep_once (ind=0, info=, tsdn=) at src/background_thread.c:307 #3 background_thread0_work (tsd=0x7fa0dfbee228) at src/background_thread.c:452 #4 background_work (ind=, tsd=0x7fa0dfbee228) at src/background_thread.c:490 #5 background_thread_entry (ind_arg=) at src/background_thread.c:522 #6 0x00007fa0e2a9de65 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fa0e060288d in clone () from /lib64/libc.so.6 Thread 60 (Thread 0x7fa0dee7e700 (LWP 124900)): #0 0x00007fa0e2aa1da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000000000a855a5 in __gthread_cond_timedwait ( __abs_timeout=0x7fa0dee6c760, __mutex=, __cond=0x7fa0df3ca138) at /opt/gcc-10.2.0/include/c++/10.2.0/x86_64-pc-linux-gnu/bits/gthr-default.h:872 #2 __wait_until_impl > > (__lock=..., __lock=..., __atime=, this=0x7fa0df3ca138) at /opt/gcc-10.2.0/include/c++/10.2.0/condition_variable:232 #3 wait_until > > (__atime=..., __lock=..., this=0x7fa0df3ca138) at /opt/gcc-10.2.0/include/c++/10.2.0/condition_variable:141 #4 wait_until >, spdlog::details::mpmc_blocking_queue::dequeue_for:: > (__p=..., __atime=..., __lock=..., this=0x7fa0df3ca138) at /opt/gcc-10.2.0/include/c++/10.2.0/condition_variable:158 #5 wait_for, spdlog::details::mpmc_blocking_queue::dequeue_for:: > ( __rtime=, __rtime=, __p=..., __lock=..., this=0x7fa0df3ca138) at /opt/gcc-10.2.0/include/c++/10.2.0/condition_variable:185 #6 dequeue_for (wait_duration=..., popped_item=..., this=0x7fa0df3ca110) at /home/couchbase/jenkins/workspace/couchbase-server-unix/third_party/spdlog/include/spdlog/details/mpmc_blocking_q.h:61 #7 spdlog::details::thread_pool::process_next_msg_ ( this=this@entry=0x7fa0df3ca110) at /home/couchbase/jenkins/workspace/couchbase-server-unix/third_party/spdlog/include/spdlog/details/thread_pool.h:193 #8 0x0000000000a85bc0 in spdlog::details::thread_pool::worker_loop_ ( this=0x7fa0df3ca110) at /home/couchbase/jenkins/workspace/couchbase-server-unix/third_party/spdlog/include/spdlog/details/thread_pool.h:184 #9 0x00007fa0e0eb9d40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #10 0x00007fa0e2a9de65 in start_thread () from /lib64/libpthread.so.0 #11 0x00007fa0e060288d in clone () from /lib64/libc.so.6 Thread 59 (Thread 0x7fa0de67d700 (LWP 124901)): #0 0x00007fa0e2aa1da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000000000a84d5b in __gthread_cond_timedwait ( __abs_timeout=0x7fa0de66b9b0, __mutex=, __cond=) at /opt/gcc-10.2.0/include/c++/10.2.0/x86_64-pc-linux-gnu/bits/gthr-default.h:872 #2 __wait_until_impl > > (__lock=..., __lock=..., __atime=, this=) at /opt/gcc-10.2.0/include/c++/10.2.0/condition_variable:232 #3 wait_until > > (__atime=..., __lock=..., this=) at /opt/gcc-10.2.0/include/c++/10.2.0/condition_variable:141 #4 wait_until >, spdlog::details::periodic_worker::periodic_worker(const std::function&, std::chrono::seconds):::: > (__p=..., __atime=..., __lock=..., this=) at /opt/gcc-10.2.0/include/c++/10.2.0/condition_variable:158 #5 wait_for, spdlog::details::periodic_worker::periodic_worker(const std::function&, std::chrono::seconds):::: > (__rtime=..., __rtime=..., __p=..., __lock=..., this=) at /opt/gcc-10.2.0/include/c++/10.2.0/condition_variable:185 #6 operator() (__closure=0x7fa0df3cb648) at /home/couchbase/jenkins/workspace/couchbase-server-unix/third_party/spdlog/include/spdlog/details/periodic_worker.h:38 #7 __invoke_impl&, std::chrono::seconds):: > ( __f=) at /opt/gcc-10.2.0/include/c++/10.2.0/bits/invoke.h:60 #8 __invoke&, std::chrono::seconds):: > ( __fn=) at /opt/gcc-10.2.0/include/c++/10.2.0/bits/invoke.h:95 #9 _M_invoke<0> (this=0x7fa0df3cb648) at /opt/gcc-10.2.0/include/c++/10.2.0/thread:264 #10 operator() (this=0x7fa0df3cb648) at /opt/gcc-10.2.0/include/c++/10.2.0/thread:271 #11 _ZNSt6thread11_State_implINS_8_InvokerISt5tupleIJZN6spdlog7details15periodic_workerC4ERKSt8functionIFvvEENSt6chrono8durationIlSt5ratioILl1ELl1EEEEEUlvE_EEEEE6_M_runEv (this=0x7fa0df3cb640) at /opt/gcc-10.2.0/include/c++/10.2.0/thread:215 #12 0x00007fa0e0eb9d40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #13 0x00007fa0e2a9de65 in start_thread () from /lib64/libpthread.so.0 #14 0x00007fa0e060288d in clone () from /lib64/libc.so.6 Thread 58 (Thread 0x7fa0dde7c700 (LWP 124902)): #0 0x00007fa0e05f7bed in poll () from /lib64/libc.so.6 #1 0x000000000058a841 in get_command (buffsize=80, buffer=0x7fa0dde6a990 "@\206\r\343\240\177") at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/stdin_check.cc:57 #2 check_stdin_thread (arg=) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/stdin_check.cc:98 #3 0x0000000000aab9e9 in run (this=0x7fa0df3c61b0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #4 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #5 0x00007fa0e2a9de65 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fa0e060288d in clone () from /lib64/libc.so.6 Thread 57 (Thread 0x7fa0dd67b700 (LWP 124903)): #0 0x00007fa0e2aa19f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00000000008c9e4b in consume_socket (thread_index=, sp=0x12e90e0, ctx=0x12e7bc0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/prometheus-cpp/prometheus-cpp-prefix/src/prometheus-cpp/3rdparty/civetweb/src/civetweb.c:18846 #2 worker_thread_run (conn=0x12e87f0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/prometheus-cpp/prometheus-cpp-prefix/src/prometheus-cpp/3rdparty/civetweb/src/civetweb.c:18987 #3 worker_thread (thread_func_param=0x12e87f0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/prometheus-cpp/prometheus-cpp-prefix/src/prometheus-cpp/3rdparty/civetweb/src/civetweb.c:19145 #4 0x00007fa0e2a9de65 in start_thread () from /lib64/libpthread.so.0 #5 0x00007fa0e060288d in clone () from /lib64/libc.so.6 Thread 56 (Thread 0x7fa0dce7a700 (LWP 124904)): #0 0x00007fa0e2aa19f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00000000008c9e4b in consume_socket (thread_index=, sp=0x12e9ac8, ctx=0x12e7bc0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/prometheus-cpp/prometheus-cpp-prefix/src/prometheus-cpp/3rdparty/civetweb/src/civetweb.c:18846 #2 worker_thread_run (conn=0x12e91d8) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/prometheus-cpp/prometheus-cpp-prefix/src/prometheus-cpp/3rdparty/civetweb/src/civetweb.c:18987 #3 worker_thread (thread_func_param=0x12e91d8) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/prometheus-cpp/prometheus-cpp-prefix/src/prometheus-cpp/3rdparty/civetweb/src/civetweb.c:19145 #4 0x00007fa0e2a9de65 in start_thread () from /lib64/libpthread.so.0 #5 0x00007fa0e060288d in clone () from /lib64/libc.so.6 Thread 55 (Thread 0x7fa0d7fff700 (LWP 124905)): #0 0x00007fa0e05f7bed in poll () from /lib64/libc.so.6 #1 0x00000000008c0db8 in master_thread_run (ctx=0x12e7bc0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/prometheus-cpp/prometheus-cpp-prefix/src/prometheus-cpp/3rdparty/civetweb/src/civetweb.c:19291 #2 master_thread (thread_func_param=0x12e7bc0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/prometheus-cpp/prometheus-cpp-prefix/src/prometheus-cpp/3rdparty/civetweb/src/civetweb.c:19386 #3 0x00007fa0e2a9de65 in start_thread () from /lib64/libpthread.so.0 #4 0x00007fa0e060288d in clone () from /lib64/libc.so.6 Thread 54 (Thread 0x7fa0d77fe700 (LWP 124906)): #0 0x00007fa0e2aa1da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00000000005bb7d6 in ExternalAuthManagerThread::run() () at /opt/gcc-10.2.0/include/c++/10.2.0/x86_64-pc-linux-gnu/bits/gthr-default.h:872 #2 0x0000000000ab4cc3 in Couchbase::Thread::thread_entry() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/thread.cc:38 #3 0x0000000000aab9e9 in run (this=0x7fa0df3c7bc0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #4 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #5 0x00007fa0e2a9de65 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fa0e060288d in clone () from /lib64/libc.so.6 Thread 53 (Thread 0x7fa0d6ffd700 (LWP 124907)): #0 0x00007fa0e2aa1da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000000000629ee7 in AuditImpl::consume_events() () at /opt/gcc-10.2.0/include/c++/10.2.0/x86_64-pc-linux-gnu/bits/gthr-default.h:872 #2 0x0000000000aab9e9 in run (this=0x7fa0df3c6660) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #3 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #4 0x00007fa0e2a9de65 in start_thread () from /lib64/libpthread.so.0 #5 0x00007fa0e060288d in clone () from /lib64/libc.so.6 Thread 52 (Thread 0x7fa0d67fc700 (LWP 124908)): #0 0x00007fa0e0602e63 in epoll_wait () from /lib64/libc.so.6 #1 0x00007fa0e1fa53c5 in epoll_dispatch (base=0x7fa0df3dd300, tv=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/epoll.c:465 #2 0x00007fa0e1f9c376 in event_base_loop (base=0x7fa0df3dd300, flags=flags@entry=0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:1998 #3 0x000000000058adcb in worker_libevent (arg=0x7fa0df137000) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/thread.cc:202 #4 0x0000000000aab9e9 in run (this=0x7fa0df3e3b60) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #5 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #6 0x00007fa0e2a9de65 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fa0e060288d in clone () from /lib64/libc.so.6 Thread 51 (Thread 0x7fa0d5ffb700 (LWP 124909)): #0 0x00007fa0e0602e63 in epoll_wait () from /lib64/libc.so.6 #1 0x00007fa0e1fa53c5 in epoll_dispatch (base=0x7fa0df3dd600, tv=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/epoll.c:465 #2 0x00007fa0e1f9c376 in event_base_loop (base=0x7fa0df3dd600, flags=flags@entry=0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:1998 #3 0x000000000058adcb in worker_libevent (arg=0x7fa0df137aa8) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/thread.cc:202 #4 0x0000000000aab9e9 in run (this=0x7fa0df3e3920) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #5 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #6 0x00007fa0e2a9de65 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fa0e060288d in clone () from /lib64/libc.so.6 Thread 50 (Thread 0x7fa0d57fa700 (LWP 124910)): #0 0x00007fa0e0602e63 in epoll_wait () from /lib64/libc.so.6 #1 0x00007fa0e1fa53c5 in epoll_dispatch (base=0x7fa0df3dd900, tv=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/epoll.c:465 #2 0x00007fa0e1f9c376 in event_base_loop (base=0x7fa0df3dd900, flags=flags@entry=0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:1998 #3 0x000000000058adcb in worker_libevent (arg=0x7fa0df138550) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/thread.cc:202 #4 0x0000000000aab9e9 in run (this=0x7fa0df3e3b00) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #5 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #6 0x00007fa0e2a9de65 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fa0e060288d in clone () from /lib64/libc.so.6 Thread 49 (Thread 0x7fa0d4ff9700 (LWP 124911)): #0 0x00007fa0e0602e63 in epoll_wait () from /lib64/libc.so.6 #1 0x00007fa0e1fa53c5 in epoll_dispatch (base=0x7fa0df3ddc00, tv=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/epoll.c:465 #2 0x00007fa0e1f9c376 in event_base_loop (base=0x7fa0df3ddc00, flags=flags@entry=0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:1998 #3 0x000000000058adcb in worker_libevent (arg=0x7fa0df138ff8) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/thread.cc:202 #4 0x0000000000aab9e9 in run (this=0x7fa0df3e3c20) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #5 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #6 0x00007fa0e2a9de65 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fa0e060288d in clone () from /lib64/libc.so.6 Thread 48 (Thread 0x7fa0cffff700 (LWP 124912)): #0 0x00007fa0e0602e63 in epoll_wait () from /lib64/libc.so.6 #1 0x00007fa0e1fa53c5 in epoll_dispatch (base=0x7fa0df3ddf00, tv=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/epoll.c:465 #2 0x00007fa0e1f9c376 in event_base_loop (base=0x7fa0df3ddf00, flags=flags@entry=0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:1998 #3 0x000000000058adcb in worker_libevent (arg=0x7fa0df139aa0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/thread.cc:202 #4 0x0000000000aab9e9 in run (this=0x7fa0df3e3aa0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #5 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #6 0x00007fa0e2a9de65 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fa0e060288d in clone () from /lib64/libc.so.6 Thread 47 (Thread 0x7fa0cf7fe700 (LWP 124913)): #0 0x00007fa0e0602e63 in epoll_wait () from /lib64/libc.so.6 #1 0x00007fa0e1fa53c5 in epoll_dispatch (base=0x7fa0df3de200, tv=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/epoll.c:465 #2 0x00007fa0e1f9c376 in event_base_loop (base=0x7fa0df3de200, flags=flags@entry=0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:1998 #3 0x000000000058adcb in worker_libevent (arg=0x7fa0df13a548) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/thread.cc:202 #4 0x0000000000aab9e9 in run (this=0x7fa0df3e3c80) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #5 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #6 0x00007fa0e2a9de65 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fa0e060288d in clone () from /lib64/libc.so.6 Thread 46 (Thread 0x7fa0ceffd700 (LWP 124914)): #0 0x00007fa0e0602e63 in epoll_wait () from /lib64/libc.so.6 #1 0x00007fa0e1fa53c5 in epoll_dispatch (base=0x7fa0df3de500, tv=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/epoll.c:465 #2 0x00007fa0e1f9c376 in event_base_loop (base=0x7fa0df3de500, flags=flags@entry=0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:1998 #3 0x000000000058adcb in worker_libevent (arg=0x7fa0df13aff0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/thread.cc:202 #4 0x0000000000aab9e9 in run (this=0x7fa0df3e3cb0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #5 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #6 0x00007fa0e2a9de65 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fa0e060288d in clone () from /lib64/libc.so.6 Thread 45 (Thread 0x7fa0ce7fc700 (LWP 124915)): #0 0x00007fa0e2aa19f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fa0e0eb48bc in __gthread_cond_wait (__mutex=, __cond=__cond@entry=0x7fa0dc46b158) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=this@entry=0x7fa0dc46b158, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000005ce701 in Executor::run() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/executor.cc:38 #4 0x0000000000ab4cc3 in Couchbase::Thread::thread_entry() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/thread.cc:38 #5 0x0000000000aab9e9 in run (this=0x7fa0df3e3a70) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007fa0e2a9de65 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fa0e060288d in clone () from /lib64/libc.so.6 Thread 44 (Thread 0x7fa0cdffb700 (LWP 124916)): #0 0x00007fa0e2aa19f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fa0e0eb48bc in __gthread_cond_wait (__mutex=, __cond=__cond@entry=0x7fa0dc46b318) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=this@entry=0x7fa0dc46b318, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000005ce701 in Executor::run() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/executor.cc:38 #4 0x0000000000ab4cc3 in Couchbase::Thread::thread_entry() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/thread.cc:38 #5 0x0000000000aab9e9 in run (this=0x7fa0df3e3c50) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007fa0e2a9de65 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fa0e060288d in clone () from /lib64/libc.so.6 Thread 43 (Thread 0x7fa0cd7fa700 (LWP 124917)): #0 0x00007fa0e2aa19f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fa0e0eb48bc in __gthread_cond_wait (__mutex=, __cond=__cond@entry=0x7fa0dc46b4d8) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=this@entry=0x7fa0dc46b4d8, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000005ce701 in Executor::run() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/executor.cc:38 #4 0x0000000000ab4cc3 in Couchbase::Thread::thread_entry() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/thread.cc:38 #5 0x0000000000aab9e9 in run (this=0x7fa0df3e3d70) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007fa0e2a9de65 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fa0e060288d in clone () from /lib64/libc.so.6 Thread 42 (Thread 0x7fa0ccff9700 (LWP 124918)): #0 0x00007fa0e2aa19f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fa0e0eb48bc in __gthread_cond_wait (__mutex=, __cond=__cond@entry=0x7fa0dc46b698) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=this@entry=0x7fa0dc46b698, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000005ce701 in Executor::run() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/executor.cc:38 #4 0x0000000000ab4cc3 in Couchbase::Thread::thread_entry() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/thread.cc:38 #5 0x0000000000aab9e9 in run (this=0x7fa0df3e3bf0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007fa0e2a9de65 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fa0e060288d in clone () from /lib64/libc.so.6 Thread 41 (Thread 0x7fa0cc7f8700 (LWP 124919)): #0 0x00007fa0e2aa19f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fa0e0eb48bc in __gthread_cond_wait (__mutex=, __cond=__cond@entry=0x7fa0dc46b858) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=this@entry=0x7fa0dc46b858, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000005ce701 in Executor::run() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/executor.cc:38 #4 0x0000000000ab4cc3 in Couchbase::Thread::thread_entry() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/thread.cc:38 #5 0x0000000000aab9e9 in run (this=0x7fa0df3e3dd0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007fa0e2a9de65 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fa0e060288d in clone () from /lib64/libc.so.6 Thread 40 (Thread 0x7fa0cbff7700 (LWP 124920)): #0 0x00007fa0e2aa19f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fa0e0eb48bc in __gthread_cond_wait (__mutex=, __cond=__cond@entry=0x7fa0dc46ba18) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=this@entry=0x7fa0dc46ba18, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000005ce701 in Executor::run() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/executor.cc:38 #4 0x0000000000ab4cc3 in Couchbase::Thread::thread_entry() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/thread.cc:38 #5 0x0000000000aab9e9 in run (this=0x7fa0df3e3b90) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007fa0e2a9de65 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fa0e060288d in clone () from /lib64/libc.so.6 Thread 39 (Thread 0x7fa0cb7f6700 (LWP 124921)): #0 0x00007fa0e2aa19f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fa0e0eb48bc in __gthread_cond_wait (__mutex=, __cond=__cond@entry=0x7fa0dc46bbd8) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=this@entry=0x7fa0dc46bbd8, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000005ce701 in Executor::run() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/executor.cc:38 #4 0x0000000000ab4cc3 in Couchbase::Thread::thread_entry() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/thread.cc:38 #5 0x0000000000aab9e9 in run (this=0x7fa0df3e3bc0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007fa0e2a9de65 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fa0e060288d in clone () from /lib64/libc.so.6 Thread 38 (Thread 0x7fa0c8fff700 (LWP 124924)): #0 0x00007fa0e2aa1da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fa0e21dde51 in background_thread_sleep (tsdn=, interval=10000000000, info=) at src/background_thread.c:255 #2 background_work_sleep_once (ind=, info=, tsdn=) at src/background_thread.c:307 #3 background_work (ind=, tsd=0x7fa0c8fee228) at src/background_thread.c:497 #4 background_thread_entry (ind_arg=) at src/background_thread.c:522 #5 0x00007fa0e2a9de65 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fa0e060288d in clone () from /lib64/libc.so.6 Thread 37 (Thread 0x7fa0b3fff700 (LWP 124925)): #0 0x00007fa0e2aa4f83 in pread64 () from /lib64/libpthread.so.0 #1 0x000000000098870b in magma::BasicFile::Read(long, unsigned long, char*, unsigned long*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/file/file_impl_linux.cc:167 #2 0x000000000098bb27 in magma::BufferedFileReader::Read(magma::File*, long, unsigned long, magma::Slice&, unsigned long&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/rw_file.cc:270 #3 0x0000000000995c75 in magma::ReadAndTransform(magma::File*, long, unsigned long, std::function >&, unsigned long&)>, std::unique_ptr >&, unsigned long&, unsigned long&, magma::BufferedFileReader*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/block_cache.cc:39 #4 0x000000000098f6de in magma::ManagedFile::Read(long, unsigned long, std::shared_ptr&, unsigned long&, unsigned long&, magma::CacheReadMode, std::function >&, unsigned long&)>, magma::BufferedFileReader*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/file_manager.cc:66 #5 0x00000000009ae7f8 in magma::SSTable::readBlock(magma::BlockPointer const&, magma::Block*, magma::ReaderStats&, bool, magma::BufferedFileReader*) () at /opt/gcc-10.2.0/include/c++/10.2.0/new:175 #6 0x00000000009b051b in magma::SSTableIterator::initIterators(magma::Slice const&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/lsm/sstable/sstable.cc:641 #7 0x00000000009bbeee in magma::JoinIterator::Next (this=0x7fa008b8e980) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/lsm/iterator.cc:160 #8 0x00000000009bd441 in Next (this=0x7f9f80978080) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/lsm/iterator.cc:78 #9 magma::MergeIterator::Next() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/lsm/iterator.cc:74 #10 0x00000000009ced98 in magma::KVReader::ReadKVs (this=0x7f9fd3229c40) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/lsm/common.cc:68 #11 0x00000000009bcda7 in magma::SnapshotIterator::readKV() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/lsm/iterator.cc:301 #12 0x00000000009bcfc5 in magma::SnapshotIterator::Next (this=) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/lsm/iterator.cc:244 #13 0x000000000094bde2 in Next (this=0x7fa02e7e2380) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/kvstore/iterator.cc:154 #14 magma::BySeqIterator::Next() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/kvstore/iterator.cc:152 #15 0x000000000084c311 in MagmaKVStore::scan(BySeqnoScanContext&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/magma-kvstore/magma-kvstore.cc:1525 #16 0x000000000070ec38 in Warmup::loadKVPairsforShard(unsigned short) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/warmup.cc:1585 #17 0x000000000071b9c1 in WarmupLoadingKVPairs::run (this=0x7fa03c991a90) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/warmup.cc:470 #18 0x00000000006a0567 in GlobalTask::execute() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/globaltask.cc:68 #19 0x000000000069b6b0 in FollyExecutorPool::TaskProxy::scheduleViaCPUPool()::{lambda()#1}::operator()() const (__closure=0x7fa0b3fed830) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/folly_executorpool.cc:190 #20 0x000000000069b6b0 in FollyExecutorPool::TaskProxy::timeoutExpired ( this=) #21 0x0000000000b0a3b0 in operator() (this=0x7fa0b3fed830) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:416 #22 folly::ThreadPoolExecutor::runTask(std::shared_ptr const&, folly::ThreadPoolExecutor::Task&&) (this=0x7fa0d4199c00, thread=..., task=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/ThreadPoolExecutor.cpp:97 #23 0x0000000000af216a in folly::CPUThreadPoolExecutor::threadRun ( this=0x7fa0d4199c00, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:265 #24 0x0000000000b0d369 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> ( __t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #25 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #26 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #27 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #28 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #29 0x0000000000699d4d in operator() (this=0x7fa0d4034b80) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/folly_executorpool.cc:48 #30 operator() (__closure=0x7fa0d4034b80) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/folly_executorpool.cc:48 #31 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #32 0x00007fa0e0eb9d40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #33 0x00007fa0e2a9de65 in start_thread () from /lib64/libpthread.so.0 #34 0x00007fa0e060288d in clone () from /lib64/libc.so.6 Thread 36 (Thread 0x7fa0b37fe700 (LWP 124926)): #0 0x00007fa0e2aa4f83 in pread64 () from /lib64/libpthread.so.0 #1 0x000000000098870b in magma::BasicFile::Read(long, unsigned long, char*, unsigned long*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/file/file_impl_linux.cc:167 #2 0x000000000098bb27 in magma::BufferedFileReader::Read(magma::File*, long, unsigned long, magma::Slice&, unsigned long&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/rw_file.cc:270 #3 0x0000000000995c75 in magma::ReadAndTransform(magma::File*, long, unsigned long, std::function >&, unsigned long&)>, std::unique_ptr >&, unsigned long&, unsigned long&, magma::BufferedFileReader*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/block_cache.cc:39 #4 0x000000000098f6de in magma::ManagedFile::Read(long, unsigned long, std::shared_ptr&, unsigned long&, unsigned long&, magma::CacheReadMode, std::function >&, unsigned long&)>, magma::BufferedFileReader*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/file_manager.cc:66 #5 0x00000000009ae7f8 in magma::SSTable::readBlock(magma::BlockPointer const&, magma::Block*, magma::ReaderStats&, bool, magma::BufferedFileReader*) () at /opt/gcc-10.2.0/include/c++/10.2.0/new:175 #6 0x00000000009b051b in magma::SSTableIterator::initIterators(magma::Slice const&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/lsm/sstable/sstable.cc:641 #7 0x00000000009bbeee in magma::JoinIterator::Next (this=0x7f9f80979340) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/lsm/iterator.cc:160 #8 0x00000000009bd441 in Next (this=0x7f9f80978740) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/lsm/iterator.cc:78 #9 magma::MergeIterator::Next() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/lsm/iterator.cc:74 #10 0x00000000009ced98 in magma::KVReader::ReadKVs (this=0x7f9f80979400) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/lsm/common.cc:68 #11 0x00000000009bcda7 in magma::SnapshotIterator::readKV() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/lsm/iterator.cc:301 #12 0x00000000009bcfc5 in magma::SnapshotIterator::Next (this=) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/lsm/iterator.cc:244 #13 0x000000000094bde2 in Next (this=0x7fa03cae6900) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/kvstore/iterator.cc:154 #14 magma::BySeqIterator::Next() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/kvstore/iterator.cc:152 #15 0x000000000084c311 in MagmaKVStore::scan(BySeqnoScanContext&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/magma-kvstore/magma-kvstore.cc:1525 #16 0x000000000070ec38 in Warmup::loadKVPairsforShard(unsigned short) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/warmup.cc:1585 #17 0x000000000071b9c1 in WarmupLoadingKVPairs::run (this=0x7fa0c857ad70) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/warmup.cc:470 #18 0x00000000006a0567 in GlobalTask::execute() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/globaltask.cc:68 #19 0x000000000069b6b0 in FollyExecutorPool::TaskProxy::scheduleViaCPUPool()::{lambda()#1}::operator()() const (__closure=0x7fa0b37ec830) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/folly_executorpool.cc:190 #20 0x000000000069b6b0 in FollyExecutorPool::TaskProxy::timeoutExpired ( this=) #21 0x0000000000b0a3b0 in operator() (this=0x7fa0b37ec830) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:416 #22 folly::ThreadPoolExecutor::runTask(std::shared_ptr const&, folly::ThreadPoolExecutor::Task&&) (this=0x7fa0d4199c00, thread=..., task=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/ThreadPoolExecutor.cpp:97 #23 0x0000000000af216a in folly::CPUThreadPoolExecutor::threadRun ( this=0x7fa0d4199c00, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:265 #24 0x0000000000b0d369 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> ( __t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #25 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #26 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #27 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #28 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #29 0x0000000000699d4d in operator() (this=0x7fa0d4034c80) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/folly_executorpool.cc:48 #30 operator() (__closure=0x7fa0d4034c80) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/folly_executorpool.cc:48 #31 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #32 0x00007fa0e0eb9d40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #33 0x00007fa0e2a9de65 in start_thread () from /lib64/libpthread.so.0 #34 0x00007fa0e060288d in clone () from /lib64/libc.so.6 Thread 35 (Thread 0x7fa0b2ffd700 (LWP 124927)): #0 0x000000000077e114 in store (__m=, __i=, this=) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/blob.cc:20 #1 RefCount (value=, this=) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/atomic.h:117 #2 RCValue (this=) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/atomic.h:141 #3 Blob (len=, start=, this=) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/blob.cc:37 #4 Blob::New ( start=start@entry=0x7fa02ce98e53 "{\"mutated\":0,\"name\":\"james\",\"body\":\"n6MN4BHUqHuA0paNDuD0ankJIexJDGoSPhKY3vQw9VUwlxAmBMRc9EbYNfF7A5ZQQX2yPuiZrD9jmkFbRHLd7Td9fW2235Rj3Qyz3Rnv23YesP6yu82eU5ke4dMIPqKxSuqP12Sz9SZ75wuw6kuwp1BCwXUs5gO23wiY"..., len=len@entry=2022) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/blob.cc:20 #5 0x00000000006aa6ef in setData (nb=2022, dta=0x7fa02ce98e53 "{\"mutated\":0,\"name\":\"james\",\"body\":\"n6MN4BHUqHuA0paNDuD0ankJIexJDGoSPhKY3vQw9VUwlxAmBMRc9EbYNfF7A5ZQQX2yPuiZrD9jmkFbRHLd7Td9fW2235Rj3Qyz3Rnv23YesP6yu82eU5ke4dMIPqKxSuqP12Sz9SZ75wuw6kuwp1BCwXUs5gO23wiY"..., this=0x7fa0b02a1b00) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/item.h:596 #6 Item::Item(DocKey const&, unsigned int, long, void const*, unsigned long, unsigned char, unsigned long, long, Vbid, unsigned long, unsigned char) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/item.cc:80 #7 0x0000000000842279 in MagmaKVStore::makeItem(Vbid, magma::Slice const&, magma::Slice const&, magma::Slice const&, ValueFilter) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/magma-kvstore/magma-kvstore.cc:1032 #8 0x000000000084c210 in MagmaKVStore::scan(BySeqnoScanContext&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/magma-kvstore/magma-kvstore.cc:1617 #9 0x000000000070ec38 in Warmup::loadKVPairsforShard(unsigned short) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/warmup.cc:1585 #10 0x000000000071b9c1 in WarmupLoadingKVPairs::run (this=0x7fa0c8579bf0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/warmup.cc:470 #11 0x00000000006a0567 in GlobalTask::execute() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/globaltask.cc:68 #12 0x000000000069b6b0 in FollyExecutorPool::TaskProxy::scheduleViaCPUPool()::{lambda()#1}::operator()() const (__closure=0x7fa0b2feb830) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/folly_executorpool.cc:190 #13 0x000000000069b6b0 in FollyExecutorPool::TaskProxy::timeoutExpired ( this=) #14 0x0000000000b0a3b0 in operator() (this=0x7fa0b2feb830) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:416 #15 folly::ThreadPoolExecutor::runTask(std::shared_ptr const&, folly::ThreadPoolExecutor::Task&&) (this=0x7fa0d4199c00, thread=..., task=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/ThreadPoolExecutor.cpp:97 #16 0x0000000000af216a in folly::CPUThreadPoolExecutor::threadRun ( this=0x7fa0d4199c00, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:265 #17 0x0000000000b0d369 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> ( __t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #18 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #19 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #20 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #21 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #22 0x0000000000699d4d in operator() (this=0x7fa0d4034cc0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/folly_executorpool.cc:48 #23 operator() (__closure=0x7fa0d4034cc0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/folly_executorpool.cc:48 #24 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #25 0x00007fa0e0eb9d40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #26 0x00007fa0e2a9de65 in start_thread () from /lib64/libpthread.so.0 #27 0x00007fa0e060288d in clone () from /lib64/libc.so.6 Thread 34 (Thread 0x7fa0b0fd7700 (LWP 124928)): #0 0x00007fa0e2aa4f83 in pread64 () from /lib64/libpthread.so.0 #1 0x000000000098870b in magma::BasicFile::Read(long, unsigned long, char*, unsigned long*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/file/file_impl_linux.cc:167 #2 0x000000000098bb27 in magma::BufferedFileReader::Read(magma::File*, long, unsigned long, magma::Slice&, unsigned long&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/rw_file.cc:270 #3 0x0000000000995c75 in magma::ReadAndTransform(magma::File*, long, unsigned long, std::function >&, unsigned long&)>, std::unique_ptr >&, unsigned long&, unsigned long&, magma::BufferedFileReader*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/block_cache.cc:39 #4 0x000000000098f6de in magma::ManagedFile::Read(long, unsigned long, std::shared_ptr&, unsigned long&, unsigned long&, magma::CacheReadMode, std::function >&, unsigned long&)>, magma::BufferedFileReader*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/file_manager.cc:66 #5 0x00000000009ae7f8 in magma::SSTable::readBlock(magma::BlockPointer const&, magma::Block*, magma::ReaderStats&, bool, magma::BufferedFileReader*) () at /opt/gcc-10.2.0/include/c++/10.2.0/new:175 #6 0x00000000009b051b in magma::SSTableIterator::initIterators(magma::Slice const&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/lsm/sstable/sstable.cc:641 #7 0x00000000009bbeee in magma::JoinIterator::Next (this=0x7f9f80978d40) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/lsm/iterator.cc:160 #8 0x00000000009bd441 in Next (this=0x7f9fd3228a40) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/lsm/iterator.cc:78 #9 magma::MergeIterator::Next() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/lsm/iterator.cc:74 #10 0x00000000009ced98 in magma::KVReader::ReadKVs (this=0x7f9f80978e00) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/lsm/common.cc:68 #11 0x00000000009bcda7 in magma::SnapshotIterator::readKV() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/lsm/iterator.cc:301 #12 0x00000000009bcfc5 in magma::SnapshotIterator::Next (this=) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/lsm/iterator.cc:244 #13 0x000000000094bde2 in Next (this=0x7fa031cd9600) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/kvstore/iterator.cc:154 #14 magma::BySeqIterator::Next() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/kvstore/iterator.cc:152 #15 0x000000000084c311 in MagmaKVStore::scan(BySeqnoScanContext&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/magma-kvstore/magma-kvstore.cc:1525 #16 0x000000000070ec38 in Warmup::loadKVPairsforShard(unsigned short) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/warmup.cc:1585 #17 0x000000000071b9c1 in WarmupLoadingKVPairs::run (this=0x7fa0c873d0d0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/warmup.cc:470 #18 0x00000000006a0567 in GlobalTask::execute() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/globaltask.cc:68 #19 0x000000000069b6b0 in FollyExecutorPool::TaskProxy::scheduleViaCPUPool()::{lambda()#1}::operator()() const (__closure=0x7fa0b0fc5830) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/folly_executorpool.cc:190 #20 0x000000000069b6b0 in FollyExecutorPool::TaskProxy::timeoutExpired ( this=) #21 0x0000000000b0a3b0 in operator() (this=0x7fa0b0fc5830) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:416 #22 folly::ThreadPoolExecutor::runTask(std::shared_ptr const&, folly::ThreadPoolExecutor::Task&&) (this=0x7fa0d4199c00, thread=..., task=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/ThreadPoolExecutor.cpp:97 #23 0x0000000000af216a in folly::CPUThreadPoolExecutor::threadRun ( this=0x7fa0d4199c00, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:265 #24 0x0000000000b0d369 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> ( __t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #25 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #26 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #27 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #28 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #29 0x0000000000699d4d in operator() (this=0x7fa0d4034d40) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/folly_executorpool.cc:48 #30 operator() (__closure=0x7fa0d4034d40) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/folly_executorpool.cc:48 #31 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #32 0x00007fa0e0eb9d40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #33 0x00007fa0e2a9de65 in start_thread () from /lib64/libpthread.so.0 #34 0x00007fa0e060288d in clone () from /lib64/libc.so.6 Thread 33 (Thread 0x7fa07bfff700 (LWP 124929)): #0 0x00007fa0e2aa4f83 in pread64 () from /lib64/libpthread.so.0 #1 0x000000000098870b in magma::BasicFile::Read(long, unsigned long, char*, unsigned long*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/file/file_impl_linux.cc:167 #2 0x000000000098bb27 in magma::BufferedFileReader::Read(magma::File*, long, unsigned long, magma::Slice&, unsigned long&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/rw_file.cc:270 #3 0x0000000000995c75 in magma::ReadAndTransform(magma::File*, long, unsigned long, std::function >&, unsigned long&)>, std::unique_ptr >&, unsigned long&, unsigned long&, magma::BufferedFileReader*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/block_cache.cc:39 #4 0x000000000098f6de in magma::ManagedFile::Read(long, unsigned long, std::shared_ptr&, unsigned long&, unsigned long&, magma::CacheReadMode, std::function >&, unsigned long&)>, magma::BufferedFileReader*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/file_manager.cc:66 #5 0x00000000009ae7f8 in magma::SSTable::readBlock(magma::BlockPointer const&, magma::Block*, magma::ReaderStats&, bool, magma::BufferedFileReader*) () at /opt/gcc-10.2.0/include/c++/10.2.0/new:175 #6 0x00000000009b051b in magma::SSTableIterator::initIterators(magma::Slice const&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/lsm/sstable/sstable.cc:641 #7 0x00000000009b17dc in magma::SSTableIterator::Seek (this=0x7fa02c5ad700, key=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/lsm/sstable/sstable.cc:685 #8 0x00000000009bd441 in Next (this=0x7f9fd3229340) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/lsm/iterator.cc:78 #9 magma::MergeIterator::Next() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/lsm/iterator.cc:74 #10 0x00000000009ced98 in magma::KVReader::ReadKVs (this=0x7f9f80977240) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/lsm/common.cc:68 #11 0x00000000009bcda7 in magma::SnapshotIterator::readKV() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/lsm/iterator.cc:301 #12 0x00000000009bcfc5 in magma::SnapshotIterator::Next (this=) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/lsm/iterator.cc:244 #13 0x000000000094bde2 in Next (this=0x7fa0339c2e80) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/kvstore/iterator.cc:154 #14 magma::BySeqIterator::Next() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/kvstore/iterator.cc:152 #15 0x000000000084c311 in MagmaKVStore::scan(BySeqnoScanContext&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/magma-kvstore/magma-kvstore.cc:1525 #16 0x000000000070ec38 in Warmup::loadKVPairsforShard(unsigned short) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/warmup.cc:1585 #17 0x000000000071b9c1 in WarmupLoadingKVPairs::run (this=0x7fa02c3e90f0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/warmup.cc:470 #18 0x00000000006a0567 in GlobalTask::execute() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/globaltask.cc:68 #19 0x000000000069b6b0 in FollyExecutorPool::TaskProxy::scheduleViaCPUPool()::{lambda()#1}::operator()() const (__closure=0x7fa07bfed830) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/folly_executorpool.cc:190 #20 0x000000000069b6b0 in FollyExecutorPool::TaskProxy::timeoutExpired ( this=) #21 0x0000000000b0a3b0 in operator() (this=0x7fa07bfed830) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:416 #22 folly::ThreadPoolExecutor::runTask(std::shared_ptr const&, folly::ThreadPoolExecutor::Task&&) (this=0x7fa0d4199c00, thread=..., task=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/ThreadPoolExecutor.cpp:97 #23 0x0000000000af216a in folly::CPUThreadPoolExecutor::threadRun ( this=0x7fa0d4199c00, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:265 #24 0x0000000000b0d369 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> ( __t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #25 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #26 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #27 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #28 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #29 0x0000000000699d4d in operator() (this=0x7fa0d4034d80) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/folly_executorpool.cc:48 #30 operator() (__closure=0x7fa0d4034d80) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/folly_executorpool.cc:48 #31 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #32 0x00007fa0e0eb9d40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #33 0x00007fa0e2a9de65 in start_thread () from /lib64/libpthread.so.0 #34 0x00007fa0e060288d in clone () from /lib64/libc.so.6 Thread 32 (Thread 0x7fa07b7fe700 (LWP 124930)): #0 0x00007fa0e066df60 in __memcmp_sse4_1 () from /lib64/libc.so.6 #1 0x00000000006d98c7 in __memcmp ( __num=, __first2=, __first1=) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/include/memcached/dockey.h:390 #2 equal (__first2=, __last1=, __first1=) at /opt/gcc-10.2.0/include/c++/10.2.0/bits/stl_algobase.h:1122 #3 __equal_aux1 ( __first2=, __last1=, __first1=) at /opt/gcc-10.2.0/include/c++/10.2.0/bits/stl_algobase.h:1156 #4 __equal_aux ( __first2=, __last1=, __first1=) at /opt/gcc-10.2.0/include/c++/10.2.0/bits/stl_algobase.h:1164 #5 equal ( __first2=, __last1=, __first1=) at /opt/gcc-10.2.0/include/c++/10.2.0/bits/stl_algobase.h:1416 #6 SerialisedDocKey::operator== (this=, rhs=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/storeddockey.cc:118 #7 0x00000000006a2ecc in HashTable::findInner(DocKey const&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/stored-value.h:1182 #8 0x00000000006a52f9 in HashTable::insertFromWarmup(Item const&, bool, bool, EvictionPolicy) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/item.h:178 #9 0x0000000000712a56 in LoadStorageKVPairCallback::callback(GetValue&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/warmup.cc:832 #10 0x000000000084c295 in MagmaKVStore::scan(BySeqnoScanContext&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/magma-kvstore/magma-kvstore.cc:1637 #11 0x000000000070ec38 in Warmup::loadKVPairsforShard(unsigned short) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/warmup.cc:1585 #12 0x000000000071b9c1 in WarmupLoadingKVPairs::run (this=0x7fa03cf26090) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/warmup.cc:470 #13 0x00000000006a0567 in GlobalTask::execute() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/globaltask.cc:68 #14 0x000000000069b6b0 in FollyExecutorPool::TaskProxy::scheduleViaCPUPool()::{lambda()#1}::operator()() const (__closure=0x7fa07b7ec830) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/folly_executorpool.cc:190 #15 0x000000000069b6b0 in FollyExecutorPool::TaskProxy::timeoutExpired ( this=) #16 0x0000000000b0a3b0 in operator() (this=0x7fa07b7ec830) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:416 #17 folly::ThreadPoolExecutor::runTask(std::shared_ptr const&, folly::ThreadPoolExecutor::Task&&) (this=0x7fa0d4199c00, thread=..., task=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/ThreadPoolExecutor.cpp:97 #18 0x0000000000af216a in folly::CPUThreadPoolExecutor::threadRun ( this=0x7fa0d4199c00, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:265 #19 0x0000000000b0d369 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> ( __t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #20 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #21 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #22 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #23 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #24 0x0000000000699d4d in operator() (this=0x7fa0d4034d00) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/folly_executorpool.cc:48 #25 operator() (__closure=0x7fa0d4034d00) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/folly_executorpool.cc:48 #26 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #27 0x00007fa0e0eb9d40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #28 0x00007fa0e2a9de65 in start_thread () from /lib64/libpthread.so.0 #29 0x00007fa0e060288d in clone () from /lib64/libc.so.6 Thread 31 (Thread 0x7fa07affd700 (LWP 124931)): #0 0x00007fa0e2aa4f83 in pread64 () from /lib64/libpthread.so.0 #1 0x000000000098870b in magma::BasicFile::Read(long, unsigned long, char*, unsigned long*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/file/file_impl_linux.cc:167 #2 0x000000000098bb27 in magma::BufferedFileReader::Read(magma::File*, long, unsigned long, magma::Slice&, unsigned long&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/rw_file.cc:270 #3 0x0000000000995c75 in magma::ReadAndTransform(magma::File*, long, unsigned long, std::function >&, unsigned long&)>, std::unique_ptr >&, unsigned long&, unsigned long&, magma::BufferedFileReader*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/block_cache.cc:39 #4 0x0000000000995e6b in magma::Cache::makeObject(magma::BlockCache::BlockID) () at /opt/gcc-10.2.0/include/c++/10.2.0/bits/std_function.h:248 #5 0x00000000009952e5 in magma::Cache::AcquireObject (this=0x7fa033b9f7c0, id=..., doCaching=false) at /opt/gcc-10.2.0/include/c++/10.2.0/x86_64-pc-linux-gnu/bits/gthr-default.h:779 #6 0x000000000098f444 in magma::ManagedFile::Read(long, unsigned long, std::shared_ptr&, unsigned long&, unsigned long&, magma::CacheReadMode, std::function >&, unsigned long&)>, magma::BufferedFileReader*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/block_cache.h:61 #7 0x00000000009ae7f8 in magma::SSTable::readBlock(magma::BlockPointer const&, magma::Block*, magma::ReaderStats&, bool, magma::BufferedFileReader*) () at /opt/gcc-10.2.0/include/c++/10.2.0/new:175 #8 0x00000000009b051b in magma::SSTableIterator::initIterators(magma::Slice const&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/lsm/sstable/sstable.cc:641 #9 0x00000000009bbeee in magma::JoinIterator::Next (this=0x7f9fd3227d80) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/lsm/iterator.cc:160 #10 0x00000000009bd441 in Next (this=0x7fa009febc40) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/lsm/iterator.cc:78 #11 magma::MergeIterator::Next() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/lsm/iterator.cc:74 #12 0x00000000009ced98 in magma::KVReader::ReadKVs (this=0x7f9fd3229ac0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/lsm/common.cc:68 #13 0x00000000009bcda7 in magma::SnapshotIterator::readKV() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/lsm/iterator.cc:301 #14 0x00000000009bcfc5 in magma::SnapshotIterator::Next (this=) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/lsm/iterator.cc:244 #15 0x000000000094bde2 in Next (this=0x7fa02c6bf700) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/kvstore/iterator.cc:154 #16 magma::BySeqIterator::Next() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/kvstore/iterator.cc:152 #17 0x000000000084c311 in MagmaKVStore::scan(BySeqnoScanContext&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/magma-kvstore/magma-kvstore.cc:1525 #18 0x000000000070ec38 in Warmup::loadKVPairsforShard(unsigned short) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/warmup.cc:1585 #19 0x000000000071b9c1 in WarmupLoadingKVPairs::run (this=0x7fa03c991b30) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/warmup.cc:470 #20 0x00000000006a0567 in GlobalTask::execute() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/globaltask.cc:68 #21 0x000000000069b6b0 in FollyExecutorPool::TaskProxy::scheduleViaCPUPool()::{lambda()#1}::operator()() const (__closure=0x7fa07afeb830) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/folly_executorpool.cc:190 #22 0x000000000069b6b0 in FollyExecutorPool::TaskProxy::timeoutExpired ( this=) #23 0x0000000000b0a3b0 in operator() (this=0x7fa07afeb830) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:416 #24 folly::ThreadPoolExecutor::runTask(std::shared_ptr const&, folly::ThreadPoolExecutor::Task&&) (this=0x7fa0d4199c00, thread=..., task=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/ThreadPoolExecutor.cpp:97 #25 0x0000000000af216a in folly::CPUThreadPoolExecutor::threadRun ( this=0x7fa0d4199c00, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:265 #26 0x0000000000b0d369 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> ( __t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #27 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #28 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #29 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #30 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #31 0x0000000000699d4d in operator() (this=0x7fa0d4034dc0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/folly_executorpool.cc:48 #32 operator() (__closure=0x7fa0d4034dc0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/folly_executorpool.cc:48 #33 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #34 0x00007fa0e0eb9d40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #35 0x00007fa0e2a9de65 in start_thread () from /lib64/libpthread.so.0 #36 0x00007fa0e060288d in clone () from /lib64/libc.so.6 Thread 30 (Thread 0x7fa07a7fc700 (LWP 124932)): #0 0x00007fa0e05f9b8d in fdatasync () from /lib64/libc.so.6 #1 0x0000000000988fcf in magma::BasicFile::Sync() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/file/file_impl_linux.cc:213 #2 0x000000000098ad08 in magma::WriteFile(magma::FileSystem, std::__cxx11::basic_string, std::allocator > const&, std::__cxx11::basic_string, std::allocator > const&, bool, bool) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/rw_file.cc:147 #3 0x00000000009c1b2d in magma::TreeSnapshot::Write() () at /opt/gcc-10.2.0/include/c++/10.2.0/bits/std_function.h:248 #4 0x000000000096ae4f in magma::CheckpointManager::prepareSnapshot(magma::TreeSnapshot&, bool) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/lsm/checkpoint.cc:285 #5 0x000000000096b266 in magma::CheckpointTransaction::CreateSnapshot(bool) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/lsm/checkpoint.cc:466 #6 0x00000000009dea41 in magma::FlushWork::CreateCheckpoint(bool) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/lsm/flush_work.cc:50 #7 0x00000000009404c8 in magma::KVStore::FlushMemTables(magma::WALOffset&, magma::FlushMode, magma::BlockingMode)::{lambda()#1}::operator()() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/kvstore/kvstore.cc:398 #8 0x0000000000940a18 in magma::KVStore::FlushMemTables(magma::WALOffset&, magma::FlushMode, magma::BlockingMode) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/kvstore/kvstore.cc:424 #9 0x00000000009014eb in magma::Magma::Impl::syncKVStore(unsigned short, bool) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/db.cc:1021 #10 0x0000000000901b40 in magma::Magma::SyncKVStore(unsigned short) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/db.cc:268 #11 0x000000000084f7cd in MagmaKVStore::initBySeqnoScanContext(std::unique_ptr, std::default_delete > >, std::unique_ptr, std::default_delete > >, Vbid, unsigned long, DocumentFilter, ValueFilter, SnapshotSource) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/include/memcached/vbucket.h:62 #12 0x000000000070ec00 in Warmup::loadKVPairsforShard(unsigned short) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/ep_engine.h:639 #13 0x000000000071b9c1 in WarmupLoadingKVPairs::run (this=0x7fa0c873d030) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/warmup.cc:470 #14 0x00000000006a0567 in GlobalTask::execute() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/globaltask.cc:68 #15 0x000000000069b6b0 in FollyExecutorPool::TaskProxy::scheduleViaCPUPool()::{lambda()#1}::operator()() const (__closure=0x7fa07a7ea830) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/folly_executorpool.cc:190 #16 0x000000000069b6b0 in FollyExecutorPool::TaskProxy::timeoutExpired ( this=) #17 0x0000000000b0a3b0 in operator() (this=0x7fa07a7ea830) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:416 #18 folly::ThreadPoolExecutor::runTask(std::shared_ptr const&, folly::ThreadPoolExecutor::Task&&) (this=0x7fa0d4199c00, thread=..., task=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/ThreadPoolExecutor.cpp:97 #19 0x0000000000af216a in folly::CPUThreadPoolExecutor::threadRun ( this=0x7fa0d4199c00, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:265 #20 0x0000000000b0d369 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> ( __t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #21 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #22 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #23 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #24 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #25 0x0000000000699d4d in operator() (this=0x7fa0d4034e00) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/folly_executorpool.cc:48 #26 operator() (__closure=0x7fa0d4034e00) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/folly_executorpool.cc:48 #27 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #28 0x00007fa0e0eb9d40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #29 0x00007fa0e2a9de65 in start_thread () from /lib64/libpthread.so.0 #30 0x00007fa0e060288d in clone () from /lib64/libc.so.6 Thread 29 (Thread 0x7fa079ffb700 (LWP 124933)): #0 0x00007fa0e05fcba9 in syscall () from /lib64/libc.so.6 #1 0x0000000000aea3bd in nativeFutexWaitImpl ( waitMask=waitMask@entry=2046727688, absSteadyTime=absSteadyTime@entry=0x7fa079fe95e8, absSystemTime=absSystemTime@entry=0x0, expected=, addr=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:123 #2 folly::detail::futexWaitImpl (futex=, expected=, absSystemTime=absSystemTime@entry=0x0, absSteadyTime=absSteadyTime@entry=0x7fa079fe9608, waitMask=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:254 #3 0x0000000000af3f5a in futexWaitImpl const, std::chrono::time_point > > > (waitMask=, deadline=..., expected=, futex=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:85 #4 folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > > ( futex=, expected=, deadline=..., waitMask=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:123 #5 0x0000000000af9dbd in folly::SaturatingSemaphore::tryWaitSlow > > (this=0x7fa0b0fd80d8, deadline=..., opt=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:164 #6 0x0000000000afa426 in folly::detail::LifoSemBase, std::atomic>::try_wait_until > > ( this=0x7fa084002140, deadline=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:179 #7 0x0000000000afdb80 in folly::UnboundedBlockingQueue::try_take_for (this=0x7fa084002100, time=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:498 #8 0x0000000000af20f7 in folly::CPUThreadPoolExecutor::threadRun ( this=0x7fa0d419a800, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:243 #9 0x0000000000b0d369 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> ( __t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #10 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #11 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #12 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #13 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #14 0x0000000000699d4d in operator() (this=0x7fa0d4034f40) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/folly_executorpool.cc:48 #15 operator() (__closure=0x7fa0d4034f40) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/folly_executorpool.cc:48 #16 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #17 0x00007fa0e0eb9d40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #18 0x00007fa0e2a9de65 in start_thread () from /lib64/libpthread.so.0 #19 0x00007fa0e060288d in clone () from /lib64/libc.so.6 Thread 28 (Thread 0x7fa0797fa700 (LWP 124934)): #0 0x00007fa0e05fcba9 in syscall () from /lib64/libc.so.6 #1 0x0000000000aea3bd in nativeFutexWaitImpl ( waitMask=waitMask@entry=2038334984, absSteadyTime=absSteadyTime@entry=0x7fa0797e85e8, absSystemTime=absSystemTime@entry=0x0, expected=, addr=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:123 #2 folly::detail::futexWaitImpl (futex=, expected=, absSystemTime=absSystemTime@entry=0x0, absSteadyTime=absSteadyTime@entry=0x7fa0797e8608, waitMask=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:254 #3 0x0000000000af3f5a in futexWaitImpl const, std::chrono::time_point > > > (waitMask=, deadline=..., expected=, futex=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:85 #4 folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > > ( futex=, expected=, deadline=..., waitMask=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:123 #5 0x0000000000af9dbd in folly::SaturatingSemaphore::tryWaitSlow > > (this=0x7fa0b0fd80f0, deadline=..., opt=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:164 #6 0x0000000000afa426 in folly::detail::LifoSemBase, std::atomic>::try_wait_until > > ( this=0x7fa084002140, deadline=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:179 #7 0x0000000000afdb80 in folly::UnboundedBlockingQueue::try_take_for (this=0x7fa084002100, time=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:498 #8 0x0000000000af20f7 in folly::CPUThreadPoolExecutor::threadRun ( this=0x7fa0d419a800, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:243 #9 0x0000000000b0d369 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> ( __t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #10 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #11 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #12 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #13 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #14 0x0000000000699d4d in operator() (this=0x7fa0d4034f80) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/folly_executorpool.cc:48 #15 operator() (__closure=0x7fa0d4034f80) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/folly_executorpool.cc:48 #16 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #17 0x00007fa0e0eb9d40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #18 0x00007fa0e2a9de65 in start_thread () from /lib64/libpthread.so.0 #19 0x00007fa0e060288d in clone () from /lib64/libc.so.6 Thread 27 (Thread 0x7fa078ff9700 (LWP 124935)): #0 0x00007fa0e05fcba9 in syscall () from /lib64/libc.so.6 #1 0x0000000000aea3bd in nativeFutexWaitImpl ( waitMask=waitMask@entry=2029942280, absSteadyTime=absSteadyTime@entry=0x7fa078fe75e8, absSystemTime=absSystemTime@entry=0x0, expected=, addr=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:123 #2 folly::detail::futexWaitImpl (futex=, expected=, absSystemTime=absSystemTime@entry=0x0, absSteadyTime=absSteadyTime@entry=0x7fa078fe7608, waitMask=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:254 #3 0x0000000000af3f5a in futexWaitImpl const, std::chrono::time_point > > > (waitMask=, deadline=..., expected=, futex=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:85 #4 folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > > ( futex=, expected=, deadline=..., waitMask=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:123 #5 0x0000000000af9dbd in folly::SaturatingSemaphore::tryWaitSlow > > (this=0x7fa0b0fd8108, deadline=..., opt=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:164 #6 0x0000000000afa426 in folly::detail::LifoSemBase, std::atomic>::try_wait_until > > ( this=0x7fa084002140, deadline=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:179 #7 0x0000000000afdb80 in folly::UnboundedBlockingQueue::try_take_for (this=0x7fa084002100, time=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:498 #8 0x0000000000af20f7 in folly::CPUThreadPoolExecutor::threadRun ( this=0x7fa0d419a800, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:243 #9 0x0000000000b0d369 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> ( __t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #10 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #11 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #12 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #13 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #14 0x0000000000699d4d in operator() (this=0x7fa0d4034fc0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/folly_executorpool.cc:48 #15 operator() (__closure=0x7fa0d4034fc0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/folly_executorpool.cc:48 #16 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #17 0x00007fa0e0eb9d40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #18 0x00007fa0e2a9de65 in start_thread () from /lib64/libpthread.so.0 #19 0x00007fa0e060288d in clone () from /lib64/libc.so.6 Thread 26 (Thread 0x7fa057fff700 (LWP 124936)): #0 0x00007fa0e05fcba9 in syscall () from /lib64/libc.so.6 #1 0x0000000000aea3bd in nativeFutexWaitImpl ( waitMask=waitMask@entry=1476318728, absSteadyTime=absSteadyTime@entry=0x7fa057fed5e8, absSystemTime=absSystemTime@entry=0x0, expected=, addr=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:123 #2 folly::detail::futexWaitImpl (futex=, expected=, absSystemTime=absSystemTime@entry=0x0, absSteadyTime=absSteadyTime@entry=0x7fa057fed608, waitMask=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:254 #3 0x0000000000af3f5a in futexWaitImpl const, std::chrono::time_point > > > (waitMask=, deadline=..., expected=, futex=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:85 #4 folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > > ( futex=, expected=, deadline=..., waitMask=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:123 #5 0x0000000000af9ef7 in folly::SaturatingSemaphore::tryWaitSlow > > (this=0x7fa0b0fd8120, deadline=..., opt=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:194 #6 0x0000000000afa426 in folly::detail::LifoSemBase, std::atomic>::try_wait_until > > ( this=0x7fa084002140, deadline=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:179 #7 0x0000000000afdb80 in folly::UnboundedBlockingQueue::try_take_for (this=0x7fa084002100, time=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:498 #8 0x0000000000af20f7 in folly::CPUThreadPoolExecutor::threadRun ( this=0x7fa0d419a800, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:243 #9 0x0000000000b0d369 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> ( __t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #10 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #11 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #12 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #13 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #14 0x0000000000699d4d in operator() (this=0x7fa0d4049040) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/folly_executorpool.cc:48 #15 operator() (__closure=0x7fa0d4049040) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/folly_executorpool.cc:48 #16 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #17 0x00007fa0e0eb9d40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #18 0x00007fa0e2a9de65 in start_thread () from /lib64/libpthread.so.0 #19 0x00007fa0e060288d in clone () from /lib64/libc.so.6 Thread 25 (Thread 0x7fa0577fe700 (LWP 124937)): #0 0x00007fa0e05fcba9 in syscall () from /lib64/libc.so.6 #1 0x0000000000aea3bd in nativeFutexWaitImpl ( waitMask=waitMask@entry=1467926024, absSteadyTime=absSteadyTime@entry=0x7fa0577ec5e8, absSystemTime=absSystemTime@entry=0x0, expected=, addr=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:123 #2 folly::detail::futexWaitImpl (futex=, expected=, absSystemTime=absSystemTime@entry=0x0, absSteadyTime=absSteadyTime@entry=0x7fa0577ec608, waitMask=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:254 #3 0x0000000000af3f5a in futexWaitImpl const, std::chrono::time_point > > > (waitMask=, deadline=..., expected=, futex=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:85 #4 folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > > ( futex=, expected=, deadline=..., waitMask=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:123 #5 0x0000000000af9dbd in folly::SaturatingSemaphore::tryWaitSlow > > (this=0x7fa0b0fd8138, deadline=..., opt=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:164 #6 0x0000000000afa426 in folly::detail::LifoSemBase, std::atomic>::try_wait_until > > ( this=0x7fa084002e40, deadline=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:179 #7 0x0000000000afdb80 in folly::UnboundedBlockingQueue::try_take_for (this=0x7fa084002e00, time=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:498 #8 0x0000000000af20f7 in folly::CPUThreadPoolExecutor::threadRun ( this=0x7fa0d419b400, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:243 #9 0x0000000000b0d369 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> ( __t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #10 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #11 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #12 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #13 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #14 0x0000000000699d4d in operator() (this=0x7fa0d4049100) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/folly_executorpool.cc:48 #15 operator() (__closure=0x7fa0d4049100) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/folly_executorpool.cc:48 #16 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #17 0x00007fa0e0eb9d40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #18 0x00007fa0e2a9de65 in start_thread () from /lib64/libpthread.so.0 #19 0x00007fa0e060288d in clone () from /lib64/libc.so.6 Thread 24 (Thread 0x7fa056ffd700 (LWP 124938)): #0 0x00007fa0e05fcba9 in syscall () from /lib64/libc.so.6 #1 0x0000000000aea3bd in nativeFutexWaitImpl ( waitMask=waitMask@entry=1459533320, absSteadyTime=absSteadyTime@entry=0x7fa056feb5e8, absSystemTime=absSystemTime@entry=0x0, expected=, addr=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:123 #2 folly::detail::futexWaitImpl (futex=, expected=, absSystemTime=absSystemTime@entry=0x0, absSteadyTime=absSteadyTime@entry=0x7fa056feb608, waitMask=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:254 #3 0x0000000000af3f5a in futexWaitImpl const, std::chrono::time_point > > > (waitMask=, deadline=..., expected=, futex=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:85 #4 folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > > ( futex=, expected=, deadline=..., waitMask=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:123 #5 0x0000000000af9ef7 in folly::SaturatingSemaphore::tryWaitSlow > > (this=0x7fa0b0fd8210, deadline=..., opt=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:194 #6 0x0000000000afa426 in folly::detail::LifoSemBase, std::atomic>::try_wait_until > > ( this=0x7fa084003340, deadline=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:179 #7 0x0000000000afdb80 in folly::UnboundedBlockingQueue::try_take_for (this=0x7fa084003300, time=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:498 #8 0x0000000000af20f7 in folly::CPUThreadPoolExecutor::threadRun ( this=0x7fa0d41db000, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:243 #9 0x0000000000b0d369 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> ( __t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #10 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #11 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #12 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #13 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #14 0x0000000000699d4d in operator() (this=0x7fa0d4049200) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/folly_executorpool.cc:48 #15 operator() (__closure=0x7fa0d4049200) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/folly_executorpool.cc:48 #16 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #17 0x00007fa0e0eb9d40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #18 0x00007fa0e2a9de65 in start_thread () from /lib64/libpthread.so.0 #19 0x00007fa0e060288d in clone () from /lib64/libc.so.6 Thread 23 (Thread 0x7fa0567fc700 (LWP 124939)): #0 0x00007fa0e05fcba9 in syscall () from /lib64/libc.so.6 #1 0x0000000000aea3bd in nativeFutexWaitImpl ( waitMask=waitMask@entry=1451140616, absSteadyTime=absSteadyTime@entry=0x7fa0567ea5e8, absSystemTime=absSystemTime@entry=0x0, expected=, addr=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:123 #2 folly::detail::futexWaitImpl (futex=, expected=, absSystemTime=absSystemTime@entry=0x0, absSteadyTime=absSteadyTime@entry=0x7fa0567ea608, waitMask=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:254 #3 0x0000000000af3f5a in futexWaitImpl const, std::chrono::time_point > > > (waitMask=, deadline=..., expected=, futex=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:85 #4 folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > > ( futex=, expected=, deadline=..., waitMask=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:123 #5 0x0000000000af9ef7 in folly::SaturatingSemaphore::tryWaitSlow > > (this=0x7fa0b0fd81f8, deadline=..., opt=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:194 #6 0x0000000000afa426 in folly::detail::LifoSemBase, std::atomic>::try_wait_until > > ( this=0x7fa084003340, deadline=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:179 #7 0x0000000000afdb80 in folly::UnboundedBlockingQueue::try_take_for (this=0x7fa084003300, time=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:498 #8 0x0000000000af20f7 in folly::CPUThreadPoolExecutor::threadRun ( this=0x7fa0d41db000, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:243 #9 0x0000000000b0d369 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> ( __t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #10 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #11 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #12 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #13 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #14 0x0000000000699d4d in operator() (this=0x7fa0d4049240) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/folly_executorpool.cc:48 #15 operator() (__closure=0x7fa0d4049240) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/folly_executorpool.cc:48 #16 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #17 0x00007fa0e0eb9d40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #18 0x00007fa0e2a9de65 in start_thread () from /lib64/libpthread.so.0 #19 0x00007fa0e060288d in clone () from /lib64/libc.so.6 Thread 22 (Thread 0x7fa055ffb700 (LWP 124940)): #0 0x00007fa0e2aa4ebd in open64 () from /lib64/libpthread.so.0 #1 0x0000000000986b45 in magma::BasicFile::Open(int) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/file/file_impl_linux.cc:96 #2 0x000000000098aba6 in magma::WriteFile(magma::FileSystem, std::__cxx11::basic_string, std::allocator > const&, std::__cxx11::basic_string, std::allocator > const&, bool, bool) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/rw_file.cc:124 #3 0x00000000009c1b2d in magma::TreeSnapshot::Write() () at /opt/gcc-10.2.0/include/c++/10.2.0/bits/std_function.h:248 #4 0x000000000096ae4f in magma::CheckpointManager::prepareSnapshot(magma::TreeSnapshot&, bool) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/lsm/checkpoint.cc:285 #5 0x000000000096b266 in magma::CheckpointTransaction::CreateSnapshot(bool) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/lsm/checkpoint.cc:466 #6 0x00000000009dea41 in magma::FlushWork::CreateCheckpoint(bool) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/lsm/flush_work.cc:50 #7 0x00000000009404c8 in magma::KVStore::FlushMemTables(magma::WALOffset&, magma::FlushMode, magma::BlockingMode)::{lambda()#1}::operator()() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/kvstore/kvstore.cc:398 #8 0x0000000000940d91 in __invoke_impl&> (__f=...) at /opt/gcc-10.2.0/include/c++/10.2.0/bits/std_function.h:289 #9 __invoke_r&> (__fn=...) at /opt/gcc-10.2.0/include/c++/10.2.0/bits/invoke.h:110 #10 std::_Function_handler >::_M_invoke(const std::_Any_data &) (__functor=...) at /opt/gcc-10.2.0/include/c++/10.2.0/bits/std_function.h:291 #11 0x000000000099653d in operator() (this=0x7fa031c888d8) at /opt/gcc-10.2.0/include/c++/10.2.0/bits/std_function.h:248 #12 magma::TimedTask::Complete(bool) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:143 #13 0x0000000000997d31 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:160 #14 0x0000000000aab9e9 in run (this=0x7fa0dc3fd390) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #15 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #16 0x00007fa0e2a9de65 in start_thread () from /lib64/libpthread.so.0 #17 0x00007fa0e060288d in clone () from /lib64/libc.so.6 Thread 21 (Thread 0x7fa0557fa700 (LWP 124941)): #0 0x00007fa0e05f9b8d in fdatasync () from /lib64/libc.so.6 #1 0x0000000000988fcf in magma::BasicFile::Sync() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/file/file_impl_linux.cc:213 #2 0x000000000098ad08 in magma::WriteFile(magma::FileSystem, std::__cxx11::basic_string, std::allocator > const&, std::__cxx11::basic_string, std::allocator > const&, bool, bool) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/rw_file.cc:147 #3 0x00000000009c1b2d in magma::TreeSnapshot::Write() () at /opt/gcc-10.2.0/include/c++/10.2.0/bits/std_function.h:248 #4 0x000000000096ae4f in magma::CheckpointManager::prepareSnapshot(magma::TreeSnapshot&, bool) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/lsm/checkpoint.cc:285 #5 0x000000000096b266 in magma::CheckpointTransaction::CreateSnapshot(bool) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/lsm/checkpoint.cc:466 #6 0x00000000009dea41 in magma::FlushWork::CreateCheckpoint(bool) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/lsm/flush_work.cc:50 #7 0x00000000009404c8 in magma::KVStore::FlushMemTables(magma::WALOffset&, magma::FlushMode, magma::BlockingMode)::{lambda()#1}::operator()() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/kvstore/kvstore.cc:398 #8 0x0000000000940d91 in __invoke_impl&> (__f=...) at /opt/gcc-10.2.0/include/c++/10.2.0/bits/std_function.h:289 #9 __invoke_r&> (__fn=...) at /opt/gcc-10.2.0/include/c++/10.2.0/bits/invoke.h:110 #10 std::_Function_handler >::_M_invoke(const std::_Any_data &) (__functor=...) at /opt/gcc-10.2.0/include/c++/10.2.0/bits/std_function.h:291 #11 0x000000000099653d in operator() (this=0x7fa031bb2dd8) at /opt/gcc-10.2.0/include/c++/10.2.0/bits/std_function.h:248 #12 magma::TimedTask::Complete(bool) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:143 #13 0x0000000000997d31 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:160 #14 0x0000000000aab9e9 in run (this=0x7fa0dc3fd3c0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #15 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #16 0x00007fa0e2a9de65 in start_thread () from /lib64/libpthread.so.0 #17 0x00007fa0e060288d in clone () from /lib64/libc.so.6 Thread 20 (Thread 0x7fa054ff9700 (LWP 124942)): #0 0x00007fa0e2aa450d in __lll_lock_wait () from /lib64/libpthread.so.0 #1 0x00007fa0e2a9fe5b in _L_lock_883 () from /lib64/libpthread.so.0 #2 0x00007fa0e2a9fd28 in pthread_mutex_lock () from /lib64/libpthread.so.0 #3 0x0000000000968c8d in __gthread_mutex_lock (__mutex=0x7fa031391458) at /opt/gcc-10.2.0/include/c++/10.2.0/x86_64-pc-linux-gnu/bits/gthr-default.h:749 #4 lock (this=0x7fa031391458) at /opt/gcc-10.2.0/include/c++/10.2.0/bits/std_mutex.h:100 #5 lock (this=) at /opt/gcc-10.2.0/include/c++/10.2.0/bits/unique_lock.h:138 #6 unique_lock (__m=..., this=) at /opt/gcc-10.2.0/include/c++/10.2.0/bits/unique_lock.h:68 #7 magma::CheckpointManager::Overhead ( ../../gdb/dwarf2read.c:3755: internal-error: dw2_find_pc_sect_symtab: Assertion `result != NULL' failed. A problem internal to GDB has been detected, further debugging may prove unreliable. Quit this debugging session? (y or n) [answered Y; input not from terminal] ../../gdb/dwarf2read.c:3755: internal-error: dw2_find_pc_sect_symtab: Assertion `result != NULL' failed. A problem internal to GDB has been detected, further debugging may prove unreliable. Create a core file of GDB? (y or n) [answered Y; input not from terminal] 2021-06-08 06:55:54,834 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-06-08 06:55:54,848 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000367.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' 2021-06-08 06:55:54,874 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000367.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-06-08 06:55:54,901 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000367.txt 2021-06-08 06:55:54,921 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.000367.txt 2021-06-08 06:55:54,940 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-06-08 06:55:54,953 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:763] 172.23.106.198: No 'sanitizers.log.*' files found 2021-06-08 06:55:54,953 | infra | DEBUG | MainThread | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.198 2021-06-08 06:55:54,994 | test | INFO | MainThread | [cluster_ready_functions:trigger_cb_collect_on_cluster:1094] Running cbcollect on node ns_1@172.23.106.168,ns_1@172.23.106.196,ns_1@172.23.106.198,ns_1@172.23.106.199 2021-06-08 06:55:55,009 | test | DEBUG | MainThread | [rest_client:_http_request:729] {'date': 'Tue, 08 Jun 2021 13:55:54 GMT', 'content-length': '2', 'server': 'Couchbase Server', 'expires': 'Thu, 01 Jan 1970 00:00:00 GMT', 'x-frame-options': 'DENY', 'x-permitted-cross-domain-policies': 'none', 'pragma': 'no-cache', 'x-content-type-options': 'nosniff', 'x-xss-protection': '1; mode=block', 'connection': 'close', 'content-type': 'application/json', 'cache-control': 'no-cache,no-store,must-revalidate', 'status': '200'} 2021-06-08 06:55:55,009 | test | DEBUG | MainThread | [rest_client:_http_request:730] [] 2021-06-08 06:55:55,016 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from couchbase_utils/cluster_utils/cluster_ready_functions.py -> trigger_cb_collect_on_cluster():L1096 2021-06-08 06:55:55,016 | infra | INFO | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: Wait for CB collect to start 2021-06-08 06:56:05,016 | test | INFO | MainThread | [cluster_ready_functions:trigger_cb_collect_on_cluster:1097] ns_1@172.23.106.168,ns_1@172.23.106.196,ns_1@172.23.106.198,ns_1@172.23.106.199 - cbcollect status: True 2021-06-08 06:56:05,016 | test | INFO | MainThread | [cluster_ready_functions:wait_for_cb_collect_to_complete:1102] Polling active_tasks to check cbcollect status 2021-06-08 06:56:05,026 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_cb_collect_to_complete:1109] CBCollectInfo Iteration 0 - running 2021-06-08 06:56:05,032 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from couchbase_utils/cluster_utils/cluster_ready_functions.py -> wait_for_cb_collect_to_complete():L1119 2021-06-08 06:56:05,032 | infra | INFO | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: CB collect still running 2021-06-08 06:56:15,042 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_cb_collect_to_complete:1109] CBCollectInfo Iteration 1 - running 2021-06-08 06:56:15,049 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from couchbase_utils/cluster_utils/cluster_ready_functions.py -> wait_for_cb_collect_to_complete():L1119 2021-06-08 06:56:15,049 | infra | INFO | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: CB collect still running 2021-06-08 06:56:25,059 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_cb_collect_to_complete:1109] CBCollectInfo Iteration 2 - running 2021-06-08 06:56:25,065 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from couchbase_utils/cluster_utils/cluster_ready_functions.py -> wait_for_cb_collect_to_complete():L1119 2021-06-08 06:56:25,065 | infra | INFO | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: CB collect still running 2021-06-08 06:56:35,075 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_cb_collect_to_complete:1109] CBCollectInfo Iteration 3 - running 2021-06-08 06:56:35,082 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from couchbase_utils/cluster_utils/cluster_ready_functions.py -> wait_for_cb_collect_to_complete():L1119 2021-06-08 06:56:35,082 | infra | INFO | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: CB collect still running 2021-06-08 06:56:45,091 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_cb_collect_to_complete:1109] CBCollectInfo Iteration 4 - running 2021-06-08 06:56:45,098 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from couchbase_utils/cluster_utils/cluster_ready_functions.py -> wait_for_cb_collect_to_complete():L1119 2021-06-08 06:56:45,098 | infra | INFO | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: CB collect still running 2021-06-08 06:56:55,107 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_cb_collect_to_complete:1109] CBCollectInfo Iteration 5 - running 2021-06-08 06:56:55,114 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from couchbase_utils/cluster_utils/cluster_ready_functions.py -> wait_for_cb_collect_to_complete():L1119 2021-06-08 06:56:55,114 | infra | INFO | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: CB collect still running 2021-06-08 06:57:05,124 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_cb_collect_to_complete:1109] CBCollectInfo Iteration 6 - running 2021-06-08 06:57:05,130 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from couchbase_utils/cluster_utils/cluster_ready_functions.py -> wait_for_cb_collect_to_complete():L1119 2021-06-08 06:57:05,130 | infra | INFO | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: CB collect still running 2021-06-08 06:57:15,140 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_cb_collect_to_complete:1109] CBCollectInfo Iteration 7 - running 2021-06-08 06:57:15,145 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from couchbase_utils/cluster_utils/cluster_ready_functions.py -> wait_for_cb_collect_to_complete():L1119 2021-06-08 06:57:15,145 | infra | INFO | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: CB collect still running 2021-06-08 06:57:25,155 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_cb_collect_to_complete:1109] CBCollectInfo Iteration 8 - running 2021-06-08 06:57:25,161 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from couchbase_utils/cluster_utils/cluster_ready_functions.py -> wait_for_cb_collect_to_complete():L1119 2021-06-08 06:57:25,163 | infra | INFO | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: CB collect still running 2021-06-08 06:57:35,171 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_cb_collect_to_complete:1109] CBCollectInfo Iteration 9 - running 2021-06-08 06:57:35,177 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from couchbase_utils/cluster_utils/cluster_ready_functions.py -> wait_for_cb_collect_to_complete():L1119 2021-06-08 06:57:35,177 | infra | INFO | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: CB collect still running 2021-06-08 06:57:45,187 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_cb_collect_to_complete:1109] CBCollectInfo Iteration 10 - running 2021-06-08 06:57:45,194 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from couchbase_utils/cluster_utils/cluster_ready_functions.py -> wait_for_cb_collect_to_complete():L1119 2021-06-08 06:57:45,194 | infra | INFO | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: CB collect still running 2021-06-08 06:57:55,203 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_cb_collect_to_complete:1109] CBCollectInfo Iteration 11 - running 2021-06-08 06:57:55,210 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from couchbase_utils/cluster_utils/cluster_ready_functions.py -> wait_for_cb_collect_to_complete():L1119 2021-06-08 06:57:55,210 | infra | INFO | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: CB collect still running 2021-06-08 06:58:05,220 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_cb_collect_to_complete:1109] CBCollectInfo Iteration 12 - running 2021-06-08 06:58:05,226 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from couchbase_utils/cluster_utils/cluster_ready_functions.py -> wait_for_cb_collect_to_complete():L1119 2021-06-08 06:58:05,226 | infra | INFO | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: CB collect still running 2021-06-08 06:58:15,236 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_cb_collect_to_complete:1109] CBCollectInfo Iteration 13 - running 2021-06-08 06:58:15,243 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from couchbase_utils/cluster_utils/cluster_ready_functions.py -> wait_for_cb_collect_to_complete():L1119 2021-06-08 06:58:15,243 | infra | INFO | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: CB collect still running 2021-06-08 06:58:25,252 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_cb_collect_to_complete:1109] CBCollectInfo Iteration 14 - running 2021-06-08 06:58:25,259 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from couchbase_utils/cluster_utils/cluster_ready_functions.py -> wait_for_cb_collect_to_complete():L1119 2021-06-08 06:58:25,259 | infra | INFO | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: CB collect still running 2021-06-08 06:58:35,267 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_cb_collect_to_complete:1109] CBCollectInfo Iteration 15 - running 2021-06-08 06:58:35,275 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from couchbase_utils/cluster_utils/cluster_ready_functions.py -> wait_for_cb_collect_to_complete():L1119 2021-06-08 06:58:35,275 | infra | INFO | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: CB collect still running 2021-06-08 06:58:45,283 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_cb_collect_to_complete:1109] CBCollectInfo Iteration 16 - running 2021-06-08 06:58:45,290 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from couchbase_utils/cluster_utils/cluster_ready_functions.py -> wait_for_cb_collect_to_complete():L1119 2021-06-08 06:58:45,290 | infra | INFO | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: CB collect still running 2021-06-08 06:58:55,299 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_cb_collect_to_complete:1109] CBCollectInfo Iteration 17 - running 2021-06-08 06:58:55,306 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from couchbase_utils/cluster_utils/cluster_ready_functions.py -> wait_for_cb_collect_to_complete():L1119 2021-06-08 06:58:55,306 | infra | INFO | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: CB collect still running 2021-06-08 06:59:05,315 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_cb_collect_to_complete:1109] CBCollectInfo Iteration 18 - completed 2021-06-08 06:59:05,325 | test | DEBUG | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1125] {u'perNode': {u'ns_1@172.23.106.168': {u'path': u'/opt/couchbase/var/lib/couchbase/tmp/collectinfo-2021-06-08T135555-ns_1@172.23.106.168.zip', u'status': u'collected'}, u'ns_1@172.23.106.199': {u'path': u'/opt/couchbase/var/lib/couchbase/tmp/collectinfo-2021-06-08T135555-ns_1@172.23.106.199.zip', u'status': u'collected'}, u'ns_1@172.23.106.198': {u'path': u'/opt/couchbase/var/lib/couchbase/tmp/collectinfo-2021-06-08T135555-ns_1@172.23.106.198.zip', u'status': u'collected'}, u'ns_1@172.23.106.196': {u'path': u'/opt/couchbase/var/lib/couchbase/tmp/collectinfo-2021-06-08T135555-ns_1@172.23.106.196.zip', u'status': u'collected'}}, u'type': u'clusterLogsCollection', u'node': u'ns_1@172.23.106.196', u'progress': 100, u'ts': u'2021-06-08 13:55:55', u'status': u'completed'} 2021-06-08 06:59:05,325 | test | INFO | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1129] ns_1@172.23.106.168: Copying cbcollect ZIP file to Client 2021-06-08 06:59:05,326 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.168 with username: root, password: couchbase 2021-06-08 06:59:05,410 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 06:59:05,624 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: ls /etc/ 2021-06-08 06:59:05,680 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 06:59:05,680 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: ls /cygdrive/c/Windows 2021-06-08 06:59:05,693 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: uname -m 2021-06-08 06:59:05,704 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: cat /proc/cpuinfo 2021-06-08 06:59:05,717 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: df -Thl 2021-06-08 06:59:05,733 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: cat /proc/meminfo 2021-06-08 06:59:05,746 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: hostname 2021-06-08 06:59:05,759 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: hostname -d 2021-06-08 06:59:06,158 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.168: rm -f /opt/couchbase/var/lib/couchbase/tmp/collectinfo-2021-06-08T135555-ns_1@172.23.106.168.zip 2021-06-08 06:59:06,180 | infra | DEBUG | MainThread | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.168 2021-06-08 06:59:06,180 | test | INFO | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1129] ns_1@172.23.106.196: Copying cbcollect ZIP file to Client 2021-06-08 06:59:06,181 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.196 with username: root, password: couchbase 2021-06-08 06:59:06,257 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 06:59:06,473 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: ls /etc/ 2021-06-08 06:59:06,523 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 06:59:06,523 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: ls /cygdrive/c/Windows 2021-06-08 06:59:06,536 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: uname -m 2021-06-08 06:59:06,546 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: cat /proc/cpuinfo 2021-06-08 06:59:06,559 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: df -Thl 2021-06-08 06:59:06,572 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: cat /proc/meminfo 2021-06-08 06:59:06,584 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: hostname 2021-06-08 06:59:06,595 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: hostname -d 2021-06-08 06:59:07,010 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.196: rm -f /opt/couchbase/var/lib/couchbase/tmp/collectinfo-2021-06-08T135555-ns_1@172.23.106.196.zip 2021-06-08 06:59:07,032 | infra | DEBUG | MainThread | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.196 2021-06-08 06:59:07,032 | test | INFO | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1129] ns_1@172.23.106.198: Copying cbcollect ZIP file to Client 2021-06-08 06:59:07,033 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.198 with username: root, password: couchbase 2021-06-08 06:59:07,223 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 06:59:07,665 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: ls /etc/ 2021-06-08 06:59:07,726 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 06:59:07,726 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: ls /cygdrive/c/Windows 2021-06-08 06:59:07,739 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: uname -m 2021-06-08 06:59:07,753 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: cat /proc/cpuinfo 2021-06-08 06:59:07,776 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: df -Thl 2021-06-08 06:59:07,792 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: cat /proc/meminfo 2021-06-08 06:59:07,803 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: hostname 2021-06-08 06:59:07,836 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: hostname -d 2021-06-08 06:59:08,346 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.198: rm -f /opt/couchbase/var/lib/couchbase/tmp/collectinfo-2021-06-08T135555-ns_1@172.23.106.198.zip 2021-06-08 06:59:08,367 | infra | DEBUG | MainThread | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.198 2021-06-08 06:59:08,367 | test | INFO | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1129] ns_1@172.23.106.199: Copying cbcollect ZIP file to Client 2021-06-08 06:59:08,368 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.199 with username: root, password: couchbase 2021-06-08 06:59:08,450 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 06:59:08,676 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: ls /etc/ 2021-06-08 06:59:08,739 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 06:59:08,739 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: ls /cygdrive/c/Windows 2021-06-08 06:59:08,753 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: uname -m 2021-06-08 06:59:08,766 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: cat /proc/cpuinfo 2021-06-08 06:59:08,779 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: df -Thl 2021-06-08 06:59:08,798 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: cat /proc/meminfo 2021-06-08 06:59:08,811 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: hostname 2021-06-08 06:59:08,823 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: hostname -d 2021-06-08 06:59:09,259 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.199: rm -f /opt/couchbase/var/lib/couchbase/tmp/collectinfo-2021-06-08T135555-ns_1@172.23.106.199.zip 2021-06-08 06:59:09,280 | infra | DEBUG | MainThread | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.199 2021-06-08 06:59:09,292 | test | CRITICAL | MainThread | [basetestcase:tearDownEverything:493] Skipping get_trace !! 2021-06-08 06:59:09,319 | infra | WARNING | MainThread | [basetestcase:tearDownEverything:498] Alerts found: [{u'msg': u'Approaching full disk warning. Usage of disk "/data" on node "172.23.106.196" is around 93%.', u'disableUIPopUp': False, u'serverTime': u'2021-06-08T06:47:43.000Z'}, {u'msg': u'Approaching full disk warning. Usage of disk "/data" on node "172.23.106.168" is around 93%.', u'disableUIPopUp': False, u'serverTime': u'2021-06-08T06:47:50.000Z'}, {u'msg': u'Approaching full disk warning. Usage of disk "/data" on node "172.23.106.198" is around 91%.', u'disableUIPopUp': False, u'serverTime': u'2021-06-08T06:47:52.000Z'}, {u'msg': u'Approaching full disk warning. Usage of disk "/data" on node "172.23.106.199" is around 95%.', u'disableUIPopUp': False, u'serverTime': u'2021-06-08T06:48:46.000Z'}, {u'msg': u'Write Commit Failure. Disk write failed for item in Bucket "default" on node 172.23.106.196.', u'disableUIPopUp': True, u'serverTime': u'2021-06-08T06:52:43.000Z'}, {u'msg': u'Write Commit Failure. Disk write failed for item in Bucket "default" on node 172.23.106.199.', u'disableUIPopUp': True, u'serverTime': u'2021-06-08T06:52:46.000Z'}, {u'msg': u'Write Commit Failure. Disk write failed for item in Bucket "default" on node 172.23.106.168.', u'disableUIPopUp': True, u'serverTime': u'2021-06-08T06:52:50.000Z'}, {u'msg': u'Write Commit Failure. Disk write failed for item in Bucket "default" on node 172.23.106.198.', u'disableUIPopUp': True, u'serverTime': u'2021-06-08T06:52:52.000Z'}] 2021-06-08 06:59:09,319 | test | DEBUG | MainThread | [basetestcase:tearDownEverything:499] Cleaning up cluster 2021-06-08 06:59:09,380 | test | DEBUG | MainThread | [bucket_ready_functions:parse_get_bucket_json:3739] Stats: {u'opsPerSec': 0, u'diskFetches': 1109697, u'dataUsed': 41722028032L, u'itemCount': 6647678, u'quotaPercentUsed': 49.19700113170875, u'storageTotals': {u'hdd': {u'usedByData': 58470071492L, u'used': 61816661605L, u'quotaTotal': 386354135040L, u'total': 386354135040L, u'free': 320673932084L}, u'ram': {u'usedByData': 17229989416L, u'used': 43256057856L, u'quotaTotal': 35026632704L, u'total': 49176707072L, u'quotaUsed': 35022438400L, u'quotaUsedPerNode': 8755609600L, u'quotaTotalPerNode': 8756658176L}}, u'diskUsed': 58470071492L, u'vbActiveNumNonResident': 1201499, u'memUsed': 17229989416L} 2021-06-08 06:59:09,380 | test | DEBUG | MainThread | [bucket_ready_functions:delete_all_buckets:1571] Deleting existing buckets [u'default'] on 172.23.106.196 2021-06-08 06:59:09,380 | test | DEBUG | MainThread | [bucket_ready_functions:delete_all_buckets:1573] Remove bucket default ... 2021-06-08 06:59:09,380 | test | DEBUG | MainThread | [bucket_ready_functions:delete_bucket:1493] Deleting existing bucket default on ip:172.23.106.196 port:8091 ssh_username:root 2021-06-08 06:59:09,431 | test | DEBUG | MainThread | [bucket_ready_functions:parse_get_bucket_json:3739] Stats: {u'opsPerSec': 0, u'diskFetches': 1109697, u'dataUsed': 41722028032L, u'itemCount': 6647678, u'quotaPercentUsed': 49.19700113170875, u'storageTotals': {u'hdd': {u'usedByData': 58470071492L, u'used': 61816661605L, u'quotaTotal': 386354135040L, u'total': 386354135040L, u'free': 320673932084L}, u'ram': {u'usedByData': 17229989416L, u'used': 43256057856L, u'quotaTotal': 35026632704L, u'total': 49176707072L, u'quotaUsed': 35022438400L, u'quotaUsedPerNode': 8755609600L, u'quotaTotalPerNode': 8756658176L}}, u'diskUsed': 58470071492L, u'vbActiveNumNonResident': 1201499, u'memUsed': 17229989416L} 2021-06-08 06:59:23,480 | test | DEBUG | MainThread | [bucket_ready_functions:delete_bucket:1513] Deleted bucket: default from 172.23.106.196 2021-06-08 06:59:23,480 | test | DEBUG | MainThread | [bucket_ready_functions:wait_for_bucket_deletion:1533] Waiting for bucket default deletion to finish 2021-06-08 06:59:23,516 | test | DEBUG | MainThread | [bucket_ready_functions:delete_all_buckets:1571] Deleting existing buckets [] on 172.23.106.199 2021-06-08 06:59:23,536 | test | DEBUG | MainThread | [bucket_ready_functions:delete_all_buckets:1571] Deleting existing buckets [] on 172.23.106.168 2021-06-08 06:59:23,553 | test | DEBUG | MainThread | [bucket_ready_functions:delete_all_buckets:1571] Deleting existing buckets [] on 172.23.106.198 2021-06-08 06:59:23,588 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:46] Checking if ns_server is running with timeout 120secs 2021-06-08 06:59:23,617 | test | DEBUG | MainThread | [cluster_ready_functions:cleanup_cluster:176] Rebalancing all nodes in order to remove nodes 2021-06-08 06:59:23,624 | test | DEBUG | MainThread | [rest_client:remove_nodes:124] Ejecting nodes '[u'ns_1@172.23.106.168', u'ns_1@172.23.106.198', u'ns_1@172.23.106.199']' from cluster 2021-06-08 06:59:23,624 | test | DEBUG | MainThread | [rest_client:rebalance:1339] Rebalance params: {'password': 'password', 'knownNodes': u'ns_1@172.23.106.168,ns_1@172.23.106.198,ns_1@172.23.106.196,ns_1@172.23.106.199', 'ejectedNodes': u'ns_1@172.23.106.168,ns_1@172.23.106.198,ns_1@172.23.106.199', 'user': 'Administrator'} 2021-06-08 06:59:23,697 | test | DEBUG | MainThread | [rest_client:rebalance:1344] Rebalance operation started 2021-06-08 06:59:23,704 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1553] Rebalance percentage: 0.00 % 2021-06-08 06:59:23,713 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1475 2021-06-08 06:59:23,713 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-06-08 06:59:26,730 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _rebalance_status_and_progress():L1557 2021-06-08 06:59:26,730 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 06:59:31,746 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1475 2021-06-08 06:59:31,746 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-06-08 06:59:34,756 | test | INFO | MainThread | [rest_client:monitorRebalance:1484] Rebalance done. Taken 11.0590000153 seconds to complete 2021-06-08 06:59:34,763 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1487 2021-06-08 06:59:34,763 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete 2021-06-08 06:59:39,815 | test | DEBUG | MainThread | [cluster_ready_functions:cleanup_cluster:223] Removed all the nodes from cluster associated with ip:172.23.106.196 port:8091 ssh_username:root ? [(u'ns_1@172.23.106.168', 8091), (u'ns_1@172.23.106.198', 8091), (u'ns_1@172.23.106.199', 8091)] 2021-06-08 06:59:39,825 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:148] Waiting for ns_server @ 172.23.106.196:8091 2021-06-08 06:59:39,825 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:46] Checking if ns_server is running with timeout 360secs 2021-06-08 06:59:39,834 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:151] ns_server @ 172.23.106.196:8091 is running ERROR 2021-06-08 06:59:39,835 | infra | INFO | MainThread | [basetestcase:tearDownEverything:514] ========== tasks in thread pool ========== 2021-06-08 06:59:39,835 | infra | INFO | MainThread | [basetestcase:tearDownEverything:516] ========================================== During the test, Remote Connections: 77, Disconnections: 78 SDK Connections: 40, Disconnections: 40 !!!!!! CRITICAL :: Shell disconnection mismatch !!!!! ====================================================================== ERROR: test_disk_full_reduce_replica (magma.magma_disk_full.MagmaDiskFull) ---------------------------------------------------------------------- Traceback (most recent call last): File "pytests/magma/magma_disk_full.py", line 31, in tearDown MagmaBaseTest.tearDown(self) File "pytests/magma/magma_base.py", line 437, in tearDown super(MagmaBaseTest, self).tearDown() File "pytests/basetestcase.py", line 453, in tearDown self.assertFalse(result, msg="Cb_log file validation failed") AssertionError: Cb_log file validation failed ====================================================================== FAIL: test_disk_full_reduce_replica (magma.magma_disk_full.MagmaDiskFull) ---------------------------------------------------------------------- Traceback (most recent call last): File "pytests/magma/magma_disk_full.py", line 498, in test_disk_full_reduce_replica self.assertTrue(rebalance_result) AssertionError: False is not true ---------------------------------------------------------------------- Ran 1 test in 1929.900s FAILED (failures=1, errors=1) summary so far suite magma.magma_disk_full.MagmaDiskFull , pass 2 , fail 1 failures so far... magma.magma_disk_full.MagmaDiskFull.test_disk_full_reduce_replica testrunner logs, diags and results are available under /data/workspace/centos-p0-magma-vset00-00-disk_full_7.0_P0/logs/testrunner-21-Jun-08_05-04-46/test_3 Logs will be stored at /data/workspace/centos-p0-magma-vset00-00-disk_full_7.0_P0/logs/testrunner-21-Jun-08_05-04-46/test_4