guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/testexec.26553.ini bucket_storage=magma,rerun=false,bucket_eviction_policy=fullEviction,randomize_value=True,enable_dp=True,log_level=debug,infra_log_level=debug,upgrade_version=7.0.0-5270 -t magma.magma_disk_full.MagmaDiskFull.test_disk_full_reduce_replica,nodes_init=4,num_items=5000000,doc_size=2048,sdk_timeout=60,replicas=3,GROUP=P0' Test Input params: {'doc_size': '2048', 'conf_file': 'conf/magma/disk_full.conf', 'upgrade_version': '7.0.0-5270', 'spec': 'disk_full', 'rerun': 'false', 'num_nodes': 4, 'GROUP': 'P0', 'enable_dp': 'True', 'sdk_timeout': '60', 'case_number': 3, 'cluster_name': 'testexec.26553', 'ini': '/tmp/testexec.26553.ini', 'replicas': '3', 'bucket_storage': 'magma', 'log_level': 'debug', 'bucket_eviction_policy': 'fullEviction', 'logs_folder': '/data/workspace/centos-p0-magma-vset00-00-disk_full_7.0_P0/logs/testrunner-21-Jun-08_10-20-08/test_3', 'nodes_init': '4', 'num_items': '5000000', 'randomize_value': 'True', 'infra_log_level': 'debug'} test_disk_full_reduce_replica (magma.magma_disk_full.MagmaDiskFull) ... 2021-06-08 11:12:57,309 | infra | INFO | MainThread | [task_manager:__init__:14] Initiating TaskManager with 30 threads 2021-06-08 11:12:57,309 | infra | DEBUG | MainThread | [cluster:__init__:29] Initiating ServerTasks 2021-06-08 11:12:57,311 | test | INFO | MainThread | [basetestcase:log_setup_status:546] ========= MagmaDiskFull setup started for test #3 test_disk_full_reduce_replica ========= 2021-06-08 11:12:57,322 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.156 with username: root, password: couchbase 2021-06-08 11:12:57,387 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 11:12:57,591 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: ls /etc/ 2021-06-08 11:12:57,644 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 11:12:57,645 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: ls /cygdrive/c/Windows 2021-06-08 11:12:57,657 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: uname -m 2021-06-08 11:12:57,668 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: cat /proc/cpuinfo 2021-06-08 11:12:57,680 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: df -Thl 2021-06-08 11:12:57,696 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: cat /proc/meminfo 2021-06-08 11:12:57,707 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: hostname 2021-06-08 11:12:57,720 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: hostname -d 2021-06-08 11:12:57,732 | infra | DEBUG | MainThread | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.156 2021-06-08 11:12:57,732 | test | INFO | MainThread | [basetestcase:log_setup_status:546] ========= BaseTestCase setup started for test #3 test_disk_full_reduce_replica ========= 2021-06-08 11:12:57,732 | test | DEBUG | MainThread | [basetestcase:setUp:294] Cleaning up cluster 2021-06-08 11:12:57,775 | test | DEBUG | MainThread | [bucket_ready_functions:delete_all_buckets:1571] Deleting existing buckets [] on 172.23.106.156 2021-06-08 11:12:57,789 | test | DEBUG | MainThread | [bucket_ready_functions:delete_all_buckets:1571] Deleting existing buckets [] on 172.23.106.125 2021-06-08 11:12:57,805 | test | DEBUG | MainThread | [bucket_ready_functions:delete_all_buckets:1571] Deleting existing buckets [] on 172.23.106.122 2021-06-08 11:12:57,822 | test | DEBUG | MainThread | [bucket_ready_functions:delete_all_buckets:1571] Deleting existing buckets [] on 172.23.106.155 2021-06-08 11:12:57,854 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:46] Checking if ns_server is running with timeout 120secs 2021-06-08 11:12:57,888 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:148] Waiting for ns_server @ 172.23.106.156:8091 2021-06-08 11:12:57,888 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:46] Checking if ns_server is running with timeout 360secs 2021-06-08 11:12:57,897 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:151] ns_server @ 172.23.106.156:8091 is running 2021-06-08 11:12:57,897 | test | INFO | MainThread | [basetestcase:setUp:320] Initializing cluster 2021-06-08 11:12:57,907 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.156 with username: root, password: couchbase 2021-06-08 11:12:57,982 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 11:12:58,184 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: ls /etc/ 2021-06-08 11:12:58,237 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 11:12:58,239 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: ls /cygdrive/c/Windows 2021-06-08 11:12:58,250 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: uname -m 2021-06-08 11:12:58,262 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: cat /proc/cpuinfo 2021-06-08 11:12:58,273 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: df -Thl 2021-06-08 11:12:58,288 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: cat /proc/meminfo 2021-06-08 11:12:58,299 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: hostname 2021-06-08 11:12:58,311 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: hostname -d 2021-06-08 11:12:58,352 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.156 with username: root, password: couchbase 2021-06-08 11:12:58,433 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 11:12:58,630 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: ls /etc/ 2021-06-08 11:12:58,681 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 11:12:58,681 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: ls /cygdrive/c/Windows 2021-06-08 11:12:58,694 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: uname -m 2021-06-08 11:12:58,706 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: cat /proc/cpuinfo 2021-06-08 11:12:58,717 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: df -Thl 2021-06-08 11:12:58,732 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: cat /proc/meminfo 2021-06-08 11:12:58,743 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: hostname 2021-06-08 11:12:58,755 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: hostname -d 2021-06-08 11:12:58,766 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:713] 172.23.106.156 - Checking if couchbase-server is installed 2021-06-08 11:12:58,785 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:746] 172.23.106.156 - The version file /opt/couchbase/ VERSION.txt exists 2021-06-08 11:12:58,818 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: cat /opt/couchbase/VERSION.txt 2021-06-08 11:12:58,829 | infra | DEBUG | MainThread | [remote_util:stop_couchbase:3770] 172.23.106.156 - Running systemd command 2021-06-08 11:12:58,829 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: systemctl stop couchbase-server.service 2021-06-08 11:13:00,555 | test | DEBUG | MainThread | [cluster_ready_functions:stop_server:417] Couchbase stopped on ip:172.23.106.156 port:8091 ssh_username:root 2021-06-08 11:13:00,555 | infra | DEBUG | MainThread | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.156 2021-06-08 11:13:00,555 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: rm -rf /data/* 2021-06-08 11:13:00,571 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: rm -rf /opt/couchbase/var/lib/couchbase/config//* 2021-06-08 11:13:00,585 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: rm -rf /opt/couchbase/var/lib/couchbase/crash//* 2021-06-08 11:13:00,598 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: rm -rf /opt/couchbase/var/lib/couchbase/config//* 2021-06-08 11:13:00,611 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.156 with username: root, password: couchbase 2021-06-08 11:13:00,677 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 11:13:00,884 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: ls /etc/ 2021-06-08 11:13:00,940 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 11:13:00,940 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: ls /cygdrive/c/Windows 2021-06-08 11:13:00,954 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: uname -m 2021-06-08 11:13:00,967 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: cat /proc/cpuinfo 2021-06-08 11:13:00,979 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: df -Thl 2021-06-08 11:13:00,994 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: cat /proc/meminfo 2021-06-08 11:13:01,006 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: hostname 2021-06-08 11:13:01,019 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: hostname -d 2021-06-08 11:13:01,030 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:713] 172.23.106.156 - Checking if couchbase-server is installed 2021-06-08 11:13:01,049 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:746] 172.23.106.156 - The version file /opt/couchbase/ VERSION.txt exists 2021-06-08 11:13:01,084 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: cat /opt/couchbase/VERSION.txt 2021-06-08 11:13:01,095 | infra | DEBUG | MainThread | [remote_util:start_couchbase:3867] 172.23.106.156 - Running systemd command 2021-06-08 11:13:01,095 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: systemctl start couchbase-server.service 2021-06-08 11:13:01,118 | test | DEBUG | MainThread | [cluster_ready_functions:start_server:431] Couchbase started on ip:172.23.106.156 port:8091 ssh_username:root 2021-06-08 11:13:01,118 | infra | DEBUG | MainThread | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.156 Got this failure java.net.ConnectException: Connection refused: /172.23.106.156:8091 during connect (<_realsocket at 0x11d23 type=client open_count=1 channel=[id: 0xb4152536, 0.0.0.0/0.0.0.0:45260] timeout=300.0>) 2021-06-08 11:13:01,124 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.106.156:8091/nodes/self error [Errno 111] Connection refused 2021-06-08 11:13:01,131 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L762 2021-06-08 11:13:01,131 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None Got this failure java.net.ConnectException: Connection refused: /172.23.106.156:8091 during connect (<_realsocket at 0x11d24 type=client open_count=1 channel=[id: 0x17adafed, 0.0.0.0/0.0.0.0:48847] timeout=300.0>) 2021-06-08 11:13:03,132 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.106.156:8091/nodes/self error [Errno 111] Connection refused 2021-06-08 11:13:03,140 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L762 2021-06-08 11:13:03,140 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None Got this failure java.net.ConnectException: Connection refused: /172.23.106.156:8091 during connect (<_realsocket at 0x11d25 type=client open_count=1 channel=[id: 0x29ca9b59, 0.0.0.0/0.0.0.0:36344] timeout=300.0>) 2021-06-08 11:13:05,141 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.106.156:8091/nodes/self error [Errno 111] Connection refused 2021-06-08 11:13:05,148 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L762 2021-06-08 11:13:05,150 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None Got this failure java.net.ConnectException: Connection refused: /172.23.106.156:8091 during connect (<_realsocket at 0x11d26 type=client open_count=1 channel=[id: 0xb1fdd251, 0.0.0.0/0.0.0.0:59605] timeout=300.0>) 2021-06-08 11:13:07,151 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.106.156:8091/nodes/self error [Errno 111] Connection refused 2021-06-08 11:13:07,157 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L762 2021-06-08 11:13:07,157 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-06-08 11:13:09,171 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:46] Checking if ns_server is running with timeout 360secs 2021-06-08 11:13:09,180 | infra | DEBUG | MainThread | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.156 2021-06-08 11:13:09,183 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.125 with username: root, password: couchbase 2021-06-08 11:13:09,249 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 11:13:09,454 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: ls /etc/ 2021-06-08 11:13:09,506 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 11:13:09,506 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: ls /cygdrive/c/Windows 2021-06-08 11:13:09,519 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: uname -m 2021-06-08 11:13:09,529 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: cat /proc/cpuinfo 2021-06-08 11:13:09,540 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: df -Thl 2021-06-08 11:13:09,555 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: cat /proc/meminfo 2021-06-08 11:13:09,565 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: hostname 2021-06-08 11:13:09,576 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: hostname -d 2021-06-08 11:13:09,618 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.125 with username: root, password: couchbase 2021-06-08 11:13:09,700 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 11:13:09,897 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: ls /etc/ 2021-06-08 11:13:09,947 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 11:13:09,947 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: ls /cygdrive/c/Windows 2021-06-08 11:13:09,960 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: uname -m 2021-06-08 11:13:09,970 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: cat /proc/cpuinfo 2021-06-08 11:13:09,982 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: df -Thl 2021-06-08 11:13:09,996 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: cat /proc/meminfo 2021-06-08 11:13:10,006 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: hostname 2021-06-08 11:13:10,017 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: hostname -d 2021-06-08 11:13:10,029 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:713] 172.23.106.125 - Checking if couchbase-server is installed 2021-06-08 11:13:10,046 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:746] 172.23.106.125 - The version file /opt/couchbase/ VERSION.txt exists 2021-06-08 11:13:10,078 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: cat /opt/couchbase/VERSION.txt 2021-06-08 11:13:10,091 | infra | DEBUG | MainThread | [remote_util:stop_couchbase:3770] 172.23.106.125 - Running systemd command 2021-06-08 11:13:10,091 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: systemctl stop couchbase-server.service 2021-06-08 11:13:11,670 | test | DEBUG | MainThread | [cluster_ready_functions:stop_server:417] Couchbase stopped on ip:172.23.106.125 port:8091 ssh_username:root 2021-06-08 11:13:11,670 | infra | DEBUG | MainThread | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.125 2021-06-08 11:13:11,670 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: rm -rf /data/* 2021-06-08 11:13:11,686 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: rm -rf /opt/couchbase/var/lib/couchbase/config//* 2021-06-08 11:13:11,698 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: rm -rf /opt/couchbase/var/lib/couchbase/crash//* 2021-06-08 11:13:11,710 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: rm -rf /opt/couchbase/var/lib/couchbase/config//* 2021-06-08 11:13:11,723 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.125 with username: root, password: couchbase 2021-06-08 11:13:11,798 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 11:13:12,000 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: ls /etc/ 2021-06-08 11:13:12,052 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 11:13:12,052 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: ls /cygdrive/c/Windows 2021-06-08 11:13:12,065 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: uname -m 2021-06-08 11:13:12,075 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: cat /proc/cpuinfo 2021-06-08 11:13:12,088 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: df -Thl 2021-06-08 11:13:12,102 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: cat /proc/meminfo 2021-06-08 11:13:12,114 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: hostname 2021-06-08 11:13:12,125 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: hostname -d 2021-06-08 11:13:12,138 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:713] 172.23.106.125 - Checking if couchbase-server is installed 2021-06-08 11:13:12,154 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:746] 172.23.106.125 - The version file /opt/couchbase/ VERSION.txt exists 2021-06-08 11:13:12,187 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: cat /opt/couchbase/VERSION.txt 2021-06-08 11:13:12,200 | infra | DEBUG | MainThread | [remote_util:start_couchbase:3867] 172.23.106.125 - Running systemd command 2021-06-08 11:13:12,200 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: systemctl start couchbase-server.service 2021-06-08 11:13:12,223 | test | DEBUG | MainThread | [cluster_ready_functions:start_server:431] Couchbase started on ip:172.23.106.125 port:8091 ssh_username:root 2021-06-08 11:13:12,223 | infra | DEBUG | MainThread | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.125 Got this failure java.net.ConnectException: Connection refused: /172.23.106.125:8091 during connect (<_realsocket at 0x11d27 type=client open_count=1 channel=[id: 0x34feab57, 0.0.0.0/0.0.0.0:49052] timeout=300.0>) 2021-06-08 11:13:12,224 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.106.125:8091/nodes/self error [Errno 111] Connection refused 2021-06-08 11:13:12,232 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L762 2021-06-08 11:13:12,232 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None Got this failure java.net.ConnectException: Connection refused: /172.23.106.125:8091 during connect (<_realsocket at 0x11d28 type=client open_count=1 channel=[id: 0x206fa069, 0.0.0.0/0.0.0.0:40011] timeout=300.0>) 2021-06-08 11:13:14,234 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.106.125:8091/nodes/self error [Errno 111] Connection refused 2021-06-08 11:13:14,240 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L762 2021-06-08 11:13:14,240 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None Got this failure java.net.ConnectException: Connection refused: /172.23.106.125:8091 during connect (<_realsocket at 0x11d29 type=client open_count=1 channel=[id: 0x197fbdf0, 0.0.0.0/0.0.0.0:54836] timeout=300.0>) 2021-06-08 11:13:16,243 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.106.125:8091/nodes/self error [Errno 111] Connection refused 2021-06-08 11:13:16,250 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L762 2021-06-08 11:13:16,250 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None Got this failure java.net.ConnectException: Connection refused: /172.23.106.125:8091 during connect (<_realsocket at 0x11d2a type=client open_count=1 channel=[id: 0x21ee0995, 0.0.0.0/0.0.0.0:37084] timeout=300.0>) 2021-06-08 11:13:18,253 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.106.125:8091/nodes/self error [Errno 111] Connection refused 2021-06-08 11:13:18,259 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L762 2021-06-08 11:13:18,259 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-06-08 11:13:20,273 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:46] Checking if ns_server is running with timeout 360secs 2021-06-08 11:13:20,282 | infra | DEBUG | MainThread | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.125 2021-06-08 11:13:20,285 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.122 with username: root, password: couchbase 2021-06-08 11:13:20,358 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 11:13:20,565 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: ls /etc/ 2021-06-08 11:13:20,621 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 11:13:20,621 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: ls /cygdrive/c/Windows 2021-06-08 11:13:20,634 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: uname -m 2021-06-08 11:13:20,645 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: cat /proc/cpuinfo 2021-06-08 11:13:20,657 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: df -Thl 2021-06-08 11:13:20,671 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: cat /proc/meminfo 2021-06-08 11:13:20,684 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: hostname 2021-06-08 11:13:20,696 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: hostname -d 2021-06-08 11:13:20,736 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.122 with username: root, password: couchbase 2021-06-08 11:13:20,802 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 11:13:21,000 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: ls /etc/ 2021-06-08 11:13:21,052 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 11:13:21,052 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: ls /cygdrive/c/Windows 2021-06-08 11:13:21,063 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: uname -m 2021-06-08 11:13:21,075 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: cat /proc/cpuinfo 2021-06-08 11:13:21,085 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: df -Thl 2021-06-08 11:13:21,099 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: cat /proc/meminfo 2021-06-08 11:13:21,111 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: hostname 2021-06-08 11:13:21,124 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: hostname -d 2021-06-08 11:13:21,135 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:713] 172.23.106.122 - Checking if couchbase-server is installed 2021-06-08 11:13:21,154 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:746] 172.23.106.122 - The version file /opt/couchbase/ VERSION.txt exists 2021-06-08 11:13:21,188 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: cat /opt/couchbase/VERSION.txt 2021-06-08 11:13:21,200 | infra | DEBUG | MainThread | [remote_util:stop_couchbase:3770] 172.23.106.122 - Running systemd command 2021-06-08 11:13:21,200 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: systemctl stop couchbase-server.service 2021-06-08 11:13:22,838 | test | DEBUG | MainThread | [cluster_ready_functions:stop_server:417] Couchbase stopped on ip:172.23.106.122 port:8091 ssh_username:root 2021-06-08 11:13:22,838 | infra | DEBUG | MainThread | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.122 2021-06-08 11:13:22,838 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: rm -rf /data/* 2021-06-08 11:13:22,855 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: rm -rf /opt/couchbase/var/lib/couchbase/config//* 2021-06-08 11:13:22,868 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: rm -rf /opt/couchbase/var/lib/couchbase/crash//* 2021-06-08 11:13:22,881 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: rm -rf /opt/couchbase/var/lib/couchbase/config//* 2021-06-08 11:13:22,895 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.122 with username: root, password: couchbase 2021-06-08 11:13:22,971 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 11:13:23,174 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: ls /etc/ 2021-06-08 11:13:23,226 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 11:13:23,226 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: ls /cygdrive/c/Windows 2021-06-08 11:13:23,237 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: uname -m 2021-06-08 11:13:23,249 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: cat /proc/cpuinfo 2021-06-08 11:13:23,260 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: df -Thl 2021-06-08 11:13:23,276 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: cat /proc/meminfo 2021-06-08 11:13:23,286 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: hostname 2021-06-08 11:13:23,298 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: hostname -d 2021-06-08 11:13:23,311 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:713] 172.23.106.122 - Checking if couchbase-server is installed 2021-06-08 11:13:23,326 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:746] 172.23.106.122 - The version file /opt/couchbase/ VERSION.txt exists 2021-06-08 11:13:23,361 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: cat /opt/couchbase/VERSION.txt 2021-06-08 11:13:23,372 | infra | DEBUG | MainThread | [remote_util:start_couchbase:3867] 172.23.106.122 - Running systemd command 2021-06-08 11:13:23,372 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: systemctl start couchbase-server.service 2021-06-08 11:13:23,395 | test | DEBUG | MainThread | [cluster_ready_functions:start_server:431] Couchbase started on ip:172.23.106.122 port:8091 ssh_username:root 2021-06-08 11:13:23,395 | infra | DEBUG | MainThread | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.122 Got this failure java.net.ConnectException: Connection refused: /172.23.106.122:8091 during connect (<_realsocket at 0x11d2b type=client open_count=1 channel=[id: 0x09ccf32b, 0.0.0.0/0.0.0.0:35232] timeout=300.0>) 2021-06-08 11:13:23,398 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.106.122:8091/nodes/self error [Errno 111] Connection refused 2021-06-08 11:13:23,405 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L762 2021-06-08 11:13:23,405 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None Got this failure java.net.ConnectException: Connection refused: /172.23.106.122:8091 during connect (<_realsocket at 0x11d2c type=client open_count=1 channel=[id: 0xe21bf920, 0.0.0.0/0.0.0.0:52816] timeout=300.0>) 2021-06-08 11:13:25,407 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.106.122:8091/nodes/self error [Errno 111] Connection refused 2021-06-08 11:13:25,414 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L762 2021-06-08 11:13:25,414 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None Got this failure java.net.ConnectException: Connection refused: /172.23.106.122:8091 during connect (<_realsocket at 0x11d2d type=client open_count=1 channel=[id: 0x1bf9d3b6, 0.0.0.0/0.0.0.0:51674] timeout=300.0>) 2021-06-08 11:13:27,417 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.106.122:8091/nodes/self error [Errno 111] Connection refused 2021-06-08 11:13:27,424 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L762 2021-06-08 11:13:27,424 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None Got this failure java.net.ConnectException: Connection refused: /172.23.106.122:8091 during connect (<_realsocket at 0x11d2e type=client open_count=1 channel=[id: 0x5d552f42, 0.0.0.0/0.0.0.0:39336] timeout=300.0>) 2021-06-08 11:13:29,427 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.106.122:8091/nodes/self error [Errno 111] Connection refused 2021-06-08 11:13:29,434 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L762 2021-06-08 11:13:29,434 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-06-08 11:13:31,447 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:46] Checking if ns_server is running with timeout 360secs 2021-06-08 11:13:31,456 | infra | DEBUG | MainThread | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.122 2021-06-08 11:13:31,457 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.155 with username: root, password: couchbase 2021-06-08 11:13:31,529 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 11:13:31,733 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: ls /etc/ 2021-06-08 11:13:31,786 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 11:13:31,786 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: ls /cygdrive/c/Windows 2021-06-08 11:13:31,799 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: uname -m 2021-06-08 11:13:31,811 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: cat /proc/cpuinfo 2021-06-08 11:13:31,822 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: df -Thl 2021-06-08 11:13:31,836 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: cat /proc/meminfo 2021-06-08 11:13:31,848 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: hostname 2021-06-08 11:13:31,861 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: hostname -d 2021-06-08 11:13:31,901 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.155 with username: root, password: couchbase 2021-06-08 11:13:31,996 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 11:13:32,194 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: ls /etc/ 2021-06-08 11:13:32,246 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 11:13:32,246 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: ls /cygdrive/c/Windows 2021-06-08 11:13:32,257 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: uname -m 2021-06-08 11:13:32,269 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: cat /proc/cpuinfo 2021-06-08 11:13:32,279 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: df -Thl 2021-06-08 11:13:32,293 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: cat /proc/meminfo 2021-06-08 11:13:32,305 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: hostname 2021-06-08 11:13:32,316 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: hostname -d 2021-06-08 11:13:32,328 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:713] 172.23.106.155 - Checking if couchbase-server is installed 2021-06-08 11:13:32,345 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:746] 172.23.106.155 - The version file /opt/couchbase/ VERSION.txt exists 2021-06-08 11:13:32,378 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: cat /opt/couchbase/VERSION.txt 2021-06-08 11:13:32,390 | infra | DEBUG | MainThread | [remote_util:stop_couchbase:3770] 172.23.106.155 - Running systemd command 2021-06-08 11:13:32,390 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: systemctl stop couchbase-server.service 2021-06-08 11:13:34,069 | test | DEBUG | MainThread | [cluster_ready_functions:stop_server:417] Couchbase stopped on ip:172.23.106.155 port:8091 ssh_username:root 2021-06-08 11:13:34,069 | infra | DEBUG | MainThread | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.155 2021-06-08 11:13:34,069 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: rm -rf /data/* 2021-06-08 11:13:34,085 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: rm -rf /opt/couchbase/var/lib/couchbase/config//* 2021-06-08 11:13:34,098 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: rm -rf /opt/couchbase/var/lib/couchbase/crash//* 2021-06-08 11:13:34,108 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: rm -rf /opt/couchbase/var/lib/couchbase/config//* 2021-06-08 11:13:34,121 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.155 with username: root, password: couchbase 2021-06-08 11:13:34,210 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 11:13:34,415 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: ls /etc/ 2021-06-08 11:13:34,467 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 11:13:34,467 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: ls /cygdrive/c/Windows 2021-06-08 11:13:34,480 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: uname -m 2021-06-08 11:13:34,490 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: cat /proc/cpuinfo 2021-06-08 11:13:34,503 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: df -Thl 2021-06-08 11:13:34,516 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: cat /proc/meminfo 2021-06-08 11:13:34,528 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: hostname 2021-06-08 11:13:34,539 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: hostname -d 2021-06-08 11:13:34,551 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:713] 172.23.106.155 - Checking if couchbase-server is installed 2021-06-08 11:13:34,568 | infra | DEBUG | MainThread | [remote_util:is_couchbase_installed:746] 172.23.106.155 - The version file /opt/couchbase/ VERSION.txt exists 2021-06-08 11:13:34,601 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: cat /opt/couchbase/VERSION.txt 2021-06-08 11:13:34,611 | infra | DEBUG | MainThread | [remote_util:start_couchbase:3867] 172.23.106.155 - Running systemd command 2021-06-08 11:13:34,611 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: systemctl start couchbase-server.service 2021-06-08 11:13:34,632 | test | DEBUG | MainThread | [cluster_ready_functions:start_server:431] Couchbase started on ip:172.23.106.155 port:8091 ssh_username:root 2021-06-08 11:13:34,632 | infra | DEBUG | MainThread | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.155 Got this failure java.net.ConnectException: Connection refused: /172.23.106.155:8091 during connect (<_realsocket at 0x11d2f type=client open_count=1 channel=[id: 0xe266db02, 0.0.0.0/0.0.0.0:51588] timeout=300.0>) 2021-06-08 11:13:34,635 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.106.155:8091/nodes/self error [Errno 111] Connection refused 2021-06-08 11:13:34,641 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L762 2021-06-08 11:13:34,641 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None Got this failure java.net.ConnectException: Connection refused: /172.23.106.155:8091 during connect (<_realsocket at 0x11d30 type=client open_count=1 channel=[id: 0x306074ea, 0.0.0.0/0.0.0.0:59548] timeout=300.0>) 2021-06-08 11:13:36,644 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.106.155:8091/nodes/self error [Errno 111] Connection refused 2021-06-08 11:13:36,651 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L762 2021-06-08 11:13:36,651 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None Got this failure java.net.ConnectException: Connection refused: /172.23.106.155:8091 during connect (<_realsocket at 0x11d31 type=client open_count=1 channel=[id: 0x43e9e9c7, 0.0.0.0/0.0.0.0:36880] timeout=300.0>) 2021-06-08 11:13:38,654 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.106.155:8091/nodes/self error [Errno 111] Connection refused 2021-06-08 11:13:38,661 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L762 2021-06-08 11:13:38,661 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None Got this failure java.net.ConnectException: Connection refused: /172.23.106.155:8091 during connect (<_realsocket at 0x11d32 type=client open_count=1 channel=[id: 0x340921ad, 0.0.0.0/0.0.0.0:50225] timeout=300.0>) 2021-06-08 11:13:40,663 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.106.155:8091/nodes/self error [Errno 111] Connection refused 2021-06-08 11:13:40,670 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _http_request():L762 2021-06-08 11:13:40,670 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-06-08 11:13:42,681 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:46] Checking if ns_server is running with timeout 360secs 2021-06-08 11:13:42,690 | infra | DEBUG | MainThread | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.155 2021-06-08 11:13:42,697 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from couchbase_utils/cluster_utils/cluster_ready_functions.py -> reset_cluster():L507 2021-06-08 11:13:42,697 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-06-08 11:13:52,700 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.156 with username: root, password: couchbase 2021-06-08 11:13:52,796 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 11:13:53,046 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: ls /etc/ 2021-06-08 11:13:53,118 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 11:13:53,118 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: ls /cygdrive/c/Windows 2021-06-08 11:13:53,137 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: uname -m 2021-06-08 11:13:53,153 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: cat /proc/cpuinfo 2021-06-08 11:13:53,171 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: df -Thl 2021-06-08 11:13:53,191 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: cat /proc/meminfo 2021-06-08 11:13:53,207 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: hostname 2021-06-08 11:13:53,224 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: hostname -d 2021-06-08 11:13:53,243 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.125 with username: root, password: couchbase 2021-06-08 11:13:53,312 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 11:13:53,523 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: ls /etc/ 2021-06-08 11:13:53,576 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 11:13:53,578 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: ls /cygdrive/c/Windows 2021-06-08 11:13:53,589 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: uname -m 2021-06-08 11:13:53,601 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: cat /proc/cpuinfo 2021-06-08 11:13:53,614 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: df -Thl 2021-06-08 11:13:53,628 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: cat /proc/meminfo 2021-06-08 11:13:53,638 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: hostname 2021-06-08 11:13:53,651 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: hostname -d 2021-06-08 11:13:53,664 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.122 with username: root, password: couchbase 2021-06-08 11:13:53,734 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 11:13:53,944 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: ls /etc/ 2021-06-08 11:13:53,996 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 11:13:53,996 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: ls /cygdrive/c/Windows 2021-06-08 11:13:54,009 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: uname -m 2021-06-08 11:13:54,020 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: cat /proc/cpuinfo 2021-06-08 11:13:54,032 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: df -Thl 2021-06-08 11:13:54,046 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: cat /proc/meminfo 2021-06-08 11:13:54,058 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: hostname 2021-06-08 11:13:54,069 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: hostname -d 2021-06-08 11:13:54,084 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.155 with username: root, password: couchbase 2021-06-08 11:13:54,161 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 11:13:54,368 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: ls /etc/ 2021-06-08 11:13:54,421 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 11:13:54,421 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: ls /cygdrive/c/Windows 2021-06-08 11:13:54,434 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: uname -m 2021-06-08 11:13:54,446 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: cat /proc/cpuinfo 2021-06-08 11:13:54,457 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: df -Thl 2021-06-08 11:13:54,470 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: cat /proc/meminfo 2021-06-08 11:13:54,483 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: hostname 2021-06-08 11:13:54,493 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: hostname -d 2021-06-08 11:13:54,506 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: rm -rf /data/* 2021-06-08 11:13:54,520 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: chown -R couchbase:couchbase /data 2021-06-08 11:13:54,543 | test | DEBUG | MainThread | [rest_client:set_data_path:2138] /nodes/self/controller/settings: path=%2Fdata&index_path=%2Fdata 2021-06-08 11:13:54,552 | test | DEBUG | MainThread | [rest_client:set_data_path:2141] Setting paths: path=%2Fdata&index_path=%2Fdata: status 1 2021-06-08 11:13:54,552 | infra | INFO | MainThread | [task_manager:add_new_task:23] Added new task: node_init_task_172.23.106.156_8091 2021-06-08 11:13:54,552 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: rm -rf /data/* 2021-06-08 11:13:54,552 | infra | INFO | pool-5-thread-1 | [task:start_task:87] Thread 'node_init_task_172.23.106.156_8091' started 2021-06-08 11:13:54,566 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: chown -R couchbase:couchbase /data 2021-06-08 11:13:54,572 | infra | DEBUG | pool-5-thread-1 | [task:wait_until:131] Wait for expected condition to get satisfied 2021-06-08 11:13:54,572 | test | DEBUG | pool-5-thread-1 | [task:call:5740] server: ip:172.23.106.156 port:8091 ssh_username:root, nodes/self: {'memcached': 11210, 'memoryFree': 11742625792L, 'eventingMemoryQuota': 256, 'rest_password': '', 'storage': [], 'ports': [], 'cbasMemoryQuota': 1839, 'hostname': u'172.23.106.156:8091', 'ftsMemoryQuota': 512, 'clusterCompatibility': 458752, 'id': u'ns_1@172.23.106.156', 'mcdMemoryReserved': 9379, 'moxi': 11211, 'rest_username': '', 'os': u'x86_64-unknown-linux-gnu', 'ip': u'172.23.106.156', 'curr_items': 0, 'mcdMemoryAllocated': 9379, 'services': [u'kv'], 'version': u'7.0.0-5270-enterprise', 'indexMemoryQuota': 512, 'uptime': u'51', 'clusterMembership': u'active', 'storageTotalRam': 11724L, 'port': u'8091', 'availableStorage': [], 'memoryTotal': 12294176768L, 'memoryQuota': 6368, 'status': u'healthy'} 2021-06-08 11:13:54,572 | test | DEBUG | pool-5-thread-1 | [rest_client:set_service_mem_quota:859] Setting RAM quotas: {'memoryQuota': 8351} 2021-06-08 11:13:54,585 | test | DEBUG | pool-5-thread-1 | [rest_client:set_indexer_storage_mode:883] settings/indexes params: storageMode=plasma 2021-06-08 11:13:54,588 | test | DEBUG | MainThread | [rest_client:set_data_path:2138] /nodes/self/controller/settings: path=%2Fdata&index_path=%2Fdata 2021-06-08 11:13:54,596 | test | DEBUG | pool-5-thread-1 | [rest_client:init_node_services:837] /node/controller/setupServices params on 172.23.106.156: 8091:services=kv&hostname=172.23.106.156&password=password&user=Administrator 2021-06-08 11:13:54,598 | test | DEBUG | MainThread | [rest_client:set_data_path:2141] Setting paths: path=%2Fdata&index_path=%2Fdata: status 1 2021-06-08 11:13:54,598 | infra | INFO | MainThread | [task_manager:add_new_task:23] Added new task: node_init_task_172.23.106.125_8091 2021-06-08 11:13:54,598 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: rm -rf /data/* 2021-06-08 11:13:54,598 | infra | INFO | pool-5-thread-2 | [task:start_task:87] Thread 'node_init_task_172.23.106.125_8091' started 2021-06-08 11:13:54,611 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: chown -R couchbase:couchbase /data 2021-06-08 11:13:54,615 | infra | DEBUG | pool-5-thread-2 | [task:wait_until:131] Wait for expected condition to get satisfied 2021-06-08 11:13:54,615 | test | DEBUG | pool-5-thread-2 | [task:call:5740] server: ip:172.23.106.125 port:8091 ssh_username:root, nodes/self: {'memcached': 11210, 'memoryFree': 11754807296L, 'eventingMemoryQuota': 256, 'rest_password': '', 'storage': [], 'ports': [], 'cbasMemoryQuota': 1839, 'hostname': u'172.23.106.125:8091', 'ftsMemoryQuota': 512, 'clusterCompatibility': 458752, 'id': u'ns_1@172.23.106.125', 'mcdMemoryReserved': 9379, 'moxi': 11211, 'rest_username': '', 'os': u'x86_64-unknown-linux-gnu', 'ip': u'172.23.106.125', 'curr_items': 0, 'mcdMemoryAllocated': 9379, 'services': [u'kv'], 'version': u'7.0.0-5270-enterprise', 'indexMemoryQuota': 512, 'uptime': u'35', 'clusterMembership': u'active', 'storageTotalRam': 11724L, 'port': u'8091', 'availableStorage': [], 'memoryTotal': 12294176768L, 'memoryQuota': 6368, 'status': u'healthy'} 2021-06-08 11:13:54,615 | test | DEBUG | pool-5-thread-2 | [rest_client:set_service_mem_quota:859] Setting RAM quotas: {'memoryQuota': 8351} 2021-06-08 11:13:54,628 | test | DEBUG | pool-5-thread-2 | [rest_client:set_indexer_storage_mode:883] settings/indexes params: storageMode=plasma 2021-06-08 11:13:54,631 | test | DEBUG | pool-5-thread-1 | [rest_client:init_cluster:770] settings/web params on 172.23.106.156:8091:password=password&port=8091&username=Administrator 2021-06-08 11:13:54,634 | test | DEBUG | MainThread | [rest_client:set_data_path:2138] /nodes/self/controller/settings: path=%2Fdata&index_path=%2Fdata 2021-06-08 11:13:54,638 | test | DEBUG | pool-5-thread-2 | [rest_client:init_cluster:770] settings/web params on 172.23.106.125:8091:password=password&port=8091&username=Administrator 2021-06-08 11:13:54,642 | test | DEBUG | MainThread | [rest_client:set_data_path:2141] Setting paths: path=%2Fdata&index_path=%2Fdata: status 1 2021-06-08 11:13:54,642 | infra | INFO | MainThread | [task_manager:add_new_task:23] Added new task: node_init_task_172.23.106.122_8091 2021-06-08 11:13:54,644 | infra | INFO | pool-5-thread-3 | [task:start_task:87] Thread 'node_init_task_172.23.106.122_8091' started 2021-06-08 11:13:54,644 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: rm -rf /data/* 2021-06-08 11:13:54,657 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: chown -R couchbase:couchbase /data 2021-06-08 11:13:54,660 | infra | DEBUG | pool-5-thread-3 | [task:wait_until:131] Wait for expected condition to get satisfied 2021-06-08 11:13:54,660 | test | DEBUG | pool-5-thread-3 | [task:call:5740] server: ip:172.23.106.122 port:8091 ssh_username:root, nodes/self: {'memcached': 11210, 'memoryFree': 11736715264L, 'eventingMemoryQuota': 256, 'rest_password': '', 'storage': [], 'ports': [], 'cbasMemoryQuota': 1839, 'hostname': u'172.23.106.122:8091', 'ftsMemoryQuota': 512, 'clusterCompatibility': 458752, 'id': u'ns_1@172.23.106.122', 'mcdMemoryReserved': 9379, 'moxi': 11211, 'rest_username': '', 'os': u'x86_64-unknown-linux-gnu', 'ip': u'172.23.106.122', 'curr_items': 0, 'mcdMemoryAllocated': 9379, 'services': [u'kv'], 'version': u'7.0.0-5270-enterprise', 'indexMemoryQuota': 512, 'uptime': u'25', 'clusterMembership': u'active', 'storageTotalRam': 11724L, 'port': u'8091', 'availableStorage': [], 'memoryTotal': 12294176768L, 'memoryQuota': 6368, 'status': u'healthy'} 2021-06-08 11:13:54,661 | test | DEBUG | pool-5-thread-3 | [rest_client:set_service_mem_quota:859] Setting RAM quotas: {'memoryQuota': 8351} 2021-06-08 11:13:54,671 | test | DEBUG | pool-5-thread-3 | [rest_client:set_indexer_storage_mode:883] settings/indexes params: storageMode=plasma 2021-06-08 11:13:54,680 | test | DEBUG | MainThread | [rest_client:set_data_path:2138] /nodes/self/controller/settings: path=%2Fdata&index_path=%2Fdata 2021-06-08 11:13:54,683 | test | DEBUG | pool-5-thread-3 | [rest_client:init_cluster:770] settings/web params on 172.23.106.122:8091:password=password&port=8091&username=Administrator 2021-06-08 11:13:54,687 | test | DEBUG | MainThread | [rest_client:set_data_path:2141] Setting paths: path=%2Fdata&index_path=%2Fdata: status 1 2021-06-08 11:13:54,687 | infra | INFO | MainThread | [task_manager:add_new_task:23] Added new task: node_init_task_172.23.106.155_8091 2021-06-08 11:13:54,687 | infra | DEBUG | MainThread | [task_manager:get_task_result:26] Getting task result for node_init_task_172.23.106.156_8091 2021-06-08 11:13:54,687 | infra | INFO | pool-5-thread-4 | [task:start_task:87] Thread 'node_init_task_172.23.106.155_8091' started 2021-06-08 11:13:54,707 | infra | DEBUG | pool-5-thread-4 | [task:wait_until:131] Wait for expected condition to get satisfied 2021-06-08 11:13:54,707 | test | DEBUG | pool-5-thread-4 | [task:call:5740] server: ip:172.23.106.155 port:8091 ssh_username:root, nodes/self: {'memcached': 11210, 'memoryFree': 11742015488L, 'eventingMemoryQuota': 256, 'rest_password': '', 'storage': [], 'ports': [], 'cbasMemoryQuota': 1839, 'hostname': u'172.23.106.155:8091', 'ftsMemoryQuota': 512, 'clusterCompatibility': 458752, 'id': u'ns_1@172.23.106.155', 'mcdMemoryReserved': 9379, 'moxi': 11211, 'rest_username': '', 'os': u'x86_64-unknown-linux-gnu', 'ip': u'172.23.106.155', 'curr_items': 0, 'mcdMemoryAllocated': 9379, 'services': [u'kv'], 'version': u'7.0.0-5270-enterprise', 'indexMemoryQuota': 512, 'uptime': u'15', 'clusterMembership': u'active', 'storageTotalRam': 11724L, 'port': u'8091', 'availableStorage': [], 'memoryTotal': 12294176768L, 'memoryQuota': 6368, 'status': u'healthy'} 2021-06-08 11:13:54,707 | test | DEBUG | pool-5-thread-4 | [rest_client:set_service_mem_quota:859] Setting RAM quotas: {'memoryQuota': 8351} 2021-06-08 11:13:54,719 | test | DEBUG | pool-5-thread-4 | [rest_client:set_indexer_storage_mode:883] settings/indexes params: storageMode=plasma 2021-06-08 11:13:54,727 | infra | DEBUG | MainThread | [task_manager:get_task_result:26] Getting task result for node_init_task_172.23.106.125_8091 2021-06-08 11:13:54,727 | infra | DEBUG | MainThread | [task_manager:get_task_result:26] Getting task result for node_init_task_172.23.106.122_8091 2021-06-08 11:13:54,730 | test | DEBUG | pool-5-thread-4 | [rest_client:init_cluster:770] settings/web params on 172.23.106.155:8091:password=password&port=8091&username=Administrator 2021-06-08 11:13:54,766 | infra | DEBUG | MainThread | [task_manager:get_task_result:26] Getting task result for node_init_task_172.23.106.155_8091 2021-06-08 11:13:54,815 | test | WARNING | MainThread | [basetestcase:_initialize_nodes:598] RAM quota was defined less than 100 MB: 2021-06-08 11:13:54,815 | test | DEBUG | MainThread | [basetestcase:_initialize_nodes:601] Node: 172.23.106.156: RAM: ['MemTotal: 12006032 kB\n', 'MemFree: 10768420 kB\n', 'MemAvailable: 11076956 kB\n', 'Buffers: 0 kB\n', 'Cached: 698664 kB\n', 'SwapCached: 0 kB\n', 'Active: 669056 kB\n', 'Inactive: 295080 kB\n', 'Active(anon): 253228 kB\n', 'Inactive(anon): 198032 kB\n', 'Active(file): 415828 kB\n', 'Inactive(file): 97048 kB\n', 'Unevictable: 15744 kB\n', 'Mlocked: 15744 kB\n', 'SwapTotal: 0 kB\n', 'SwapFree: 0 kB\n', 'Dirty: 48 kB\n', 'Writeback: 0 kB\n', 'AnonPages: 281280 kB\n', 'Mapped: 88288 kB\n', 'Shmem: 182148 kB\n', 'Slab: 126432 kB\n', 'SReclaimable: 59416 kB\n', 'SUnreclaim: 67016 kB\n', 'KernelStack: 8608 kB\n', 'PageTables: 8116 kB\n', 'NFS_Unstable: 0 kB\n', 'Bounce: 0 kB\n', 'WritebackTmp: 0 kB\n', 'CommitLimit: 6003016 kB\n', 'Committed_AS: 1545000 kB\n', 'VmallocTotal: 34359738367 kB\n', 'VmallocUsed: 76372 kB\n', 'VmallocChunk: 34359651324 kB\n', 'HardwareCorrupted: 0 kB\n', 'AnonHugePages: 10240 kB\n', 'CmaTotal: 0 kB\n', 'CmaFree: 0 kB\n', 'HugePages_Total: 0\n', 'HugePages_Free: 0\n', 'HugePages_Rsvd: 0\n', 'HugePages_Surp: 0\n', 'Hugepagesize: 2048 kB\n', 'DirectMap4k: 149504 kB\n', 'DirectMap2M: 6029312 kB\n', 'DirectMap1G: 6291456 kB\n'] 2021-06-08 11:13:54,815 | test | DEBUG | MainThread | [basetestcase:_initialize_nodes:601] Node: 172.23.106.125: RAM: ['MemTotal: 12006032 kB\n', 'MemFree: 10636600 kB\n', 'MemAvailable: 10962328 kB\n', 'Buffers: 0 kB\n', 'Cached: 840352 kB\n', 'SwapCached: 0 kB\n', 'Active: 689684 kB\n', 'Inactive: 396556 kB\n', 'Active(anon): 234300 kB\n', 'Inactive(anon): 327980 kB\n', 'Active(file): 455384 kB\n', 'Inactive(file): 68576 kB\n', 'Unevictable: 8044 kB\n', 'Mlocked: 8044 kB\n', 'SwapTotal: 0 kB\n', 'SwapFree: 0 kB\n', 'Dirty: 124 kB\n', 'Writeback: 0 kB\n', 'AnonPages: 253780 kB\n', 'Mapped: 96160 kB\n', 'Shmem: 312752 kB\n', 'Slab: 144800 kB\n', 'SReclaimable: 71632 kB\n', 'SUnreclaim: 73168 kB\n', 'KernelStack: 8672 kB\n', 'PageTables: 7928 kB\n', 'NFS_Unstable: 0 kB\n', 'Bounce: 0 kB\n', 'WritebackTmp: 0 kB\n', 'CommitLimit: 6003016 kB\n', 'Committed_AS: 1669596 kB\n', 'VmallocTotal: 34359738367 kB\n', 'VmallocUsed: 77396 kB\n', 'VmallocChunk: 34359651324 kB\n', 'HardwareCorrupted: 0 kB\n', 'AnonHugePages: 6144 kB\n', 'CmaTotal: 0 kB\n', 'CmaFree: 0 kB\n', 'HugePages_Total: 0\n', 'HugePages_Free: 0\n', 'HugePages_Rsvd: 0\n', 'HugePages_Surp: 0\n', 'Hugepagesize: 2048 kB\n', 'DirectMap4k: 155648 kB\n', 'DirectMap2M: 4974592 kB\n', 'DirectMap1G: 7340032 kB\n'] 2021-06-08 11:13:54,815 | test | DEBUG | MainThread | [basetestcase:_initialize_nodes:601] Node: 172.23.106.122: RAM: ['MemTotal: 12006032 kB\n', 'MemFree: 10627548 kB\n', 'MemAvailable: 10947396 kB\n', 'Buffers: 0 kB\n', 'Cached: 837424 kB\n', 'SwapCached: 0 kB\n', 'Active: 692232 kB\n', 'Inactive: 400032 kB\n', 'Active(anon): 241940 kB\n', 'Inactive(anon): 329768 kB\n', 'Active(file): 450292 kB\n', 'Inactive(file): 70264 kB\n', 'Unevictable: 12820 kB\n', 'Mlocked: 12820 kB\n', 'SwapTotal: 0 kB\n', 'SwapFree: 0 kB\n', 'Dirty: 32 kB\n', 'Writeback: 0 kB\n', 'AnonPages: 267828 kB\n', 'Mapped: 96204 kB\n', 'Shmem: 313228 kB\n', 'Slab: 140584 kB\n', 'SReclaimable: 66680 kB\n', 'SUnreclaim: 73904 kB\n', 'KernelStack: 8512 kB\n', 'PageTables: 7456 kB\n', 'NFS_Unstable: 0 kB\n', 'Bounce: 0 kB\n', 'WritebackTmp: 0 kB\n', 'CommitLimit: 6003016 kB\n', 'Committed_AS: 1670752 kB\n', 'VmallocTotal: 34359738367 kB\n', 'VmallocUsed: 76372 kB\n', 'VmallocChunk: 34359651324 kB\n', 'HardwareCorrupted: 0 kB\n', 'AnonHugePages: 8192 kB\n', 'CmaTotal: 0 kB\n', 'CmaFree: 0 kB\n', 'HugePages_Total: 0\n', 'HugePages_Free: 0\n', 'HugePages_Rsvd: 0\n', 'HugePages_Surp: 0\n', 'Hugepagesize: 2048 kB\n', 'DirectMap4k: 147456 kB\n', 'DirectMap2M: 8128512 kB\n', 'DirectMap1G: 4194304 kB\n'] 2021-06-08 11:13:54,815 | test | DEBUG | MainThread | [basetestcase:_initialize_nodes:601] Node: 172.23.106.155: RAM: ['MemTotal: 12006032 kB\n', 'MemFree: 10442112 kB\n', 'MemAvailable: 10759864 kB\n', 'Buffers: 0 kB\n', 'Cached: 1006292 kB\n', 'SwapCached: 0 kB\n', 'Active: 739936 kB\n', 'Inactive: 533828 kB\n', 'Active(anon): 285812 kB\n', 'Inactive(anon): 471676 kB\n', 'Active(file): 454124 kB\n', 'Inactive(file): 62152 kB\n', 'Unevictable: 9616 kB\n', 'Mlocked: 9616 kB\n', 'SwapTotal: 0 kB\n', 'SwapFree: 0 kB\n', 'Dirty: 844 kB\n', 'Writeback: 0 kB\n', 'AnonPages: 276948 kB\n', 'Mapped: 67080 kB\n', 'Shmem: 486372 kB\n', 'Slab: 146388 kB\n', 'SReclaimable: 71048 kB\n', 'SUnreclaim: 75340 kB\n', 'KernelStack: 8592 kB\n', 'PageTables: 7596 kB\n', 'NFS_Unstable: 0 kB\n', 'Bounce: 0 kB\n', 'WritebackTmp: 0 kB\n', 'CommitLimit: 6003016 kB\n', 'Committed_AS: 1854720 kB\n', 'VmallocTotal: 34359738367 kB\n', 'VmallocUsed: 77396 kB\n', 'VmallocChunk: 34359651324 kB\n', 'HardwareCorrupted: 0 kB\n', 'AnonHugePages: 2048 kB\n', 'CmaTotal: 0 kB\n', 'CmaFree: 0 kB\n', 'HugePages_Total: 0\n', 'HugePages_Free: 0\n', 'HugePages_Rsvd: 0\n', 'HugePages_Surp: 0\n', 'Hugepagesize: 2048 kB\n', 'DirectMap4k: 145408 kB\n', 'DirectMap2M: 4984832 kB\n', 'DirectMap1G: 7340032 kB\n'] 2021-06-08 11:13:54,815 | infra | DEBUG | MainThread | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.156 2021-06-08 11:13:54,815 | infra | DEBUG | MainThread | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.125 2021-06-08 11:13:54,816 | infra | DEBUG | MainThread | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.122 2021-06-08 11:13:54,816 | infra | DEBUG | MainThread | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.155 2021-06-08 11:13:54,819 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.156 with username: root, password: couchbase 2021-06-08 11:13:54,905 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 11:13:55,118 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: ls /etc/ 2021-06-08 11:13:55,171 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 11:13:55,171 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: ls /cygdrive/c/Windows 2021-06-08 11:13:55,183 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: uname -m 2021-06-08 11:13:55,194 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: cat /proc/cpuinfo 2021-06-08 11:13:55,207 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: df -Thl 2021-06-08 11:13:55,221 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: cat /proc/meminfo 2021-06-08 11:13:55,233 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: hostname 2021-06-08 11:13:55,244 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: hostname -d 2021-06-08 11:13:55,256 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: grep ^COUCHBASE_NUM_VBUCKETS /opt/couchbase/bin/couchbase-server | cut -d "=" -f 2 2021-06-08 11:13:55,269 | test | DEBUG | MainThread | [cluster_ready_functions:change_env_variables:301] Current vBuckets on node ip:172.23.106.156 port:8091 ssh_username:root: [] 2021-06-08 11:13:55,269 | infra | DEBUG | MainThread | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.156 2021-06-08 11:13:55,269 | test | DEBUG | MainThread | [cluster_ready_functions:change_env_variables:318] ========= CHANGED ENVIRONMENT SETTING =========== 2021-06-08 11:13:55,272 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.125 with username: root, password: couchbase 2021-06-08 11:13:55,345 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 11:13:55,552 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: ls /etc/ 2021-06-08 11:13:55,605 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 11:13:55,605 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: ls /cygdrive/c/Windows 2021-06-08 11:13:55,618 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: uname -m 2021-06-08 11:13:55,628 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: cat /proc/cpuinfo 2021-06-08 11:13:55,641 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: df -Thl 2021-06-08 11:13:55,654 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: cat /proc/meminfo 2021-06-08 11:13:55,667 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: hostname 2021-06-08 11:13:55,677 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: hostname -d 2021-06-08 11:13:55,690 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: grep ^COUCHBASE_NUM_VBUCKETS /opt/couchbase/bin/couchbase-server | cut -d "=" -f 2 2021-06-08 11:13:55,703 | test | DEBUG | MainThread | [cluster_ready_functions:change_env_variables:301] Current vBuckets on node ip:172.23.106.125 port:8091 ssh_username:root: [] 2021-06-08 11:13:55,703 | infra | DEBUG | MainThread | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.125 2021-06-08 11:13:55,703 | test | DEBUG | MainThread | [cluster_ready_functions:change_env_variables:318] ========= CHANGED ENVIRONMENT SETTING =========== 2021-06-08 11:13:55,704 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.122 with username: root, password: couchbase 2021-06-08 11:13:55,767 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 11:13:55,967 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: ls /etc/ 2021-06-08 11:13:56,016 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 11:13:56,016 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: ls /cygdrive/c/Windows 2021-06-08 11:13:56,029 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: uname -m 2021-06-08 11:13:56,039 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: cat /proc/cpuinfo 2021-06-08 11:13:56,051 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: df -Thl 2021-06-08 11:13:56,063 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: cat /proc/meminfo 2021-06-08 11:13:56,075 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: hostname 2021-06-08 11:13:56,085 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: hostname -d 2021-06-08 11:13:56,098 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: grep ^COUCHBASE_NUM_VBUCKETS /opt/couchbase/bin/couchbase-server | cut -d "=" -f 2 2021-06-08 11:13:56,108 | test | DEBUG | MainThread | [cluster_ready_functions:change_env_variables:301] Current vBuckets on node ip:172.23.106.122 port:8091 ssh_username:root: [] 2021-06-08 11:13:56,108 | infra | DEBUG | MainThread | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.122 2021-06-08 11:13:56,109 | test | DEBUG | MainThread | [cluster_ready_functions:change_env_variables:318] ========= CHANGED ENVIRONMENT SETTING =========== 2021-06-08 11:13:56,111 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.155 with username: root, password: couchbase 2021-06-08 11:13:56,180 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 11:13:56,378 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: ls /etc/ 2021-06-08 11:13:56,430 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 11:13:56,430 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: ls /cygdrive/c/Windows 2021-06-08 11:13:56,444 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: uname -m 2021-06-08 11:13:56,454 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: cat /proc/cpuinfo 2021-06-08 11:13:56,466 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: df -Thl 2021-06-08 11:13:56,480 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: cat /proc/meminfo 2021-06-08 11:13:56,493 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: hostname 2021-06-08 11:13:56,505 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: hostname -d 2021-06-08 11:13:56,516 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: grep ^COUCHBASE_NUM_VBUCKETS /opt/couchbase/bin/couchbase-server | cut -d "=" -f 2 2021-06-08 11:13:56,529 | test | DEBUG | MainThread | [cluster_ready_functions:change_env_variables:301] Current vBuckets on node ip:172.23.106.155 port:8091 ssh_username:root: [] 2021-06-08 11:13:56,529 | infra | DEBUG | MainThread | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.155 2021-06-08 11:13:56,529 | test | DEBUG | MainThread | [cluster_ready_functions:change_env_variables:318] ========= CHANGED ENVIRONMENT SETTING =========== 2021-06-08 11:13:56,530 | test | DEBUG | MainThread | [cluster_ready_functions:change_env_variables:320] Wait for all the services to come up after change_env_vars update 2021-06-08 11:13:56,536 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from couchbase_utils/cluster_utils/cluster_ready_functions.py -> change_env_variables():L322 2021-06-08 11:13:56,536 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 10 seconds. Reason: None 2021-06-08 11:14:06,536 | test | INFO | MainThread | [basetestcase:setUp:342] Couchbase Cluster: C1, Nodes: 172.23.106.156, 172.23.106.125, 172.23.106.122, 172.23.106.155 initialized 2021-06-08 11:14:06,539 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.156 with username: root, password: couchbase 2021-06-08 11:14:06,625 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 11:14:06,834 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: ls /etc/ 2021-06-08 11:14:06,885 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 11:14:06,887 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: ls /cygdrive/c/Windows 2021-06-08 11:14:06,898 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: uname -m 2021-06-08 11:14:06,911 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: cat /proc/cpuinfo 2021-06-08 11:14:06,921 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: df -Thl 2021-06-08 11:14:06,937 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: cat /proc/meminfo 2021-06-08 11:14:06,948 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: hostname 2021-06-08 11:14:06,960 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: hostname -d 2021-06-08 11:14:06,973 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: echo 'y' | /opt/couchbase/bin/couchbase-cli enable-developer-preview --enable -c 172.23.106.156:8091 -u Administrator -p password 2021-06-08 11:14:07,246 | infra | DEBUG | MainThread | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.156 2021-06-08 11:14:07,250 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.125 with username: root, password: couchbase 2021-06-08 11:14:07,336 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 11:14:07,538 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: ls /etc/ 2021-06-08 11:14:07,589 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 11:14:07,589 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: ls /cygdrive/c/Windows 2021-06-08 11:14:07,601 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: uname -m 2021-06-08 11:14:07,612 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: cat /proc/cpuinfo 2021-06-08 11:14:07,624 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: df -Thl 2021-06-08 11:14:07,638 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: cat /proc/meminfo 2021-06-08 11:14:07,650 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: hostname 2021-06-08 11:14:07,661 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: hostname -d 2021-06-08 11:14:07,673 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: echo 'y' | /opt/couchbase/bin/couchbase-cli enable-developer-preview --enable -c 172.23.106.125:8091 -u Administrator -p password 2021-06-08 11:14:07,892 | infra | DEBUG | MainThread | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.125 2021-06-08 11:14:07,895 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.122 with username: root, password: couchbase 2021-06-08 11:14:07,977 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 11:14:08,181 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: ls /etc/ 2021-06-08 11:14:08,233 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 11:14:08,233 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: ls /cygdrive/c/Windows 2021-06-08 11:14:08,244 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: uname -m 2021-06-08 11:14:08,256 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: cat /proc/cpuinfo 2021-06-08 11:14:08,266 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: df -Thl 2021-06-08 11:14:08,280 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: cat /proc/meminfo 2021-06-08 11:14:08,292 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: hostname 2021-06-08 11:14:08,303 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: hostname -d 2021-06-08 11:14:08,316 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: echo 'y' | /opt/couchbase/bin/couchbase-cli enable-developer-preview --enable -c 172.23.106.122:8091 -u Administrator -p password 2021-06-08 11:14:08,526 | infra | DEBUG | MainThread | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.122 2021-06-08 11:14:08,529 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.155 with username: root, password: couchbase 2021-06-08 11:14:08,598 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 11:14:08,798 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: ls /etc/ 2021-06-08 11:14:08,849 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 11:14:08,849 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: ls /cygdrive/c/Windows 2021-06-08 11:14:08,864 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: uname -m 2021-06-08 11:14:08,875 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: cat /proc/cpuinfo 2021-06-08 11:14:08,887 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: df -Thl 2021-06-08 11:14:08,901 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: cat /proc/meminfo 2021-06-08 11:14:08,913 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: hostname 2021-06-08 11:14:08,924 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: hostname -d 2021-06-08 11:14:08,937 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: echo 'y' | /opt/couchbase/bin/couchbase-cli enable-developer-preview --enable -c 172.23.106.155:8091 -u Administrator -p password 2021-06-08 11:14:09,167 | infra | DEBUG | MainThread | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.155 2021-06-08 11:14:09,178 | test | INFO | MainThread | [basetestcase:log_setup_status:546] ========= BaseTestCase setup finished for test #3 test_disk_full_reduce_replica ========= 2021-06-08 11:14:09,210 | test | DEBUG | MainThread | [bucket_ready_functions:add_rbac_user:3934] **** Add built-in 'cbadminbucket' user to node 172.23.106.156 **** 2021-06-08 11:14:09,292 | test | DEBUG | MainThread | [bucket_ready_functions:add_rbac_user:3938] **** Add 'admin' role to 'cbadminbucket' user **** 2021-06-08 11:14:09,312 | test | DEBUG | MainThread | [rest_client:init_cluster:770] settings/web params on 172.23.106.156:8091:password=password&port=8091&username=Administrator 2021-06-08 11:14:09,385 | infra | INFO | MainThread | [task_manager:add_new_task:23] Added new task: Rebalance_task_IN=[172.23.106.125,172.23.106.122,172.23.106.155]_OUT=[]_1623176049.37 2021-06-08 11:14:09,387 | infra | INFO | pool-5-thread-5 | [task:start_task:87] Thread 'Rebalance_task_IN=[172.23.106.125,172.23.106.122,172.23.106.155]_OUT=[]_1623176049.37' started 2021-06-08 11:14:09,387 | infra | DEBUG | MainThread | [task_manager:get_task_result:26] Getting task result for Rebalance_task_IN=[172.23.106.125,172.23.106.122,172.23.106.155]_OUT=[]_1623176049.37 2021-06-08 11:14:09,388 | test | DEBUG | pool-5-thread-5 | [rest_client:add_node:1089] Adding remote node 172.23.106.125:8091 to cluster 172.23.106.156:8091 2021-06-08 11:14:12,433 | test | DEBUG | pool-5-thread-5 | [rest_client:add_node:1089] Adding remote node 172.23.106.122:8091 to cluster 172.23.106.156:8091 2021-06-08 11:14:15,489 | test | DEBUG | pool-5-thread-5 | [rest_client:add_node:1089] Adding remote node 172.23.106.155:8091 to cluster 172.23.106.156:8091 2021-06-08 11:14:18,526 | test | DEBUG | pool-5-thread-5 | [rest_client:rebalance:1339] Rebalance params: {'password': 'password', 'knownNodes': u'ns_1@172.23.106.155,ns_1@172.23.106.156,ns_1@172.23.106.122,ns_1@172.23.106.125', 'ejectedNodes': '', 'user': 'Administrator'} 2021-06-08 11:14:18,581 | test | DEBUG | pool-5-thread-5 | [rest_client:rebalance:1344] Rebalance operation started 2021-06-08 11:14:18,582 | test | INFO | pool-5-thread-5 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.106.156 | kv | 7.0.0-5270-enterprise | 1.48148148148 | Cluster node | | 172.23.106.125 | ['kv'] | | | <--- IN --- | | 172.23.106.122 | ['kv'] | | | <--- IN --- | | 172.23.106.155 | ['kv'] | | | <--- IN --- | +----------------+----------+-----------------------+---------------+--------------+ 2021-06-08 11:14:18,592 | test | DEBUG | pool-5-thread-5 | [rest_client:_rebalance_status_and_progress:1553] Rebalance percentage: 0.00 % 2021-06-08 11:14:18,592 | test | INFO | pool-5-thread-5 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-06-08 11:14:18,605 | infra | DEBUG | pool-5-thread-5 | [task:check:382] Wait before next rebalance progress check 2021-06-08 11:14:18,607 | infra | DEBUG | pool-5-thread-5 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> check():L383 2021-06-08 11:14:18,607 | infra | DEBUG | pool-5-thread-5 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 11:14:23,618 | infra | DEBUG | pool-5-thread-5 | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _rebalance_status_and_progress():L1557 2021-06-08 11:14:23,618 | infra | DEBUG | pool-5-thread-5 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 11:14:28,628 | test | INFO | pool-5-thread-5 | [task:check:355] Rebalance - status: none, progress: 100 2021-06-08 11:14:28,641 | test | INFO | pool-5-thread-5 | [task:check:414] Rebalance completed with progress: 100% in 10.0599999428 sec 2021-06-08 11:14:28,641 | infra | INFO | pool-5-thread-5 | [task:complete_task:102] Thread 'Rebalance_task_IN=[172.23.106.125,172.23.106.122,172.23.106.155]_OUT=[]_1623176049.37' completed 2021-06-08 11:14:28,660 | test | DEBUG | MainThread | [bucket_ready_functions:async_create_bucket:1462] Creating bucket: default 2021-06-08 11:14:28,661 | infra | INFO | MainThread | [task_manager:add_new_task:23] Added new task: bucket_default_create_task 2021-06-08 11:14:28,661 | infra | DEBUG | MainThread | [task_manager:get_task_result:26] Getting task result for bucket_default_create_task 2021-06-08 11:14:28,714 | infra | INFO | pool-5-thread-6 | [BucketOperations_Rest:create_bucket:342] Creating 'couchbase' bucket default 2021-06-08 11:14:28,714 | infra | DEBUG | pool-5-thread-6 | [BucketOperations_Rest:create_bucket:344] http://172.23.106.156:8091/pools/default/buckets with param: storageBackend=magma&viewFragmentationThreshold%255Bpercentage%255D=50&fragmentationPercentage=50&replicaIndex=1&maxTTL=0&indexCompactionMode=circular&databaseFragmentationThreshold%255Bpercentage%255D=50&flushEnabled=0¶llelDBAndViewCompaction=false&durabilityMinLevel=none&compressionMode=off&purgeInterval=3&bucketType=couchbase&autoCompactionDefined=false&conflictResolutionType=seqno&name=default&replicaNumber=3&ramQuotaMB=8350&threadsNumber=3&evictionPolicy=fullEviction 2021-06-08 11:14:28,750 | infra | DEBUG | pool-5-thread-6 | [BucketOperations_Rest:create_bucket:368] 0.03 seconds to create bucket default 2021-06-08 11:14:28,750 | infra | INFO | pool-5-thread-6 | [task:complete_task:102] Thread 'bucket_default_create_task' completed 2021-06-08 11:14:28,750 | test | DEBUG | MainThread | [bucket_ready_functions:create_bucket:1474] Wait for memcached to accept cbstats or any other bucket request connections 2021-06-08 11:14:28,755 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from couchbase_utils/bucket_utils/bucket_ready_functions.py -> create_bucket():L1476 2021-06-08 11:14:28,755 | test | DEBUG | MainThread | [common_lib:sleep:24] Sleep 2 seconds. Reason: None 2021-06-08 11:14:30,802 | test | DEBUG | MainThread | [cluster_ready_functions:get_nodes_from_services_map:597] All nodes in cluster: [ip:172.23.106.155 port:8091 ssh_username:root, ip:172.23.106.156 port:8091 ssh_username:root, ip:172.23.106.122 port:8091 ssh_username:root, ip:172.23.106.125 port:8091 ssh_username:root] 2021-06-08 11:14:30,802 | test | DEBUG | MainThread | [bucket_ready_functions:_wait_warmup_completed:3884] Waiting for bucket default to complete warm-up 2021-06-08 11:14:30,803 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.156 with username: root, password: couchbase 2021-06-08 11:14:30,881 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 11:14:31,092 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: ls /etc/ 2021-06-08 11:14:31,151 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 11:14:31,151 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: ls /cygdrive/c/Windows 2021-06-08 11:14:31,165 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: uname -m 2021-06-08 11:14:31,178 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: cat /proc/cpuinfo 2021-06-08 11:14:31,191 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: df -Thl 2021-06-08 11:14:31,207 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: cat /proc/meminfo 2021-06-08 11:14:31,220 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: hostname 2021-06-08 11:14:31,230 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: hostname -d 2021-06-08 11:14:31,243 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:14:31,437 | infra | DEBUG | MainThread | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.156 2021-06-08 11:14:31,441 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.125 with username: root, password: couchbase 2021-06-08 11:14:31,507 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 11:14:31,711 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: ls /etc/ 2021-06-08 11:14:31,763 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 11:14:31,763 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: ls /cygdrive/c/Windows 2021-06-08 11:14:31,776 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: uname -m 2021-06-08 11:14:31,786 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: cat /proc/cpuinfo 2021-06-08 11:14:31,798 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: df -Thl 2021-06-08 11:14:31,812 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: cat /proc/meminfo 2021-06-08 11:14:31,823 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: hostname 2021-06-08 11:14:31,835 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: hostname -d 2021-06-08 11:14:31,846 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:14:32,033 | infra | DEBUG | MainThread | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.125 2021-06-08 11:14:32,036 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.122 with username: root, password: couchbase 2021-06-08 11:14:32,122 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 11:14:32,328 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: ls /etc/ 2021-06-08 11:14:32,380 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 11:14:32,380 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: ls /cygdrive/c/Windows 2021-06-08 11:14:32,392 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: uname -m 2021-06-08 11:14:32,404 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: cat /proc/cpuinfo 2021-06-08 11:14:32,415 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: df -Thl 2021-06-08 11:14:32,430 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: cat /proc/meminfo 2021-06-08 11:14:32,441 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: hostname 2021-06-08 11:14:32,453 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: hostname -d 2021-06-08 11:14:32,467 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:14:32,654 | infra | DEBUG | MainThread | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.122 2021-06-08 11:14:32,657 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.155 with username: root, password: couchbase 2021-06-08 11:14:32,736 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 11:14:32,940 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: ls /etc/ 2021-06-08 11:14:32,994 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 11:14:32,996 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: ls /cygdrive/c/Windows 2021-06-08 11:14:33,007 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: uname -m 2021-06-08 11:14:33,019 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: cat /proc/cpuinfo 2021-06-08 11:14:33,030 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: df -Thl 2021-06-08 11:14:33,046 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: cat /proc/meminfo 2021-06-08 11:14:33,059 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: hostname 2021-06-08 11:14:33,071 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: hostname -d 2021-06-08 11:14:33,082 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:14:33,273 | infra | DEBUG | MainThread | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.155 2021-06-08 11:14:33,273 | test | INFO | MainThread | [magma_base:setUp:120] Scopes list is ['_default'] 2021-06-08 11:14:33,275 | test | DEBUG | MainThread | [magma_base:setUp:135] Collections list == ['_default'] 2021-06-08 11:14:33,276 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.156 with username: root, password: couchbase 2021-06-08 11:14:33,349 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 11:14:33,552 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: ls /etc/ 2021-06-08 11:14:33,607 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 11:14:33,608 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: ls /cygdrive/c/Windows 2021-06-08 11:14:33,619 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: uname -m 2021-06-08 11:14:33,631 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: cat /proc/cpuinfo 2021-06-08 11:14:33,644 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: df -Thl 2021-06-08 11:14:33,658 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: cat /proc/meminfo 2021-06-08 11:14:33,671 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: hostname 2021-06-08 11:14:33,683 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: hostname -d 2021-06-08 11:14:33,694 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: du -cm /data/default/magma.*/kv* | tail -1 | awk '{print $1}' 2021-06-08 11:14:33,846 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: du -cm /data/default/magma.*/wal | tail -1 | awk '{print $1}' 2021-06-08 11:14:33,862 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: du -cm /data/default/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2021-06-08 11:14:33,943 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: du -cm /data/default/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2021-06-08 11:14:34,019 | infra | DEBUG | MainThread | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.156 2021-06-08 11:14:34,022 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.125 with username: root, password: couchbase 2021-06-08 11:14:34,095 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 11:14:34,299 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: ls /etc/ 2021-06-08 11:14:34,351 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 11:14:34,352 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: ls /cygdrive/c/Windows 2021-06-08 11:14:34,365 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: uname -m 2021-06-08 11:14:34,377 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: cat /proc/cpuinfo 2021-06-08 11:14:34,388 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: df -Thl 2021-06-08 11:14:34,403 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: cat /proc/meminfo 2021-06-08 11:14:34,414 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: hostname 2021-06-08 11:14:34,427 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: hostname -d 2021-06-08 11:14:34,440 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: du -cm /data/default/magma.*/kv* | tail -1 | awk '{print $1}' 2021-06-08 11:14:34,589 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: du -cm /data/default/magma.*/wal | tail -1 | awk '{print $1}' 2021-06-08 11:14:34,602 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: du -cm /data/default/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2021-06-08 11:14:34,677 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: du -cm /data/default/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2021-06-08 11:14:34,752 | infra | DEBUG | MainThread | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.125 2021-06-08 11:14:34,755 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.122 with username: root, password: couchbase 2021-06-08 11:14:34,823 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 11:14:35,023 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: ls /etc/ 2021-06-08 11:14:35,075 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 11:14:35,075 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: ls /cygdrive/c/Windows 2021-06-08 11:14:35,086 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: uname -m 2021-06-08 11:14:35,098 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: cat /proc/cpuinfo 2021-06-08 11:14:35,109 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: df -Thl 2021-06-08 11:14:35,124 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: cat /proc/meminfo 2021-06-08 11:14:35,135 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: hostname 2021-06-08 11:14:35,147 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: hostname -d 2021-06-08 11:14:35,160 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: du -cm /data/default/magma.*/kv* | tail -1 | awk '{print $1}' 2021-06-08 11:14:35,311 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: du -cm /data/default/magma.*/wal | tail -1 | awk '{print $1}' 2021-06-08 11:14:35,325 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: du -cm /data/default/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2021-06-08 11:14:35,398 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: du -cm /data/default/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2021-06-08 11:14:35,473 | infra | DEBUG | MainThread | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.122 2021-06-08 11:14:35,476 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.155 with username: root, password: couchbase 2021-06-08 11:14:35,542 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 11:14:35,746 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: ls /etc/ 2021-06-08 11:14:35,798 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 11:14:35,798 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: ls /cygdrive/c/Windows 2021-06-08 11:14:35,809 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: uname -m 2021-06-08 11:14:35,821 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: cat /proc/cpuinfo 2021-06-08 11:14:35,832 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: df -Thl 2021-06-08 11:14:35,846 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: cat /proc/meminfo 2021-06-08 11:14:35,858 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: hostname 2021-06-08 11:14:35,869 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: hostname -d 2021-06-08 11:14:35,881 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: du -cm /data/default/magma.*/kv* | tail -1 | awk '{print $1}' 2021-06-08 11:14:36,038 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: du -cm /data/default/magma.*/wal | tail -1 | awk '{print $1}' 2021-06-08 11:14:36,052 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: du -cm /data/default/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2021-06-08 11:14:36,128 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: du -cm /data/default/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2021-06-08 11:14:36,203 | infra | DEBUG | MainThread | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.155 2021-06-08 11:14:36,203 | test | INFO | MainThread | [magma_base:get_disk_usage:815] Disk usage stats for bucekt default is below 2021-06-08 11:14:36,203 | test | INFO | MainThread | [magma_base:get_disk_usage:816] Total Disk usage for kvstore is 48MB 2021-06-08 11:14:36,368 | test | INFO | MainThread | [magma_base:get_bucket_dgm:767] Active Resident Threshold of default is 100 2021-06-08 11:14:36,368 | test | DEBUG | MainThread | [magma_base:get_disk_usage:818] Total Disk usage for wal is 32MB 2021-06-08 11:14:36,368 | test | DEBUG | MainThread | [magma_base:get_disk_usage:819] Total Disk usage for keyTree is 16MB 2021-06-08 11:14:36,368 | test | DEBUG | MainThread | [magma_base:get_disk_usage:820] Total Disk usage for seqTree is 16MB 2021-06-08 11:14:36,368 | test | INFO | MainThread | [magma_base:setUp:178] Empty magma bucket disk usage: 48 2021-06-08 11:14:36,368 | test | INFO | MainThread | [magma_base:setUp:243] Creating SDK clients for client_pool 2021-06-08 11:14:36,368 | test | DEBUG | MainThread | [sdk_client3:_SDKClient__create_conn:237] Creating SDK connection for 'default' 2021-06-08 11:14:36,717 | test | DEBUG | MainThread | [sdk_client3:_SDKClient__create_conn:237] Creating SDK connection for 'default' 2021-06-08 11:14:37,046 | test | DEBUG | MainThread | [sdk_client3:_SDKClient__create_conn:237] Creating SDK connection for 'default' 2021-06-08 11:14:37,407 | test | DEBUG | MainThread | [sdk_client3:_SDKClient__create_conn:237] Creating SDK connection for 'default' 2021-06-08 11:14:37,858 | test | DEBUG | MainThread | [sdk_client3:_SDKClient__create_conn:237] Creating SDK connection for 'default' 2021-06-08 11:14:38,187 | test | DEBUG | MainThread | [sdk_client3:_SDKClient__create_conn:237] Creating SDK connection for 'default' 2021-06-08 11:14:38,618 | test | DEBUG | MainThread | [sdk_client3:_SDKClient__create_conn:237] Creating SDK connection for 'default' 2021-06-08 11:14:39,058 | test | DEBUG | MainThread | [sdk_client3:_SDKClient__create_conn:237] Creating SDK connection for 'default' 2021-06-08 11:14:39,427 | test | DEBUG | MainThread | [sdk_client3:_SDKClient__create_conn:237] Creating SDK connection for 'default' 2021-06-08 11:14:39,765 | test | DEBUG | MainThread | [sdk_client3:_SDKClient__create_conn:237] Creating SDK connection for 'default' 2021-06-08 11:14:40,207 | test | DEBUG | MainThread | [sdk_client3:_SDKClient__create_conn:237] Creating SDK connection for 'default' 2021-06-08 11:14:40,647 | test | DEBUG | MainThread | [sdk_client3:_SDKClient__create_conn:237] Creating SDK connection for 'default' 2021-06-08 11:14:40,977 | test | DEBUG | MainThread | [sdk_client3:_SDKClient__create_conn:237] Creating SDK connection for 'default' 2021-06-08 11:14:41,309 | test | DEBUG | MainThread | [sdk_client3:_SDKClient__create_conn:237] Creating SDK connection for 'default' 2021-06-08 11:14:41,638 | test | DEBUG | MainThread | [sdk_client3:_SDKClient__create_conn:237] Creating SDK connection for 'default' 2021-06-08 11:14:42,088 | test | DEBUG | MainThread | [sdk_client3:_SDKClient__create_conn:237] Creating SDK connection for 'default' 2021-06-08 11:14:42,417 | test | DEBUG | MainThread | [sdk_client3:_SDKClient__create_conn:237] Creating SDK connection for 'default' 2021-06-08 11:14:42,749 | test | DEBUG | MainThread | [sdk_client3:_SDKClient__create_conn:237] Creating SDK connection for 'default' 2021-06-08 11:14:43,088 | test | DEBUG | MainThread | [sdk_client3:_SDKClient__create_conn:237] Creating SDK connection for 'default' 2021-06-08 11:14:43,477 | test | DEBUG | MainThread | [sdk_client3:_SDKClient__create_conn:237] Creating SDK connection for 'default' 2021-06-08 11:14:43,815 | test | DEBUG | MainThread | [magma_base:initial_load:350] initial_items_in_each_collection 5000000 2021-06-08 11:14:43,818 | infra | DEBUG | MainThread | [cluster:async_load_gen_docs_from_spec:105] Initializing mutation task for given spec 2021-06-08 11:14:43,819 | infra | INFO | MainThread | [task_manager:add_new_task:23] Added new task: MutateDocsFromSpecTask_1623176083.82 2021-06-08 11:14:43,819 | infra | DEBUG | MainThread | [task_manager:get_task_result:26] Getting task result for MutateDocsFromSpecTask_1623176083.82 2021-06-08 11:14:43,819 | infra | INFO | pool-5-thread-7 | [task:start_task:87] Thread 'MutateDocsFromSpecTask_1623176083.82' started 2021-06-08 11:14:43,846 | infra | INFO | pool-5-thread-7 | [task_manager:add_new_task:23] Added new task: PrintBucketStats_default_1623176083.83 2021-06-08 11:14:43,848 | infra | INFO | pool-5-thread-7 | [task_manager:add_new_task:23] Added new task: LoadDocs_MutateDocsFromSpecTask_1623176083.82_default__default__default_ttl=0_create__0_250000_1623176083.83 2021-06-08 11:14:43,848 | infra | INFO | pool-5-thread-7 | [task_manager:add_new_task:23] Added new task: LoadDocs_MutateDocsFromSpecTask_1623176083.82_default__default__default_ttl=0_create__250000_500000_1623176083.83 2021-06-08 11:14:43,848 | infra | INFO | pool-5-thread-7 | [task_manager:add_new_task:23] Added new task: LoadDocs_MutateDocsFromSpecTask_1623176083.82_default__default__default_ttl=0_create__500000_750000_1623176083.83 2021-06-08 11:14:43,848 | infra | INFO | pool-5-thread-7 | [task_manager:add_new_task:23] Added new task: LoadDocs_MutateDocsFromSpecTask_1623176083.82_default__default__default_ttl=0_create__750000_1000000_1623176083.83 2021-06-08 11:14:43,848 | infra | INFO | pool-5-thread-7 | [task_manager:add_new_task:23] Added new task: LoadDocs_MutateDocsFromSpecTask_1623176083.82_default__default__default_ttl=0_create__1000000_1250000_1623176083.83 2021-06-08 11:14:43,848 | infra | INFO | pool-5-thread-7 | [task_manager:add_new_task:23] Added new task: LoadDocs_MutateDocsFromSpecTask_1623176083.82_default__default__default_ttl=0_create__1250000_1500000_1623176083.83 2021-06-08 11:14:43,848 | infra | INFO | pool-5-thread-7 | [task_manager:add_new_task:23] Added new task: LoadDocs_MutateDocsFromSpecTask_1623176083.82_default__default__default_ttl=0_create__1500000_1750000_1623176083.83 2021-06-08 11:14:43,848 | infra | INFO | pool-5-thread-7 | [task_manager:add_new_task:23] Added new task: LoadDocs_MutateDocsFromSpecTask_1623176083.82_default__default__default_ttl=0_create__1750000_2000000_1623176083.83 2021-06-08 11:14:43,849 | infra | INFO | pool-5-thread-7 | [task_manager:add_new_task:23] Added new task: LoadDocs_MutateDocsFromSpecTask_1623176083.82_default__default__default_ttl=0_create__2000000_2250000_1623176083.83 2021-06-08 11:14:43,849 | infra | INFO | pool-5-thread-7 | [task_manager:add_new_task:23] Added new task: LoadDocs_MutateDocsFromSpecTask_1623176083.82_default__default__default_ttl=0_create__2250000_2500000_1623176083.83 2021-06-08 11:14:43,849 | infra | INFO | pool-5-thread-7 | [task_manager:add_new_task:23] Added new task: LoadDocs_MutateDocsFromSpecTask_1623176083.82_default__default__default_ttl=0_create__2500000_2750000_1623176083.83 2021-06-08 11:14:43,849 | infra | INFO | pool-5-thread-7 | [task_manager:add_new_task:23] Added new task: LoadDocs_MutateDocsFromSpecTask_1623176083.82_default__default__default_ttl=0_create__2750000_3000000_1623176083.83 2021-06-08 11:14:43,851 | infra | INFO | pool-5-thread-7 | [task_manager:add_new_task:23] Added new task: LoadDocs_MutateDocsFromSpecTask_1623176083.82_default__default__default_ttl=0_create__3000000_3250000_1623176083.83 2021-06-08 11:14:43,851 | infra | INFO | pool-5-thread-7 | [task_manager:add_new_task:23] Added new task: LoadDocs_MutateDocsFromSpecTask_1623176083.82_default__default__default_ttl=0_create__3250000_3500000_1623176083.83 2021-06-08 11:14:43,851 | infra | INFO | pool-5-thread-7 | [task_manager:add_new_task:23] Added new task: LoadDocs_MutateDocsFromSpecTask_1623176083.82_default__default__default_ttl=0_create__3500000_3750000_1623176083.83 2021-06-08 11:14:43,851 | infra | INFO | pool-5-thread-7 | [task_manager:add_new_task:23] Added new task: LoadDocs_MutateDocsFromSpecTask_1623176083.82_default__default__default_ttl=0_create__3750000_4000000_1623176083.83 2021-06-08 11:14:43,851 | infra | INFO | pool-5-thread-7 | [task_manager:add_new_task:23] Added new task: LoadDocs_MutateDocsFromSpecTask_1623176083.82_default__default__default_ttl=0_create__4000000_4250000_1623176083.83 2021-06-08 11:14:43,851 | infra | INFO | pool-5-thread-7 | [task_manager:add_new_task:23] Added new task: LoadDocs_MutateDocsFromSpecTask_1623176083.82_default__default__default_ttl=0_create__4250000_4500000_1623176083.83 2021-06-08 11:14:43,851 | infra | INFO | pool-5-thread-7 | [task_manager:add_new_task:23] Added new task: LoadDocs_MutateDocsFromSpecTask_1623176083.82_default__default__default_ttl=0_create__4500000_4750000_1623176083.83 2021-06-08 11:14:43,851 | infra | INFO | pool-5-thread-7 | [task_manager:add_new_task:23] Added new task: LoadDocs_MutateDocsFromSpecTask_1623176083.82_default__default__default_ttl=0_create__4750000_5000000_1623176083.83 2021-06-08 11:14:43,852 | infra | DEBUG | pool-5-thread-7 | [task_manager:get_task_result:26] Getting task result for LoadDocs_MutateDocsFromSpecTask_1623176083.82_default__default__default_ttl=0_create__0_250000_1623176083.83 2021-06-08 11:14:43,854 | infra | INFO | pool-5-thread-8 | [task:start_task:87] Thread 'PrintBucketStats_default_1623176083.83' started 2021-06-08 11:14:43,854 | infra | INFO | pool-5-thread-9 | [task:start_task:87] Thread 'LoadDocs_MutateDocsFromSpecTask_1623176083.82_default__default__default_ttl=0_create__0_250000_1623176083.83' started 2021-06-08 11:14:43,854 | infra | INFO | pool-5-thread-10 | [task:start_task:87] Thread 'LoadDocs_MutateDocsFromSpecTask_1623176083.82_default__default__default_ttl=0_create__250000_500000_1623176083.83' started 2021-06-08 11:14:43,855 | infra | INFO | pool-5-thread-11 | [task:start_task:87] Thread 'LoadDocs_MutateDocsFromSpecTask_1623176083.82_default__default__default_ttl=0_create__500000_750000_1623176083.83' started 2021-06-08 11:14:43,855 | infra | INFO | pool-5-thread-12 | [task:start_task:87] Thread 'LoadDocs_MutateDocsFromSpecTask_1623176083.82_default__default__default_ttl=0_create__750000_1000000_1623176083.83' started 2021-06-08 11:14:43,855 | infra | INFO | pool-5-thread-13 | [task:start_task:87] Thread 'LoadDocs_MutateDocsFromSpecTask_1623176083.82_default__default__default_ttl=0_create__1000000_1250000_1623176083.83' started 2021-06-08 11:14:43,855 | infra | INFO | pool-5-thread-14 | [task:start_task:87] Thread 'LoadDocs_MutateDocsFromSpecTask_1623176083.82_default__default__default_ttl=0_create__1250000_1500000_1623176083.83' started 2021-06-08 11:14:43,855 | infra | INFO | pool-5-thread-15 | [task:start_task:87] Thread 'LoadDocs_MutateDocsFromSpecTask_1623176083.82_default__default__default_ttl=0_create__1500000_1750000_1623176083.83' started 2021-06-08 11:14:43,857 | infra | INFO | pool-5-thread-20 | [task:start_task:87] Thread 'LoadDocs_MutateDocsFromSpecTask_1623176083.82_default__default__default_ttl=0_create__2750000_3000000_1623176083.83' started 2021-06-08 11:14:43,857 | infra | INFO | pool-5-thread-21 | [task:start_task:87] Thread 'LoadDocs_MutateDocsFromSpecTask_1623176083.82_default__default__default_ttl=0_create__3000000_3250000_1623176083.83' started 2021-06-08 11:14:43,855 | infra | INFO | pool-5-thread-16 | [task:start_task:87] Thread 'LoadDocs_MutateDocsFromSpecTask_1623176083.82_default__default__default_ttl=0_create__1750000_2000000_1623176083.83' started 2021-06-08 11:14:43,857 | infra | INFO | pool-5-thread-22 | [task:start_task:87] Thread 'LoadDocs_MutateDocsFromSpecTask_1623176083.82_default__default__default_ttl=0_create__3250000_3500000_1623176083.83' started 2021-06-08 11:14:43,857 | infra | INFO | pool-5-thread-24 | [task:start_task:87] Thread 'LoadDocs_MutateDocsFromSpecTask_1623176083.82_default__default__default_ttl=0_create__3750000_4000000_1623176083.83' started 2021-06-08 11:14:43,857 | infra | INFO | pool-5-thread-25 | [task:start_task:87] Thread 'LoadDocs_MutateDocsFromSpecTask_1623176083.82_default__default__default_ttl=0_create__4000000_4250000_1623176083.83' started 2021-06-08 11:14:43,855 | infra | INFO | pool-5-thread-17 | [task:start_task:87] Thread 'LoadDocs_MutateDocsFromSpecTask_1623176083.82_default__default__default_ttl=0_create__2000000_2250000_1623176083.83' started 2021-06-08 11:14:43,857 | infra | INFO | pool-5-thread-18 | [task:start_task:87] Thread 'LoadDocs_MutateDocsFromSpecTask_1623176083.82_default__default__default_ttl=0_create__2250000_2500000_1623176083.83' started 2021-06-08 11:14:43,857 | infra | INFO | pool-5-thread-19 | [task:start_task:87] Thread 'LoadDocs_MutateDocsFromSpecTask_1623176083.82_default__default__default_ttl=0_create__2500000_2750000_1623176083.83' started 2021-06-08 11:14:43,857 | infra | INFO | pool-5-thread-23 | [task:start_task:87] Thread 'LoadDocs_MutateDocsFromSpecTask_1623176083.82_default__default__default_ttl=0_create__3500000_3750000_1623176083.83' started 2021-06-08 11:14:43,857 | infra | INFO | pool-5-thread-26 | [task:start_task:87] Thread 'LoadDocs_MutateDocsFromSpecTask_1623176083.82_default__default__default_ttl=0_create__4250000_4500000_1623176083.83' started 2021-06-08 11:14:43,857 | infra | INFO | pool-5-thread-27 | [task:start_task:87] Thread 'LoadDocs_MutateDocsFromSpecTask_1623176083.82_default__default__default_ttl=0_create__4500000_4750000_1623176083.83' started 2021-06-08 11:14:43,857 | infra | INFO | pool-5-thread-28 | [task:start_task:87] Thread 'LoadDocs_MutateDocsFromSpecTask_1623176083.82_default__default__default_ttl=0_create__4750000_5000000_1623176083.83' started 2021-06-08 11:14:44,151 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:14:44,154 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:14:44,154 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:14:45,447 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:14:45,450 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:14:45,450 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:14:46,756 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:14:46,757 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:14:46,759 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:14:48,085 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:14:48,088 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:14:48,088 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:14:49,392 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:14:49,395 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:14:49,395 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:14:50,697 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:14:50,700 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:14:50,700 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:14:52,042 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 82749.633187 2021-06-08 11:14:52,045 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:14:52,045 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:14:53,447 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 82749.633187 2021-06-08 11:14:53,450 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:14:53,450 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:14:54,812 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 82749.633187 2021-06-08 11:14:54,815 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:14:54,815 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:14:56,174 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 82749.633187 2021-06-08 11:14:56,177 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:14:56,178 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:14:57,556 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 82749.633187 2021-06-08 11:14:57,559 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:14:57,559 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:14:58,891 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 82749.633187 2021-06-08 11:14:58,901 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:14:58,901 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:15:00,290 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 82749.633187 2021-06-08 11:15:00,296 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:15:00,296 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:15:01,611 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 112744.074252 2021-06-08 11:15:01,612 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:15:01,612 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:15:02,964 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 112744.074252 2021-06-08 11:15:02,967 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:15:02,967 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:15:04,344 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 112744.074252 2021-06-08 11:15:04,346 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:15:04,346 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:15:05,671 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 112744.074252 2021-06-08 11:15:05,674 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:15:05,674 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:15:07,013 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 112744.074252 2021-06-08 11:15:07,015 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:15:07,015 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:15:08,325 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 112744.074252 2021-06-08 11:15:08,328 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:15:08,328 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:15:09,691 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 112744.074252 2021-06-08 11:15:09,694 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:15:09,696 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:15:10,957 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 112744.074252 2021-06-08 11:15:10,960 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:15:10,960 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:15:12,263 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 93791.511762 2021-06-08 11:15:12,266 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:15:12,267 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:15:13,661 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 93791.511762 2021-06-08 11:15:13,664 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:15:13,664 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:15:15,104 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 93791.511762 2021-06-08 11:15:15,107 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:15:15,107 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:15:16,505 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 93791.511762 2021-06-08 11:15:16,509 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:15:16,509 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:15:17,857 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 93791.511762 2021-06-08 11:15:17,859 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:15:17,859 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:15:19,201 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 93791.511762 2021-06-08 11:15:19,203 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:15:19,203 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:15:20,663 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 93791.511762 2021-06-08 11:15:20,667 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:15:20,667 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:15:22,209 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 109498.200000 2021-06-08 11:15:22,211 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:15:22,213 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:15:23,703 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 109498.200000 2021-06-08 11:15:23,707 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:15:23,707 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:15:25,069 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 109498.200000 2021-06-08 11:15:25,071 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:15:25,072 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:15:26,381 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 109498.200000 2021-06-08 11:15:26,382 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:15:26,382 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:15:27,657 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 109498.200000 2021-06-08 11:15:27,657 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:15:27,657 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:15:29,033 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 109498.200000 2021-06-08 11:15:29,036 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:15:29,036 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:15:29,934 | infra | DEBUG | pool-5-thread-17 | [task:batch_create:495] Sleep before reading the doc for verification 2021-06-08 11:15:30,032 | infra | DEBUG | pool-5-thread-13 | [task:batch_create:495] Sleep before reading the doc for verification 2021-06-08 11:15:30,049 | infra | DEBUG | pool-5-thread-9 | [task:batch_create:495] Sleep before reading the doc for verification 2021-06-08 11:15:30,098 | infra | DEBUG | pool-5-thread-27 | [task:batch_create:495] Sleep before reading the doc for verification 2021-06-08 11:15:30,107 | infra | DEBUG | pool-5-thread-21 | [task:batch_create:495] Sleep before reading the doc for verification 2021-06-08 11:15:30,122 | infra | DEBUG | pool-5-thread-26 | [task:batch_create:495] Sleep before reading the doc for verification 2021-06-08 11:15:30,128 | infra | DEBUG | pool-5-thread-25 | [task:batch_create:495] Sleep before reading the doc for verification 2021-06-08 11:15:30,134 | infra | DEBUG | pool-5-thread-24 | [task:batch_create:495] Sleep before reading the doc for verification 2021-06-08 11:15:30,138 | infra | DEBUG | pool-5-thread-18 | [task:batch_create:495] Sleep before reading the doc for verification 2021-06-08 11:15:30,144 | infra | DEBUG | pool-5-thread-10 | [task:batch_create:495] Sleep before reading the doc for verification 2021-06-08 11:15:30,145 | infra | DEBUG | pool-5-thread-20 | [task:batch_create:495] Sleep before reading the doc for verification 2021-06-08 11:15:30,148 | infra | DEBUG | pool-5-thread-12 | [task:batch_create:495] Sleep before reading the doc for verification 2021-06-08 11:15:30,158 | infra | DEBUG | pool-5-thread-14 | [task:batch_create:495] Sleep before reading the doc for verification 2021-06-08 11:15:30,160 | infra | DEBUG | pool-5-thread-23 | [task:batch_create:495] Sleep before reading the doc for verification 2021-06-08 11:15:30,170 | infra | DEBUG | pool-5-thread-28 | [task:batch_create:495] Sleep before reading the doc for verification 2021-06-08 11:15:30,177 | infra | DEBUG | pool-5-thread-15 | [task:batch_create:495] Sleep before reading the doc for verification 2021-06-08 11:15:30,180 | infra | DEBUG | pool-5-thread-11 | [task:batch_create:495] Sleep before reading the doc for verification 2021-06-08 11:15:30,180 | infra | DEBUG | pool-5-thread-16 | [task:batch_create:495] Sleep before reading the doc for verification 2021-06-08 11:15:30,181 | infra | DEBUG | pool-5-thread-19 | [task:batch_create:495] Sleep before reading the doc for verification 2021-06-08 11:15:30,184 | infra | DEBUG | pool-5-thread-22 | [task:batch_create:495] Sleep before reading the doc for verification 2021-06-08 11:15:30,352 | infra | DEBUG | pool-5-thread-13 | [task:batch_create:495] Sleep before reading the doc for verification 2021-06-08 11:15:30,355 | infra | DEBUG | pool-5-thread-17 | [task:batch_create:495] Sleep before reading the doc for verification 2021-06-08 11:15:30,367 | infra | DEBUG | pool-5-thread-9 | [task:batch_create:495] Sleep before reading the doc for verification 2021-06-08 11:15:30,381 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 109498.200000 2021-06-08 11:15:30,384 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:15:30,384 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:15:30,493 | infra | DEBUG | pool-5-thread-21 | [task:batch_create:495] Sleep before reading the doc for verification 2021-06-08 11:15:31,690 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 63591.700000 2021-06-08 11:15:31,691 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:15:31,691 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:15:33,053 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 63591.700000 2021-06-08 11:15:33,055 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:15:33,055 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:15:34,346 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 63591.700000 2021-06-08 11:15:34,348 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:15:34,348 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:15:35,678 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 63591.700000 2021-06-08 11:15:35,680 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:15:35,680 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:15:36,953 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 63591.700000 2021-06-08 11:15:36,954 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:15:36,954 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:15:38,187 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 63591.700000 2021-06-08 11:15:38,188 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:15:38,188 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:15:39,440 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 63591.700000 2021-06-08 11:15:39,443 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:15:39,443 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:15:40,687 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 63591.700000 2021-06-08 11:15:40,688 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:15:40,688 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:15:41,911 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 10325.800000 2021-06-08 11:15:41,913 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:15:41,913 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:15:43,164 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 10325.800000 2021-06-08 11:15:43,165 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:15:43,165 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:15:44,424 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 10325.800000 2021-06-08 11:15:44,426 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:15:44,426 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:15:45,674 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 10325.800000 2021-06-08 11:15:45,674 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:15:45,676 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:15:46,921 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 10325.800000 2021-06-08 11:15:46,923 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:15:46,923 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:15:48,194 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 10325.800000 2021-06-08 11:15:48,197 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:15:48,197 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:15:49,434 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 10325.800000 2021-06-08 11:15:49,436 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:15:49,436 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:15:50,677 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 10325.800000 2021-06-08 11:15:50,678 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:15:50,680 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:15:51,930 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 8836.000000 2021-06-08 11:15:51,931 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:15:51,931 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:15:53,174 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 8836.000000 2021-06-08 11:15:53,177 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:15:53,177 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:15:54,430 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 8836.000000 2021-06-08 11:15:54,433 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:15:54,433 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:15:55,667 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 8836.000000 2021-06-08 11:15:55,670 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:15:55,670 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:15:56,905 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 8836.000000 2021-06-08 11:15:56,907 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:15:56,907 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:15:58,142 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 8836.000000 2021-06-08 11:15:58,144 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:15:58,144 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:15:59,388 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 8836.000000 2021-06-08 11:15:59,390 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:15:59,390 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:16:00,631 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 8836.000000 2021-06-08 11:16:00,634 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:16:00,634 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:16:01,862 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 8996.700000 2021-06-08 11:16:01,865 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:16:01,865 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:16:03,108 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 8996.700000 2021-06-08 11:16:03,108 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:16:03,109 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:16:04,358 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 8996.700000 2021-06-08 11:16:04,359 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:16:04,361 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:16:05,601 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 8996.700000 2021-06-08 11:16:05,602 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:16:05,602 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:16:06,851 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 8996.700000 2021-06-08 11:16:06,851 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:16:06,851 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:16:07,413 | infra | DEBUG | pool-5-thread-26 | [task:batch_create:495] Sleep before reading the doc for verification 2021-06-08 11:16:07,447 | infra | DEBUG | pool-5-thread-25 | [task:batch_create:495] Sleep before reading the doc for verification 2021-06-08 11:16:07,450 | infra | DEBUG | pool-5-thread-27 | [task:batch_create:495] Sleep before reading the doc for verification 2021-06-08 11:16:08,069 | infra | DEBUG | pool-5-thread-23 | [task:batch_create:495] Sleep before reading the doc for verification 2021-06-08 11:16:08,089 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 8996.700000 2021-06-08 11:16:08,092 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:16:08,092 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:16:09,331 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 8996.700000 2021-06-08 11:16:09,332 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:16:09,332 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:16:10,565 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 8996.700000 2021-06-08 11:16:10,566 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:16:10,568 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:16:11,801 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 8035.600000 2021-06-08 11:16:11,802 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:16:11,802 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:16:13,035 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 8035.600000 2021-06-08 11:16:13,036 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:16:13,036 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:16:14,276 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 17952.000000 2021-06-08 11:16:14,279 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:16:14,279 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:16:15,516 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 17952.000000 2021-06-08 11:16:15,519 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:16:15,519 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:16:16,750 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 17952.000000 2021-06-08 11:16:16,753 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:16:16,753 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:16:17,986 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 17952.000000 2021-06-08 11:16:17,987 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:16:17,987 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:16:19,210 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 17952.000000 2021-06-08 11:16:19,213 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:16:19,213 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:16:20,450 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 17952.000000 2021-06-08 11:16:20,451 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:16:20,451 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:16:21,684 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 7614.600000 2021-06-08 11:16:21,687 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:16:21,687 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:16:22,943 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 7614.600000 2021-06-08 11:16:22,944 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:16:22,944 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:16:24,183 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 7614.600000 2021-06-08 11:16:24,184 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:16:24,184 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:16:25,405 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 7614.600000 2021-06-08 11:16:25,407 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:16:25,407 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:16:26,638 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 7614.600000 2021-06-08 11:16:26,640 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:16:26,640 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:16:27,880 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 7614.600000 2021-06-08 11:16:27,881 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:16:27,881 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:16:29,108 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 7614.600000 2021-06-08 11:16:29,111 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:16:29,111 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:16:30,349 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 7614.600000 2021-06-08 11:16:30,351 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:16:30,351 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:16:31,579 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 7614.600000 2021-06-08 11:16:31,582 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:16:31,582 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:16:32,812 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 7614.600000 2021-06-08 11:16:32,812 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:16:32,812 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:16:34,043 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 11789.400000 2021-06-08 11:16:34,046 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:16:34,046 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:16:35,278 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 11789.400000 2021-06-08 11:16:35,279 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:16:35,279 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:16:36,522 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 11789.400000 2021-06-08 11:16:36,523 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:16:36,523 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:16:37,757 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 11789.400000 2021-06-08 11:16:37,759 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:16:37,759 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:16:39,009 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 11789.400000 2021-06-08 11:16:39,009 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:16:39,010 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:16:40,269 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 11789.400000 2021-06-08 11:16:40,269 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:16:40,270 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:16:41,526 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 7676.000000 2021-06-08 11:16:41,528 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:16:41,528 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:16:42,752 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 7676.000000 2021-06-08 11:16:42,753 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:16:42,753 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:16:43,987 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 7676.000000 2021-06-08 11:16:43,990 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:16:43,990 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:16:45,219 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 7676.000000 2021-06-08 11:16:45,220 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:16:45,220 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:16:46,454 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 7676.000000 2021-06-08 11:16:46,457 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:16:46,457 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:16:47,726 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 7676.000000 2021-06-08 11:16:47,727 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:16:47,727 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:16:49,029 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 7676.000000 2021-06-08 11:16:49,032 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:16:49,032 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:16:50,262 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 7676.000000 2021-06-08 11:16:50,263 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:16:50,263 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:16:51,489 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 7771.900000 2021-06-08 11:16:51,490 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:16:51,490 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:16:52,759 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 10161.300000 2021-06-08 11:16:52,760 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:16:52,760 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:16:54,058 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 10161.300000 2021-06-08 11:16:54,059 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:16:54,059 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:16:55,323 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 10161.300000 2021-06-08 11:16:55,325 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:16:55,325 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:16:56,551 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 10161.300000 2021-06-08 11:16:56,552 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:16:56,552 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:16:57,798 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 10161.300000 2021-06-08 11:16:57,799 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:16:57,799 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:16:59,025 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 10161.300000 2021-06-08 11:16:59,026 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:16:59,026 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:17:00,305 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 10161.300000 2021-06-08 11:17:00,306 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:17:00,306 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:17:01,573 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 9969.400000 2021-06-08 11:17:01,575 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:17:01,575 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:17:02,805 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 7172.900000 2021-06-08 11:17:02,805 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:17:02,805 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:17:04,040 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 7172.900000 2021-06-08 11:17:04,042 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:17:04,042 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:17:04,440 | infra | DEBUG | pool-5-thread-10 | [task:batch_create:495] Sleep before reading the doc for verification 2021-06-08 11:17:04,510 | infra | DEBUG | pool-5-thread-20 | [task:batch_create:495] Sleep before reading the doc for verification 2021-06-08 11:17:04,513 | infra | DEBUG | pool-5-thread-11 | [task:batch_create:495] Sleep before reading the doc for verification 2021-06-08 11:17:04,601 | infra | DEBUG | pool-5-thread-18 | [task:batch_create:495] Sleep before reading the doc for verification 2021-06-08 11:17:04,621 | infra | DEBUG | pool-5-thread-9 | [task:batch_create:495] Sleep before reading the doc for verification 2021-06-08 11:17:04,628 | infra | DEBUG | pool-5-thread-12 | [task:batch_create:495] Sleep before reading the doc for verification 2021-06-08 11:17:04,631 | infra | DEBUG | pool-5-thread-22 | [task:batch_create:495] Sleep before reading the doc for verification 2021-06-08 11:17:04,631 | infra | DEBUG | pool-5-thread-13 | [task:batch_create:495] Sleep before reading the doc for verification 2021-06-08 11:17:04,668 | infra | DEBUG | pool-5-thread-21 | [task:batch_create:495] Sleep before reading the doc for verification 2021-06-08 11:17:04,671 | infra | DEBUG | pool-5-thread-24 | [task:batch_create:495] Sleep before reading the doc for verification 2021-06-08 11:17:04,673 | infra | DEBUG | pool-5-thread-19 | [task:batch_create:495] Sleep before reading the doc for verification 2021-06-08 11:17:04,686 | infra | DEBUG | pool-5-thread-16 | [task:batch_create:495] Sleep before reading the doc for verification 2021-06-08 11:17:04,690 | infra | DEBUG | pool-5-thread-28 | [task:batch_create:495] Sleep before reading the doc for verification 2021-06-08 11:17:04,694 | infra | DEBUG | pool-5-thread-14 | [task:batch_create:495] Sleep before reading the doc for verification 2021-06-08 11:17:04,697 | infra | DEBUG | pool-5-thread-17 | [task:batch_create:495] Sleep before reading the doc for verification 2021-06-08 11:17:04,701 | infra | DEBUG | pool-5-thread-15 | [task:batch_create:495] Sleep before reading the doc for verification 2021-06-08 11:17:05,377 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 7172.900000 2021-06-08 11:17:05,378 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:17:05,378 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:17:06,651 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 7172.900000 2021-06-08 11:17:06,654 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:17:06,654 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:17:07,954 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 7172.900000 2021-06-08 11:17:07,957 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:17:07,957 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:17:09,184 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 7172.900000 2021-06-08 11:17:09,187 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:17:09,187 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:17:10,437 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 7172.900000 2021-06-08 11:17:10,440 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:17:10,440 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:17:11,753 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 9872.400000 2021-06-08 11:17:11,756 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:17:11,756 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:17:13,026 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 15583.600000 2021-06-08 11:17:13,029 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:17:13,029 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:17:14,423 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 25650.700000 2021-06-08 11:17:14,424 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:17:14,424 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:17:15,694 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 25650.700000 2021-06-08 11:17:15,694 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:17:15,696 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:17:17,023 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 25650.700000 2021-06-08 11:17:17,026 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:17:17,026 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:17:18,285 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 25650.700000 2021-06-08 11:17:18,286 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:17:18,286 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:17:19,516 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 25650.700000 2021-06-08 11:17:19,517 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:17:19,517 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:17:20,061 | infra | INFO | pool-5-thread-23 | [task:complete_task:102] Thread 'LoadDocs_MutateDocsFromSpecTask_1623176083.82_default__default__default_ttl=0_create__3500000_3750000_1623176083.83' completed 2021-06-08 11:17:20,792 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 25650.700000 2021-06-08 11:17:20,795 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:17:20,795 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:17:22,013 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 26416.900000 2021-06-08 11:17:22,015 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:17:22,016 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:17:23,255 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 26416.900000 2021-06-08 11:17:23,256 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:17:23,256 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:17:24,492 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 26416.900000 2021-06-08 11:17:24,493 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:17:24,493 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:17:25,822 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 26416.900000 2021-06-08 11:17:25,825 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:17:25,825 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:17:26,621 | infra | INFO | pool-5-thread-21 | [task:complete_task:102] Thread 'LoadDocs_MutateDocsFromSpecTask_1623176083.82_default__default__default_ttl=0_create__3000000_3250000_1623176083.83' completed 2021-06-08 11:17:27,161 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 26416.900000 2021-06-08 11:17:27,163 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:17:27,163 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:17:28,401 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 26416.900000 2021-06-08 11:17:28,401 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:17:28,401 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:17:29,631 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 26416.900000 2021-06-08 11:17:29,632 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:17:29,632 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:17:30,855 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 26416.900000 2021-06-08 11:17:30,855 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:17:30,855 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:17:32,095 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 24682.300000 2021-06-08 11:17:32,096 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:17:32,096 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:17:33,316 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 23822.900000 2021-06-08 11:17:33,319 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:17:33,319 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:17:34,551 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 24902.500000 2021-06-08 11:17:34,552 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:17:34,552 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:17:35,290 | infra | INFO | pool-5-thread-26 | [task:complete_task:102] Thread 'LoadDocs_MutateDocsFromSpecTask_1623176083.82_default__default__default_ttl=0_create__4250000_4500000_1623176083.83' completed 2021-06-08 11:17:35,295 | infra | INFO | pool-5-thread-13 | [task:complete_task:102] Thread 'LoadDocs_MutateDocsFromSpecTask_1623176083.82_default__default__default_ttl=0_create__1000000_1250000_1623176083.83' completed 2021-06-08 11:17:35,743 | infra | INFO | pool-5-thread-12 | [task:complete_task:102] Thread 'LoadDocs_MutateDocsFromSpecTask_1623176083.82_default__default__default_ttl=0_create__750000_1000000_1623176083.83' completed 2021-06-08 11:17:35,898 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 24902.500000 2021-06-08 11:17:35,901 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:17:35,901 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:17:37,147 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 24902.500000 2021-06-08 11:17:37,148 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:17:37,148 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:17:37,642 | infra | INFO | pool-5-thread-10 | [task:complete_task:102] Thread 'LoadDocs_MutateDocsFromSpecTask_1623176083.82_default__default__default_ttl=0_create__250000_500000_1623176083.83' completed 2021-06-08 11:17:37,684 | infra | INFO | pool-5-thread-16 | [task:complete_task:102] Thread 'LoadDocs_MutateDocsFromSpecTask_1623176083.82_default__default__default_ttl=0_create__1750000_2000000_1623176083.83' completed 2021-06-08 11:17:37,691 | infra | INFO | pool-5-thread-28 | [task:complete_task:102] Thread 'LoadDocs_MutateDocsFromSpecTask_1623176083.82_default__default__default_ttl=0_create__4750000_5000000_1623176083.83' completed 2021-06-08 11:17:37,701 | infra | INFO | pool-5-thread-24 | [task:complete_task:102] Thread 'LoadDocs_MutateDocsFromSpecTask_1623176083.82_default__default__default_ttl=0_create__3750000_4000000_1623176083.83' completed 2021-06-08 11:17:38,380 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 24902.500000 2021-06-08 11:17:38,381 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:17:38,381 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:17:39,632 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 24902.500000 2021-06-08 11:17:39,634 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:17:39,634 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:17:40,868 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 24902.500000 2021-06-08 11:17:40,868 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:17:40,868 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:17:42,105 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 17066.400000 2021-06-08 11:17:42,107 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:17:42,108 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:17:42,536 | infra | INFO | pool-5-thread-9 | [task:complete_task:102] Thread 'LoadDocs_MutateDocsFromSpecTask_1623176083.82_default__default__default_ttl=0_create__0_250000_1623176083.83' completed 2021-06-08 11:17:42,536 | infra | DEBUG | pool-5-thread-7 | [task:call:3816] Items loaded in task LoadDocs_MutateDocsFromSpecTask_1623176083.82_default__default__default_ttl=0_create__0_250000_1623176083.83 are 250000 2021-06-08 11:17:42,536 | test | ERROR | pool-5-thread-7 | [task:call:3845] Failed to load 767 docs from 0 to 250000 2021-06-08 11:17:42,536 | infra | DEBUG | pool-5-thread-7 | [task_manager:get_task_result:26] Getting task result for LoadDocs_MutateDocsFromSpecTask_1623176083.82_default__default__default_ttl=0_create__250000_500000_1623176083.83 2021-06-08 11:17:42,538 | infra | DEBUG | pool-5-thread-7 | [task:call:3816] Items loaded in task LoadDocs_MutateDocsFromSpecTask_1623176083.82_default__default__default_ttl=0_create__250000_500000_1623176083.83 are 250000 2021-06-08 11:17:42,538 | test | ERROR | pool-5-thread-7 | [task:call:3845] Failed to load 649 docs from 250000 to 500000 2021-06-08 11:17:42,538 | infra | INFO | pool-5-thread-17 | [task:complete_task:102] Thread 'LoadDocs_MutateDocsFromSpecTask_1623176083.82_default__default__default_ttl=0_create__2000000_2250000_1623176083.83' completed 2021-06-08 11:17:42,538 | infra | DEBUG | pool-5-thread-7 | [task_manager:get_task_result:26] Getting task result for LoadDocs_MutateDocsFromSpecTask_1623176083.82_default__default__default_ttl=0_create__500000_750000_1623176083.83 2021-06-08 11:17:42,539 | infra | INFO | pool-5-thread-25 | [task:complete_task:102] Thread 'LoadDocs_MutateDocsFromSpecTask_1623176083.82_default__default__default_ttl=0_create__4000000_4250000_1623176083.83' completed 2021-06-08 11:17:42,586 | infra | INFO | pool-5-thread-18 | [task:complete_task:102] Thread 'LoadDocs_MutateDocsFromSpecTask_1623176083.82_default__default__default_ttl=0_create__2250000_2500000_1623176083.83' completed 2021-06-08 11:17:42,589 | infra | INFO | pool-5-thread-20 | [task:complete_task:102] Thread 'LoadDocs_MutateDocsFromSpecTask_1623176083.82_default__default__default_ttl=0_create__2750000_3000000_1623176083.83' completed 2021-06-08 11:17:42,601 | infra | INFO | pool-5-thread-11 | [task:complete_task:102] Thread 'LoadDocs_MutateDocsFromSpecTask_1623176083.82_default__default__default_ttl=0_create__500000_750000_1623176083.83' completed 2021-06-08 11:17:42,601 | infra | DEBUG | pool-5-thread-7 | [task:call:3816] Items loaded in task LoadDocs_MutateDocsFromSpecTask_1623176083.82_default__default__default_ttl=0_create__500000_750000_1623176083.83 are 250000 2021-06-08 11:17:42,601 | test | ERROR | pool-5-thread-7 | [task:call:3845] Failed to load 745 docs from 500000 to 750000 2021-06-08 11:17:42,601 | infra | DEBUG | pool-5-thread-7 | [task_manager:get_task_result:26] Getting task result for LoadDocs_MutateDocsFromSpecTask_1623176083.82_default__default__default_ttl=0_create__750000_1000000_1623176083.83 2021-06-08 11:17:42,601 | infra | DEBUG | pool-5-thread-7 | [task:call:3816] Items loaded in task LoadDocs_MutateDocsFromSpecTask_1623176083.82_default__default__default_ttl=0_create__750000_1000000_1623176083.83 are 250000 2021-06-08 11:17:42,601 | test | ERROR | pool-5-thread-7 | [task:call:3845] Failed to load 755 docs from 750000 to 1000000 2021-06-08 11:17:42,601 | infra | DEBUG | pool-5-thread-7 | [task_manager:get_task_result:26] Getting task result for LoadDocs_MutateDocsFromSpecTask_1623176083.82_default__default__default_ttl=0_create__1000000_1250000_1623176083.83 2021-06-08 11:17:42,602 | infra | DEBUG | pool-5-thread-7 | [task:call:3816] Items loaded in task LoadDocs_MutateDocsFromSpecTask_1623176083.82_default__default__default_ttl=0_create__1000000_1250000_1623176083.83 are 250000 2021-06-08 11:17:42,602 | test | ERROR | pool-5-thread-7 | [task:call:3845] Failed to load 780 docs from 1000000 to 1250000 2021-06-08 11:17:42,602 | infra | DEBUG | pool-5-thread-7 | [task_manager:get_task_result:26] Getting task result for LoadDocs_MutateDocsFromSpecTask_1623176083.82_default__default__default_ttl=0_create__1250000_1500000_1623176083.83 2021-06-08 11:17:43,335 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 17066.400000 2021-06-08 11:17:43,336 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:17:43,336 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:17:43,918 | infra | INFO | pool-5-thread-19 | [task:complete_task:102] Thread 'LoadDocs_MutateDocsFromSpecTask_1623176083.82_default__default__default_ttl=0_create__2500000_2750000_1623176083.83' completed 2021-06-08 11:17:43,921 | infra | INFO | pool-5-thread-27 | [task:complete_task:102] Thread 'LoadDocs_MutateDocsFromSpecTask_1623176083.82_default__default__default_ttl=0_create__4500000_4750000_1623176083.83' completed 2021-06-08 11:17:44,569 | test | DEBUG | pool-5-thread-8 | [task:record_bucket_ops:3495] Ops rate for 'default': 17066.400000 2021-06-08 11:17:44,572 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:17:44,572 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:17:44,960 | infra | INFO | pool-5-thread-14 | [task:complete_task:102] Thread 'LoadDocs_MutateDocsFromSpecTask_1623176083.82_default__default__default_ttl=0_create__1250000_1500000_1623176083.83' completed 2021-06-08 11:17:44,960 | infra | DEBUG | pool-5-thread-7 | [task:call:3816] Items loaded in task LoadDocs_MutateDocsFromSpecTask_1623176083.82_default__default__default_ttl=0_create__1250000_1500000_1623176083.83 are 250000 2021-06-08 11:17:44,960 | test | ERROR | pool-5-thread-7 | [task:call:3845] Failed to load 755 docs from 1250000 to 1500000 2021-06-08 11:17:44,960 | infra | DEBUG | pool-5-thread-7 | [task_manager:get_task_result:26] Getting task result for LoadDocs_MutateDocsFromSpecTask_1623176083.82_default__default__default_ttl=0_create__1500000_1750000_1623176083.83 2021-06-08 11:17:44,961 | infra | INFO | pool-5-thread-15 | [task:complete_task:102] Thread 'LoadDocs_MutateDocsFromSpecTask_1623176083.82_default__default__default_ttl=0_create__1500000_1750000_1623176083.83' completed 2021-06-08 11:17:44,963 | infra | DEBUG | pool-5-thread-7 | [task:call:3816] Items loaded in task LoadDocs_MutateDocsFromSpecTask_1623176083.82_default__default__default_ttl=0_create__1500000_1750000_1623176083.83 are 250000 2021-06-08 11:17:44,963 | test | ERROR | pool-5-thread-7 | [task:call:3845] Failed to load 710 docs from 1500000 to 1750000 2021-06-08 11:17:44,963 | infra | DEBUG | pool-5-thread-7 | [task_manager:get_task_result:26] Getting task result for LoadDocs_MutateDocsFromSpecTask_1623176083.82_default__default__default_ttl=0_create__1750000_2000000_1623176083.83 2021-06-08 11:17:44,963 | infra | DEBUG | pool-5-thread-7 | [task:call:3816] Items loaded in task LoadDocs_MutateDocsFromSpecTask_1623176083.82_default__default__default_ttl=0_create__1750000_2000000_1623176083.83 are 250000 2021-06-08 11:17:44,963 | test | ERROR | pool-5-thread-7 | [task:call:3845] Failed to load 756 docs from 1750000 to 2000000 2021-06-08 11:17:44,963 | infra | DEBUG | pool-5-thread-7 | [task_manager:get_task_result:26] Getting task result for LoadDocs_MutateDocsFromSpecTask_1623176083.82_default__default__default_ttl=0_create__2000000_2250000_1623176083.83 2021-06-08 11:17:44,963 | infra | DEBUG | pool-5-thread-7 | [task:call:3816] Items loaded in task LoadDocs_MutateDocsFromSpecTask_1623176083.82_default__default__default_ttl=0_create__2000000_2250000_1623176083.83 are 250000 2021-06-08 11:17:44,964 | test | ERROR | pool-5-thread-7 | [task:call:3845] Failed to load 762 docs from 2000000 to 2250000 2021-06-08 11:17:44,964 | infra | DEBUG | pool-5-thread-7 | [task_manager:get_task_result:26] Getting task result for LoadDocs_MutateDocsFromSpecTask_1623176083.82_default__default__default_ttl=0_create__2250000_2500000_1623176083.83 2021-06-08 11:17:44,964 | infra | DEBUG | pool-5-thread-7 | [task:call:3816] Items loaded in task LoadDocs_MutateDocsFromSpecTask_1623176083.82_default__default__default_ttl=0_create__2250000_2500000_1623176083.83 are 250000 2021-06-08 11:17:44,964 | test | ERROR | pool-5-thread-7 | [task:call:3845] Failed to load 568 docs from 2250000 to 2500000 2021-06-08 11:17:44,964 | infra | DEBUG | pool-5-thread-7 | [task_manager:get_task_result:26] Getting task result for LoadDocs_MutateDocsFromSpecTask_1623176083.82_default__default__default_ttl=0_create__2500000_2750000_1623176083.83 2021-06-08 11:17:44,966 | infra | DEBUG | pool-5-thread-7 | [task:call:3816] Items loaded in task LoadDocs_MutateDocsFromSpecTask_1623176083.82_default__default__default_ttl=0_create__2500000_2750000_1623176083.83 are 250000 2021-06-08 11:17:44,966 | test | ERROR | pool-5-thread-7 | [task:call:3845] Failed to load 756 docs from 2500000 to 2750000 2021-06-08 11:17:44,966 | infra | DEBUG | pool-5-thread-7 | [task_manager:get_task_result:26] Getting task result for LoadDocs_MutateDocsFromSpecTask_1623176083.82_default__default__default_ttl=0_create__2750000_3000000_1623176083.83 2021-06-08 11:17:44,966 | infra | DEBUG | pool-5-thread-7 | [task:call:3816] Items loaded in task LoadDocs_MutateDocsFromSpecTask_1623176083.82_default__default__default_ttl=0_create__2750000_3000000_1623176083.83 are 250000 2021-06-08 11:17:44,967 | test | ERROR | pool-5-thread-7 | [task:call:3845] Failed to load 599 docs from 2750000 to 3000000 2021-06-08 11:17:44,967 | infra | DEBUG | pool-5-thread-7 | [task_manager:get_task_result:26] Getting task result for LoadDocs_MutateDocsFromSpecTask_1623176083.82_default__default__default_ttl=0_create__3000000_3250000_1623176083.83 2021-06-08 11:17:44,967 | infra | DEBUG | pool-5-thread-7 | [task:call:3816] Items loaded in task LoadDocs_MutateDocsFromSpecTask_1623176083.82_default__default__default_ttl=0_create__3000000_3250000_1623176083.83 are 250000 2021-06-08 11:17:44,967 | test | ERROR | pool-5-thread-7 | [task:call:3845] Failed to load 1255 docs from 3000000 to 3250000 2021-06-08 11:17:44,967 | infra | DEBUG | pool-5-thread-7 | [task_manager:get_task_result:26] Getting task result for LoadDocs_MutateDocsFromSpecTask_1623176083.82_default__default__default_ttl=0_create__3250000_3500000_1623176083.83 2021-06-08 11:17:45,035 | infra | INFO | pool-5-thread-22 | [task:complete_task:102] Thread 'LoadDocs_MutateDocsFromSpecTask_1623176083.82_default__default__default_ttl=0_create__3250000_3500000_1623176083.83' completed 2021-06-08 11:17:45,036 | infra | DEBUG | pool-5-thread-7 | [task:call:3816] Items loaded in task LoadDocs_MutateDocsFromSpecTask_1623176083.82_default__default__default_ttl=0_create__3250000_3500000_1623176083.83 are 250000 2021-06-08 11:17:45,036 | test | ERROR | pool-5-thread-7 | [task:call:3845] Failed to load 714 docs from 3250000 to 3500000 2021-06-08 11:17:45,036 | infra | DEBUG | pool-5-thread-7 | [task_manager:get_task_result:26] Getting task result for LoadDocs_MutateDocsFromSpecTask_1623176083.82_default__default__default_ttl=0_create__3500000_3750000_1623176083.83 2021-06-08 11:17:45,036 | infra | DEBUG | pool-5-thread-7 | [task:call:3816] Items loaded in task LoadDocs_MutateDocsFromSpecTask_1623176083.82_default__default__default_ttl=0_create__3500000_3750000_1623176083.83 are 250000 2021-06-08 11:17:45,036 | test | ERROR | pool-5-thread-7 | [task:call:3845] Failed to load 708 docs from 3500000 to 3750000 2021-06-08 11:17:45,036 | infra | DEBUG | pool-5-thread-7 | [task_manager:get_task_result:26] Getting task result for LoadDocs_MutateDocsFromSpecTask_1623176083.82_default__default__default_ttl=0_create__3750000_4000000_1623176083.83 2021-06-08 11:17:45,036 | infra | DEBUG | pool-5-thread-7 | [task:call:3816] Items loaded in task LoadDocs_MutateDocsFromSpecTask_1623176083.82_default__default__default_ttl=0_create__3750000_4000000_1623176083.83 are 250000 2021-06-08 11:17:45,036 | test | ERROR | pool-5-thread-7 | [task:call:3845] Failed to load 755 docs from 3750000 to 4000000 2021-06-08 11:17:45,038 | infra | DEBUG | pool-5-thread-7 | [task_manager:get_task_result:26] Getting task result for LoadDocs_MutateDocsFromSpecTask_1623176083.82_default__default__default_ttl=0_create__4000000_4250000_1623176083.83 2021-06-08 11:17:45,038 | infra | DEBUG | pool-5-thread-7 | [task:call:3816] Items loaded in task LoadDocs_MutateDocsFromSpecTask_1623176083.82_default__default__default_ttl=0_create__4000000_4250000_1623176083.83 are 250000 2021-06-08 11:17:45,039 | test | ERROR | pool-5-thread-7 | [task:call:3845] Failed to load 1000 docs from 4000000 to 4250000 2021-06-08 11:17:45,039 | infra | DEBUG | pool-5-thread-7 | [task_manager:get_task_result:26] Getting task result for LoadDocs_MutateDocsFromSpecTask_1623176083.82_default__default__default_ttl=0_create__4250000_4500000_1623176083.83 2021-06-08 11:17:45,039 | infra | DEBUG | pool-5-thread-7 | [task:call:3816] Items loaded in task LoadDocs_MutateDocsFromSpecTask_1623176083.82_default__default__default_ttl=0_create__4250000_4500000_1623176083.83 are 250000 2021-06-08 11:17:45,039 | test | ERROR | pool-5-thread-7 | [task:call:3845] Failed to load 784 docs from 4250000 to 4500000 2021-06-08 11:17:45,039 | infra | DEBUG | pool-5-thread-7 | [task_manager:get_task_result:26] Getting task result for LoadDocs_MutateDocsFromSpecTask_1623176083.82_default__default__default_ttl=0_create__4500000_4750000_1623176083.83 2021-06-08 11:17:45,039 | infra | DEBUG | pool-5-thread-7 | [task:call:3816] Items loaded in task LoadDocs_MutateDocsFromSpecTask_1623176083.82_default__default__default_ttl=0_create__4500000_4750000_1623176083.83 are 250000 2021-06-08 11:17:45,039 | test | ERROR | pool-5-thread-7 | [task:call:3845] Failed to load 1000 docs from 4500000 to 4750000 2021-06-08 11:17:45,039 | infra | DEBUG | pool-5-thread-7 | [task_manager:get_task_result:26] Getting task result for LoadDocs_MutateDocsFromSpecTask_1623176083.82_default__default__default_ttl=0_create__4750000_5000000_1623176083.83 2021-06-08 11:17:45,040 | infra | DEBUG | pool-5-thread-7 | [task:call:3816] Items loaded in task LoadDocs_MutateDocsFromSpecTask_1623176083.82_default__default__default_ttl=0_create__4750000_5000000_1623176083.83 are 250000 2021-06-08 11:17:45,040 | test | ERROR | pool-5-thread-7 | [task:call:3845] Failed to load 756 docs from 4750000 to 5000000 2021-06-08 11:17:45,040 | infra | DEBUG | pool-5-thread-7 | [task_manager:get_task_result:26] Getting task result for PrintBucketStats_default_1623176083.83 2021-06-08 11:17:45,572 | test | INFO | pool-5-thread-8 | [table_view:display:72] Ops trend for bucket 'default' +-----------+---------------------------------------------------------------------------------------------------------------------------+------------+ | Min | Trend | Max | +-----------+---------------------------------------------------------------------------------------------------------------------------+------------+ | 0.000 | ******.........................................................................*******...........................*******X | 112744.074 | | 93791.512 | ..................................................................................*******..............******X | 109498.200 | | 63591.700 | .......................................................*******X | 63591.700 | | 10325.800 | ........*******X | 10325.800 | | 8836.000 | ......***************X | 8996.700 | | 8035.600 | ......**........*****X | 17952.000 | | 7614.600 | .....**********....*****X | 11789.400 | | 7676.000 | .....*********...******X | 10161.300 | | 9969.400 | .......X | 9969.400 | | 7172.900 | .....*******..*.....*.........******.*******X | 26416.900 | | 24682.300 | ....................X | 24682.300 | | 23822.900 | ....................*.*****X | 24902.500 | | 17066.400 | ..............**X | 17066.400 | +-----------+---------------------------------------------------------------------------------------------------------------------------+------------+ 2021-06-08 11:17:45,573 | infra | INFO | pool-5-thread-8 | [task:complete_task:102] Thread 'PrintBucketStats_default_1623176083.83' completed 2021-06-08 11:17:45,573 | infra | DEBUG | pool-5-thread-7 | [task:call:3861] ========= Tasks in loadgen pool======= 2021-06-08 11:17:45,573 | infra | WARNING | pool-5-thread-7 | [task_manager:print_tasks_in_pool:83] Task 'MutateDocsFromSpecTask_1623176083.82' not completed 2021-06-08 11:17:45,573 | infra | DEBUG | pool-5-thread-7 | [task:call:3863] ====================================== 2021-06-08 11:17:45,573 | infra | DEBUG | pool-5-thread-7 | [task:call:3866] Task 'LoadDocs_MutateDocsFromSpecTask_1623176083.82_default__default__default_ttl=0_create__0_250000_1623176083.83' complete. Loaded 250000 items 2021-06-08 11:17:45,575 | infra | DEBUG | pool-5-thread-7 | [task:call:3866] Task 'LoadDocs_MutateDocsFromSpecTask_1623176083.82_default__default__default_ttl=0_create__250000_500000_1623176083.83' complete. Loaded 250000 items 2021-06-08 11:17:45,575 | infra | DEBUG | pool-5-thread-7 | [task:call:3866] Task 'LoadDocs_MutateDocsFromSpecTask_1623176083.82_default__default__default_ttl=0_create__500000_750000_1623176083.83' complete. Loaded 250000 items 2021-06-08 11:17:45,575 | infra | DEBUG | pool-5-thread-7 | [task:call:3866] Task 'LoadDocs_MutateDocsFromSpecTask_1623176083.82_default__default__default_ttl=0_create__750000_1000000_1623176083.83' complete. Loaded 250000 items 2021-06-08 11:17:45,575 | infra | DEBUG | pool-5-thread-7 | [task:call:3866] Task 'LoadDocs_MutateDocsFromSpecTask_1623176083.82_default__default__default_ttl=0_create__1000000_1250000_1623176083.83' complete. Loaded 250000 items 2021-06-08 11:17:45,575 | infra | DEBUG | pool-5-thread-7 | [task:call:3866] Task 'LoadDocs_MutateDocsFromSpecTask_1623176083.82_default__default__default_ttl=0_create__1250000_1500000_1623176083.83' complete. Loaded 250000 items 2021-06-08 11:17:45,575 | infra | DEBUG | pool-5-thread-7 | [task:call:3866] Task 'LoadDocs_MutateDocsFromSpecTask_1623176083.82_default__default__default_ttl=0_create__1500000_1750000_1623176083.83' complete. Loaded 250000 items 2021-06-08 11:17:45,575 | infra | DEBUG | pool-5-thread-7 | [task:call:3866] Task 'LoadDocs_MutateDocsFromSpecTask_1623176083.82_default__default__default_ttl=0_create__1750000_2000000_1623176083.83' complete. Loaded 250000 items 2021-06-08 11:17:45,575 | infra | DEBUG | pool-5-thread-7 | [task:call:3866] Task 'LoadDocs_MutateDocsFromSpecTask_1623176083.82_default__default__default_ttl=0_create__2000000_2250000_1623176083.83' complete. Loaded 250000 items 2021-06-08 11:17:45,575 | infra | DEBUG | pool-5-thread-7 | [task:call:3866] Task 'LoadDocs_MutateDocsFromSpecTask_1623176083.82_default__default__default_ttl=0_create__2250000_2500000_1623176083.83' complete. Loaded 250000 items 2021-06-08 11:17:45,575 | infra | DEBUG | pool-5-thread-7 | [task:call:3866] Task 'LoadDocs_MutateDocsFromSpecTask_1623176083.82_default__default__default_ttl=0_create__2500000_2750000_1623176083.83' complete. Loaded 250000 items 2021-06-08 11:17:45,575 | infra | DEBUG | pool-5-thread-7 | [task:call:3866] Task 'LoadDocs_MutateDocsFromSpecTask_1623176083.82_default__default__default_ttl=0_create__2750000_3000000_1623176083.83' complete. Loaded 250000 items 2021-06-08 11:17:45,575 | infra | DEBUG | pool-5-thread-7 | [task:call:3866] Task 'LoadDocs_MutateDocsFromSpecTask_1623176083.82_default__default__default_ttl=0_create__3000000_3250000_1623176083.83' complete. Loaded 250000 items 2021-06-08 11:17:45,575 | infra | DEBUG | pool-5-thread-7 | [task:call:3866] Task 'LoadDocs_MutateDocsFromSpecTask_1623176083.82_default__default__default_ttl=0_create__3250000_3500000_1623176083.83' complete. Loaded 250000 items 2021-06-08 11:17:45,575 | infra | DEBUG | pool-5-thread-7 | [task:call:3866] Task 'LoadDocs_MutateDocsFromSpecTask_1623176083.82_default__default__default_ttl=0_create__3500000_3750000_1623176083.83' complete. Loaded 250000 items 2021-06-08 11:17:45,575 | infra | DEBUG | pool-5-thread-7 | [task:call:3866] Task 'LoadDocs_MutateDocsFromSpecTask_1623176083.82_default__default__default_ttl=0_create__3750000_4000000_1623176083.83' complete. Loaded 250000 items 2021-06-08 11:17:45,575 | infra | DEBUG | pool-5-thread-7 | [task:call:3866] Task 'LoadDocs_MutateDocsFromSpecTask_1623176083.82_default__default__default_ttl=0_create__4000000_4250000_1623176083.83' complete. Loaded 250000 items 2021-06-08 11:17:45,575 | infra | DEBUG | pool-5-thread-7 | [task:call:3866] Task 'LoadDocs_MutateDocsFromSpecTask_1623176083.82_default__default__default_ttl=0_create__4250000_4500000_1623176083.83' complete. Loaded 250000 items 2021-06-08 11:17:45,575 | infra | DEBUG | pool-5-thread-7 | [task:call:3866] Task 'LoadDocs_MutateDocsFromSpecTask_1623176083.82_default__default__default_ttl=0_create__4500000_4750000_1623176083.83' complete. Loaded 250000 items 2021-06-08 11:17:45,575 | infra | DEBUG | pool-5-thread-7 | [task:call:3866] Task 'LoadDocs_MutateDocsFromSpecTask_1623176083.82_default__default__default_ttl=0_create__4750000_5000000_1623176083.83' complete. Loaded 250000 items 2021-06-08 11:17:45,575 | infra | INFO | pool-5-thread-7 | [task:complete_task:102] Thread 'MutateDocsFromSpecTask_1623176083.82' completed 2021-06-08 11:18:02,802 | test | DEBUG | MainThread | [cluster_ready_functions:get_nodes_from_services_map:597] All nodes in cluster: [ip:172.23.106.155 port:8091 ssh_username:root, ip:172.23.106.156 port:8091 ssh_username:root, ip:172.23.106.122 port:8091 ssh_username:root, ip:172.23.106.125 port:8091 ssh_username:root] 2021-06-08 11:18:02,805 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.156 with username: root, password: couchbase 2021-06-08 11:18:02,897 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 11:18:03,111 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: ls /etc/ 2021-06-08 11:18:03,165 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 11:18:03,167 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: ls /cygdrive/c/Windows 2021-06-08 11:18:03,178 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: uname -m 2021-06-08 11:18:03,190 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: cat /proc/cpuinfo 2021-06-08 11:18:03,201 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: df -Thl 2021-06-08 11:18:03,220 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: cat /proc/meminfo 2021-06-08 11:18:03,234 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: hostname 2021-06-08 11:18:03,247 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: hostname -d 2021-06-08 11:18:03,262 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:559] Starting StatsWaitTask for num_items_for_persistence on bucket default 2021-06-08 11:18:03,263 | infra | INFO | MainThread | [task_manager:add_new_task:23] Added new task: StatsWaitTask_default_num_items_for_persistence_1623176283.26 2021-06-08 11:18:03,265 | infra | INFO | pool-5-thread-29 | [task:start_task:87] Thread 'StatsWaitTask_default_num_items_for_persistence_1623176283.26' started 2021-06-08 11:18:03,265 | infra | DEBUG | pool-5-thread-29 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 11:18:03,266 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.125 with username: root, password: couchbase 2021-06-08 11:18:03,342 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 11:18:03,571 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: ls /etc/ 2021-06-08 11:18:03,630 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 11:18:03,630 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: ls /cygdrive/c/Windows 2021-06-08 11:18:03,642 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: uname -m 2021-06-08 11:18:03,654 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: cat /proc/cpuinfo 2021-06-08 11:18:03,667 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: df -Thl 2021-06-08 11:18:03,683 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: cat /proc/meminfo 2021-06-08 11:18:03,694 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: hostname 2021-06-08 11:18:03,707 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: hostname -d 2021-06-08 11:18:03,719 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:559] Starting StatsWaitTask for num_items_for_persistence on bucket default 2021-06-08 11:18:03,720 | infra | INFO | MainThread | [task_manager:add_new_task:23] Added new task: StatsWaitTask_default_num_items_for_persistence_1623176283.72 2021-06-08 11:18:03,720 | infra | INFO | pool-5-thread-30 | [task:start_task:87] Thread 'StatsWaitTask_default_num_items_for_persistence_1623176283.72' started 2021-06-08 11:18:03,720 | infra | DEBUG | pool-5-thread-30 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 11:18:03,723 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.122 with username: root, password: couchbase 2021-06-08 11:18:03,806 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 11:18:04,023 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: ls /etc/ 2021-06-08 11:18:04,082 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 11:18:04,082 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: ls /cygdrive/c/Windows 2021-06-08 11:18:04,098 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: uname -m 2021-06-08 11:18:04,111 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: cat /proc/cpuinfo 2021-06-08 11:18:04,124 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: df -Thl 2021-06-08 11:18:04,140 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: cat /proc/meminfo 2021-06-08 11:18:04,153 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: hostname 2021-06-08 11:18:04,165 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: hostname -d 2021-06-08 11:18:04,183 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:559] Starting StatsWaitTask for num_items_for_persistence on bucket default 2021-06-08 11:18:04,184 | infra | INFO | MainThread | [task_manager:add_new_task:23] Added new task: StatsWaitTask_default_num_items_for_persistence_1623176284.18 2021-06-08 11:18:04,184 | infra | INFO | pool-5-thread-2 | [task:start_task:87] Thread 'StatsWaitTask_default_num_items_for_persistence_1623176284.18' started 2021-06-08 11:18:04,184 | infra | DEBUG | pool-5-thread-2 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 11:18:04,187 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.155 with username: root, password: couchbase 2021-06-08 11:18:04,266 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 11:18:04,484 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: ls /etc/ 2021-06-08 11:18:04,546 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 11:18:04,546 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: ls /cygdrive/c/Windows 2021-06-08 11:18:04,561 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: uname -m 2021-06-08 11:18:04,573 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: cat /proc/cpuinfo 2021-06-08 11:18:04,585 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: df -Thl 2021-06-08 11:18:04,601 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: cat /proc/meminfo 2021-06-08 11:18:04,615 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: hostname 2021-06-08 11:18:04,628 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: hostname -d 2021-06-08 11:18:04,642 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:559] Starting StatsWaitTask for num_items_for_persistence on bucket default 2021-06-08 11:18:04,642 | infra | INFO | MainThread | [task_manager:add_new_task:23] Added new task: StatsWaitTask_default_num_items_for_persistence_1623176284.64 2021-06-08 11:18:04,642 | infra | INFO | pool-5-thread-1 | [task:start_task:87] Thread 'StatsWaitTask_default_num_items_for_persistence_1623176284.64' started 2021-06-08 11:18:04,642 | infra | DEBUG | MainThread | [task_manager:get_task_result:26] Getting task result for StatsWaitTask_default_num_items_for_persistence_1623176283.26 2021-06-08 11:18:04,644 | infra | DEBUG | pool-5-thread-1 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 11:18:05,183 | test | DEBUG | pool-5-thread-29 | [task:_compare:2649] Comparing 2585926 == 0 2021-06-08 11:18:05,184 | test | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 2585926 == 0. Received: {'172.23.106.156': 2585926} for bucket 'default' 2021-06-08 11:18:05,184 | infra | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 11:18:05,186 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 11:18:05,186 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 11:18:05,203 | test | DEBUG | pool-5-thread-30 | [task:_compare:2649] Comparing 279741 == 0 2021-06-08 11:18:05,203 | test | DEBUG | pool-5-thread-30 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 279741 == 0. Received: {'172.23.106.125': 279741} for bucket 'default' 2021-06-08 11:18:05,203 | infra | DEBUG | pool-5-thread-30 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 11:18:05,206 | infra | DEBUG | pool-5-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 11:18:05,206 | infra | DEBUG | pool-5-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 11:18:05,456 | test | DEBUG | pool-5-thread-2 | [task:_compare:2649] Comparing 231660 == 0 2021-06-08 11:18:05,456 | test | DEBUG | pool-5-thread-2 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 231660 == 0. Received: {'172.23.106.122': 231660} for bucket 'default' 2021-06-08 11:18:05,456 | infra | DEBUG | pool-5-thread-2 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 11:18:05,457 | infra | DEBUG | pool-5-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 11:18:05,457 | infra | DEBUG | pool-5-thread-2 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 11:18:05,943 | test | DEBUG | pool-5-thread-1 | [task:_compare:2649] Comparing 233627 == 0 2021-06-08 11:18:05,943 | test | DEBUG | pool-5-thread-1 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 233627 == 0. Received: {'172.23.106.155': 233627} for bucket 'default' 2021-06-08 11:18:05,943 | infra | DEBUG | pool-5-thread-1 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 11:18:05,944 | infra | DEBUG | pool-5-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 11:18:05,944 | infra | DEBUG | pool-5-thread-1 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 11:18:10,187 | infra | DEBUG | pool-5-thread-29 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 11:18:10,207 | infra | DEBUG | pool-5-thread-30 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 11:18:10,459 | infra | DEBUG | pool-5-thread-2 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 11:18:10,946 | infra | DEBUG | pool-5-thread-1 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 11:18:11,459 | test | DEBUG | pool-5-thread-30 | [task:_compare:2649] Comparing 166585 == 0 2021-06-08 11:18:11,459 | test | DEBUG | pool-5-thread-30 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 166585 == 0. Received: {'172.23.106.125': 166585} for bucket 'default' 2021-06-08 11:18:11,459 | infra | DEBUG | pool-5-thread-30 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 11:18:11,460 | infra | DEBUG | pool-5-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 11:18:11,460 | infra | DEBUG | pool-5-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 11:18:11,763 | test | DEBUG | pool-5-thread-2 | [task:_compare:2649] Comparing 113741 == 0 2021-06-08 11:18:11,765 | test | DEBUG | pool-5-thread-2 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 113741 == 0. Received: {'172.23.106.122': 113741} for bucket 'default' 2021-06-08 11:18:11,765 | infra | DEBUG | pool-5-thread-2 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 11:18:11,766 | infra | DEBUG | pool-5-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 11:18:11,766 | infra | DEBUG | pool-5-thread-2 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 11:18:12,040 | test | DEBUG | pool-5-thread-29 | [task:_compare:2649] Comparing 2577047 == 0 2021-06-08 11:18:12,042 | test | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 2577047 == 0. Received: {'172.23.106.156': 2577047} for bucket 'default' 2021-06-08 11:18:12,042 | infra | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 11:18:12,043 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 11:18:12,043 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 11:18:12,184 | test | DEBUG | pool-5-thread-1 | [task:_compare:2649] Comparing 119845 == 0 2021-06-08 11:18:12,184 | test | DEBUG | pool-5-thread-1 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 119845 == 0. Received: {'172.23.106.155': 119845} for bucket 'default' 2021-06-08 11:18:12,184 | infra | DEBUG | pool-5-thread-1 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 11:18:12,186 | infra | DEBUG | pool-5-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 11:18:12,186 | infra | DEBUG | pool-5-thread-1 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 11:18:16,461 | infra | DEBUG | pool-5-thread-30 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 11:18:16,767 | infra | DEBUG | pool-5-thread-2 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 11:18:17,043 | infra | DEBUG | pool-5-thread-29 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 11:18:17,187 | infra | DEBUG | pool-5-thread-1 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 11:18:17,697 | test | DEBUG | pool-5-thread-30 | [task:_compare:2649] Comparing 64903 == 0 2021-06-08 11:18:17,698 | test | DEBUG | pool-5-thread-30 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 64903 == 0. Received: {'172.23.106.125': 64903} for bucket 'default' 2021-06-08 11:18:17,698 | infra | DEBUG | pool-5-thread-30 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 11:18:17,700 | infra | DEBUG | pool-5-thread-30 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 11:18:17,700 | infra | DEBUG | pool-5-thread-30 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 11:18:17,913 | test | DEBUG | pool-5-thread-2 | [task:_compare:2649] Comparing 12794 == 0 2021-06-08 11:18:17,914 | test | DEBUG | pool-5-thread-2 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 12794 == 0. Received: {'172.23.106.122': 12794} for bucket 'default' 2021-06-08 11:18:17,914 | infra | DEBUG | pool-5-thread-2 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 11:18:17,915 | infra | DEBUG | pool-5-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 11:18:17,915 | infra | DEBUG | pool-5-thread-2 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 11:18:18,332 | test | DEBUG | pool-5-thread-1 | [task:_compare:2649] Comparing 10411 == 0 2021-06-08 11:18:18,332 | test | DEBUG | pool-5-thread-1 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 10411 == 0. Received: {'172.23.106.155': 10411} for bucket 'default' 2021-06-08 11:18:18,332 | infra | DEBUG | pool-5-thread-1 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 11:18:18,334 | infra | DEBUG | pool-5-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 11:18:18,334 | infra | DEBUG | pool-5-thread-1 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 11:18:18,842 | test | DEBUG | pool-5-thread-29 | [task:_compare:2649] Comparing 2500989 == 0 2021-06-08 11:18:18,842 | test | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 2500989 == 0. Received: {'172.23.106.156': 2500989} for bucket 'default' 2021-06-08 11:18:18,842 | infra | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 11:18:18,845 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 11:18:18,845 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 11:18:22,700 | infra | DEBUG | pool-5-thread-30 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 11:18:22,915 | infra | DEBUG | pool-5-thread-2 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 11:18:23,335 | infra | DEBUG | pool-5-thread-1 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 11:18:23,845 | infra | DEBUG | pool-5-thread-29 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 11:18:23,849 | test | DEBUG | pool-5-thread-30 | [task:_compare:2649] Comparing 0 == 0 2021-06-08 11:18:23,849 | test | DEBUG | pool-5-thread-30 | [task:_get_checkpoint_stats_and_compare:2638] Ready: num_items_for_persistence 0 == 0. Received: {'172.23.106.125': 0} for bucket 'default' 2021-06-08 11:18:23,849 | infra | INFO | pool-5-thread-30 | [task:complete_task:102] Thread 'StatsWaitTask_default_num_items_for_persistence_1623176283.72' completed 2021-06-08 11:18:24,055 | test | DEBUG | pool-5-thread-2 | [task:_compare:2649] Comparing 0 == 0 2021-06-08 11:18:24,055 | test | DEBUG | pool-5-thread-2 | [task:_get_checkpoint_stats_and_compare:2638] Ready: num_items_for_persistence 0 == 0. Received: {'172.23.106.122': 0} for bucket 'default' 2021-06-08 11:18:24,055 | infra | INFO | pool-5-thread-2 | [task:complete_task:102] Thread 'StatsWaitTask_default_num_items_for_persistence_1623176284.18' completed 2021-06-08 11:18:24,463 | test | DEBUG | pool-5-thread-1 | [task:_compare:2649] Comparing 0 == 0 2021-06-08 11:18:24,463 | test | DEBUG | pool-5-thread-1 | [task:_get_checkpoint_stats_and_compare:2638] Ready: num_items_for_persistence 0 == 0. Received: {'172.23.106.155': 0} for bucket 'default' 2021-06-08 11:18:24,463 | infra | INFO | pool-5-thread-1 | [task:complete_task:102] Thread 'StatsWaitTask_default_num_items_for_persistence_1623176284.64' completed 2021-06-08 11:18:25,726 | test | DEBUG | pool-5-thread-29 | [task:_compare:2649] Comparing 2422264 == 0 2021-06-08 11:18:25,726 | test | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 2422264 == 0. Received: {'172.23.106.156': 2422264} for bucket 'default' 2021-06-08 11:18:25,726 | infra | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 11:18:25,729 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 11:18:25,729 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 11:18:30,730 | infra | DEBUG | pool-5-thread-29 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 11:18:32,641 | test | DEBUG | pool-5-thread-29 | [task:_compare:2649] Comparing 2305068 == 0 2021-06-08 11:18:32,641 | test | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 2305068 == 0. Received: {'172.23.106.156': 2305068} for bucket 'default' 2021-06-08 11:18:32,642 | infra | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 11:18:32,644 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 11:18:32,644 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 11:18:37,644 | infra | DEBUG | pool-5-thread-29 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 11:18:39,519 | test | DEBUG | pool-5-thread-29 | [task:_compare:2649] Comparing 2198521 == 0 2021-06-08 11:18:39,520 | test | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 2198521 == 0. Received: {'172.23.106.156': 2198521} for bucket 'default' 2021-06-08 11:18:39,520 | infra | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 11:18:39,523 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 11:18:39,523 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 11:18:44,523 | infra | DEBUG | pool-5-thread-29 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 11:18:46,305 | test | DEBUG | pool-5-thread-29 | [task:_compare:2649] Comparing 2081830 == 0 2021-06-08 11:18:46,306 | test | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 2081830 == 0. Received: {'172.23.106.156': 2081830} for bucket 'default' 2021-06-08 11:18:46,306 | infra | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 11:18:46,309 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 11:18:46,309 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 11:18:51,309 | infra | DEBUG | pool-5-thread-29 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 11:18:53,246 | test | DEBUG | pool-5-thread-29 | [task:_compare:2649] Comparing 1989479 == 0 2021-06-08 11:18:53,246 | test | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 1989479 == 0. Received: {'172.23.106.156': 1989479} for bucket 'default' 2021-06-08 11:18:53,246 | infra | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 11:18:53,247 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 11:18:53,247 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 11:18:58,249 | infra | DEBUG | pool-5-thread-29 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 11:18:59,940 | test | DEBUG | pool-5-thread-29 | [task:_compare:2649] Comparing 1911958 == 0 2021-06-08 11:18:59,940 | test | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 1911958 == 0. Received: {'172.23.106.156': 1911958} for bucket 'default' 2021-06-08 11:18:59,940 | infra | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 11:18:59,943 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 11:18:59,943 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 11:19:04,943 | infra | DEBUG | pool-5-thread-29 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 11:19:06,619 | test | DEBUG | pool-5-thread-29 | [task:_compare:2649] Comparing 1861406 == 0 2021-06-08 11:19:06,621 | test | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 1861406 == 0. Received: {'172.23.106.156': 1861406} for bucket 'default' 2021-06-08 11:19:06,621 | infra | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 11:19:06,622 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 11:19:06,622 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 11:19:11,622 | infra | DEBUG | pool-5-thread-29 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 11:19:14,026 | test | DEBUG | pool-5-thread-29 | [task:_compare:2649] Comparing 1787876 == 0 2021-06-08 11:19:14,026 | test | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 1787876 == 0. Received: {'172.23.106.156': 1787876} for bucket 'default' 2021-06-08 11:19:14,026 | infra | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 11:19:14,029 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 11:19:14,029 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 11:19:19,029 | infra | DEBUG | pool-5-thread-29 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 11:19:20,674 | test | DEBUG | pool-5-thread-29 | [task:_compare:2649] Comparing 1727166 == 0 2021-06-08 11:19:20,676 | test | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 1727166 == 0. Received: {'172.23.106.156': 1727166} for bucket 'default' 2021-06-08 11:19:20,676 | infra | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 11:19:20,677 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 11:19:20,677 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 11:19:25,677 | infra | DEBUG | pool-5-thread-29 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 11:19:27,303 | test | DEBUG | pool-5-thread-29 | [task:_compare:2649] Comparing 1654670 == 0 2021-06-08 11:19:27,303 | test | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 1654670 == 0. Received: {'172.23.106.156': 1654670} for bucket 'default' 2021-06-08 11:19:27,303 | infra | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 11:19:27,305 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 11:19:27,305 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 11:19:32,306 | infra | DEBUG | pool-5-thread-29 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 11:19:33,961 | test | DEBUG | pool-5-thread-29 | [task:_compare:2649] Comparing 1591206 == 0 2021-06-08 11:19:33,961 | test | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 1591206 == 0. Received: {'172.23.106.156': 1591206} for bucket 'default' 2021-06-08 11:19:33,961 | infra | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 11:19:33,963 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 11:19:33,963 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 11:19:38,963 | infra | DEBUG | pool-5-thread-29 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 11:19:40,617 | test | DEBUG | pool-5-thread-29 | [task:_compare:2649] Comparing 1529018 == 0 2021-06-08 11:19:40,618 | test | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 1529018 == 0. Received: {'172.23.106.156': 1529018} for bucket 'default' 2021-06-08 11:19:40,618 | infra | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 11:19:40,619 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 11:19:40,619 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 11:19:45,619 | infra | DEBUG | pool-5-thread-29 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 11:19:47,224 | test | DEBUG | pool-5-thread-29 | [task:_compare:2649] Comparing 1466396 == 0 2021-06-08 11:19:47,224 | test | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 1466396 == 0. Received: {'172.23.106.156': 1466396} for bucket 'default' 2021-06-08 11:19:47,224 | infra | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 11:19:47,226 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 11:19:47,226 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 11:19:52,226 | infra | DEBUG | pool-5-thread-29 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 11:19:53,842 | test | DEBUG | pool-5-thread-29 | [task:_compare:2649] Comparing 1392198 == 0 2021-06-08 11:19:53,842 | test | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 1392198 == 0. Received: {'172.23.106.156': 1392198} for bucket 'default' 2021-06-08 11:19:53,842 | infra | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 11:19:53,844 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 11:19:53,844 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 11:19:58,845 | infra | DEBUG | pool-5-thread-29 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 11:20:00,457 | test | DEBUG | pool-5-thread-29 | [task:_compare:2649] Comparing 1318926 == 0 2021-06-08 11:20:00,457 | test | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 1318926 == 0. Received: {'172.23.106.156': 1318926} for bucket 'default' 2021-06-08 11:20:00,457 | infra | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 11:20:00,460 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 11:20:00,460 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 11:20:05,460 | infra | DEBUG | pool-5-thread-29 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 11:20:07,040 | test | DEBUG | pool-5-thread-29 | [task:_compare:2649] Comparing 1271889 == 0 2021-06-08 11:20:07,042 | test | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 1271889 == 0. Received: {'172.23.106.156': 1271889} for bucket 'default' 2021-06-08 11:20:07,042 | infra | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 11:20:07,043 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 11:20:07,043 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 11:20:12,043 | infra | DEBUG | pool-5-thread-29 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 11:20:16,411 | test | DEBUG | pool-5-thread-29 | [task:_compare:2649] Comparing 1162534 == 0 2021-06-08 11:20:16,411 | test | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 1162534 == 0. Received: {'172.23.106.156': 1162534} for bucket 'default' 2021-06-08 11:20:16,411 | infra | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 11:20:16,414 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 11:20:16,414 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 11:20:21,414 | infra | DEBUG | pool-5-thread-29 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 11:20:24,357 | test | DEBUG | pool-5-thread-29 | [task:_compare:2649] Comparing 1117723 == 0 2021-06-08 11:20:24,357 | test | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 1117723 == 0. Received: {'172.23.106.156': 1117723} for bucket 'default' 2021-06-08 11:20:24,357 | infra | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 11:20:24,358 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 11:20:24,358 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 11:20:29,358 | infra | DEBUG | pool-5-thread-29 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 11:20:30,874 | test | DEBUG | pool-5-thread-29 | [task:_compare:2649] Comparing 1064145 == 0 2021-06-08 11:20:30,874 | test | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 1064145 == 0. Received: {'172.23.106.156': 1064145} for bucket 'default' 2021-06-08 11:20:30,874 | infra | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 11:20:30,875 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 11:20:30,875 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 11:20:35,877 | infra | DEBUG | pool-5-thread-29 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 11:20:37,380 | test | DEBUG | pool-5-thread-29 | [task:_compare:2649] Comparing 1016124 == 0 2021-06-08 11:20:37,380 | test | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 1016124 == 0. Received: {'172.23.106.156': 1016124} for bucket 'default' 2021-06-08 11:20:37,380 | infra | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 11:20:37,381 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 11:20:37,381 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 11:20:42,382 | infra | DEBUG | pool-5-thread-29 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 11:20:51,885 | test | DEBUG | pool-5-thread-29 | [task:_compare:2649] Comparing 903024 == 0 2021-06-08 11:20:51,885 | test | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 903024 == 0. Received: {'172.23.106.156': 903024} for bucket 'default' 2021-06-08 11:20:51,885 | infra | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 11:20:51,888 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 11:20:51,888 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 11:20:56,888 | infra | DEBUG | pool-5-thread-29 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 11:20:58,368 | test | DEBUG | pool-5-thread-29 | [task:_compare:2649] Comparing 871756 == 0 2021-06-08 11:20:58,369 | test | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 871756 == 0. Received: {'172.23.106.156': 871756} for bucket 'default' 2021-06-08 11:20:58,369 | infra | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 11:20:58,371 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 11:20:58,371 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 11:21:03,371 | infra | DEBUG | pool-5-thread-29 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 11:21:13,085 | test | DEBUG | pool-5-thread-29 | [task:_compare:2649] Comparing 726546 == 0 2021-06-08 11:21:13,085 | test | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 726546 == 0. Received: {'172.23.106.156': 726546} for bucket 'default' 2021-06-08 11:21:13,085 | infra | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 11:21:13,086 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 11:21:13,086 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 11:21:18,088 | infra | DEBUG | pool-5-thread-29 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 11:21:19,539 | test | DEBUG | pool-5-thread-29 | [task:_compare:2649] Comparing 717343 == 0 2021-06-08 11:21:19,539 | test | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 717343 == 0. Received: {'172.23.106.156': 717343} for bucket 'default' 2021-06-08 11:21:19,539 | infra | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 11:21:19,542 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 11:21:19,542 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 11:21:24,542 | infra | DEBUG | pool-5-thread-29 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 11:21:26,413 | test | DEBUG | pool-5-thread-29 | [task:_compare:2649] Comparing 681077 == 0 2021-06-08 11:21:26,413 | test | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 681077 == 0. Received: {'172.23.106.156': 681077} for bucket 'default' 2021-06-08 11:21:26,413 | infra | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 11:21:26,414 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 11:21:26,414 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 11:21:31,414 | infra | DEBUG | pool-5-thread-29 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 11:21:32,869 | test | DEBUG | pool-5-thread-29 | [task:_compare:2649] Comparing 627688 == 0 2021-06-08 11:21:32,871 | test | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 627688 == 0. Received: {'172.23.106.156': 627688} for bucket 'default' 2021-06-08 11:21:32,871 | infra | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 11:21:32,872 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 11:21:32,872 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 11:21:37,872 | infra | DEBUG | pool-5-thread-29 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 11:21:39,301 | test | DEBUG | pool-5-thread-29 | [task:_compare:2649] Comparing 585200 == 0 2021-06-08 11:21:39,301 | test | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 585200 == 0. Received: {'172.23.106.156': 585200} for bucket 'default' 2021-06-08 11:21:39,301 | infra | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 11:21:39,302 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 11:21:39,302 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 11:21:44,303 | infra | DEBUG | pool-5-thread-29 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 11:21:45,750 | test | DEBUG | pool-5-thread-29 | [task:_compare:2649] Comparing 567414 == 0 2021-06-08 11:21:45,750 | test | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 567414 == 0. Received: {'172.23.106.156': 567414} for bucket 'default' 2021-06-08 11:21:45,750 | infra | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 11:21:45,752 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 11:21:45,752 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 11:21:50,753 | infra | DEBUG | pool-5-thread-29 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 11:21:55,016 | test | DEBUG | pool-5-thread-29 | [task:_compare:2649] Comparing 505966 == 0 2021-06-08 11:21:55,016 | test | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 505966 == 0. Received: {'172.23.106.156': 505966} for bucket 'default' 2021-06-08 11:21:55,016 | infra | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 11:21:55,017 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 11:21:55,019 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 11:22:00,019 | infra | DEBUG | pool-5-thread-29 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 11:22:01,433 | test | DEBUG | pool-5-thread-29 | [task:_compare:2649] Comparing 471789 == 0 2021-06-08 11:22:01,434 | test | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 471789 == 0. Received: {'172.23.106.156': 471789} for bucket 'default' 2021-06-08 11:22:01,434 | infra | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 11:22:01,434 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 11:22:01,436 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 11:22:06,436 | infra | DEBUG | pool-5-thread-29 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 11:22:07,861 | test | DEBUG | pool-5-thread-29 | [task:_compare:2649] Comparing 446327 == 0 2021-06-08 11:22:07,861 | test | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 446327 == 0. Received: {'172.23.106.156': 446327} for bucket 'default' 2021-06-08 11:22:07,861 | infra | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 11:22:07,864 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 11:22:07,865 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 11:22:12,865 | infra | DEBUG | pool-5-thread-29 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 11:22:15,134 | test | DEBUG | pool-5-thread-29 | [task:_compare:2649] Comparing 414742 == 0 2021-06-08 11:22:15,135 | test | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 414742 == 0. Received: {'172.23.106.156': 414742} for bucket 'default' 2021-06-08 11:22:15,135 | infra | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 11:22:15,138 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 11:22:15,138 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 11:22:20,138 | infra | DEBUG | pool-5-thread-29 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 11:22:21,520 | test | DEBUG | pool-5-thread-29 | [task:_compare:2649] Comparing 387113 == 0 2021-06-08 11:22:21,520 | test | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 387113 == 0. Received: {'172.23.106.156': 387113} for bucket 'default' 2021-06-08 11:22:21,520 | infra | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 11:22:21,523 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 11:22:21,523 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 11:22:26,523 | infra | DEBUG | pool-5-thread-29 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 11:22:31,895 | test | DEBUG | pool-5-thread-29 | [task:_compare:2649] Comparing 339636 == 0 2021-06-08 11:22:31,895 | test | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 339636 == 0. Received: {'172.23.106.156': 339636} for bucket 'default' 2021-06-08 11:22:31,895 | infra | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 11:22:31,898 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 11:22:31,898 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 11:22:36,898 | infra | DEBUG | pool-5-thread-29 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 11:22:38,246 | test | DEBUG | pool-5-thread-29 | [task:_compare:2649] Comparing 316678 == 0 2021-06-08 11:22:38,246 | test | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 316678 == 0. Received: {'172.23.106.156': 316678} for bucket 'default' 2021-06-08 11:22:38,246 | infra | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 11:22:38,249 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 11:22:38,249 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 11:22:43,249 | infra | DEBUG | pool-5-thread-29 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 11:22:46,165 | test | DEBUG | pool-5-thread-29 | [task:_compare:2649] Comparing 288222 == 0 2021-06-08 11:22:46,167 | test | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 288222 == 0. Received: {'172.23.106.156': 288222} for bucket 'default' 2021-06-08 11:22:46,167 | infra | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 11:22:46,168 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 11:22:46,168 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 11:22:51,168 | infra | DEBUG | pool-5-thread-29 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 11:22:56,414 | test | DEBUG | pool-5-thread-29 | [task:_compare:2649] Comparing 245230 == 0 2021-06-08 11:22:56,414 | test | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 245230 == 0. Received: {'172.23.106.156': 245230} for bucket 'default' 2021-06-08 11:22:56,414 | infra | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 11:22:56,417 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 11:22:56,417 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 11:23:01,417 | infra | DEBUG | pool-5-thread-29 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 11:23:02,730 | test | DEBUG | pool-5-thread-29 | [task:_compare:2649] Comparing 224602 == 0 2021-06-08 11:23:02,730 | test | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 224602 == 0. Received: {'172.23.106.156': 224602} for bucket 'default' 2021-06-08 11:23:02,730 | infra | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 11:23:02,732 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 11:23:02,732 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 11:23:07,733 | infra | DEBUG | pool-5-thread-29 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 11:23:09,078 | test | DEBUG | pool-5-thread-29 | [task:_compare:2649] Comparing 149273 == 0 2021-06-08 11:23:09,078 | test | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 149273 == 0. Received: {'172.23.106.156': 149273} for bucket 'default' 2021-06-08 11:23:09,078 | infra | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 11:23:09,081 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 11:23:09,081 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 11:23:14,081 | infra | DEBUG | pool-5-thread-29 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 11:23:15,388 | test | DEBUG | pool-5-thread-29 | [task:_compare:2649] Comparing 83248 == 0 2021-06-08 11:23:15,388 | test | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 83248 == 0. Received: {'172.23.106.156': 83248} for bucket 'default' 2021-06-08 11:23:15,388 | infra | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 11:23:15,390 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 11:23:15,390 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 11:23:20,390 | infra | DEBUG | pool-5-thread-29 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 11:23:21,595 | test | DEBUG | pool-5-thread-29 | [task:_compare:2649] Comparing 25475 == 0 2021-06-08 11:23:21,595 | test | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 25475 == 0. Received: {'172.23.106.156': 25475} for bucket 'default' 2021-06-08 11:23:21,595 | infra | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 11:23:21,596 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 11:23:21,598 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 11:23:26,598 | infra | DEBUG | pool-5-thread-29 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 11:23:27,733 | test | DEBUG | pool-5-thread-29 | [task:_compare:2649] Comparing 7509 == 0 2021-06-08 11:23:27,733 | test | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2631] Not Ready: num_items_for_persistence 7509 == 0. Received: {'172.23.106.156': 7509} for bucket 'default' 2021-06-08 11:23:27,733 | infra | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2635] Wait before next StatWaitTask check 2021-06-08 11:23:27,736 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_checkpoint_stats_and_compare():L2636 2021-06-08 11:23:27,736 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 11:23:32,736 | infra | DEBUG | pool-5-thread-29 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default checkpoint 2021-06-08 11:23:33,881 | test | DEBUG | pool-5-thread-29 | [task:_compare:2649] Comparing 0 == 0 2021-06-08 11:23:33,881 | test | DEBUG | pool-5-thread-29 | [task:_get_checkpoint_stats_and_compare:2638] Ready: num_items_for_persistence 0 == 0. Received: {'172.23.106.156': 0} for bucket 'default' 2021-06-08 11:23:33,881 | infra | INFO | pool-5-thread-29 | [task:complete_task:102] Thread 'StatsWaitTask_default_num_items_for_persistence_1623176283.26' completed 2021-06-08 11:23:33,881 | infra | DEBUG | MainThread | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.156 2021-06-08 11:23:33,882 | infra | DEBUG | MainThread | [task_manager:get_task_result:26] Getting task result for StatsWaitTask_default_num_items_for_persistence_1623176283.72 2021-06-08 11:23:33,882 | infra | DEBUG | MainThread | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.125 2021-06-08 11:23:33,884 | infra | DEBUG | MainThread | [task_manager:get_task_result:26] Getting task result for StatsWaitTask_default_num_items_for_persistence_1623176284.18 2021-06-08 11:23:33,884 | infra | DEBUG | MainThread | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.122 2021-06-08 11:23:33,884 | infra | DEBUG | MainThread | [task_manager:get_task_result:26] Getting task result for StatsWaitTask_default_num_items_for_persistence_1623176284.64 2021-06-08 11:23:33,884 | infra | DEBUG | MainThread | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.155 2021-06-08 11:23:33,930 | test | DEBUG | MainThread | [bucket_ready_functions:parse_get_bucket_json:3739] Stats: {u'opsPerSec': 0, u'diskFetches': 908, u'dataUsed': 30979452928L, u'itemCount': 5000000, u'quotaPercentUsed': 74.43808220960423, u'storageTotals': {u'hdd': {u'usedByData': 36743955279L, u'used': 39601298840L, u'quotaTotal': 386354135040L, u'total': 386354135040L, u'free': 332264556136L}, u'ram': {u'usedByData': 26070031488L, u'used': 39681593344L, u'quotaTotal': 35026632704L, u'total': 49176707072L, u'quotaUsed': 35022438400L, u'quotaUsedPerNode': 8755609600L, u'quotaTotalPerNode': 8756658176L}}, u'diskUsed': 36743955279L, u'vbActiveNumNonResident': 0, u'memUsed': 26070031488L} 2021-06-08 11:23:33,931 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.156 with username: root, password: couchbase 2021-06-08 11:23:34,030 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 11:23:34,243 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: ls /etc/ 2021-06-08 11:23:34,299 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 11:23:34,299 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: ls /cygdrive/c/Windows 2021-06-08 11:23:34,312 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: uname -m 2021-06-08 11:23:34,323 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: cat /proc/cpuinfo 2021-06-08 11:23:34,335 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: df -Thl 2021-06-08 11:23:34,352 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: cat /proc/meminfo 2021-06-08 11:23:34,364 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: hostname 2021-06-08 11:23:34,375 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: hostname -d 2021-06-08 11:23:34,388 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: du -cm /data/default/magma.*/kv* | tail -1 | awk '{print $1}' 2021-06-08 11:23:34,596 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: du -cm /data/default/magma.*/wal | tail -1 | awk '{print $1}' 2021-06-08 11:23:34,611 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: du -cm /data/default/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2021-06-08 11:23:34,703 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: du -cm /data/default/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2021-06-08 11:23:34,796 | infra | DEBUG | MainThread | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.156 2021-06-08 11:23:34,799 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.125 with username: root, password: couchbase 2021-06-08 11:23:34,875 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 11:23:35,085 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: ls /etc/ 2021-06-08 11:23:35,141 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 11:23:35,141 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: ls /cygdrive/c/Windows 2021-06-08 11:23:35,154 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: uname -m 2021-06-08 11:23:35,164 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: cat /proc/cpuinfo 2021-06-08 11:23:35,177 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: df -Thl 2021-06-08 11:23:35,190 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: cat /proc/meminfo 2021-06-08 11:23:35,203 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: hostname 2021-06-08 11:23:35,214 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: hostname -d 2021-06-08 11:23:35,226 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: du -cm /data/default/magma.*/kv* | tail -1 | awk '{print $1}' 2021-06-08 11:23:35,444 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: du -cm /data/default/magma.*/wal | tail -1 | awk '{print $1}' 2021-06-08 11:23:35,459 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: du -cm /data/default/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2021-06-08 11:23:35,555 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: du -cm /data/default/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2021-06-08 11:23:35,651 | infra | DEBUG | MainThread | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.125 2021-06-08 11:23:35,654 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.122 with username: root, password: couchbase 2021-06-08 11:23:35,732 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 11:23:35,944 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: ls /etc/ 2021-06-08 11:23:36,003 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 11:23:36,003 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: ls /cygdrive/c/Windows 2021-06-08 11:23:36,016 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: uname -m 2021-06-08 11:23:36,028 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: cat /proc/cpuinfo 2021-06-08 11:23:36,039 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: df -Thl 2021-06-08 11:23:36,055 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: cat /proc/meminfo 2021-06-08 11:23:36,066 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: hostname 2021-06-08 11:23:36,078 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: hostname -d 2021-06-08 11:23:36,092 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: du -cm /data/default/magma.*/kv* | tail -1 | awk '{print $1}' 2021-06-08 11:23:36,322 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: du -cm /data/default/magma.*/wal | tail -1 | awk '{print $1}' 2021-06-08 11:23:36,336 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: du -cm /data/default/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2021-06-08 11:23:36,437 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: du -cm /data/default/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2021-06-08 11:23:36,533 | infra | DEBUG | MainThread | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.122 2021-06-08 11:23:36,536 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.155 with username: root, password: couchbase 2021-06-08 11:23:36,615 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 11:23:36,823 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: ls /etc/ 2021-06-08 11:23:36,878 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 11:23:36,878 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: ls /cygdrive/c/Windows 2021-06-08 11:23:36,891 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: uname -m 2021-06-08 11:23:36,903 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: cat /proc/cpuinfo 2021-06-08 11:23:36,914 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: df -Thl 2021-06-08 11:23:36,930 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: cat /proc/meminfo 2021-06-08 11:23:36,941 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: hostname 2021-06-08 11:23:36,953 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: hostname -d 2021-06-08 11:23:36,967 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: du -cm /data/default/magma.*/kv* | tail -1 | awk '{print $1}' 2021-06-08 11:23:37,193 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: du -cm /data/default/magma.*/wal | tail -1 | awk '{print $1}' 2021-06-08 11:23:37,207 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: du -cm /data/default/magma.*/kv*/rev*/key* | tail -1 | awk '{print $1}' 2021-06-08 11:23:37,305 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: du -cm /data/default/magma.*/kv*/rev*/seq* | tail -1 | awk '{print $1}' 2021-06-08 11:23:37,398 | infra | DEBUG | MainThread | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.155 2021-06-08 11:23:37,400 | test | INFO | MainThread | [magma_base:get_disk_usage:815] Disk usage stats for bucekt default is below 2021-06-08 11:23:37,400 | test | INFO | MainThread | [magma_base:get_disk_usage:816] Total Disk usage for kvstore is 30026MB 2021-06-08 11:23:37,641 | test | INFO | MainThread | [magma_base:get_bucket_dgm:767] Active Resident Threshold of default is 100 2021-06-08 11:23:37,641 | test | DEBUG | MainThread | [magma_base:get_disk_usage:818] Total Disk usage for wal is 5801MB 2021-06-08 11:23:37,642 | test | DEBUG | MainThread | [magma_base:get_disk_usage:819] Total Disk usage for keyTree is 746MB 2021-06-08 11:23:37,642 | test | DEBUG | MainThread | [magma_base:get_disk_usage:820] Total Disk usage for seqTree is 29072MB 2021-06-08 11:23:37,642 | test | INFO | MainThread | [magma_base:initial_load:359] For bucket default disk usage after initial creation is 30026MB 2021-06-08 11:23:37,642 | test | INFO | MainThread | [magma_base:setUp:261] ==========Finished magma base setup======== 2021-06-08 11:23:37,645 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.156 with username: root, password: couchbase 2021-06-08 11:23:37,729 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 11:23:37,934 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: ls /etc/ 2021-06-08 11:23:37,993 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 11:23:37,993 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: ls /cygdrive/c/Windows 2021-06-08 11:23:38,006 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: uname -m 2021-06-08 11:23:38,017 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: cat /proc/cpuinfo 2021-06-08 11:23:38,029 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: df -Thl 2021-06-08 11:23:38,046 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: cat /proc/meminfo 2021-06-08 11:23:38,059 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: hostname 2021-06-08 11:23:38,071 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: hostname -d 2021-06-08 11:23:38,085 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: rm -rf /data/full_disk_* 2021-06-08 11:23:38,096 | infra | DEBUG | MainThread | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.156 2021-06-08 11:23:38,102 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/magma/magma_disk_full.py -> free_disk():L109 2021-06-08 11:23:38,102 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: Wait for files to clean up from the disk 2021-06-08 11:23:48,104 | test | DEBUG | MainThread | [rest_client:update_autofailover_settings:1942] settings/autoFailover params: failoverServerGroup=false&failoverOnDataDiskIssues%5Benabled%5D=false&maxCount=1&timeout=600&enabled=false&canAbortRebalance=false 2021-06-08 11:23:48,124 | infra | DEBUG | Thread-17 | [remote_util:connect:206] Connecting to 172.23.106.156 with username: root, password: couchbase 2021-06-08 11:23:48,125 | infra | DEBUG | Thread-18 | [remote_util:connect:206] Connecting to 172.23.106.125 with username: root, password: couchbase 2021-06-08 11:23:48,125 | infra | DEBUG | Thread-20 | [remote_util:connect:206] Connecting to 172.23.106.155 with username: root, password: couchbase 2021-06-08 11:23:48,125 | infra | DEBUG | Thread-19 | [remote_util:connect:206] Connecting to 172.23.106.122 with username: root, password: couchbase 2021-06-08 11:23:48,198 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 11:23:48,203 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 11:23:48,203 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 11:23:48,203 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 11:23:48,404 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: ls /etc/ 2021-06-08 11:23:48,405 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: ls /etc/ 2021-06-08 11:23:48,411 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: ls /etc/ 2021-06-08 11:23:48,411 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: ls /etc/ 2021-06-08 11:23:48,460 | infra | DEBUG | Thread-17 | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 11:23:48,460 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: ls /cygdrive/c/Windows 2021-06-08 11:23:48,463 | infra | DEBUG | Thread-19 | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 11:23:48,463 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: ls /cygdrive/c/Windows 2021-06-08 11:23:48,467 | infra | DEBUG | Thread-18 | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 11:23:48,467 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: ls /cygdrive/c/Windows 2021-06-08 11:23:48,467 | infra | DEBUG | Thread-20 | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 11:23:48,467 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: ls /cygdrive/c/Windows 2021-06-08 11:23:48,473 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: uname -m 2021-06-08 11:23:48,476 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: uname -m 2021-06-08 11:23:48,480 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: uname -m 2021-06-08 11:23:48,482 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: uname -m 2021-06-08 11:23:48,483 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: cat /proc/cpuinfo 2021-06-08 11:23:48,489 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: cat /proc/cpuinfo 2021-06-08 11:23:48,492 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: cat /proc/cpuinfo 2021-06-08 11:23:48,493 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: cat /proc/cpuinfo 2021-06-08 11:23:48,494 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: df -Thl 2021-06-08 11:23:48,500 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: df -Thl 2021-06-08 11:23:48,503 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: df -Thl 2021-06-08 11:23:48,506 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: df -Thl 2021-06-08 11:23:48,510 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: cat /proc/meminfo 2021-06-08 11:23:48,516 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: cat /proc/meminfo 2021-06-08 11:23:48,519 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: cat /proc/meminfo 2021-06-08 11:23:48,522 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: cat /proc/meminfo 2021-06-08 11:23:48,523 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: hostname 2021-06-08 11:23:48,529 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: hostname 2021-06-08 11:23:48,532 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: hostname 2021-06-08 11:23:48,535 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: hostname 2021-06-08 11:23:48,535 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: hostname -d 2021-06-08 11:23:48,542 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: hostname -d 2021-06-08 11:23:48,543 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: hostname -d 2021-06-08 11:23:48,546 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: hostname -d 2021-06-08 11:23:48,548 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: df -lBM /data 2021-06-08 11:23:48,555 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: df -lBM /data 2021-06-08 11:23:48,556 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: df -lBM /data 2021-06-08 11:23:48,559 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: df -lBM /data 2021-06-08 11:23:48,562 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_79481MB_1623176628.56 bs=1024M count=1 2021-06-08 11:23:48,562 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: dd if=/dev/zero of=/data/full_disk_79481MB_1623176628.56 bs=1024M count=1 2021-06-08 11:23:48,568 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_84040MB_1623176628.57 bs=1024M count=1 2021-06-08 11:23:48,568 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: dd if=/dev/zero of=/data/full_disk_84040MB_1623176628.57 bs=1024M count=1 2021-06-08 11:23:48,569 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_84031MB_1623176628.57 bs=1024M count=1 2021-06-08 11:23:48,569 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: dd if=/dev/zero of=/data/full_disk_84031MB_1623176628.57 bs=1024M count=1 2021-06-08 11:23:48,572 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_84035MB_1623176628.57 bs=1024M count=1 2021-06-08 11:23:48,572 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: dd if=/dev/zero of=/data/full_disk_84035MB_1623176628.57 bs=1024M count=1 2021-06-08 11:23:55,953 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 7.26226 s, 148 MB/s 2021-06-08 11:23:55,954 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_83007MB_1623176635.95 bs=1024M count=1 2021-06-08 11:23:55,954 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: dd if=/dev/zero of=/data/full_disk_83007MB_1623176635.95 bs=1024M count=1 2021-06-08 11:23:56,946 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 8.24328 s, 130 MB/s 2021-06-08 11:23:56,947 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_83016MB_1623176636.95 bs=1024M count=1 2021-06-08 11:23:56,947 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: dd if=/dev/zero of=/data/full_disk_83016MB_1623176636.95 bs=1024M count=1 2021-06-08 11:23:57,644 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 8.92735 s, 120 MB/s 2021-06-08 11:23:57,644 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_83011MB_1623176637.64 bs=1024M count=1 2021-06-08 11:23:57,644 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: dd if=/dev/zero of=/data/full_disk_83011MB_1623176637.64 bs=1024M count=1 2021-06-08 11:23:57,826 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 9.15093 s, 117 MB/s 2021-06-08 11:23:57,826 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_78457MB_1623176637.83 bs=1024M count=1 2021-06-08 11:23:57,828 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: dd if=/dev/zero of=/data/full_disk_78457MB_1623176637.83 bs=1024M count=1 2021-06-08 11:24:15,809 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 19.7145 s, 54.5 MB/s 2021-06-08 11:24:15,811 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_81983MB_1623176655.81 bs=1024M count=1 2021-06-08 11:24:15,811 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: dd if=/dev/zero of=/data/full_disk_81983MB_1623176655.81 bs=1024M count=1 2021-06-08 11:24:16,914 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 19.8398 s, 54.1 MB/s 2021-06-08 11:24:16,915 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_81992MB_1623176656.91 bs=1024M count=1 2021-06-08 11:24:16,915 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: dd if=/dev/zero of=/data/full_disk_81992MB_1623176656.91 bs=1024M count=1 2021-06-08 11:24:17,375 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 19.5847 s, 54.8 MB/s 2021-06-08 11:24:17,375 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_81987MB_1623176657.38 bs=1024M count=1 2021-06-08 11:24:17,375 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: dd if=/dev/zero of=/data/full_disk_81987MB_1623176657.38 bs=1024M count=1 2021-06-08 11:24:20,322 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 22.3817 s, 48.0 MB/s 2021-06-08 11:24:20,323 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_77433MB_1623176660.32 bs=1024M count=1 2021-06-08 11:24:20,323 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: dd if=/dev/zero of=/data/full_disk_77433MB_1623176660.32 bs=1024M count=1 2021-06-08 11:24:29,944 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 9.50723 s, 113 MB/s 2021-06-08 11:24:29,944 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_76409MB_1623176669.94 bs=1024M count=1 2021-06-08 11:24:29,946 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: dd if=/dev/zero of=/data/full_disk_76409MB_1623176669.94 bs=1024M count=1 2021-06-08 11:24:34,427 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 4.37063 s, 246 MB/s 2021-06-08 11:24:34,427 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_75385MB_1623176674.43 bs=1024M count=1 2021-06-08 11:24:34,428 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: dd if=/dev/zero of=/data/full_disk_75385MB_1623176674.43 bs=1024M count=1 2021-06-08 11:24:34,631 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 18.6896 s, 57.5 MB/s 2021-06-08 11:24:34,631 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_80959MB_1623176674.63 bs=1024M count=1 2021-06-08 11:24:34,631 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: dd if=/dev/zero of=/data/full_disk_80959MB_1623176674.63 bs=1024M count=1 2021-06-08 11:24:35,573 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 18.5355 s, 57.9 MB/s 2021-06-08 11:24:35,575 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_80968MB_1623176675.58 bs=1024M count=1 2021-06-08 11:24:35,575 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: dd if=/dev/zero of=/data/full_disk_80968MB_1623176675.58 bs=1024M count=1 2021-06-08 11:24:35,819 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 1.28527 s, 835 MB/s 2021-06-08 11:24:35,819 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_74361MB_1623176675.82 bs=1024M count=1 2021-06-08 11:24:35,819 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: dd if=/dev/zero of=/data/full_disk_74361MB_1623176675.82 bs=1024M count=1 2021-06-08 11:24:36,038 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 18.5294 s, 57.9 MB/s 2021-06-08 11:24:36,038 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_80963MB_1623176676.04 bs=1024M count=1 2021-06-08 11:24:36,038 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: dd if=/dev/zero of=/data/full_disk_80963MB_1623176676.04 bs=1024M count=1 2021-06-08 11:24:37,325 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 1.40001 s, 767 MB/s 2021-06-08 11:24:37,325 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_73337MB_1623176677.33 bs=1024M count=1 2021-06-08 11:24:37,325 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: dd if=/dev/zero of=/data/full_disk_73337MB_1623176677.33 bs=1024M count=1 2021-06-08 11:24:40,844 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 3.39908 s, 316 MB/s 2021-06-08 11:24:40,844 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_72313MB_1623176680.84 bs=1024M count=1 2021-06-08 11:24:40,844 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: dd if=/dev/zero of=/data/full_disk_72313MB_1623176680.84 bs=1024M count=1 2021-06-08 11:24:46,769 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 5.81372 s, 185 MB/s 2021-06-08 11:24:46,769 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_71289MB_1623176686.77 bs=1024M count=1 2021-06-08 11:24:46,769 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: dd if=/dev/zero of=/data/full_disk_71289MB_1623176686.77 bs=1024M count=1 2021-06-08 11:24:50,575 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 15.8056 s, 67.9 MB/s 2021-06-08 11:24:50,576 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_79935MB_1623176690.58 bs=1024M count=1 2021-06-08 11:24:50,576 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: dd if=/dev/zero of=/data/full_disk_79935MB_1623176690.58 bs=1024M count=1 2021-06-08 11:24:50,971 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 15.2521 s, 70.4 MB/s 2021-06-08 11:24:50,971 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_79944MB_1623176690.97 bs=1024M count=1 2021-06-08 11:24:50,971 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: dd if=/dev/zero of=/data/full_disk_79944MB_1623176690.97 bs=1024M count=1 2021-06-08 11:24:51,003 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 14.8358 s, 72.4 MB/s 2021-06-08 11:24:51,003 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_79939MB_1623176691.0 bs=1024M count=1 2021-06-08 11:24:51,003 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: dd if=/dev/zero of=/data/full_disk_79939MB_1623176691.0 bs=1024M count=1 2021-06-08 11:24:51,141 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 4.26978 s, 251 MB/s 2021-06-08 11:24:51,141 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_70265MB_1623176691.14 bs=1024M count=1 2021-06-08 11:24:51,141 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: dd if=/dev/zero of=/data/full_disk_70265MB_1623176691.14 bs=1024M count=1 2021-06-08 11:24:54,621 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 3.36419 s, 319 MB/s 2021-06-08 11:24:54,621 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_69241MB_1623176694.62 bs=1024M count=1 2021-06-08 11:24:54,622 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: dd if=/dev/zero of=/data/full_disk_69241MB_1623176694.62 bs=1024M count=1 2021-06-08 11:24:58,457 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 3.72262 s, 288 MB/s 2021-06-08 11:24:58,457 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_68217MB_1623176698.46 bs=1024M count=1 2021-06-08 11:24:58,457 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: dd if=/dev/zero of=/data/full_disk_68217MB_1623176698.46 bs=1024M count=1 2021-06-08 11:25:02,286 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 11.5758 s, 92.8 MB/s 2021-06-08 11:25:02,286 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_78911MB_1623176702.29 bs=1024M count=1 2021-06-08 11:25:02,286 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: dd if=/dev/zero of=/data/full_disk_78911MB_1623176702.29 bs=1024M count=1 2021-06-08 11:25:02,750 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 11.6196 s, 92.4 MB/s 2021-06-08 11:25:02,750 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_78915MB_1623176702.75 bs=1024M count=1 2021-06-08 11:25:02,750 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: dd if=/dev/zero of=/data/full_disk_78915MB_1623176702.75 bs=1024M count=1 2021-06-08 11:25:05,667 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 7.10373 s, 151 MB/s 2021-06-08 11:25:05,667 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_67193MB_1623176705.67 bs=1024M count=1 2021-06-08 11:25:05,667 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: dd if=/dev/zero of=/data/full_disk_67193MB_1623176705.67 bs=1024M count=1 2021-06-08 11:25:08,450 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 2.67725 s, 401 MB/s 2021-06-08 11:25:08,451 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_66169MB_1623176708.45 bs=1024M count=1 2021-06-08 11:25:08,451 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: dd if=/dev/zero of=/data/full_disk_66169MB_1623176708.45 bs=1024M count=1 2021-06-08 11:25:09,207 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 6.78229 s, 158 MB/s 2021-06-08 11:25:09,207 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_77887MB_1623176709.21 bs=1024M count=1 2021-06-08 11:25:09,207 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: dd if=/dev/zero of=/data/full_disk_77887MB_1623176709.21 bs=1024M count=1 2021-06-08 11:25:10,598 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 1.25458 s, 856 MB/s 2021-06-08 11:25:10,598 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_76863MB_1623176710.6 bs=1024M count=1 2021-06-08 11:25:10,598 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: dd if=/dev/zero of=/data/full_disk_76863MB_1623176710.6 bs=1024M count=1 2021-06-08 11:25:11,743 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 20.6222 s, 52.1 MB/s 2021-06-08 11:25:11,743 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_78920MB_1623176711.74 bs=1024M count=1 2021-06-08 11:25:11,743 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: dd if=/dev/zero of=/data/full_disk_78920MB_1623176711.74 bs=1024M count=1 2021-06-08 11:25:12,204 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 1.47614 s, 727 MB/s 2021-06-08 11:25:12,204 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_75839MB_1623176712.2 bs=1024M count=1 2021-06-08 11:25:12,204 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: dd if=/dev/zero of=/data/full_disk_75839MB_1623176712.2 bs=1024M count=1 2021-06-08 11:25:12,430 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 3.85771 s, 278 MB/s 2021-06-08 11:25:12,430 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_65145MB_1623176712.43 bs=1024M count=1 2021-06-08 11:25:12,430 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: dd if=/dev/zero of=/data/full_disk_65145MB_1623176712.43 bs=1024M count=1 2021-06-08 11:25:13,670 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 1.32552 s, 810 MB/s 2021-06-08 11:25:13,670 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_74815MB_1623176713.67 bs=1024M count=1 2021-06-08 11:25:13,670 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: dd if=/dev/zero of=/data/full_disk_74815MB_1623176713.67 bs=1024M count=1 2021-06-08 11:25:14,545 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 2.00566 s, 535 MB/s 2021-06-08 11:25:14,545 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_64121MB_1623176714.55 bs=1024M count=1 2021-06-08 11:25:14,545 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: dd if=/dev/zero of=/data/full_disk_64121MB_1623176714.55 bs=1024M count=1 2021-06-08 11:25:15,996 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 1.34477 s, 798 MB/s 2021-06-08 11:25:15,996 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_63097MB_1623176716.0 bs=1024M count=1 2021-06-08 11:25:15,996 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: dd if=/dev/zero of=/data/full_disk_63097MB_1623176716.0 bs=1024M count=1 2021-06-08 11:25:18,102 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 4.29844 s, 250 MB/s 2021-06-08 11:25:18,104 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_73791MB_1623176718.1 bs=1024M count=1 2021-06-08 11:25:18,104 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: dd if=/dev/zero of=/data/full_disk_73791MB_1623176718.1 bs=1024M count=1 2021-06-08 11:25:18,756 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 2.64057 s, 407 MB/s 2021-06-08 11:25:18,756 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_62073MB_1623176718.76 bs=1024M count=1 2021-06-08 11:25:18,756 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: dd if=/dev/zero of=/data/full_disk_62073MB_1623176718.76 bs=1024M count=1 2021-06-08 11:25:23,528 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 3.15651 s, 340 MB/s 2021-06-08 11:25:23,528 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_61049MB_1623176723.53 bs=1024M count=1 2021-06-08 11:25:23,529 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: dd if=/dev/zero of=/data/full_disk_61049MB_1623176723.53 bs=1024M count=1 2021-06-08 11:25:26,730 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 3.08815 s, 348 MB/s 2021-06-08 11:25:26,732 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_60025MB_1623176726.73 bs=1024M count=1 2021-06-08 11:25:26,732 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: dd if=/dev/zero of=/data/full_disk_60025MB_1623176726.73 bs=1024M count=1 2021-06-08 11:25:30,480 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 3.62951 s, 296 MB/s 2021-06-08 11:25:30,480 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_59001MB_1623176730.48 bs=1024M count=1 2021-06-08 11:25:30,480 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: dd if=/dev/zero of=/data/full_disk_59001MB_1623176730.48 bs=1024M count=1 2021-06-08 11:25:32,217 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 13.9801 s, 76.8 MB/s 2021-06-08 11:25:32,217 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_72767MB_1623176732.22 bs=1024M count=1 2021-06-08 11:25:32,219 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: dd if=/dev/zero of=/data/full_disk_72767MB_1623176732.22 bs=1024M count=1 2021-06-08 11:25:34,630 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 4.03004 s, 266 MB/s 2021-06-08 11:25:34,631 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_57977MB_1623176734.63 bs=1024M count=1 2021-06-08 11:25:34,631 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: dd if=/dev/zero of=/data/full_disk_57977MB_1623176734.63 bs=1024M count=1 2021-06-08 11:25:35,065 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 2.71161 s, 396 MB/s 2021-06-08 11:25:35,065 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_71743MB_1623176735.07 bs=1024M count=1 2021-06-08 11:25:35,065 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: dd if=/dev/zero of=/data/full_disk_71743MB_1623176735.07 bs=1024M count=1 2021-06-08 11:25:35,469 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 23.5944 s, 45.5 MB/s 2021-06-08 11:25:35,469 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_77896MB_1623176735.47 bs=1024M count=1 2021-06-08 11:25:35,470 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: dd if=/dev/zero of=/data/full_disk_77896MB_1623176735.47 bs=1024M count=1 2021-06-08 11:25:38,477 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 3.72463 s, 288 MB/s 2021-06-08 11:25:38,477 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_56953MB_1623176738.48 bs=1024M count=1 2021-06-08 11:25:38,477 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: dd if=/dev/zero of=/data/full_disk_56953MB_1623176738.48 bs=1024M count=1 2021-06-08 11:25:38,569 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 2.979 s, 360 MB/s 2021-06-08 11:25:38,569 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_76872MB_1623176738.57 bs=1024M count=1 2021-06-08 11:25:38,569 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: dd if=/dev/zero of=/data/full_disk_76872MB_1623176738.57 bs=1024M count=1 2021-06-08 11:25:40,017 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 35.4528 s, 30.3 MB/s 2021-06-08 11:25:40,019 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_77891MB_1623176740.02 bs=1024M count=1 2021-06-08 11:25:40,019 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: dd if=/dev/zero of=/data/full_disk_77891MB_1623176740.02 bs=1024M count=1 2021-06-08 11:25:41,507 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 2.91711 s, 368 MB/s 2021-06-08 11:25:41,507 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_55929MB_1623176741.51 bs=1024M count=1 2021-06-08 11:25:41,507 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: dd if=/dev/zero of=/data/full_disk_55929MB_1623176741.51 bs=1024M count=1 2021-06-08 11:25:44,632 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 2.9823 s, 360 MB/s 2021-06-08 11:25:44,634 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_54905MB_1623176744.63 bs=1024M count=1 2021-06-08 11:25:44,634 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: dd if=/dev/zero of=/data/full_disk_54905MB_1623176744.63 bs=1024M count=1 2021-06-08 11:25:47,693 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 2.94541 s, 365 MB/s 2021-06-08 11:25:47,694 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_53881MB_1623176747.69 bs=1024M count=1 2021-06-08 11:25:47,694 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: dd if=/dev/zero of=/data/full_disk_53881MB_1623176747.69 bs=1024M count=1 2021-06-08 11:25:49,266 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 14.0736 s, 76.3 MB/s 2021-06-08 11:25:49,266 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_70719MB_1623176749.27 bs=1024M count=1 2021-06-08 11:25:49,267 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: dd if=/dev/zero of=/data/full_disk_70719MB_1623176749.27 bs=1024M count=1 2021-06-08 11:25:51,240 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 1.85288 s, 579 MB/s 2021-06-08 11:25:51,242 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_69695MB_1623176751.24 bs=1024M count=1 2021-06-08 11:25:51,242 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: dd if=/dev/zero of=/data/full_disk_69695MB_1623176751.24 bs=1024M count=1 2021-06-08 11:25:51,250 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 3.07892 s, 349 MB/s 2021-06-08 11:25:51,250 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_52857MB_1623176751.25 bs=1024M count=1 2021-06-08 11:25:51,250 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: dd if=/dev/zero of=/data/full_disk_52857MB_1623176751.25 bs=1024M count=1 2021-06-08 11:25:54,220 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 2.84746 s, 377 MB/s 2021-06-08 11:25:54,220 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_51833MB_1623176754.22 bs=1024M count=1 2021-06-08 11:25:54,220 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: dd if=/dev/zero of=/data/full_disk_51833MB_1623176754.22 bs=1024M count=1 2021-06-08 11:25:57,164 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 18.4725 s, 58.1 MB/s 2021-06-08 11:25:57,164 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_75848MB_1623176757.16 bs=1024M count=1 2021-06-08 11:25:57,164 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: dd if=/dev/zero of=/data/full_disk_75848MB_1623176757.16 bs=1024M count=1 2021-06-08 11:25:57,506 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 2.78383 s, 386 MB/s 2021-06-08 11:25:57,506 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_50809MB_1623176757.51 bs=1024M count=1 2021-06-08 11:25:57,506 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: dd if=/dev/zero of=/data/full_disk_50809MB_1623176757.51 bs=1024M count=1 2021-06-08 11:25:58,973 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 1.35501 s, 792 MB/s 2021-06-08 11:25:58,974 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_49785MB_1623176758.97 bs=1024M count=1 2021-06-08 11:25:58,974 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: dd if=/dev/zero of=/data/full_disk_49785MB_1623176758.97 bs=1024M count=1 2021-06-08 11:26:00,561 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 1.45861 s, 736 MB/s 2021-06-08 11:26:00,561 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_48761MB_1623176760.56 bs=1024M count=1 2021-06-08 11:26:00,561 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: dd if=/dev/zero of=/data/full_disk_48761MB_1623176760.56 bs=1024M count=1 2021-06-08 11:26:02,023 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 1.33976 s, 801 MB/s 2021-06-08 11:26:02,023 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_47737MB_1623176762.02 bs=1024M count=1 2021-06-08 11:26:02,023 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: dd if=/dev/zero of=/data/full_disk_47737MB_1623176762.02 bs=1024M count=1 2021-06-08 11:26:03,515 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 1.3673 s, 785 MB/s 2021-06-08 11:26:03,516 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_46713MB_1623176763.52 bs=1024M count=1 2021-06-08 11:26:03,516 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: dd if=/dev/zero of=/data/full_disk_46713MB_1623176763.52 bs=1024M count=1 2021-06-08 11:26:05,848 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 1.72528 s, 622 MB/s 2021-06-08 11:26:05,848 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_45689MB_1623176765.85 bs=1024M count=1 2021-06-08 11:26:05,848 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: dd if=/dev/zero of=/data/full_disk_45689MB_1623176765.85 bs=1024M count=1 2021-06-08 11:26:08,516 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 17.1362 s, 62.7 MB/s 2021-06-08 11:26:08,516 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_68671MB_1623176768.52 bs=1024M count=1 2021-06-08 11:26:08,516 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: dd if=/dev/zero of=/data/full_disk_68671MB_1623176768.52 bs=1024M count=1 2021-06-08 11:26:11,026 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 13.7393 s, 78.2 MB/s 2021-06-08 11:26:11,026 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_74824MB_1623176771.03 bs=1024M count=1 2021-06-08 11:26:11,026 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: dd if=/dev/zero of=/data/full_disk_74824MB_1623176771.03 bs=1024M count=1 2021-06-08 11:26:12,651 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 1.48393 s, 724 MB/s 2021-06-08 11:26:12,651 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_73800MB_1623176772.65 bs=1024M count=1 2021-06-08 11:26:12,651 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: dd if=/dev/zero of=/data/full_disk_73800MB_1623176772.65 bs=1024M count=1 2021-06-08 11:26:23,335 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 17.175 s, 62.5 MB/s 2021-06-08 11:26:23,335 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_44665MB_1623176783.34 bs=1024M count=1 2021-06-08 11:26:23,336 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: dd if=/dev/zero of=/data/full_disk_44665MB_1623176783.34 bs=1024M count=1 2021-06-08 11:26:23,536 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 14.8931 s, 72.1 MB/s 2021-06-08 11:26:23,538 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_67647MB_1623176783.54 bs=1024M count=1 2021-06-08 11:26:23,538 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: dd if=/dev/zero of=/data/full_disk_67647MB_1623176783.54 bs=1024M count=1 2021-06-08 11:26:24,571 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 11.7915 s, 91.1 MB/s 2021-06-08 11:26:24,571 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_72776MB_1623176784.57 bs=1024M count=1 2021-06-08 11:26:24,571 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: dd if=/dev/zero of=/data/full_disk_72776MB_1623176784.57 bs=1024M count=1 2021-06-08 11:26:24,822 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 1.36101 s, 789 MB/s 2021-06-08 11:26:24,822 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_43641MB_1623176784.82 bs=1024M count=1 2021-06-08 11:26:24,822 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: dd if=/dev/zero of=/data/full_disk_43641MB_1623176784.82 bs=1024M count=1 2021-06-08 11:26:26,414 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 1.46781 s, 732 MB/s 2021-06-08 11:26:26,414 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_42617MB_1623176786.41 bs=1024M count=1 2021-06-08 11:26:26,414 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: dd if=/dev/zero of=/data/full_disk_42617MB_1623176786.41 bs=1024M count=1 2021-06-08 11:26:28,342 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 1.80178 s, 596 MB/s 2021-06-08 11:26:28,342 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_41593MB_1623176788.34 bs=1024M count=1 2021-06-08 11:26:28,342 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: dd if=/dev/zero of=/data/full_disk_41593MB_1623176788.34 bs=1024M count=1 2021-06-08 11:26:32,094 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 3.63501 s, 295 MB/s 2021-06-08 11:26:32,094 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_40569MB_1623176792.09 bs=1024M count=1 2021-06-08 11:26:32,094 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: dd if=/dev/zero of=/data/full_disk_40569MB_1623176792.09 bs=1024M count=1 2021-06-08 11:26:32,447 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 52.2963 s, 20.5 MB/s 2021-06-08 11:26:32,447 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_76867MB_1623176792.45 bs=1024M count=1 2021-06-08 11:26:32,447 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: dd if=/dev/zero of=/data/full_disk_76867MB_1623176792.45 bs=1024M count=1 2021-06-08 11:26:35,578 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 3.37336 s, 318 MB/s 2021-06-08 11:26:35,578 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_39545MB_1623176795.58 bs=1024M count=1 2021-06-08 11:26:35,578 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: dd if=/dev/zero of=/data/full_disk_39545MB_1623176795.58 bs=1024M count=1 2021-06-08 11:26:36,443 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 11.7277 s, 91.6 MB/s 2021-06-08 11:26:36,444 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_71752MB_1623176796.44 bs=1024M count=1 2021-06-08 11:26:36,444 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: dd if=/dev/zero of=/data/full_disk_71752MB_1623176796.44 bs=1024M count=1 2021-06-08 11:26:37,144 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 13.4639 s, 79.7 MB/s 2021-06-08 11:26:37,144 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_66623MB_1623176797.14 bs=1024M count=1 2021-06-08 11:26:37,144 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: dd if=/dev/zero of=/data/full_disk_66623MB_1623176797.14 bs=1024M count=1 2021-06-08 11:26:37,924 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 1.3486 s, 796 MB/s 2021-06-08 11:26:37,924 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_70728MB_1623176797.92 bs=1024M count=1 2021-06-08 11:26:37,926 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: dd if=/dev/zero of=/data/full_disk_70728MB_1623176797.92 bs=1024M count=1 2021-06-08 11:26:39,236 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 3.52666 s, 304 MB/s 2021-06-08 11:26:39,236 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_38521MB_1623176799.24 bs=1024M count=1 2021-06-08 11:26:39,236 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: dd if=/dev/zero of=/data/full_disk_38521MB_1623176799.24 bs=1024M count=1 2021-06-08 11:26:41,871 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 2.51703 s, 427 MB/s 2021-06-08 11:26:41,871 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_37497MB_1623176801.87 bs=1024M count=1 2021-06-08 11:26:41,871 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: dd if=/dev/zero of=/data/full_disk_37497MB_1623176801.87 bs=1024M count=1 2021-06-08 11:26:43,562 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 1.55061 s, 692 MB/s 2021-06-08 11:26:43,562 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_36473MB_1623176803.56 bs=1024M count=1 2021-06-08 11:26:43,562 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: dd if=/dev/zero of=/data/full_disk_36473MB_1623176803.56 bs=1024M count=1 2021-06-08 11:26:45,342 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 8.07317 s, 133 MB/s 2021-06-08 11:26:45,342 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_65599MB_1623176805.34 bs=1024M count=1 2021-06-08 11:26:45,344 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: dd if=/dev/zero of=/data/full_disk_65599MB_1623176805.34 bs=1024M count=1 2021-06-08 11:26:46,509 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 2.8202 s, 381 MB/s 2021-06-08 11:26:46,510 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_35449MB_1623176806.51 bs=1024M count=1 2021-06-08 11:26:46,510 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: dd if=/dev/zero of=/data/full_disk_35449MB_1623176806.51 bs=1024M count=1 2021-06-08 11:26:54,759 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 8.11412 s, 132 MB/s 2021-06-08 11:26:54,759 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_34425MB_1623176814.76 bs=1024M count=1 2021-06-08 11:26:54,759 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: dd if=/dev/zero of=/data/full_disk_34425MB_1623176814.76 bs=1024M count=1 2021-06-08 11:26:56,319 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 1.41165 s, 761 MB/s 2021-06-08 11:26:56,319 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_33401MB_1623176816.32 bs=1024M count=1 2021-06-08 11:26:56,319 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: dd if=/dev/zero of=/data/full_disk_33401MB_1623176816.32 bs=1024M count=1 2021-06-08 11:26:56,332 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 18.2708 s, 58.8 MB/s 2021-06-08 11:26:56,332 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_69704MB_1623176816.33 bs=1024M count=1 2021-06-08 11:26:56,332 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: dd if=/dev/zero of=/data/full_disk_69704MB_1623176816.33 bs=1024M count=1 2021-06-08 11:26:57,862 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 1.41026 s, 761 MB/s 2021-06-08 11:26:57,862 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_32377MB_1623176817.86 bs=1024M count=1 2021-06-08 11:26:57,864 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: dd if=/dev/zero of=/data/full_disk_32377MB_1623176817.86 bs=1024M count=1 2021-06-08 11:26:59,157 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 2.69962 s, 398 MB/s 2021-06-08 11:26:59,158 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_68680MB_1623176819.16 bs=1024M count=1 2021-06-08 11:26:59,158 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: dd if=/dev/zero of=/data/full_disk_68680MB_1623176819.16 bs=1024M count=1 2021-06-08 11:26:59,405 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 1.38794 s, 774 MB/s 2021-06-08 11:26:59,405 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_31353MB_1623176819.41 bs=1024M count=1 2021-06-08 11:26:59,405 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: dd if=/dev/zero of=/data/full_disk_31353MB_1623176819.41 bs=1024M count=1 2021-06-08 11:27:00,726 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 1.4217 s, 755 MB/s 2021-06-08 11:27:00,726 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_67656MB_1623176820.73 bs=1024M count=1 2021-06-08 11:27:00,726 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: dd if=/dev/zero of=/data/full_disk_67656MB_1623176820.73 bs=1024M count=1 2021-06-08 11:27:01,401 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 1.85098 s, 580 MB/s 2021-06-08 11:27:01,401 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_30329MB_1623176821.4 bs=1024M count=1 2021-06-08 11:27:01,401 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: dd if=/dev/zero of=/data/full_disk_30329MB_1623176821.4 bs=1024M count=1 2021-06-08 11:27:02,944 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 1.39689 s, 769 MB/s 2021-06-08 11:27:02,944 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_29305MB_1623176822.94 bs=1024M count=1 2021-06-08 11:27:02,946 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: dd if=/dev/zero of=/data/full_disk_29305MB_1623176822.94 bs=1024M count=1 2021-06-08 11:27:05,253 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 4.3865 s, 245 MB/s 2021-06-08 11:27:05,253 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_66632MB_1623176825.25 bs=1024M count=1 2021-06-08 11:27:05,253 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: dd if=/dev/zero of=/data/full_disk_66632MB_1623176825.25 bs=1024M count=1 2021-06-08 11:27:10,655 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 38.0795 s, 28.2 MB/s 2021-06-08 11:27:10,657 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_75843MB_1623176830.66 bs=1024M count=1 2021-06-08 11:27:10,657 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: dd if=/dev/zero of=/data/full_disk_75843MB_1623176830.66 bs=1024M count=1 2021-06-08 11:27:19,572 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 16.4765 s, 65.2 MB/s 2021-06-08 11:27:19,572 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_28281MB_1623176839.57 bs=1024M count=1 2021-06-08 11:27:19,572 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: dd if=/dev/zero of=/data/full_disk_28281MB_1623176839.57 bs=1024M count=1 2021-06-08 11:27:21,763 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 2.05743 s, 522 MB/s 2021-06-08 11:27:21,765 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_27257MB_1623176841.76 bs=1024M count=1 2021-06-08 11:27:21,765 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: dd if=/dev/zero of=/data/full_disk_27257MB_1623176841.76 bs=1024M count=1 2021-06-08 11:27:22,844 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 37.3819 s, 28.7 MB/s 2021-06-08 11:27:22,845 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_64575MB_1623176842.84 bs=1024M count=1 2021-06-08 11:27:22,845 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: dd if=/dev/zero of=/data/full_disk_64575MB_1623176842.84 bs=1024M count=1 2021-06-08 11:27:26,997 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 5.10241 s, 210 MB/s 2021-06-08 11:27:26,997 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_26233MB_1623176847.0 bs=1024M count=1 2021-06-08 11:27:26,999 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: dd if=/dev/zero of=/data/full_disk_26233MB_1623176847.0 bs=1024M count=1 2021-06-08 11:27:27,430 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 22.0379 s, 48.7 MB/s 2021-06-08 11:27:27,430 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_65608MB_1623176847.43 bs=1024M count=1 2021-06-08 11:27:27,430 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: dd if=/dev/zero of=/data/full_disk_65608MB_1623176847.43 bs=1024M count=1 2021-06-08 11:27:30,494 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 19.7137 s, 54.5 MB/s 2021-06-08 11:27:30,496 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_74819MB_1623176850.5 bs=1024M count=1 2021-06-08 11:27:30,496 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: dd if=/dev/zero of=/data/full_disk_74819MB_1623176850.5 bs=1024M count=1 2021-06-08 11:27:31,384 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 4.24972 s, 253 MB/s 2021-06-08 11:27:31,385 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_25209MB_1623176851.39 bs=1024M count=1 2021-06-08 11:27:31,385 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: dd if=/dev/zero of=/data/full_disk_25209MB_1623176851.39 bs=1024M count=1 2021-06-08 11:27:34,983 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 3.46163 s, 310 MB/s 2021-06-08 11:27:34,983 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_24185MB_1623176854.98 bs=1024M count=1 2021-06-08 11:27:34,983 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: dd if=/dev/zero of=/data/full_disk_24185MB_1623176854.98 bs=1024M count=1 2021-06-08 11:27:36,789 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 13.8063 s, 77.8 MB/s 2021-06-08 11:27:36,789 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_63551MB_1623176856.79 bs=1024M count=1 2021-06-08 11:27:36,790 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: dd if=/dev/zero of=/data/full_disk_63551MB_1623176856.79 bs=1024M count=1 2021-06-08 11:27:36,796 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 1.65511 s, 649 MB/s 2021-06-08 11:27:36,796 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_23161MB_1623176856.8 bs=1024M count=1 2021-06-08 11:27:36,796 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: dd if=/dev/zero of=/data/full_disk_23161MB_1623176856.8 bs=1024M count=1 2021-06-08 11:27:39,003 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 11.4302 s, 93.9 MB/s 2021-06-08 11:27:39,003 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_64584MB_1623176859.0 bs=1024M count=1 2021-06-08 11:27:39,003 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: dd if=/dev/zero of=/data/full_disk_64584MB_1623176859.0 bs=1024M count=1 2021-06-08 11:27:43,532 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 6.59764 s, 163 MB/s 2021-06-08 11:27:43,533 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_22137MB_1623176863.53 bs=1024M count=1 2021-06-08 11:27:43,533 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: dd if=/dev/zero of=/data/full_disk_22137MB_1623176863.53 bs=1024M count=1 2021-06-08 11:27:45,865 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 6.73199 s, 159 MB/s 2021-06-08 11:27:45,865 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_63560MB_1623176865.87 bs=1024M count=1 2021-06-08 11:27:45,865 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: dd if=/dev/zero of=/data/full_disk_63560MB_1623176865.87 bs=1024M count=1 2021-06-08 11:27:45,984 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 15.3367 s, 70.0 MB/s 2021-06-08 11:27:45,984 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_73795MB_1623176865.98 bs=1024M count=1 2021-06-08 11:27:45,984 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: dd if=/dev/zero of=/data/full_disk_73795MB_1623176865.98 bs=1024M count=1 2021-06-08 11:27:48,826 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 5.13614 s, 209 MB/s 2021-06-08 11:27:48,828 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_21113MB_1623176868.83 bs=1024M count=1 2021-06-08 11:27:48,828 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: dd if=/dev/zero of=/data/full_disk_21113MB_1623176868.83 bs=1024M count=1 2021-06-08 11:27:49,082 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 12.1617 s, 88.3 MB/s 2021-06-08 11:27:49,082 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_62527MB_1623176869.08 bs=1024M count=1 2021-06-08 11:27:49,082 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: dd if=/dev/zero of=/data/full_disk_62527MB_1623176869.08 bs=1024M count=1 2021-06-08 11:27:55,336 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 9.21796 s, 116 MB/s 2021-06-08 11:27:55,336 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_72771MB_1623176875.34 bs=1024M count=1 2021-06-08 11:27:55,336 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: dd if=/dev/zero of=/data/full_disk_72771MB_1623176875.34 bs=1024M count=1 2021-06-08 11:27:56,476 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 10.4858 s, 102 MB/s 2021-06-08 11:27:56,476 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_62536MB_1623176876.48 bs=1024M count=1 2021-06-08 11:27:56,476 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: dd if=/dev/zero of=/data/full_disk_62536MB_1623176876.48 bs=1024M count=1 2021-06-08 11:27:58,726 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 9.51976 s, 113 MB/s 2021-06-08 11:27:58,726 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_61503MB_1623176878.73 bs=1024M count=1 2021-06-08 11:27:58,726 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: dd if=/dev/zero of=/data/full_disk_61503MB_1623176878.73 bs=1024M count=1 2021-06-08 11:27:59,121 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 10.1597 s, 106 MB/s 2021-06-08 11:27:59,122 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_20089MB_1623176879.12 bs=1024M count=1 2021-06-08 11:27:59,122 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: dd if=/dev/zero of=/data/full_disk_20089MB_1623176879.12 bs=1024M count=1 2021-06-08 11:28:03,694 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 4.41483 s, 243 MB/s 2021-06-08 11:28:03,694 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_19065MB_1623176883.69 bs=1024M count=1 2021-06-08 11:28:03,694 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: dd if=/dev/zero of=/data/full_disk_19065MB_1623176883.69 bs=1024M count=1 2021-06-08 11:28:06,369 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 10.9068 s, 98.4 MB/s 2021-06-08 11:28:06,369 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_71747MB_1623176886.37 bs=1024M count=1 2021-06-08 11:28:06,371 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: dd if=/dev/zero of=/data/full_disk_71747MB_1623176886.37 bs=1024M count=1 2021-06-08 11:28:08,010 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 10.512 s, 102 MB/s 2021-06-08 11:28:08,012 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_61512MB_1623176888.01 bs=1024M count=1 2021-06-08 11:28:08,012 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: dd if=/dev/zero of=/data/full_disk_61512MB_1623176888.01 bs=1024M count=1 2021-06-08 11:28:08,049 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 1.55046 s, 693 MB/s 2021-06-08 11:28:08,049 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_70723MB_1623176888.05 bs=1024M count=1 2021-06-08 11:28:08,049 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: dd if=/dev/zero of=/data/full_disk_70723MB_1623176888.05 bs=1024M count=1 2021-06-08 11:28:08,331 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 4.49751 s, 239 MB/s 2021-06-08 11:28:08,331 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_18041MB_1623176888.33 bs=1024M count=1 2021-06-08 11:28:08,331 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: dd if=/dev/zero of=/data/full_disk_18041MB_1623176888.33 bs=1024M count=1 2021-06-08 11:28:10,867 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 12.0183 s, 89.3 MB/s 2021-06-08 11:28:10,868 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_60479MB_1623176890.87 bs=1024M count=1 2021-06-08 11:28:10,868 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: dd if=/dev/zero of=/data/full_disk_60479MB_1623176890.87 bs=1024M count=1 2021-06-08 11:28:12,026 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 3.82714 s, 281 MB/s 2021-06-08 11:28:12,026 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_69699MB_1623176892.03 bs=1024M count=1 2021-06-08 11:28:12,026 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: dd if=/dev/zero of=/data/full_disk_69699MB_1623176892.03 bs=1024M count=1 2021-06-08 11:28:13,354 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 4.88507 s, 220 MB/s 2021-06-08 11:28:13,354 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_17017MB_1623176893.35 bs=1024M count=1 2021-06-08 11:28:13,355 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: dd if=/dev/zero of=/data/full_disk_17017MB_1623176893.35 bs=1024M count=1 2021-06-08 11:28:13,545 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 1.37162 s, 783 MB/s 2021-06-08 11:28:13,545 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_68675MB_1623176893.55 bs=1024M count=1 2021-06-08 11:28:13,545 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: dd if=/dev/zero of=/data/full_disk_68675MB_1623176893.55 bs=1024M count=1 2021-06-08 11:28:15,061 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 1.55072 s, 692 MB/s 2021-06-08 11:28:15,061 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_15993MB_1623176895.06 bs=1024M count=1 2021-06-08 11:28:15,061 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: dd if=/dev/zero of=/data/full_disk_15993MB_1623176895.06 bs=1024M count=1 2021-06-08 11:28:15,875 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 2.19533 s, 489 MB/s 2021-06-08 11:28:15,875 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_67651MB_1623176895.88 bs=1024M count=1 2021-06-08 11:28:15,875 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: dd if=/dev/zero of=/data/full_disk_67651MB_1623176895.88 bs=1024M count=1 2021-06-08 11:28:18,469 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 2.46585 s, 435 MB/s 2021-06-08 11:28:18,470 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_66627MB_1623176898.47 bs=1024M count=1 2021-06-08 11:28:18,470 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: dd if=/dev/zero of=/data/full_disk_66627MB_1623176898.47 bs=1024M count=1 2021-06-08 11:28:20,959 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 5.75693 s, 187 MB/s 2021-06-08 11:28:20,959 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_14969MB_1623176900.96 bs=1024M count=1 2021-06-08 11:28:20,959 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: dd if=/dev/zero of=/data/full_disk_14969MB_1623176900.96 bs=1024M count=1 2021-06-08 11:28:25,436 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 6.84056 s, 157 MB/s 2021-06-08 11:28:25,436 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_65603MB_1623176905.44 bs=1024M count=1 2021-06-08 11:28:25,436 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: dd if=/dev/zero of=/data/full_disk_65603MB_1623176905.44 bs=1024M count=1 2021-06-08 11:28:28,940 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 7.82806 s, 137 MB/s 2021-06-08 11:28:28,940 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_13945MB_1623176908.94 bs=1024M count=1 2021-06-08 11:28:28,940 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: dd if=/dev/zero of=/data/full_disk_13945MB_1623176908.94 bs=1024M count=1 2021-06-08 11:28:32,709 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 3.62069 s, 297 MB/s 2021-06-08 11:28:32,710 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_12921MB_1623176912.71 bs=1024M count=1 2021-06-08 11:28:32,710 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: dd if=/dev/zero of=/data/full_disk_12921MB_1623176912.71 bs=1024M count=1 2021-06-08 11:28:35,278 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 2.42016 s, 444 MB/s 2021-06-08 11:28:35,278 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_11897MB_1623176915.28 bs=1024M count=1 2021-06-08 11:28:35,278 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: dd if=/dev/zero of=/data/full_disk_11897MB_1623176915.28 bs=1024M count=1 2021-06-08 11:28:41,473 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 33.3237 s, 32.2 MB/s 2021-06-08 11:28:41,474 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_60488MB_1623176921.47 bs=1024M count=1 2021-06-08 11:28:41,474 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: dd if=/dev/zero of=/data/full_disk_60488MB_1623176921.47 bs=1024M count=1 2021-06-08 11:28:41,928 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 30.9205 s, 34.7 MB/s 2021-06-08 11:28:41,928 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_59455MB_1623176921.93 bs=1024M count=1 2021-06-08 11:28:41,930 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: dd if=/dev/zero of=/data/full_disk_59455MB_1623176921.93 bs=1024M count=1 2021-06-08 11:28:49,006 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 6.94815 s, 155 MB/s 2021-06-08 11:28:49,006 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_58431MB_1623176929.01 bs=1024M count=1 2021-06-08 11:28:49,006 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: dd if=/dev/zero of=/data/full_disk_58431MB_1623176929.01 bs=1024M count=1 2021-06-08 11:28:50,012 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 1.74266 s, 616 MB/s 2021-06-08 11:28:50,012 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_10873MB_1623176930.01 bs=1024M count=1 2021-06-08 11:28:50,012 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: dd if=/dev/zero of=/data/full_disk_10873MB_1623176930.01 bs=1024M count=1 2021-06-08 11:28:51,555 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 1.38947 s, 773 MB/s 2021-06-08 11:28:51,555 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_9849MB_1623176931.56 bs=1024M count=1 2021-06-08 11:28:51,555 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: dd if=/dev/zero of=/data/full_disk_9849MB_1623176931.56 bs=1024M count=1 2021-06-08 11:28:52,905 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 27.3422 s, 39.3 MB/s 2021-06-08 11:28:52,905 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_64579MB_1623176932.91 bs=1024M count=1 2021-06-08 11:28:52,907 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: dd if=/dev/zero of=/data/full_disk_64579MB_1623176932.91 bs=1024M count=1 2021-06-08 11:28:55,663 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 3.96896 s, 271 MB/s 2021-06-08 11:28:55,664 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_8825MB_1623176935.66 bs=1024M count=1 2021-06-08 11:28:55,664 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: dd if=/dev/zero of=/data/full_disk_8825MB_1623176935.66 bs=1024M count=1 2021-06-08 11:28:58,459 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 9.31351 s, 115 MB/s 2021-06-08 11:28:58,460 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_57407MB_1623176938.46 bs=1024M count=1 2021-06-08 11:28:58,460 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: dd if=/dev/zero of=/data/full_disk_57407MB_1623176938.46 bs=1024M count=1 2021-06-08 11:29:02,941 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 4.34478 s, 247 MB/s 2021-06-08 11:29:02,941 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_56383MB_1623176942.94 bs=1024M count=1 2021-06-08 11:29:02,941 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: dd if=/dev/zero of=/data/full_disk_56383MB_1623176942.94 bs=1024M count=1 2021-06-08 11:29:03,674 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 7.89156 s, 136 MB/s 2021-06-08 11:29:03,676 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_7801MB_1623176943.68 bs=1024M count=1 2021-06-08 11:29:03,676 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: dd if=/dev/zero of=/data/full_disk_7801MB_1623176943.68 bs=1024M count=1 2021-06-08 11:29:04,773 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 1.69219 s, 635 MB/s 2021-06-08 11:29:04,773 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_55359MB_1623176944.77 bs=1024M count=1 2021-06-08 11:29:04,773 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: dd if=/dev/zero of=/data/full_disk_55359MB_1623176944.77 bs=1024M count=1 2021-06-08 11:29:08,559 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 4.75032 s, 226 MB/s 2021-06-08 11:29:08,559 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_6777MB_1623176948.56 bs=1024M count=1 2021-06-08 11:29:08,559 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: dd if=/dev/zero of=/data/full_disk_6777MB_1623176948.56 bs=1024M count=1 2021-06-08 11:29:10,801 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 29.1983 s, 36.8 MB/s 2021-06-08 11:29:10,801 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_59464MB_1623176950.8 bs=1024M count=1 2021-06-08 11:29:10,802 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: dd if=/dev/zero of=/data/full_disk_59464MB_1623176950.8 bs=1024M count=1 2021-06-08 11:29:12,897 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 1.9538 s, 550 MB/s 2021-06-08 11:29:12,898 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_58440MB_1623176952.9 bs=1024M count=1 2021-06-08 11:29:12,898 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: dd if=/dev/zero of=/data/full_disk_58440MB_1623176952.9 bs=1024M count=1 2021-06-08 11:29:15,661 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 6.98093 s, 154 MB/s 2021-06-08 11:29:15,661 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_5753MB_1623176955.66 bs=1024M count=1 2021-06-08 11:29:15,661 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: dd if=/dev/zero of=/data/full_disk_5753MB_1623176955.66 bs=1024M count=1 2021-06-08 11:29:20,720 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 7.68378 s, 140 MB/s 2021-06-08 11:29:20,721 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_57416MB_1623176960.72 bs=1024M count=1 2021-06-08 11:29:20,721 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: dd if=/dev/zero of=/data/full_disk_57416MB_1623176960.72 bs=1024M count=1 2021-06-08 11:29:24,766 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 19.8634 s, 54.1 MB/s 2021-06-08 11:29:24,766 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_54335MB_1623176964.77 bs=1024M count=1 2021-06-08 11:29:24,766 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: dd if=/dev/zero of=/data/full_disk_54335MB_1623176964.77 bs=1024M count=1 2021-06-08 11:29:25,825 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 10.0143 s, 107 MB/s 2021-06-08 11:29:25,826 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_4729MB_1623176965.83 bs=1024M count=1 2021-06-08 11:29:25,826 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: dd if=/dev/zero of=/data/full_disk_4729MB_1623176965.83 bs=1024M count=1 2021-06-08 11:29:28,790 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 3.88048 s, 277 MB/s 2021-06-08 11:29:28,790 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_53311MB_1623176968.79 bs=1024M count=1 2021-06-08 11:29:28,792 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: dd if=/dev/zero of=/data/full_disk_53311MB_1623176968.79 bs=1024M count=1 2021-06-08 11:29:32,802 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 6.83943 s, 157 MB/s 2021-06-08 11:29:32,802 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_3705MB_1623176972.8 bs=1024M count=1 2021-06-08 11:29:32,802 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: dd if=/dev/zero of=/data/full_disk_3705MB_1623176972.8 bs=1024M count=1 2021-06-08 11:29:36,319 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 3.39058 s, 317 MB/s 2021-06-08 11:29:36,319 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_2681MB_1623176976.32 bs=1024M count=1 2021-06-08 11:29:36,319 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: dd if=/dev/zero of=/data/full_disk_2681MB_1623176976.32 bs=1024M count=1 2021-06-08 11:29:38,540 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 9.60795 s, 112 MB/s 2021-06-08 11:29:38,542 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_52287MB_1623176978.54 bs=1024M count=1 2021-06-08 11:29:38,542 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: dd if=/dev/zero of=/data/full_disk_52287MB_1623176978.54 bs=1024M count=1 2021-06-08 11:29:41,526 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 48.493 s, 22.1 MB/s 2021-06-08 11:29:41,526 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_63555MB_1623176981.53 bs=1024M count=1 2021-06-08 11:29:41,526 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: dd if=/dev/zero of=/data/full_disk_63555MB_1623176981.53 bs=1024M count=1 2021-06-08 11:29:42,138 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 3.45844 s, 310 MB/s 2021-06-08 11:29:42,138 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_51263MB_1623176982.14 bs=1024M count=1 2021-06-08 11:29:42,138 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: dd if=/dev/zero of=/data/full_disk_51263MB_1623176982.14 bs=1024M count=1 2021-06-08 11:29:45,693 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 3.42445 s, 314 MB/s 2021-06-08 11:29:45,693 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_50239MB_1623176985.69 bs=1024M count=1 2021-06-08 11:29:45,693 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: dd if=/dev/zero of=/data/full_disk_50239MB_1623176985.69 bs=1024M count=1 2021-06-08 11:29:45,809 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 9.36809 s, 115 MB/s 2021-06-08 11:29:45,809 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_1657MB_1623176985.81 bs=1024M count=1 2021-06-08 11:29:45,809 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: dd if=/dev/zero of=/data/full_disk_1657MB_1623176985.81 bs=1024M count=1 2021-06-08 11:29:48,865 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 28.006 s, 38.3 MB/s 2021-06-08 11:29:48,867 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_56392MB_1623176988.87 bs=1024M count=1 2021-06-08 11:29:48,867 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: dd if=/dev/zero of=/data/full_disk_56392MB_1623176988.87 bs=1024M count=1 2021-06-08 11:29:50,253 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 4.41672 s, 243 MB/s 2021-06-08 11:29:50,253 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_49215MB_1623176990.25 bs=1024M count=1 2021-06-08 11:29:50,253 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: dd if=/dev/zero of=/data/full_disk_49215MB_1623176990.25 bs=1024M count=1 2021-06-08 11:29:50,368 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 4.43187 s, 242 MB/s 2021-06-08 11:29:50,368 | test | DEBUG | Thread-17 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_633MB_1623176990.37 bs=633M count=1 2021-06-08 11:29:50,368 | infra | DEBUG | Thread-17 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: dd if=/dev/zero of=/data/full_disk_633MB_1623176990.37 bs=633M count=1 2021-06-08 11:29:57,414 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 7.02279 s, 153 MB/s 2021-06-08 11:29:57,414 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_48191MB_1623176997.41 bs=1024M count=1 2021-06-08 11:29:57,414 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: dd if=/dev/zero of=/data/full_disk_48191MB_1623176997.41 bs=1024M count=1 2021-06-08 11:29:59,313 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 17.638 s, 60.9 MB/s 2021-06-08 11:29:59,315 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_62531MB_1623176999.32 bs=1024M count=1 2021-06-08 11:29:59,315 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: dd if=/dev/zero of=/data/full_disk_62531MB_1623176999.32 bs=1024M count=1 2021-06-08 11:30:02,150 | test | ERROR | Thread-17 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 663748608 bytes (664 MB) copied, 11.6916 s, 56.8 MB/s 2021-06-08 11:30:02,150 | infra | DEBUG | Thread-17 | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.156 2021-06-08 11:30:03,559 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 5.99849 s, 179 MB/s 2021-06-08 11:30:03,559 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_47167MB_1623177003.56 bs=1024M count=1 2021-06-08 11:30:03,559 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: dd if=/dev/zero of=/data/full_disk_47167MB_1623177003.56 bs=1024M count=1 2021-06-08 11:30:10,009 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 21.022 s, 51.1 MB/s 2021-06-08 11:30:10,010 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_55368MB_1623177010.01 bs=1024M count=1 2021-06-08 11:30:10,010 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: dd if=/dev/zero of=/data/full_disk_55368MB_1623177010.01 bs=1024M count=1 2021-06-08 11:30:16,786 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 15.8967 s, 67.5 MB/s 2021-06-08 11:30:16,786 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_61507MB_1623177016.79 bs=1024M count=1 2021-06-08 11:30:16,786 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: dd if=/dev/zero of=/data/full_disk_61507MB_1623177016.79 bs=1024M count=1 2021-06-08 11:30:17,698 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 7.56425 s, 142 MB/s 2021-06-08 11:30:17,698 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_54344MB_1623177017.7 bs=1024M count=1 2021-06-08 11:30:17,698 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: dd if=/dev/zero of=/data/full_disk_54344MB_1623177017.7 bs=1024M count=1 2021-06-08 11:30:19,367 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 15.6696 s, 68.5 MB/s 2021-06-08 11:30:19,367 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_46143MB_1623177019.37 bs=1024M count=1 2021-06-08 11:30:19,367 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: dd if=/dev/zero of=/data/full_disk_46143MB_1623177019.37 bs=1024M count=1 2021-06-08 11:30:27,714 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 9.87298 s, 109 MB/s 2021-06-08 11:30:27,714 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_53320MB_1623177027.71 bs=1024M count=1 2021-06-08 11:30:27,716 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: dd if=/dev/zero of=/data/full_disk_53320MB_1623177027.71 bs=1024M count=1 2021-06-08 11:30:34,009 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 14.5099 s, 74.0 MB/s 2021-06-08 11:30:34,009 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_45119MB_1623177034.01 bs=1024M count=1 2021-06-08 11:30:34,010 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: dd if=/dev/zero of=/data/full_disk_45119MB_1623177034.01 bs=1024M count=1 2021-06-08 11:30:35,621 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 7.7744 s, 138 MB/s 2021-06-08 11:30:35,621 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_52296MB_1623177035.62 bs=1024M count=1 2021-06-08 11:30:35,621 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: dd if=/dev/zero of=/data/full_disk_52296MB_1623177035.62 bs=1024M count=1 2021-06-08 11:30:40,378 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 4.60899 s, 233 MB/s 2021-06-08 11:30:40,378 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_51272MB_1623177040.38 bs=1024M count=1 2021-06-08 11:30:40,378 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: dd if=/dev/zero of=/data/full_disk_51272MB_1623177040.38 bs=1024M count=1 2021-06-08 11:30:45,628 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 5.11816 s, 210 MB/s 2021-06-08 11:30:45,628 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_50248MB_1623177045.63 bs=1024M count=1 2021-06-08 11:30:45,628 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: dd if=/dev/zero of=/data/full_disk_50248MB_1623177045.63 bs=1024M count=1 2021-06-08 11:30:46,105 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 29.134 s, 36.9 MB/s 2021-06-08 11:30:46,105 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_60483MB_1623177046.11 bs=1024M count=1 2021-06-08 11:30:46,107 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: dd if=/dev/zero of=/data/full_disk_60483MB_1623177046.11 bs=1024M count=1 2021-06-08 11:30:50,808 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 16.6738 s, 64.4 MB/s 2021-06-08 11:30:50,808 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_44095MB_1623177050.81 bs=1024M count=1 2021-06-08 11:30:50,808 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: dd if=/dev/zero of=/data/full_disk_44095MB_1623177050.81 bs=1024M count=1 2021-06-08 11:30:51,562 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 5.79082 s, 185 MB/s 2021-06-08 11:30:51,562 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_49224MB_1623177051.56 bs=1024M count=1 2021-06-08 11:30:51,562 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: dd if=/dev/zero of=/data/full_disk_49224MB_1623177051.56 bs=1024M count=1 2021-06-08 11:31:01,976 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 10.2838 s, 104 MB/s 2021-06-08 11:31:01,977 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_48200MB_1623177061.98 bs=1024M count=1 2021-06-08 11:31:01,977 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: dd if=/dev/zero of=/data/full_disk_48200MB_1623177061.98 bs=1024M count=1 2021-06-08 11:31:08,595 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 22.3374 s, 48.1 MB/s 2021-06-08 11:31:08,595 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_59459MB_1623177068.6 bs=1024M count=1 2021-06-08 11:31:08,595 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: dd if=/dev/zero of=/data/full_disk_59459MB_1623177068.6 bs=1024M count=1 2021-06-08 11:31:09,033 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 6.91075 s, 155 MB/s 2021-06-08 11:31:09,033 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_47176MB_1623177069.03 bs=1024M count=1 2021-06-08 11:31:09,035 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: dd if=/dev/zero of=/data/full_disk_47176MB_1623177069.03 bs=1024M count=1 2021-06-08 11:31:09,421 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 18.4828 s, 58.1 MB/s 2021-06-08 11:31:09,421 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_43071MB_1623177069.42 bs=1024M count=1 2021-06-08 11:31:09,421 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: dd if=/dev/zero of=/data/full_disk_43071MB_1623177069.42 bs=1024M count=1 2021-06-08 11:31:14,426 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 5.25164 s, 204 MB/s 2021-06-08 11:31:14,427 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_46152MB_1623177074.43 bs=1024M count=1 2021-06-08 11:31:14,427 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: dd if=/dev/zero of=/data/full_disk_46152MB_1623177074.43 bs=1024M count=1 2021-06-08 11:31:15,921 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 1.36484 s, 787 MB/s 2021-06-08 11:31:15,921 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_45128MB_1623177075.92 bs=1024M count=1 2021-06-08 11:31:15,921 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: dd if=/dev/zero of=/data/full_disk_45128MB_1623177075.92 bs=1024M count=1 2021-06-08 11:31:24,914 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 15.3701 s, 69.9 MB/s 2021-06-08 11:31:24,914 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_42047MB_1623177084.91 bs=1024M count=1 2021-06-08 11:31:24,914 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: dd if=/dev/zero of=/data/full_disk_42047MB_1623177084.91 bs=1024M count=1 2021-06-08 11:31:26,131 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 10.0877 s, 106 MB/s 2021-06-08 11:31:26,132 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_44104MB_1623177086.13 bs=1024M count=1 2021-06-08 11:31:26,132 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: dd if=/dev/zero of=/data/full_disk_44104MB_1623177086.13 bs=1024M count=1 2021-06-08 11:31:27,163 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 18.4189 s, 58.3 MB/s 2021-06-08 11:31:27,163 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_58435MB_1623177087.16 bs=1024M count=1 2021-06-08 11:31:27,163 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: dd if=/dev/zero of=/data/full_disk_58435MB_1623177087.16 bs=1024M count=1 2021-06-08 11:31:34,500 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 9.45913 s, 114 MB/s 2021-06-08 11:31:34,500 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_41023MB_1623177094.5 bs=1024M count=1 2021-06-08 11:31:34,500 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: dd if=/dev/zero of=/data/full_disk_41023MB_1623177094.5 bs=1024M count=1 2021-06-08 11:31:36,066 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 1.42445 s, 754 MB/s 2021-06-08 11:31:36,068 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_39999MB_1623177096.07 bs=1024M count=1 2021-06-08 11:31:36,068 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: dd if=/dev/zero of=/data/full_disk_39999MB_1623177096.07 bs=1024M count=1 2021-06-08 11:31:38,711 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 12.4576 s, 86.2 MB/s 2021-06-08 11:31:38,711 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_43080MB_1623177098.71 bs=1024M count=1 2021-06-08 11:31:38,713 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: dd if=/dev/zero of=/data/full_disk_43080MB_1623177098.71 bs=1024M count=1 2021-06-08 11:31:44,246 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 16.937 s, 63.4 MB/s 2021-06-08 11:31:44,246 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_57411MB_1623177104.25 bs=1024M count=1 2021-06-08 11:31:44,247 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: dd if=/dev/zero of=/data/full_disk_57411MB_1623177104.25 bs=1024M count=1 2021-06-08 11:31:47,835 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 11.6249 s, 92.4 MB/s 2021-06-08 11:31:47,835 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_38975MB_1623177107.84 bs=1024M count=1 2021-06-08 11:31:47,835 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: dd if=/dev/zero of=/data/full_disk_38975MB_1623177107.84 bs=1024M count=1 2021-06-08 11:31:52,096 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 4.11931 s, 261 MB/s 2021-06-08 11:31:52,096 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_37951MB_1623177112.1 bs=1024M count=1 2021-06-08 11:31:52,096 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: dd if=/dev/zero of=/data/full_disk_37951MB_1623177112.1 bs=1024M count=1 2021-06-08 11:31:54,822 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 2.58499 s, 415 MB/s 2021-06-08 11:31:54,822 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_36927MB_1623177114.82 bs=1024M count=1 2021-06-08 11:31:54,822 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: dd if=/dev/zero of=/data/full_disk_36927MB_1623177114.82 bs=1024M count=1 2021-06-08 11:32:05,384 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 26.5314 s, 40.5 MB/s 2021-06-08 11:32:05,384 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_42056MB_1623177125.38 bs=1024M count=1 2021-06-08 11:32:05,384 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: dd if=/dev/zero of=/data/full_disk_42056MB_1623177125.38 bs=1024M count=1 2021-06-08 11:32:06,010 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 21.618 s, 49.7 MB/s 2021-06-08 11:32:06,010 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_56387MB_1623177126.01 bs=1024M count=1 2021-06-08 11:32:06,010 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: dd if=/dev/zero of=/data/full_disk_56387MB_1623177126.01 bs=1024M count=1 2021-06-08 11:32:06,357 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 11.3943 s, 94.2 MB/s 2021-06-08 11:32:06,357 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_35903MB_1623177126.36 bs=1024M count=1 2021-06-08 11:32:06,357 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: dd if=/dev/zero of=/data/full_disk_35903MB_1623177126.36 bs=1024M count=1 2021-06-08 11:32:14,348 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 7.85252 s, 137 MB/s 2021-06-08 11:32:14,348 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_34879MB_1623177134.35 bs=1024M count=1 2021-06-08 11:32:14,348 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: dd if=/dev/zero of=/data/full_disk_34879MB_1623177134.35 bs=1024M count=1 2021-06-08 11:32:19,194 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 4.70733 s, 228 MB/s 2021-06-08 11:32:19,194 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_33855MB_1623177139.19 bs=1024M count=1 2021-06-08 11:32:19,194 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: dd if=/dev/zero of=/data/full_disk_33855MB_1623177139.19 bs=1024M count=1 2021-06-08 11:32:26,517 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 7.18678 s, 149 MB/s 2021-06-08 11:32:26,519 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_32831MB_1623177146.52 bs=1024M count=1 2021-06-08 11:32:26,519 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: dd if=/dev/zero of=/data/full_disk_32831MB_1623177146.52 bs=1024M count=1 2021-06-08 11:32:28,107 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 22.5792 s, 47.6 MB/s 2021-06-08 11:32:28,107 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_41032MB_1623177148.11 bs=1024M count=1 2021-06-08 11:32:28,107 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: dd if=/dev/zero of=/data/full_disk_41032MB_1623177148.11 bs=1024M count=1 2021-06-08 11:32:28,533 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 22.3837 s, 48.0 MB/s 2021-06-08 11:32:28,535 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_55363MB_1623177148.54 bs=1024M count=1 2021-06-08 11:32:28,535 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: dd if=/dev/zero of=/data/full_disk_55363MB_1623177148.54 bs=1024M count=1 2021-06-08 11:32:33,092 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 6.43329 s, 167 MB/s 2021-06-08 11:32:33,092 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_31807MB_1623177153.09 bs=1024M count=1 2021-06-08 11:32:33,092 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: dd if=/dev/zero of=/data/full_disk_31807MB_1623177153.09 bs=1024M count=1 2021-06-08 11:32:41,454 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 8.22377 s, 131 MB/s 2021-06-08 11:32:41,454 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_30783MB_1623177161.45 bs=1024M count=1 2021-06-08 11:32:41,454 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: dd if=/dev/zero of=/data/full_disk_30783MB_1623177161.45 bs=1024M count=1 2021-06-08 11:32:42,851 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 14.6106 s, 73.5 MB/s 2021-06-08 11:32:42,852 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_40008MB_1623177162.85 bs=1024M count=1 2021-06-08 11:32:42,852 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: dd if=/dev/zero of=/data/full_disk_40008MB_1623177162.85 bs=1024M count=1 2021-06-08 11:32:43,736 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 15.0553 s, 71.3 MB/s 2021-06-08 11:32:43,736 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_54339MB_1623177163.74 bs=1024M count=1 2021-06-08 11:32:43,736 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: dd if=/dev/zero of=/data/full_disk_54339MB_1623177163.74 bs=1024M count=1 2021-06-08 11:32:52,424 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 10.8316 s, 99.1 MB/s 2021-06-08 11:32:52,424 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_29759MB_1623177172.42 bs=1024M count=1 2021-06-08 11:32:52,424 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: dd if=/dev/zero of=/data/full_disk_29759MB_1623177172.42 bs=1024M count=1 2021-06-08 11:32:52,674 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 8.79768 s, 122 MB/s 2021-06-08 11:32:52,674 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_53315MB_1623177172.67 bs=1024M count=1 2021-06-08 11:32:52,674 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: dd if=/dev/zero of=/data/full_disk_53315MB_1623177172.67 bs=1024M count=1 2021-06-08 11:32:55,128 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 12.1271 s, 88.5 MB/s 2021-06-08 11:32:55,128 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_38984MB_1623177175.13 bs=1024M count=1 2021-06-08 11:32:55,128 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: dd if=/dev/zero of=/data/full_disk_38984MB_1623177175.13 bs=1024M count=1 2021-06-08 11:32:59,174 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 6.60782 s, 162 MB/s 2021-06-08 11:32:59,174 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_28735MB_1623177179.17 bs=1024M count=1 2021-06-08 11:32:59,174 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: dd if=/dev/zero of=/data/full_disk_28735MB_1623177179.17 bs=1024M count=1 2021-06-08 11:33:01,398 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 8.59431 s, 125 MB/s 2021-06-08 11:33:01,400 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_52291MB_1623177181.4 bs=1024M count=1 2021-06-08 11:33:01,400 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: dd if=/dev/zero of=/data/full_disk_52291MB_1623177181.4 bs=1024M count=1 2021-06-08 11:33:01,862 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 2.55883 s, 420 MB/s 2021-06-08 11:33:01,862 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_27711MB_1623177181.86 bs=1024M count=1 2021-06-08 11:33:01,862 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: dd if=/dev/zero of=/data/full_disk_27711MB_1623177181.86 bs=1024M count=1 2021-06-08 11:33:06,071 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 3.75134 s, 286 MB/s 2021-06-08 11:33:06,072 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_26687MB_1623177186.07 bs=1024M count=1 2021-06-08 11:33:06,072 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: dd if=/dev/zero of=/data/full_disk_26687MB_1623177186.07 bs=1024M count=1 2021-06-08 11:33:09,740 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 3.54135 s, 303 MB/s 2021-06-08 11:33:09,740 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_25663MB_1623177189.74 bs=1024M count=1 2021-06-08 11:33:09,740 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: dd if=/dev/zero of=/data/full_disk_25663MB_1623177189.74 bs=1024M count=1 2021-06-08 11:33:14,684 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 19.4268 s, 55.3 MB/s 2021-06-08 11:33:14,684 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_37960MB_1623177194.68 bs=1024M count=1 2021-06-08 11:33:14,684 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: dd if=/dev/zero of=/data/full_disk_37960MB_1623177194.68 bs=1024M count=1 2021-06-08 11:33:16,447 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 1.63913 s, 655 MB/s 2021-06-08 11:33:16,447 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_36936MB_1623177196.45 bs=1024M count=1 2021-06-08 11:33:16,447 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: dd if=/dev/zero of=/data/full_disk_36936MB_1623177196.45 bs=1024M count=1 2021-06-08 11:33:19,891 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 3.29764 s, 326 MB/s 2021-06-08 11:33:19,891 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_35912MB_1623177199.89 bs=1024M count=1 2021-06-08 11:33:19,891 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: dd if=/dev/zero of=/data/full_disk_35912MB_1623177199.89 bs=1024M count=1 2021-06-08 11:33:20,917 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 19.369 s, 55.4 MB/s 2021-06-08 11:33:20,918 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_51267MB_1623177200.92 bs=1024M count=1 2021-06-08 11:33:20,918 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: dd if=/dev/zero of=/data/full_disk_51267MB_1623177200.92 bs=1024M count=1 2021-06-08 11:33:24,022 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 2.94834 s, 364 MB/s 2021-06-08 11:33:24,022 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_50243MB_1623177204.02 bs=1024M count=1 2021-06-08 11:33:24,023 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: dd if=/dev/zero of=/data/full_disk_50243MB_1623177204.02 bs=1024M count=1 2021-06-08 11:33:24,854 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 4.83467 s, 222 MB/s 2021-06-08 11:33:24,854 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_34888MB_1623177204.85 bs=1024M count=1 2021-06-08 11:33:24,855 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: dd if=/dev/zero of=/data/full_disk_34888MB_1623177204.85 bs=1024M count=1 2021-06-08 11:33:26,213 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 2.04487 s, 525 MB/s 2021-06-08 11:33:26,213 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_49219MB_1623177206.21 bs=1024M count=1 2021-06-08 11:33:26,213 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: dd if=/dev/zero of=/data/full_disk_49219MB_1623177206.21 bs=1024M count=1 2021-06-08 11:33:30,525 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 5.54617 s, 194 MB/s 2021-06-08 11:33:30,525 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_33864MB_1623177210.53 bs=1024M count=1 2021-06-08 11:33:30,526 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: dd if=/dev/zero of=/data/full_disk_33864MB_1623177210.53 bs=1024M count=1 2021-06-08 11:33:31,463 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 21.5993 s, 49.7 MB/s 2021-06-08 11:33:31,463 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_24639MB_1623177211.46 bs=1024M count=1 2021-06-08 11:33:31,463 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: dd if=/dev/zero of=/data/full_disk_24639MB_1623177211.46 bs=1024M count=1 2021-06-08 11:33:32,585 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 1.91294 s, 561 MB/s 2021-06-08 11:33:32,585 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_32840MB_1623177212.59 bs=1024M count=1 2021-06-08 11:33:32,585 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: dd if=/dev/zero of=/data/full_disk_32840MB_1623177212.59 bs=1024M count=1 2021-06-08 11:33:33,497 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 7.13181 s, 151 MB/s 2021-06-08 11:33:33,499 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_48195MB_1623177213.5 bs=1024M count=1 2021-06-08 11:33:33,499 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: dd if=/dev/zero of=/data/full_disk_48195MB_1623177213.5 bs=1024M count=1 2021-06-08 11:33:34,813 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 2.10046 s, 511 MB/s 2021-06-08 11:33:34,815 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_31816MB_1623177214.82 bs=1024M count=1 2021-06-08 11:33:34,815 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: dd if=/dev/zero of=/data/full_disk_31816MB_1623177214.82 bs=1024M count=1 2021-06-08 11:33:49,470 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 14.5298 s, 73.9 MB/s 2021-06-08 11:33:49,470 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_30792MB_1623177229.47 bs=1024M count=1 2021-06-08 11:33:49,470 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: dd if=/dev/zero of=/data/full_disk_30792MB_1623177229.47 bs=1024M count=1 2021-06-08 11:33:50,924 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 17.2886 s, 62.1 MB/s 2021-06-08 11:33:50,926 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_47171MB_1623177230.92 bs=1024M count=1 2021-06-08 11:33:50,926 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: dd if=/dev/zero of=/data/full_disk_47171MB_1623177230.92 bs=1024M count=1 2021-06-08 11:33:58,223 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 6.89124 s, 156 MB/s 2021-06-08 11:33:58,223 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_46147MB_1623177238.22 bs=1024M count=1 2021-06-08 11:33:58,223 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: dd if=/dev/zero of=/data/full_disk_46147MB_1623177238.22 bs=1024M count=1 2021-06-08 11:33:59,009 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 27.4194 s, 39.2 MB/s 2021-06-08 11:33:59,009 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_23615MB_1623177239.01 bs=1024M count=1 2021-06-08 11:33:59,009 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: dd if=/dev/zero of=/data/full_disk_23615MB_1623177239.01 bs=1024M count=1 2021-06-08 11:34:04,181 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 14.5841 s, 73.6 MB/s 2021-06-08 11:34:04,181 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_29768MB_1623177244.18 bs=1024M count=1 2021-06-08 11:34:04,181 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: dd if=/dev/zero of=/data/full_disk_29768MB_1623177244.18 bs=1024M count=1 2021-06-08 11:34:06,872 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 7.71402 s, 139 MB/s 2021-06-08 11:34:06,874 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_45123MB_1623177246.87 bs=1024M count=1 2021-06-08 11:34:06,874 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: dd if=/dev/zero of=/data/full_disk_45123MB_1623177246.87 bs=1024M count=1 2021-06-08 11:34:08,030 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 8.87632 s, 121 MB/s 2021-06-08 11:34:08,030 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_22591MB_1623177248.03 bs=1024M count=1 2021-06-08 11:34:08,030 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: dd if=/dev/zero of=/data/full_disk_22591MB_1623177248.03 bs=1024M count=1 2021-06-08 11:34:08,625 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 4.29091 s, 250 MB/s 2021-06-08 11:34:08,625 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_28744MB_1623177248.63 bs=1024M count=1 2021-06-08 11:34:08,625 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: dd if=/dev/zero of=/data/full_disk_28744MB_1623177248.63 bs=1024M count=1 2021-06-08 11:34:10,414 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 1.64799 s, 652 MB/s 2021-06-08 11:34:10,414 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_27720MB_1623177250.41 bs=1024M count=1 2021-06-08 11:34:10,414 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: dd if=/dev/zero of=/data/full_disk_27720MB_1623177250.41 bs=1024M count=1 2021-06-08 11:34:21,960 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 14.9411 s, 71.9 MB/s 2021-06-08 11:34:21,960 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_44099MB_1623177261.96 bs=1024M count=1 2021-06-08 11:34:21,960 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: dd if=/dev/zero of=/data/full_disk_44099MB_1623177261.96 bs=1024M count=1 2021-06-08 11:34:23,230 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 12.6725 s, 84.7 MB/s 2021-06-08 11:34:23,230 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_26696MB_1623177263.23 bs=1024M count=1 2021-06-08 11:34:23,230 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: dd if=/dev/zero of=/data/full_disk_26696MB_1623177263.23 bs=1024M count=1 2021-06-08 11:34:27,234 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 19.0725 s, 56.3 MB/s 2021-06-08 11:34:27,234 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_21567MB_1623177267.23 bs=1024M count=1 2021-06-08 11:34:27,234 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: dd if=/dev/zero of=/data/full_disk_21567MB_1623177267.23 bs=1024M count=1 2021-06-08 11:34:29,539 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 6.16876 s, 174 MB/s 2021-06-08 11:34:29,539 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_25672MB_1623177269.54 bs=1024M count=1 2021-06-08 11:34:29,539 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: dd if=/dev/zero of=/data/full_disk_25672MB_1623177269.54 bs=1024M count=1 2021-06-08 11:34:33,193 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 11.0946 s, 96.8 MB/s 2021-06-08 11:34:33,194 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_43075MB_1623177273.19 bs=1024M count=1 2021-06-08 11:34:33,194 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: dd if=/dev/zero of=/data/full_disk_43075MB_1623177273.19 bs=1024M count=1 2021-06-08 11:34:35,473 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 5.79432 s, 185 MB/s 2021-06-08 11:34:35,473 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_24648MB_1623177275.47 bs=1024M count=1 2021-06-08 11:34:35,473 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: dd if=/dev/zero of=/data/full_disk_24648MB_1623177275.47 bs=1024M count=1 2021-06-08 11:34:36,407 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 9.02646 s, 119 MB/s 2021-06-08 11:34:36,407 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_20543MB_1623177276.41 bs=1024M count=1 2021-06-08 11:34:36,407 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: dd if=/dev/zero of=/data/full_disk_20543MB_1623177276.41 bs=1024M count=1 2021-06-08 11:34:37,914 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 1.37792 s, 779 MB/s 2021-06-08 11:34:37,914 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_19519MB_1623177277.91 bs=1024M count=1 2021-06-08 11:34:37,914 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: dd if=/dev/zero of=/data/full_disk_19519MB_1623177277.91 bs=1024M count=1 2021-06-08 11:34:41,819 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 3.75969 s, 286 MB/s 2021-06-08 11:34:41,819 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_18495MB_1623177281.82 bs=1024M count=1 2021-06-08 11:34:41,819 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: dd if=/dev/zero of=/data/full_disk_18495MB_1623177281.82 bs=1024M count=1 2021-06-08 11:34:47,812 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 14.4708 s, 74.2 MB/s 2021-06-08 11:34:47,812 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_42051MB_1623177287.81 bs=1024M count=1 2021-06-08 11:34:47,812 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: dd if=/dev/zero of=/data/full_disk_42051MB_1623177287.81 bs=1024M count=1 2021-06-08 11:34:49,595 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 13.9979 s, 76.7 MB/s 2021-06-08 11:34:49,595 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_23624MB_1623177289.6 bs=1024M count=1 2021-06-08 11:34:49,595 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: dd if=/dev/zero of=/data/full_disk_23624MB_1623177289.6 bs=1024M count=1 2021-06-08 11:34:50,592 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 8.63214 s, 124 MB/s 2021-06-08 11:34:50,592 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_17471MB_1623177290.59 bs=1024M count=1 2021-06-08 11:34:50,592 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: dd if=/dev/zero of=/data/full_disk_17471MB_1623177290.59 bs=1024M count=1 2021-06-08 11:34:55,315 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 4.59244 s, 234 MB/s 2021-06-08 11:34:55,315 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_16447MB_1623177295.32 bs=1024M count=1 2021-06-08 11:34:55,315 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: dd if=/dev/zero of=/data/full_disk_16447MB_1623177295.32 bs=1024M count=1 2021-06-08 11:34:59,151 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 3.71583 s, 289 MB/s 2021-06-08 11:34:59,151 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_15423MB_1623177299.15 bs=1024M count=1 2021-06-08 11:34:59,151 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: dd if=/dev/zero of=/data/full_disk_15423MB_1623177299.15 bs=1024M count=1 2021-06-08 11:35:00,512 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 12.5506 s, 85.6 MB/s 2021-06-08 11:35:00,512 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_41027MB_1623177300.51 bs=1024M count=1 2021-06-08 11:35:00,513 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: dd if=/dev/zero of=/data/full_disk_41027MB_1623177300.51 bs=1024M count=1 2021-06-08 11:35:03,625 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 4.32986 s, 248 MB/s 2021-06-08 11:35:03,625 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_14399MB_1623177303.63 bs=1024M count=1 2021-06-08 11:35:03,625 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: dd if=/dev/zero of=/data/full_disk_14399MB_1623177303.63 bs=1024M count=1 2021-06-08 11:35:04,365 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 14.1298 s, 76.0 MB/s 2021-06-08 11:35:04,365 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_22600MB_1623177304.37 bs=1024M count=1 2021-06-08 11:35:04,365 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: dd if=/dev/zero of=/data/full_disk_22600MB_1623177304.37 bs=1024M count=1 2021-06-08 11:35:08,328 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 4.56245 s, 235 MB/s 2021-06-08 11:35:08,328 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_13375MB_1623177308.33 bs=1024M count=1 2021-06-08 11:35:08,328 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: dd if=/dev/zero of=/data/full_disk_13375MB_1623177308.33 bs=1024M count=1 2021-06-08 11:35:12,640 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 4.17044 s, 257 MB/s 2021-06-08 11:35:12,641 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_12351MB_1623177312.64 bs=1024M count=1 2021-06-08 11:35:12,641 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: dd if=/dev/zero of=/data/full_disk_12351MB_1623177312.64 bs=1024M count=1 2021-06-08 11:35:14,976 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 14.3166 s, 75.0 MB/s 2021-06-08 11:35:14,976 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_40003MB_1623177314.98 bs=1024M count=1 2021-06-08 11:35:14,976 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: dd if=/dev/zero of=/data/full_disk_40003MB_1623177314.98 bs=1024M count=1 2021-06-08 11:35:16,688 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 3.92184 s, 274 MB/s 2021-06-08 11:35:16,690 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_11327MB_1623177316.69 bs=1024M count=1 2021-06-08 11:35:16,690 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: dd if=/dev/zero of=/data/full_disk_11327MB_1623177316.69 bs=1024M count=1 2021-06-08 11:35:20,697 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 3.88225 s, 277 MB/s 2021-06-08 11:35:20,697 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_10303MB_1623177320.7 bs=1024M count=1 2021-06-08 11:35:20,697 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: dd if=/dev/zero of=/data/full_disk_10303MB_1623177320.7 bs=1024M count=1 2021-06-08 11:35:22,265 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 17.7738 s, 60.4 MB/s 2021-06-08 11:35:22,265 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_21576MB_1623177322.27 bs=1024M count=1 2021-06-08 11:35:22,265 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: dd if=/dev/zero of=/data/full_disk_21576MB_1623177322.27 bs=1024M count=1 2021-06-08 11:35:25,187 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 4.36383 s, 246 MB/s 2021-06-08 11:35:25,188 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_9279MB_1623177325.19 bs=1024M count=1 2021-06-08 11:35:25,188 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: dd if=/dev/zero of=/data/full_disk_9279MB_1623177325.19 bs=1024M count=1 2021-06-08 11:35:27,078 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 11.9631 s, 89.8 MB/s 2021-06-08 11:35:27,078 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_38979MB_1623177327.08 bs=1024M count=1 2021-06-08 11:35:27,078 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: dd if=/dev/zero of=/data/full_disk_38979MB_1623177327.08 bs=1024M count=1 2021-06-08 11:35:28,384 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 3.0498 s, 352 MB/s 2021-06-08 11:35:28,384 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_8255MB_1623177328.38 bs=1024M count=1 2021-06-08 11:35:28,384 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: dd if=/dev/zero of=/data/full_disk_8255MB_1623177328.38 bs=1024M count=1 2021-06-08 11:35:32,374 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 3.84753 s, 279 MB/s 2021-06-08 11:35:32,374 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_7231MB_1623177332.37 bs=1024M count=1 2021-06-08 11:35:32,374 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: dd if=/dev/zero of=/data/full_disk_7231MB_1623177332.37 bs=1024M count=1 2021-06-08 11:35:34,660 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 2.13818 s, 502 MB/s 2021-06-08 11:35:34,661 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_6207MB_1623177334.66 bs=1024M count=1 2021-06-08 11:35:34,661 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: dd if=/dev/zero of=/data/full_disk_6207MB_1623177334.66 bs=1024M count=1 2021-06-08 11:35:37,227 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 14.815 s, 72.5 MB/s 2021-06-08 11:35:37,227 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_20552MB_1623177337.23 bs=1024M count=1 2021-06-08 11:35:37,229 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: dd if=/dev/zero of=/data/full_disk_20552MB_1623177337.23 bs=1024M count=1 2021-06-08 11:35:38,744 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 1.37514 s, 781 MB/s 2021-06-08 11:35:38,746 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_19528MB_1623177338.75 bs=1024M count=1 2021-06-08 11:35:38,746 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: dd if=/dev/zero of=/data/full_disk_19528MB_1623177338.75 bs=1024M count=1 2021-06-08 11:35:40,354 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 1.47776 s, 727 MB/s 2021-06-08 11:35:40,354 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_18504MB_1623177340.35 bs=1024M count=1 2021-06-08 11:35:40,354 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: dd if=/dev/zero of=/data/full_disk_18504MB_1623177340.35 bs=1024M count=1 2021-06-08 11:35:40,444 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 5.64145 s, 190 MB/s 2021-06-08 11:35:40,444 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_5183MB_1623177340.44 bs=1024M count=1 2021-06-08 11:35:40,444 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: dd if=/dev/zero of=/data/full_disk_5183MB_1623177340.44 bs=1024M count=1 2021-06-08 11:35:44,358 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 3.85868 s, 278 MB/s 2021-06-08 11:35:44,359 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_17480MB_1623177344.36 bs=1024M count=1 2021-06-08 11:35:44,359 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: dd if=/dev/zero of=/data/full_disk_17480MB_1623177344.36 bs=1024M count=1 2021-06-08 11:35:50,901 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 6.40804 s, 168 MB/s 2021-06-08 11:35:50,901 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_16456MB_1623177350.9 bs=1024M count=1 2021-06-08 11:35:50,901 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: dd if=/dev/zero of=/data/full_disk_16456MB_1623177350.9 bs=1024M count=1 2021-06-08 11:35:51,013 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 23.807 s, 45.1 MB/s 2021-06-08 11:35:51,013 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_37955MB_1623177351.01 bs=1024M count=1 2021-06-08 11:35:51,013 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: dd if=/dev/zero of=/data/full_disk_37955MB_1623177351.01 bs=1024M count=1 2021-06-08 11:35:57,078 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 6.02908 s, 178 MB/s 2021-06-08 11:35:57,078 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_15432MB_1623177357.08 bs=1024M count=1 2021-06-08 11:35:57,078 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: dd if=/dev/zero of=/data/full_disk_15432MB_1623177357.08 bs=1024M count=1 2021-06-08 11:36:01,436 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 4.21622 s, 255 MB/s 2021-06-08 11:36:01,436 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_14408MB_1623177361.44 bs=1024M count=1 2021-06-08 11:36:01,436 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: dd if=/dev/zero of=/data/full_disk_14408MB_1623177361.44 bs=1024M count=1 2021-06-08 11:36:01,630 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 21.0405 s, 51.0 MB/s 2021-06-08 11:36:01,631 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_4159MB_1623177361.63 bs=1024M count=1 2021-06-08 11:36:01,631 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: dd if=/dev/zero of=/data/full_disk_4159MB_1623177361.63 bs=1024M count=1 2021-06-08 11:36:04,234 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 13.0812 s, 82.1 MB/s 2021-06-08 11:36:04,236 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_36931MB_1623177364.24 bs=1024M count=1 2021-06-08 11:36:04,236 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: dd if=/dev/zero of=/data/full_disk_36931MB_1623177364.24 bs=1024M count=1 2021-06-08 11:36:04,986 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 3.40686 s, 315 MB/s 2021-06-08 11:36:04,986 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_13384MB_1623177364.99 bs=1024M count=1 2021-06-08 11:36:04,986 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: dd if=/dev/zero of=/data/full_disk_13384MB_1623177364.99 bs=1024M count=1 2021-06-08 11:36:09,341 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 4.20858 s, 255 MB/s 2021-06-08 11:36:09,342 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_12360MB_1623177369.34 bs=1024M count=1 2021-06-08 11:36:09,342 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: dd if=/dev/zero of=/data/full_disk_12360MB_1623177369.34 bs=1024M count=1 2021-06-08 11:36:13,506 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 4.02044 s, 267 MB/s 2021-06-08 11:36:13,506 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_11336MB_1623177373.51 bs=1024M count=1 2021-06-08 11:36:13,506 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: dd if=/dev/zero of=/data/full_disk_11336MB_1623177373.51 bs=1024M count=1 2021-06-08 11:36:17,308 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 3.67027 s, 293 MB/s 2021-06-08 11:36:17,308 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_10312MB_1623177377.31 bs=1024M count=1 2021-06-08 11:36:17,308 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: dd if=/dev/zero of=/data/full_disk_10312MB_1623177377.31 bs=1024M count=1 2021-06-08 11:36:18,265 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 13.8763 s, 77.4 MB/s 2021-06-08 11:36:18,265 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_35907MB_1623177378.27 bs=1024M count=1 2021-06-08 11:36:18,265 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: dd if=/dev/zero of=/data/full_disk_35907MB_1623177378.27 bs=1024M count=1 2021-06-08 11:36:18,854 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 17.0797 s, 62.9 MB/s 2021-06-08 11:36:18,855 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_3135MB_1623177378.86 bs=1024M count=1 2021-06-08 11:36:18,855 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: dd if=/dev/zero of=/data/full_disk_3135MB_1623177378.86 bs=1024M count=1 2021-06-08 11:36:24,276 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 5.87405 s, 183 MB/s 2021-06-08 11:36:24,276 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_34883MB_1623177384.28 bs=1024M count=1 2021-06-08 11:36:24,276 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: dd if=/dev/zero of=/data/full_disk_34883MB_1623177384.28 bs=1024M count=1 2021-06-08 11:36:25,592 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 6.59359 s, 163 MB/s 2021-06-08 11:36:25,592 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_2111MB_1623177385.59 bs=1024M count=1 2021-06-08 11:36:25,592 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: dd if=/dev/zero of=/data/full_disk_2111MB_1623177385.59 bs=1024M count=1 2021-06-08 11:36:25,825 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 8.3871 s, 128 MB/s 2021-06-08 11:36:25,826 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_9288MB_1623177385.83 bs=1024M count=1 2021-06-08 11:36:25,826 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: dd if=/dev/zero of=/data/full_disk_9288MB_1623177385.83 bs=1024M count=1 2021-06-08 11:36:27,407 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 1.42677 s, 753 MB/s 2021-06-08 11:36:27,407 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_8264MB_1623177387.41 bs=1024M count=1 2021-06-08 11:36:27,407 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: dd if=/dev/zero of=/data/full_disk_8264MB_1623177387.41 bs=1024M count=1 2021-06-08 11:36:29,734 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 2.18574 s, 491 MB/s 2021-06-08 11:36:29,734 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_7240MB_1623177389.73 bs=1024M count=1 2021-06-08 11:36:29,734 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: dd if=/dev/zero of=/data/full_disk_7240MB_1623177389.73 bs=1024M count=1 2021-06-08 11:36:33,976 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 4.10742 s, 261 MB/s 2021-06-08 11:36:33,976 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_6216MB_1623177393.98 bs=1024M count=1 2021-06-08 11:36:33,977 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: dd if=/dev/zero of=/data/full_disk_6216MB_1623177393.98 bs=1024M count=1 2021-06-08 11:36:37,631 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 3.52831 s, 304 MB/s 2021-06-08 11:36:37,631 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_5192MB_1623177397.63 bs=1024M count=1 2021-06-08 11:36:37,631 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: dd if=/dev/zero of=/data/full_disk_5192MB_1623177397.63 bs=1024M count=1 2021-06-08 11:36:39,453 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 15.0474 s, 71.4 MB/s 2021-06-08 11:36:39,453 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_33859MB_1623177399.45 bs=1024M count=1 2021-06-08 11:36:39,453 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: dd if=/dev/zero of=/data/full_disk_33859MB_1623177399.45 bs=1024M count=1 2021-06-08 11:36:40,002 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 14.2667 s, 75.3 MB/s 2021-06-08 11:36:40,002 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_1087MB_1623177400.0 bs=1024M count=1 2021-06-08 11:36:40,002 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: dd if=/dev/zero of=/data/full_disk_1087MB_1623177400.0 bs=1024M count=1 2021-06-08 11:36:48,072 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 10.3109 s, 104 MB/s 2021-06-08 11:36:48,073 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_4168MB_1623177408.07 bs=1024M count=1 2021-06-08 11:36:48,073 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: dd if=/dev/zero of=/data/full_disk_4168MB_1623177408.07 bs=1024M count=1 2021-06-08 11:36:49,831 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 9.69813 s, 111 MB/s 2021-06-08 11:36:49,832 | test | DEBUG | Thread-18 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_63MB_1623177409.83 bs=63M count=1 2021-06-08 11:36:49,832 | infra | DEBUG | Thread-18 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: dd if=/dev/zero of=/data/full_disk_63MB_1623177409.83 bs=63M count=1 2021-06-08 11:36:49,927 | test | ERROR | Thread-18 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 66060288 bytes (66 MB) copied, 0.0739659 s, 893 MB/s 2021-06-08 11:36:49,927 | infra | DEBUG | Thread-18 | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.125 2021-06-08 11:36:50,657 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 11.0499 s, 97.2 MB/s 2021-06-08 11:36:50,657 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_32835MB_1623177410.66 bs=1024M count=1 2021-06-08 11:36:50,657 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: dd if=/dev/zero of=/data/full_disk_32835MB_1623177410.66 bs=1024M count=1 2021-06-08 11:36:58,565 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 7.77314 s, 138 MB/s 2021-06-08 11:36:58,566 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_31811MB_1623177418.57 bs=1024M count=1 2021-06-08 11:36:58,566 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: dd if=/dev/zero of=/data/full_disk_31811MB_1623177418.57 bs=1024M count=1 2021-06-08 11:37:00,088 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 11.8652 s, 90.5 MB/s 2021-06-08 11:37:00,088 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_3144MB_1623177420.09 bs=1024M count=1 2021-06-08 11:37:00,088 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: dd if=/dev/zero of=/data/full_disk_3144MB_1623177420.09 bs=1024M count=1 2021-06-08 11:37:02,809 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 4.08968 s, 263 MB/s 2021-06-08 11:37:02,811 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_30787MB_1623177422.81 bs=1024M count=1 2021-06-08 11:37:02,811 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: dd if=/dev/zero of=/data/full_disk_30787MB_1623177422.81 bs=1024M count=1 2021-06-08 11:37:05,510 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 2.53403 s, 424 MB/s 2021-06-08 11:37:05,512 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_29763MB_1623177425.51 bs=1024M count=1 2021-06-08 11:37:05,512 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: dd if=/dev/zero of=/data/full_disk_29763MB_1623177425.51 bs=1024M count=1 2021-06-08 11:37:14,865 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 2.96446 s, 362 MB/s 2021-06-08 11:37:14,865 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_28739MB_1623177434.87 bs=1024M count=1 2021-06-08 11:37:14,865 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: dd if=/dev/zero of=/data/full_disk_28739MB_1623177434.87 bs=1024M count=1 2021-06-08 11:37:15,440 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 15.2158 s, 70.6 MB/s 2021-06-08 11:37:15,441 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_2120MB_1623177435.44 bs=1024M count=1 2021-06-08 11:37:15,441 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: dd if=/dev/zero of=/data/full_disk_2120MB_1623177435.44 bs=1024M count=1 2021-06-08 11:37:16,342 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 1.34504 s, 798 MB/s 2021-06-08 11:37:16,342 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_27715MB_1623177436.34 bs=1024M count=1 2021-06-08 11:37:16,342 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: dd if=/dev/zero of=/data/full_disk_27715MB_1623177436.34 bs=1024M count=1 2021-06-08 11:37:19,647 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 3.15281 s, 341 MB/s 2021-06-08 11:37:19,648 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_26691MB_1623177439.65 bs=1024M count=1 2021-06-08 11:37:19,648 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: dd if=/dev/zero of=/data/full_disk_26691MB_1623177439.65 bs=1024M count=1 2021-06-08 11:37:25,056 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 5.26153 s, 204 MB/s 2021-06-08 11:37:25,056 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_25667MB_1623177445.06 bs=1024M count=1 2021-06-08 11:37:25,056 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: dd if=/dev/zero of=/data/full_disk_25667MB_1623177445.06 bs=1024M count=1 2021-06-08 11:37:28,752 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 13.1619 s, 81.6 MB/s 2021-06-08 11:37:28,752 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_1096MB_1623177448.75 bs=1024M count=1 2021-06-08 11:37:28,752 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: dd if=/dev/zero of=/data/full_disk_1096MB_1623177448.75 bs=1024M count=1 2021-06-08 11:37:30,959 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 5.76313 s, 186 MB/s 2021-06-08 11:37:30,959 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_24643MB_1623177450.96 bs=1024M count=1 2021-06-08 11:37:30,959 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: dd if=/dev/zero of=/data/full_disk_24643MB_1623177450.96 bs=1024M count=1 2021-06-08 11:37:34,832 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 5.94479 s, 181 MB/s 2021-06-08 11:37:34,832 | test | DEBUG | Thread-19 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_72MB_1623177454.83 bs=72M count=1 2021-06-08 11:37:34,832 | infra | DEBUG | Thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: dd if=/dev/zero of=/data/full_disk_72MB_1623177454.83 bs=72M count=1 2021-06-08 11:37:34,947 | test | ERROR | Thread-19 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 75497472 bytes (75 MB) copied, 0.0897647 s, 841 MB/s 2021-06-08 11:37:34,947 | infra | DEBUG | Thread-19 | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.122 2021-06-08 11:37:37,282 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 6.19107 s, 173 MB/s 2021-06-08 11:37:37,282 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_23619MB_1623177457.28 bs=1024M count=1 2021-06-08 11:37:37,282 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: dd if=/dev/zero of=/data/full_disk_23619MB_1623177457.28 bs=1024M count=1 2021-06-08 11:37:43,065 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 5.64997 s, 190 MB/s 2021-06-08 11:37:43,065 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_22595MB_1623177463.07 bs=1024M count=1 2021-06-08 11:37:43,065 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: dd if=/dev/zero of=/data/full_disk_22595MB_1623177463.07 bs=1024M count=1 2021-06-08 11:37:48,507 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 5.29091 s, 203 MB/s 2021-06-08 11:37:48,509 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_21571MB_1623177468.51 bs=1024M count=1 2021-06-08 11:37:48,509 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: dd if=/dev/zero of=/data/full_disk_21571MB_1623177468.51 bs=1024M count=1 2021-06-08 11:37:52,069 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 3.42276 s, 314 MB/s 2021-06-08 11:37:52,069 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_20547MB_1623177472.07 bs=1024M count=1 2021-06-08 11:37:52,069 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: dd if=/dev/zero of=/data/full_disk_20547MB_1623177472.07 bs=1024M count=1 2021-06-08 11:38:00,361 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 8.14319 s, 132 MB/s 2021-06-08 11:38:00,361 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_19523MB_1623177480.36 bs=1024M count=1 2021-06-08 11:38:00,361 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: dd if=/dev/zero of=/data/full_disk_19523MB_1623177480.36 bs=1024M count=1 2021-06-08 11:38:03,730 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 3.22075 s, 333 MB/s 2021-06-08 11:38:03,730 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_18499MB_1623177483.73 bs=1024M count=1 2021-06-08 11:38:03,730 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: dd if=/dev/zero of=/data/full_disk_18499MB_1623177483.73 bs=1024M count=1 2021-06-08 11:38:07,683 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 3.81345 s, 282 MB/s 2021-06-08 11:38:07,684 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_17475MB_1623177487.68 bs=1024M count=1 2021-06-08 11:38:07,684 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: dd if=/dev/zero of=/data/full_disk_17475MB_1623177487.68 bs=1024M count=1 2021-06-08 11:38:11,042 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 3.20833 s, 335 MB/s 2021-06-08 11:38:11,042 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_16451MB_1623177491.04 bs=1024M count=1 2021-06-08 11:38:11,042 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: dd if=/dev/zero of=/data/full_disk_16451MB_1623177491.04 bs=1024M count=1 2021-06-08 11:38:14,400 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 3.21952 s, 334 MB/s 2021-06-08 11:38:14,401 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_15427MB_1623177494.4 bs=1024M count=1 2021-06-08 11:38:14,401 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: dd if=/dev/zero of=/data/full_disk_15427MB_1623177494.4 bs=1024M count=1 2021-06-08 11:38:18,292 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 3.7613 s, 285 MB/s 2021-06-08 11:38:18,292 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_14403MB_1623177498.29 bs=1024M count=1 2021-06-08 11:38:18,292 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: dd if=/dev/zero of=/data/full_disk_14403MB_1623177498.29 bs=1024M count=1 2021-06-08 11:38:21,645 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 3.20115 s, 335 MB/s 2021-06-08 11:38:21,645 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_13379MB_1623177501.65 bs=1024M count=1 2021-06-08 11:38:21,645 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: dd if=/dev/zero of=/data/full_disk_13379MB_1623177501.65 bs=1024M count=1 2021-06-08 11:38:24,966 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 3.18312 s, 337 MB/s 2021-06-08 11:38:24,966 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_12355MB_1623177504.97 bs=1024M count=1 2021-06-08 11:38:24,966 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: dd if=/dev/zero of=/data/full_disk_12355MB_1623177504.97 bs=1024M count=1 2021-06-08 11:38:26,476 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 1.37773 s, 779 MB/s 2021-06-08 11:38:26,476 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_11331MB_1623177506.48 bs=1024M count=1 2021-06-08 11:38:26,476 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: dd if=/dev/zero of=/data/full_disk_11331MB_1623177506.48 bs=1024M count=1 2021-06-08 11:38:27,953 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 1.34772 s, 797 MB/s 2021-06-08 11:38:27,953 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_10307MB_1623177507.95 bs=1024M count=1 2021-06-08 11:38:27,953 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: dd if=/dev/zero of=/data/full_disk_10307MB_1623177507.95 bs=1024M count=1 2021-06-08 11:38:29,440 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 1.35609 s, 792 MB/s 2021-06-08 11:38:29,440 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_9283MB_1623177509.44 bs=1024M count=1 2021-06-08 11:38:29,440 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: dd if=/dev/zero of=/data/full_disk_9283MB_1623177509.44 bs=1024M count=1 2021-06-08 11:38:30,891 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 1.31925 s, 814 MB/s 2021-06-08 11:38:30,892 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_8259MB_1623177510.89 bs=1024M count=1 2021-06-08 11:38:30,892 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: dd if=/dev/zero of=/data/full_disk_8259MB_1623177510.89 bs=1024M count=1 2021-06-08 11:38:32,338 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 1.31666 s, 816 MB/s 2021-06-08 11:38:32,338 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_7235MB_1623177512.34 bs=1024M count=1 2021-06-08 11:38:32,338 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: dd if=/dev/zero of=/data/full_disk_7235MB_1623177512.34 bs=1024M count=1 2021-06-08 11:38:43,519 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 11.0224 s, 97.4 MB/s 2021-06-08 11:38:43,520 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_6211MB_1623177523.52 bs=1024M count=1 2021-06-08 11:38:43,520 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: dd if=/dev/zero of=/data/full_disk_6211MB_1623177523.52 bs=1024M count=1 2021-06-08 11:38:45,173 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 1.51046 s, 711 MB/s 2021-06-08 11:38:45,174 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_5187MB_1623177525.17 bs=1024M count=1 2021-06-08 11:38:45,174 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: dd if=/dev/zero of=/data/full_disk_5187MB_1623177525.17 bs=1024M count=1 2021-06-08 11:38:49,141 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 3.83709 s, 280 MB/s 2021-06-08 11:38:49,141 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_4163MB_1623177529.14 bs=1024M count=1 2021-06-08 11:38:49,142 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: dd if=/dev/zero of=/data/full_disk_4163MB_1623177529.14 bs=1024M count=1 2021-06-08 11:38:52,523 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 3.24894 s, 330 MB/s 2021-06-08 11:38:52,523 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_3139MB_1623177532.52 bs=1024M count=1 2021-06-08 11:38:52,523 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: dd if=/dev/zero of=/data/full_disk_3139MB_1623177532.52 bs=1024M count=1 2021-06-08 11:38:56,585 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 3.93178 s, 273 MB/s 2021-06-08 11:38:56,585 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_2115MB_1623177536.59 bs=1024M count=1 2021-06-08 11:38:56,585 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: dd if=/dev/zero of=/data/full_disk_2115MB_1623177536.59 bs=1024M count=1 2021-06-08 11:39:00,533 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 3.81843 s, 281 MB/s 2021-06-08 11:39:00,535 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_1091MB_1623177540.54 bs=1024M count=1 2021-06-08 11:39:00,535 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: dd if=/dev/zero of=/data/full_disk_1091MB_1623177540.54 bs=1024M count=1 2021-06-08 11:39:04,967 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB) copied, 4.29652 s, 250 MB/s 2021-06-08 11:39:04,967 | test | DEBUG | Thread-20 | [magma_disk_full:fill_disk:89] dd if=/dev/zero of=/data/full_disk_67MB_1623177544.97 bs=67M count=1 2021-06-08 11:39:04,967 | infra | DEBUG | Thread-20 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: dd if=/dev/zero of=/data/full_disk_67MB_1623177544.97 bs=67M count=1 2021-06-08 11:39:05,072 | test | ERROR | Thread-20 | [magma_disk_full:fill_disk:93] 1+0 records in 1+0 records out 70254592 bytes (70 MB) copied, 0.0819132 s, 858 MB/s 2021-06-08 11:39:05,072 | infra | DEBUG | Thread-20 | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.155 2021-06-08 11:39:05,075 | infra | INFO | MainThread | [task_manager:add_new_task:23] Added new task: LoadDocsGen_default__default__default_create_5000000_6000000_1623177545.08 2021-06-08 11:39:05,075 | infra | INFO | pool-5-thread-3 | [task:start_task:87] Thread 'LoadDocsGen_default__default__default_create_5000000_6000000_1623177545.08' started 2021-06-08 11:39:05,075 | infra | DEBUG | MainThread | [task_manager:get_task_result:26] Getting task result for LoadDocsGen_default__default__default_create_5000000_6000000_1623177545.08 2021-06-08 11:39:05,107 | infra | INFO | pool-5-thread-3 | [task_manager:add_new_task:23] Added new task: PrintBucketStats_default_1623177545.09 2021-06-08 11:39:05,107 | infra | INFO | pool-5-thread-4 | [task:start_task:87] Thread 'PrintBucketStats_default_1623177545.09' started 2021-06-08 11:39:05,108 | infra | INFO | pool-5-thread-3 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623177545.08_create__5000000_5050000_1623177545.09 2021-06-08 11:39:05,108 | infra | INFO | pool-5-thread-5 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623177545.08_create__5000000_5050000_1623177545.09' started 2021-06-08 11:39:05,108 | infra | INFO | pool-5-thread-6 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623177545.08_create__5050000_5100000_1623177545.09' started 2021-06-08 11:39:05,108 | infra | INFO | pool-5-thread-3 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623177545.08_create__5050000_5100000_1623177545.09 2021-06-08 11:39:05,108 | infra | INFO | pool-5-thread-3 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623177545.08_create__5100000_5150000_1623177545.09 2021-06-08 11:39:05,108 | infra | INFO | pool-5-thread-23 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623177545.08_create__5100000_5150000_1623177545.09' started 2021-06-08 11:39:05,108 | infra | INFO | pool-5-thread-3 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623177545.08_create__5150000_5200000_1623177545.09 2021-06-08 11:39:05,108 | infra | INFO | pool-5-thread-21 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623177545.08_create__5150000_5200000_1623177545.09' started 2021-06-08 11:39:05,108 | infra | INFO | pool-5-thread-3 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623177545.08_create__5200000_5250000_1623177545.09 2021-06-08 11:39:05,108 | infra | INFO | pool-5-thread-26 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623177545.08_create__5200000_5250000_1623177545.09' started 2021-06-08 11:39:05,108 | infra | INFO | pool-5-thread-3 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623177545.08_create__5250000_5300000_1623177545.09 2021-06-08 11:39:05,108 | infra | INFO | pool-5-thread-13 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623177545.08_create__5250000_5300000_1623177545.09' started 2021-06-08 11:39:05,108 | infra | INFO | pool-5-thread-3 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623177545.08_create__5300000_5350000_1623177545.09 2021-06-08 11:39:05,108 | infra | INFO | pool-5-thread-12 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623177545.08_create__5300000_5350000_1623177545.09' started 2021-06-08 11:39:05,108 | infra | INFO | pool-5-thread-3 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623177545.08_create__5350000_5400000_1623177545.09 2021-06-08 11:39:05,109 | infra | INFO | pool-5-thread-10 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623177545.08_create__5350000_5400000_1623177545.09' started 2021-06-08 11:39:05,109 | infra | INFO | pool-5-thread-3 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623177545.08_create__5400000_5450000_1623177545.09 2021-06-08 11:39:05,109 | infra | INFO | pool-5-thread-3 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623177545.08_create__5450000_5500000_1623177545.09 2021-06-08 11:39:05,109 | infra | INFO | pool-5-thread-16 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623177545.08_create__5400000_5450000_1623177545.09' started 2021-06-08 11:39:05,109 | infra | INFO | pool-5-thread-28 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623177545.08_create__5450000_5500000_1623177545.09' started 2021-06-08 11:39:05,109 | infra | INFO | pool-5-thread-3 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623177545.08_create__5500000_5550000_1623177545.09 2021-06-08 11:39:05,109 | infra | INFO | pool-5-thread-24 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623177545.08_create__5500000_5550000_1623177545.09' started 2021-06-08 11:39:05,111 | infra | INFO | pool-5-thread-3 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623177545.08_create__5550000_5600000_1623177545.09 2021-06-08 11:39:05,111 | infra | INFO | pool-5-thread-9 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623177545.08_create__5550000_5600000_1623177545.09' started 2021-06-08 11:39:05,111 | infra | INFO | pool-5-thread-3 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623177545.08_create__5600000_5650000_1623177545.09 2021-06-08 11:39:05,111 | infra | INFO | pool-5-thread-17 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623177545.08_create__5600000_5650000_1623177545.09' started 2021-06-08 11:39:05,111 | infra | INFO | pool-5-thread-3 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623177545.08_create__5650000_5700000_1623177545.09 2021-06-08 11:39:05,112 | infra | INFO | pool-5-thread-3 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623177545.08_create__5700000_5750000_1623177545.09 2021-06-08 11:39:05,112 | infra | INFO | pool-5-thread-18 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623177545.08_create__5700000_5750000_1623177545.09' started 2021-06-08 11:39:05,112 | infra | INFO | pool-5-thread-3 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623177545.08_create__5750000_5800000_1623177545.09 2021-06-08 11:39:05,112 | infra | INFO | pool-5-thread-3 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623177545.08_create__5800000_5850000_1623177545.09 2021-06-08 11:39:05,112 | infra | INFO | pool-5-thread-11 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623177545.08_create__5800000_5850000_1623177545.09' started 2021-06-08 11:39:05,112 | infra | INFO | pool-5-thread-19 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623177545.08_create__5850000_5900000_1623177545.09' started 2021-06-08 11:39:05,112 | infra | INFO | pool-5-thread-25 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623177545.08_create__5650000_5700000_1623177545.09' started 2021-06-08 11:39:05,112 | infra | INFO | pool-5-thread-20 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623177545.08_create__5750000_5800000_1623177545.09' started 2021-06-08 11:39:05,112 | infra | INFO | pool-5-thread-3 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623177545.08_create__5850000_5900000_1623177545.09 2021-06-08 11:39:05,114 | infra | INFO | pool-5-thread-3 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623177545.08_create__5900000_5950000_1623177545.09 2021-06-08 11:39:05,114 | infra | INFO | pool-5-thread-3 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623177545.08_create__5950000_6000000_1623177545.09 2021-06-08 11:39:05,114 | infra | INFO | pool-5-thread-14 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623177545.08_create__5950000_6000000_1623177545.09' started 2021-06-08 11:39:05,115 | infra | DEBUG | pool-5-thread-3 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623177545.08_create__5000000_5050000_1623177545.09 2021-06-08 11:39:05,114 | infra | INFO | pool-5-thread-27 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623177545.08_create__5900000_5950000_1623177545.09' started 2021-06-08 11:39:05,359 | test | DEBUG | pool-5-thread-4 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:39:05,359 | test | DEBUG | pool-5-thread-4 | [task:print_ep_queue_size:3508] ep_queue_size for default: 0 2021-06-08 11:39:05,361 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:39:05,361 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:39:06,595 | test | DEBUG | pool-5-thread-4 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:39:06,596 | test | DEBUG | pool-5-thread-4 | [task:print_ep_queue_size:3508] ep_queue_size for default: 0 2021-06-08 11:39:06,598 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:39:06,598 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:39:07,825 | test | DEBUG | pool-5-thread-4 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:39:07,825 | test | DEBUG | pool-5-thread-4 | [task:print_ep_queue_size:3508] ep_queue_size for default: 0 2021-06-08 11:39:07,825 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:39:07,825 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:39:09,043 | test | DEBUG | pool-5-thread-4 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:39:09,043 | test | DEBUG | pool-5-thread-4 | [task:print_ep_queue_size:3508] ep_queue_size for default: 0 2021-06-08 11:39:09,046 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:39:09,046 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:39:10,282 | test | DEBUG | pool-5-thread-4 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:39:10,282 | test | DEBUG | pool-5-thread-4 | [task:print_ep_queue_size:3508] ep_queue_size for default: 0 2021-06-08 11:39:10,285 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:39:10,285 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:39:11,506 | test | DEBUG | pool-5-thread-4 | [task:record_bucket_ops:3495] Ops rate for 'default': 2302.400000 2021-06-08 11:39:11,506 | test | DEBUG | pool-5-thread-4 | [task:print_ep_queue_size:3508] ep_queue_size for default: 670 2021-06-08 11:39:11,507 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:39:11,507 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:39:12,734 | test | DEBUG | pool-5-thread-4 | [task:record_bucket_ops:3495] Ops rate for 'default': 2302.400000 2021-06-08 11:39:12,734 | test | DEBUG | pool-5-thread-4 | [task:print_ep_queue_size:3508] ep_queue_size for default: 670 2021-06-08 11:39:12,736 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:39:12,736 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:39:13,973 | test | DEBUG | pool-5-thread-4 | [task:record_bucket_ops:3495] Ops rate for 'default': 2302.400000 2021-06-08 11:39:13,974 | test | DEBUG | pool-5-thread-4 | [task:print_ep_queue_size:3508] ep_queue_size for default: 670 2021-06-08 11:39:13,976 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:39:13,976 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:39:15,194 | test | DEBUG | pool-5-thread-4 | [task:record_bucket_ops:3495] Ops rate for 'default': 2302.400000 2021-06-08 11:39:15,194 | test | DEBUG | pool-5-thread-4 | [task:print_ep_queue_size:3508] ep_queue_size for default: 670 2021-06-08 11:39:15,196 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:39:15,196 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:39:16,413 | test | DEBUG | pool-5-thread-4 | [task:record_bucket_ops:3495] Ops rate for 'default': 2302.400000 2021-06-08 11:39:16,414 | test | DEBUG | pool-5-thread-4 | [task:print_ep_queue_size:3508] ep_queue_size for default: 670 2021-06-08 11:39:16,414 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:39:16,414 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:39:17,628 | test | DEBUG | pool-5-thread-4 | [task:record_bucket_ops:3495] Ops rate for 'default': 2302.400000 2021-06-08 11:39:17,628 | test | DEBUG | pool-5-thread-4 | [task:print_ep_queue_size:3508] ep_queue_size for default: 670 2021-06-08 11:39:17,631 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:39:17,631 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:39:18,841 | test | DEBUG | pool-5-thread-4 | [task:record_bucket_ops:3495] Ops rate for 'default': 2302.400000 2021-06-08 11:39:18,841 | test | DEBUG | pool-5-thread-4 | [task:print_ep_queue_size:3508] ep_queue_size for default: 670 2021-06-08 11:39:18,842 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:39:18,842 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:39:20,048 | test | DEBUG | pool-5-thread-4 | [task:record_bucket_ops:3495] Ops rate for 'default': 2302.400000 2021-06-08 11:39:20,048 | test | DEBUG | pool-5-thread-4 | [task:print_ep_queue_size:3508] ep_queue_size for default: 670 2021-06-08 11:39:20,049 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:39:20,049 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:39:21,259 | test | DEBUG | pool-5-thread-4 | [task:record_bucket_ops:3495] Ops rate for 'default': 697.600000 2021-06-08 11:39:21,259 | test | DEBUG | pool-5-thread-4 | [task:print_ep_queue_size:3508] ep_queue_size for default: 6049 2021-06-08 11:39:21,260 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:39:21,260 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:39:22,482 | test | DEBUG | pool-5-thread-4 | [task:record_bucket_ops:3495] Ops rate for 'default': 697.600000 2021-06-08 11:39:22,482 | test | DEBUG | pool-5-thread-4 | [task:print_ep_queue_size:3508] ep_queue_size for default: 6049 2021-06-08 11:39:22,483 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:39:22,483 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:39:23,697 | test | DEBUG | pool-5-thread-4 | [task:record_bucket_ops:3495] Ops rate for 'default': 697.600000 2021-06-08 11:39:23,697 | test | DEBUG | pool-5-thread-4 | [task:print_ep_queue_size:3508] ep_queue_size for default: 6049 2021-06-08 11:39:23,697 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:39:23,697 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:39:24,921 | test | DEBUG | pool-5-thread-4 | [task:record_bucket_ops:3495] Ops rate for 'default': 697.600000 2021-06-08 11:39:24,921 | test | DEBUG | pool-5-thread-4 | [task:print_ep_queue_size:3508] ep_queue_size for default: 6049 2021-06-08 11:39:24,921 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:39:24,923 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:39:26,132 | test | DEBUG | pool-5-thread-4 | [task:record_bucket_ops:3495] Ops rate for 'default': 697.600000 2021-06-08 11:39:26,132 | test | DEBUG | pool-5-thread-4 | [task:print_ep_queue_size:3508] ep_queue_size for default: 6049 2021-06-08 11:39:26,134 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:39:26,134 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:39:27,345 | test | DEBUG | pool-5-thread-4 | [task:record_bucket_ops:3495] Ops rate for 'default': 697.600000 2021-06-08 11:39:27,345 | test | DEBUG | pool-5-thread-4 | [task:print_ep_queue_size:3508] ep_queue_size for default: 6049 2021-06-08 11:39:27,346 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:39:27,346 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:39:28,568 | test | DEBUG | pool-5-thread-4 | [task:record_bucket_ops:3495] Ops rate for 'default': 697.600000 2021-06-08 11:39:28,569 | test | DEBUG | pool-5-thread-4 | [task:print_ep_queue_size:3508] ep_queue_size for default: 6049 2021-06-08 11:39:28,569 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:39:28,569 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:39:29,776 | test | DEBUG | pool-5-thread-4 | [task:record_bucket_ops:3495] Ops rate for 'default': 697.600000 2021-06-08 11:39:29,776 | test | DEBUG | pool-5-thread-4 | [task:print_ep_queue_size:3508] ep_queue_size for default: 6049 2021-06-08 11:39:29,779 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:39:29,779 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:39:30,987 | test | DEBUG | pool-5-thread-4 | [task:record_bucket_ops:3495] Ops rate for 'default': 697.600000 2021-06-08 11:39:30,989 | test | DEBUG | pool-5-thread-4 | [task:print_ep_queue_size:3508] ep_queue_size for default: 6049 2021-06-08 11:39:30,990 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:39:30,990 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:39:32,209 | test | DEBUG | pool-5-thread-4 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:39:32,209 | test | DEBUG | pool-5-thread-4 | [task:print_ep_queue_size:3508] ep_queue_size for default: 6023 2021-06-08 11:39:32,210 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:39:32,210 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:39:33,427 | test | DEBUG | pool-5-thread-4 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:39:33,427 | test | DEBUG | pool-5-thread-4 | [task:print_ep_queue_size:3508] ep_queue_size for default: 6023 2021-06-08 11:39:33,428 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:39:33,428 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:39:34,648 | test | DEBUG | pool-5-thread-4 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:39:34,648 | test | DEBUG | pool-5-thread-4 | [task:print_ep_queue_size:3508] ep_queue_size for default: 6023 2021-06-08 11:39:34,650 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:39:34,650 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:39:35,865 | test | DEBUG | pool-5-thread-4 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:39:35,867 | test | DEBUG | pool-5-thread-4 | [task:print_ep_queue_size:3508] ep_queue_size for default: 6023 2021-06-08 11:39:35,868 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:39:35,868 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:39:37,082 | test | DEBUG | pool-5-thread-4 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:39:37,082 | test | DEBUG | pool-5-thread-4 | [task:print_ep_queue_size:3508] ep_queue_size for default: 6023 2021-06-08 11:39:37,084 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:39:37,084 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:39:38,296 | test | DEBUG | pool-5-thread-4 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:39:38,296 | test | DEBUG | pool-5-thread-4 | [task:print_ep_queue_size:3508] ep_queue_size for default: 6023 2021-06-08 11:39:38,299 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:39:38,299 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:39:39,513 | test | DEBUG | pool-5-thread-4 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:39:39,513 | test | DEBUG | pool-5-thread-4 | [task:print_ep_queue_size:3508] ep_queue_size for default: 6023 2021-06-08 11:39:39,513 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:39:39,515 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:39:40,736 | test | DEBUG | pool-5-thread-4 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:39:40,737 | test | DEBUG | pool-5-thread-4 | [task:print_ep_queue_size:3508] ep_queue_size for default: 6023 2021-06-08 11:39:40,739 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:39:40,739 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:39:41,953 | test | DEBUG | pool-5-thread-4 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:39:41,953 | test | DEBUG | pool-5-thread-4 | [task:print_ep_queue_size:3508] ep_queue_size for default: 6023 2021-06-08 11:39:41,956 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:39:41,956 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:39:43,184 | test | DEBUG | pool-5-thread-4 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:39:43,184 | test | DEBUG | pool-5-thread-4 | [task:print_ep_queue_size:3508] ep_queue_size for default: 6023 2021-06-08 11:39:43,186 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:39:43,186 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:39:44,394 | test | DEBUG | pool-5-thread-4 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:39:44,394 | test | DEBUG | pool-5-thread-4 | [task:print_ep_queue_size:3508] ep_queue_size for default: 6023 2021-06-08 11:39:44,394 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:39:44,395 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:39:45,608 | test | DEBUG | pool-5-thread-4 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:39:45,609 | test | DEBUG | pool-5-thread-4 | [task:print_ep_queue_size:3508] ep_queue_size for default: 6023 2021-06-08 11:39:45,611 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:39:45,611 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:39:46,832 | test | DEBUG | pool-5-thread-4 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:39:46,832 | test | DEBUG | pool-5-thread-4 | [task:print_ep_queue_size:3508] ep_queue_size for default: 6023 2021-06-08 11:39:46,834 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:39:46,834 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:39:48,059 | test | DEBUG | pool-5-thread-4 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:39:48,059 | test | DEBUG | pool-5-thread-4 | [task:print_ep_queue_size:3508] ep_queue_size for default: 6023 2021-06-08 11:39:48,059 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:39:48,061 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:39:49,275 | test | DEBUG | pool-5-thread-4 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:39:49,275 | test | DEBUG | pool-5-thread-4 | [task:print_ep_queue_size:3508] ep_queue_size for default: 6023 2021-06-08 11:39:49,276 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:39:49,276 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:39:50,492 | test | DEBUG | pool-5-thread-4 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:39:50,492 | test | DEBUG | pool-5-thread-4 | [task:print_ep_queue_size:3508] ep_queue_size for default: 6023 2021-06-08 11:39:50,493 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:39:50,493 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:39:51,710 | test | DEBUG | pool-5-thread-4 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:39:51,711 | test | DEBUG | pool-5-thread-4 | [task:print_ep_queue_size:3508] ep_queue_size for default: 6023 2021-06-08 11:39:51,713 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:39:51,713 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:39:52,937 | test | DEBUG | pool-5-thread-4 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:39:52,937 | test | DEBUG | pool-5-thread-4 | [task:print_ep_queue_size:3508] ep_queue_size for default: 6023 2021-06-08 11:39:52,938 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:39:52,938 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:39:54,154 | test | DEBUG | pool-5-thread-4 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:39:54,155 | test | DEBUG | pool-5-thread-4 | [task:print_ep_queue_size:3508] ep_queue_size for default: 6023 2021-06-08 11:39:54,157 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:39:54,157 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:39:55,369 | test | DEBUG | pool-5-thread-4 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:39:55,371 | test | DEBUG | pool-5-thread-4 | [task:print_ep_queue_size:3508] ep_queue_size for default: 6023 2021-06-08 11:39:55,371 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:39:55,371 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:39:56,589 | test | DEBUG | pool-5-thread-4 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:39:56,591 | test | DEBUG | pool-5-thread-4 | [task:print_ep_queue_size:3508] ep_queue_size for default: 6023 2021-06-08 11:39:56,592 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:39:56,592 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:39:57,813 | test | DEBUG | pool-5-thread-4 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:39:57,815 | test | DEBUG | pool-5-thread-4 | [task:print_ep_queue_size:3508] ep_queue_size for default: 6023 2021-06-08 11:39:57,815 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:39:57,815 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:39:59,035 | test | DEBUG | pool-5-thread-4 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:39:59,035 | test | DEBUG | pool-5-thread-4 | [task:print_ep_queue_size:3508] ep_queue_size for default: 6023 2021-06-08 11:39:59,036 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:39:59,036 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:40:00,256 | test | DEBUG | pool-5-thread-4 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:40:00,256 | test | DEBUG | pool-5-thread-4 | [task:print_ep_queue_size:3508] ep_queue_size for default: 6023 2021-06-08 11:40:00,259 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:40:00,259 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:40:01,473 | test | DEBUG | pool-5-thread-4 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:40:01,473 | test | DEBUG | pool-5-thread-4 | [task:print_ep_queue_size:3508] ep_queue_size for default: 6023 2021-06-08 11:40:01,474 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:40:01,474 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:40:02,690 | test | DEBUG | pool-5-thread-4 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:40:02,690 | test | DEBUG | pool-5-thread-4 | [task:print_ep_queue_size:3508] ep_queue_size for default: 6023 2021-06-08 11:40:02,691 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:40:02,691 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:40:03,905 | test | DEBUG | pool-5-thread-4 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:40:03,905 | test | DEBUG | pool-5-thread-4 | [task:print_ep_queue_size:3508] ep_queue_size for default: 6023 2021-06-08 11:40:03,907 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:40:03,907 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:40:05,121 | test | DEBUG | pool-5-thread-4 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:40:05,122 | test | DEBUG | pool-5-thread-4 | [task:print_ep_queue_size:3508] ep_queue_size for default: 6023 2021-06-08 11:40:05,124 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:40:05,124 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:40:06,335 | test | DEBUG | pool-5-thread-4 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:40:06,335 | test | DEBUG | pool-5-thread-4 | [task:print_ep_queue_size:3508] ep_queue_size for default: 6023 2021-06-08 11:40:06,336 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:40:06,338 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:40:07,551 | test | DEBUG | pool-5-thread-4 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:40:07,551 | test | DEBUG | pool-5-thread-4 | [task:print_ep_queue_size:3508] ep_queue_size for default: 6023 2021-06-08 11:40:07,552 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:40:07,552 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:40:08,793 | test | DEBUG | pool-5-thread-4 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:40:08,793 | test | DEBUG | pool-5-thread-4 | [task:print_ep_queue_size:3508] ep_queue_size for default: 6023 2021-06-08 11:40:08,796 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:40:08,796 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:40:10,016 | test | DEBUG | pool-5-thread-4 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:40:10,017 | test | DEBUG | pool-5-thread-4 | [task:print_ep_queue_size:3508] ep_queue_size for default: 6023 2021-06-08 11:40:10,019 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:40:10,019 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:40:11,237 | test | DEBUG | pool-5-thread-4 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:40:11,237 | test | DEBUG | pool-5-thread-4 | [task:print_ep_queue_size:3508] ep_queue_size for default: 6015 2021-06-08 11:40:11,240 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:40:11,240 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:40:12,467 | test | DEBUG | pool-5-thread-4 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:40:12,467 | test | DEBUG | pool-5-thread-4 | [task:print_ep_queue_size:3508] ep_queue_size for default: 6015 2021-06-08 11:40:12,467 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:40:12,469 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:40:13,703 | test | DEBUG | pool-5-thread-4 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:40:13,703 | test | DEBUG | pool-5-thread-4 | [task:print_ep_queue_size:3508] ep_queue_size for default: 6015 2021-06-08 11:40:13,704 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:40:13,706 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:40:14,937 | test | DEBUG | pool-5-thread-4 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:40:14,937 | test | DEBUG | pool-5-thread-4 | [task:print_ep_queue_size:3508] ep_queue_size for default: 6015 2021-06-08 11:40:14,937 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:40:14,938 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:40:16,181 | test | DEBUG | pool-5-thread-4 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:40:16,183 | test | DEBUG | pool-5-thread-4 | [task:print_ep_queue_size:3508] ep_queue_size for default: 6015 2021-06-08 11:40:16,184 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:40:16,184 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:40:17,401 | test | DEBUG | pool-5-thread-4 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:40:17,403 | test | DEBUG | pool-5-thread-4 | [task:print_ep_queue_size:3508] ep_queue_size for default: 6015 2021-06-08 11:40:17,404 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:40:17,404 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:40:18,638 | test | DEBUG | pool-5-thread-4 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:40:18,640 | test | DEBUG | pool-5-thread-4 | [task:print_ep_queue_size:3508] ep_queue_size for default: 6015 2021-06-08 11:40:18,641 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:40:18,641 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:40:19,924 | test | DEBUG | pool-5-thread-4 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:40:19,926 | test | DEBUG | pool-5-thread-4 | [task:print_ep_queue_size:3508] ep_queue_size for default: 6015 2021-06-08 11:40:19,927 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:40:19,927 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:40:21,201 | test | DEBUG | pool-5-thread-4 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:40:21,203 | test | DEBUG | pool-5-thread-4 | [task:print_ep_queue_size:3508] ep_queue_size for default: 6015 2021-06-08 11:40:21,204 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:40:21,204 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:40:22,569 | test | DEBUG | pool-5-thread-4 | [task:record_bucket_ops:3495] Ops rate for 'default': 8871.100000 2021-06-08 11:40:22,569 | test | DEBUG | pool-5-thread-4 | [task:print_ep_queue_size:3508] ep_queue_size for default: 229864 2021-06-08 11:40:22,572 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:40:22,573 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:40:23,831 | test | DEBUG | pool-5-thread-4 | [task:record_bucket_ops:3495] Ops rate for 'default': 8871.100000 2021-06-08 11:40:23,832 | test | DEBUG | pool-5-thread-4 | [task:print_ep_queue_size:3508] ep_queue_size for default: 229864 2021-06-08 11:40:23,832 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:40:23,832 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:40:25,049 | test | DEBUG | pool-5-thread-4 | [task:record_bucket_ops:3495] Ops rate for 'default': 8871.100000 2021-06-08 11:40:25,049 | test | DEBUG | pool-5-thread-4 | [task:print_ep_queue_size:3508] ep_queue_size for default: 229864 2021-06-08 11:40:25,051 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:40:25,051 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:40:26,272 | test | DEBUG | pool-5-thread-4 | [task:record_bucket_ops:3495] Ops rate for 'default': 8871.100000 2021-06-08 11:40:26,272 | test | DEBUG | pool-5-thread-4 | [task:print_ep_queue_size:3508] ep_queue_size for default: 229864 2021-06-08 11:40:26,273 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:40:26,273 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:40:27,499 | test | DEBUG | pool-5-thread-4 | [task:record_bucket_ops:3495] Ops rate for 'default': 8871.100000 2021-06-08 11:40:27,499 | test | DEBUG | pool-5-thread-4 | [task:print_ep_queue_size:3508] ep_queue_size for default: 229864 2021-06-08 11:40:27,500 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:40:27,500 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:40:28,717 | test | DEBUG | pool-5-thread-4 | [task:record_bucket_ops:3495] Ops rate for 'default': 8871.100000 2021-06-08 11:40:28,717 | test | DEBUG | pool-5-thread-4 | [task:print_ep_queue_size:3508] ep_queue_size for default: 229864 2021-06-08 11:40:28,720 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:40:28,720 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:40:29,938 | test | DEBUG | pool-5-thread-4 | [task:record_bucket_ops:3495] Ops rate for 'default': 8871.100000 2021-06-08 11:40:29,938 | test | DEBUG | pool-5-thread-4 | [task:print_ep_queue_size:3508] ep_queue_size for default: 229864 2021-06-08 11:40:29,940 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:40:29,940 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:40:31,157 | test | DEBUG | pool-5-thread-4 | [task:record_bucket_ops:3495] Ops rate for 'default': 8871.100000 2021-06-08 11:40:31,157 | test | DEBUG | pool-5-thread-4 | [task:print_ep_queue_size:3508] ep_queue_size for default: 229864 2021-06-08 11:40:31,160 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:40:31,160 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:40:32,378 | test | DEBUG | pool-5-thread-4 | [task:record_bucket_ops:3495] Ops rate for 'default': 12728.300000 2021-06-08 11:40:32,378 | test | DEBUG | pool-5-thread-4 | [task:print_ep_queue_size:3508] ep_queue_size for default: 785115 2021-06-08 11:40:32,380 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:40:32,380 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:40:33,601 | test | DEBUG | pool-5-thread-4 | [task:record_bucket_ops:3495] Ops rate for 'default': 12728.300000 2021-06-08 11:40:33,601 | test | DEBUG | pool-5-thread-4 | [task:print_ep_queue_size:3508] ep_queue_size for default: 785115 2021-06-08 11:40:33,602 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:40:33,604 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:40:34,822 | test | DEBUG | pool-5-thread-4 | [task:record_bucket_ops:3495] Ops rate for 'default': 12728.300000 2021-06-08 11:40:34,822 | test | DEBUG | pool-5-thread-4 | [task:print_ep_queue_size:3508] ep_queue_size for default: 785115 2021-06-08 11:40:34,825 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:40:34,825 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:40:36,045 | test | DEBUG | pool-5-thread-4 | [task:record_bucket_ops:3495] Ops rate for 'default': 12728.300000 2021-06-08 11:40:36,045 | test | DEBUG | pool-5-thread-4 | [task:print_ep_queue_size:3508] ep_queue_size for default: 785115 2021-06-08 11:40:36,046 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:40:36,046 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:40:37,257 | test | DEBUG | pool-5-thread-4 | [task:record_bucket_ops:3495] Ops rate for 'default': 12728.300000 2021-06-08 11:40:37,259 | test | DEBUG | pool-5-thread-4 | [task:print_ep_queue_size:3508] ep_queue_size for default: 785115 2021-06-08 11:40:37,259 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:40:37,259 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:40:38,476 | test | DEBUG | pool-5-thread-4 | [task:record_bucket_ops:3495] Ops rate for 'default': 12728.300000 2021-06-08 11:40:38,477 | test | DEBUG | pool-5-thread-4 | [task:print_ep_queue_size:3508] ep_queue_size for default: 785115 2021-06-08 11:40:38,479 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:40:38,479 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:40:39,696 | test | DEBUG | pool-5-thread-4 | [task:record_bucket_ops:3495] Ops rate for 'default': 12728.300000 2021-06-08 11:40:39,696 | test | DEBUG | pool-5-thread-4 | [task:print_ep_queue_size:3508] ep_queue_size for default: 785115 2021-06-08 11:40:39,697 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:40:39,697 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:40:40,917 | test | DEBUG | pool-5-thread-4 | [task:record_bucket_ops:3495] Ops rate for 'default': 12728.300000 2021-06-08 11:40:40,917 | test | DEBUG | pool-5-thread-4 | [task:print_ep_queue_size:3508] ep_queue_size for default: 785115 2021-06-08 11:40:40,918 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:40:40,918 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:40:42,144 | test | DEBUG | pool-5-thread-4 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:40:42,144 | test | DEBUG | pool-5-thread-4 | [task:print_ep_queue_size:3508] ep_queue_size for default: 784681 2021-06-08 11:40:42,145 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:40:42,145 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:40:43,368 | test | DEBUG | pool-5-thread-4 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:40:43,368 | test | DEBUG | pool-5-thread-4 | [task:print_ep_queue_size:3508] ep_queue_size for default: 784681 2021-06-08 11:40:43,368 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:40:43,369 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:40:44,585 | test | DEBUG | pool-5-thread-4 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:40:44,585 | test | DEBUG | pool-5-thread-4 | [task:print_ep_queue_size:3508] ep_queue_size for default: 784681 2021-06-08 11:40:44,588 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:40:44,588 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:40:45,806 | test | DEBUG | pool-5-thread-4 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:40:45,806 | test | DEBUG | pool-5-thread-4 | [task:print_ep_queue_size:3508] ep_queue_size for default: 784681 2021-06-08 11:40:45,809 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:40:45,809 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:40:47,029 | test | DEBUG | pool-5-thread-4 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:40:47,029 | test | DEBUG | pool-5-thread-4 | [task:print_ep_queue_size:3508] ep_queue_size for default: 784681 2021-06-08 11:40:47,030 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:40:47,030 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:40:48,256 | test | DEBUG | pool-5-thread-4 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:40:48,257 | test | DEBUG | pool-5-thread-4 | [task:print_ep_queue_size:3508] ep_queue_size for default: 784681 2021-06-08 11:40:48,259 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:40:48,259 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:40:49,479 | test | DEBUG | pool-5-thread-4 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:40:49,479 | test | DEBUG | pool-5-thread-4 | [task:print_ep_queue_size:3508] ep_queue_size for default: 784681 2021-06-08 11:40:49,480 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:40:49,480 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:40:50,701 | test | DEBUG | pool-5-thread-4 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:40:50,701 | test | DEBUG | pool-5-thread-4 | [task:print_ep_queue_size:3508] ep_queue_size for default: 784681 2021-06-08 11:40:50,703 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:40:50,703 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:40:51,926 | test | DEBUG | pool-5-thread-4 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:40:51,926 | test | DEBUG | pool-5-thread-4 | [task:print_ep_queue_size:3508] ep_queue_size for default: 784681 2021-06-08 11:40:51,927 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:40:51,927 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:40:53,147 | test | DEBUG | pool-5-thread-4 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:40:53,147 | test | DEBUG | pool-5-thread-4 | [task:print_ep_queue_size:3508] ep_queue_size for default: 784681 2021-06-08 11:40:53,148 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:40:53,148 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:40:54,375 | test | DEBUG | pool-5-thread-4 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:40:54,375 | test | DEBUG | pool-5-thread-4 | [task:print_ep_queue_size:3508] ep_queue_size for default: 784681 2021-06-08 11:40:54,377 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:40:54,377 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:40:55,596 | test | DEBUG | pool-5-thread-4 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:40:55,596 | test | DEBUG | pool-5-thread-4 | [task:print_ep_queue_size:3508] ep_queue_size for default: 784681 2021-06-08 11:40:55,598 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:40:55,598 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:40:56,815 | test | DEBUG | pool-5-thread-4 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:40:56,815 | test | DEBUG | pool-5-thread-4 | [task:print_ep_queue_size:3508] ep_queue_size for default: 784681 2021-06-08 11:40:56,818 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:40:56,818 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:40:58,033 | test | DEBUG | pool-5-thread-4 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:40:58,033 | test | DEBUG | pool-5-thread-4 | [task:print_ep_queue_size:3508] ep_queue_size for default: 784681 2021-06-08 11:40:58,035 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:40:58,036 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:40:59,252 | test | DEBUG | pool-5-thread-4 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:40:59,252 | test | DEBUG | pool-5-thread-4 | [task:print_ep_queue_size:3508] ep_queue_size for default: 784681 2021-06-08 11:40:59,253 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:40:59,253 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:41:00,473 | test | DEBUG | pool-5-thread-4 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:41:00,474 | test | DEBUG | pool-5-thread-4 | [task:print_ep_queue_size:3508] ep_queue_size for default: 784681 2021-06-08 11:41:00,476 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:41:00,476 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:41:01,696 | test | DEBUG | pool-5-thread-4 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:41:01,696 | test | DEBUG | pool-5-thread-4 | [task:print_ep_queue_size:3508] ep_queue_size for default: 784681 2021-06-08 11:41:01,697 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:41:01,697 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:41:02,915 | test | DEBUG | pool-5-thread-4 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:41:02,915 | test | DEBUG | pool-5-thread-4 | [task:print_ep_queue_size:3508] ep_queue_size for default: 784681 2021-06-08 11:41:02,917 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:41:02,917 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:41:04,138 | test | DEBUG | pool-5-thread-4 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:41:04,140 | test | DEBUG | pool-5-thread-4 | [task:print_ep_queue_size:3508] ep_queue_size for default: 784681 2021-06-08 11:41:04,141 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:41:04,141 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:41:05,349 | test | DEBUG | pool-5-thread-4 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:41:05,351 | test | DEBUG | pool-5-thread-4 | [task:print_ep_queue_size:3508] ep_queue_size for default: 784681 2021-06-08 11:41:05,351 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:41:05,351 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:41:06,569 | test | DEBUG | pool-5-thread-4 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:41:06,569 | test | DEBUG | pool-5-thread-4 | [task:print_ep_queue_size:3508] ep_queue_size for default: 784681 2021-06-08 11:41:06,571 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:41:06,571 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:41:07,786 | test | DEBUG | pool-5-thread-4 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:41:07,786 | test | DEBUG | pool-5-thread-4 | [task:print_ep_queue_size:3508] ep_queue_size for default: 784681 2021-06-08 11:41:07,789 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:41:07,789 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:41:09,006 | test | DEBUG | pool-5-thread-4 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:41:09,006 | test | DEBUG | pool-5-thread-4 | [task:print_ep_queue_size:3508] ep_queue_size for default: 784681 2021-06-08 11:41:09,007 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:41:09,009 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:41:10,226 | test | DEBUG | pool-5-thread-4 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:41:10,226 | test | DEBUG | pool-5-thread-4 | [task:print_ep_queue_size:3508] ep_queue_size for default: 784681 2021-06-08 11:41:10,226 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:41:10,227 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:41:11,447 | test | DEBUG | pool-5-thread-4 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:41:11,447 | test | DEBUG | pool-5-thread-4 | [task:print_ep_queue_size:3508] ep_queue_size for default: 784681 2021-06-08 11:41:11,447 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:41:11,448 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:41:12,665 | test | DEBUG | pool-5-thread-4 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:41:12,665 | test | DEBUG | pool-5-thread-4 | [task:print_ep_queue_size:3508] ep_queue_size for default: 784681 2021-06-08 11:41:12,667 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:41:12,667 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:41:13,884 | test | DEBUG | pool-5-thread-4 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:41:13,884 | test | DEBUG | pool-5-thread-4 | [task:print_ep_queue_size:3508] ep_queue_size for default: 784681 2021-06-08 11:41:13,885 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:41:13,885 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:41:15,117 | test | DEBUG | pool-5-thread-4 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:41:15,117 | test | DEBUG | pool-5-thread-4 | [task:print_ep_queue_size:3508] ep_queue_size for default: 784681 2021-06-08 11:41:15,118 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:41:15,118 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:41:16,345 | test | DEBUG | pool-5-thread-4 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:41:16,345 | test | DEBUG | pool-5-thread-4 | [task:print_ep_queue_size:3508] ep_queue_size for default: 784681 2021-06-08 11:41:16,346 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:41:16,346 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:41:17,569 | test | DEBUG | pool-5-thread-4 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:41:17,569 | test | DEBUG | pool-5-thread-4 | [task:print_ep_queue_size:3508] ep_queue_size for default: 784681 2021-06-08 11:41:17,571 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:41:17,572 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:41:18,796 | test | DEBUG | pool-5-thread-4 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:41:18,796 | test | DEBUG | pool-5-thread-4 | [task:print_ep_queue_size:3508] ep_queue_size for default: 784681 2021-06-08 11:41:18,799 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:41:18,799 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:41:20,019 | test | DEBUG | pool-5-thread-4 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:41:20,020 | test | DEBUG | pool-5-thread-4 | [task:print_ep_queue_size:3508] ep_queue_size for default: 784681 2021-06-08 11:41:20,022 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:41:20,022 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:41:21,247 | test | DEBUG | pool-5-thread-4 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:41:21,247 | test | DEBUG | pool-5-thread-4 | [task:print_ep_queue_size:3508] ep_queue_size for default: 784681 2021-06-08 11:41:21,250 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:41:21,250 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:41:22,471 | test | DEBUG | pool-5-thread-4 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:41:22,471 | test | DEBUG | pool-5-thread-4 | [task:print_ep_queue_size:3508] ep_queue_size for default: 784681 2021-06-08 11:41:22,473 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:41:22,473 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:41:23,815 | test | DEBUG | pool-5-thread-4 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:41:23,815 | test | DEBUG | pool-5-thread-4 | [task:print_ep_queue_size:3508] ep_queue_size for default: 784681 2021-06-08 11:41:23,818 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:41:23,818 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:41:25,053 | test | DEBUG | pool-5-thread-4 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:41:25,055 | test | DEBUG | pool-5-thread-4 | [task:print_ep_queue_size:3508] ep_queue_size for default: 784681 2021-06-08 11:41:25,055 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:41:25,056 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:41:26,280 | test | DEBUG | pool-5-thread-4 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:41:26,282 | test | DEBUG | pool-5-thread-4 | [task:print_ep_queue_size:3508] ep_queue_size for default: 784681 2021-06-08 11:41:26,282 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:41:26,282 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:41:27,506 | test | DEBUG | pool-5-thread-4 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:41:27,507 | test | DEBUG | pool-5-thread-4 | [task:print_ep_queue_size:3508] ep_queue_size for default: 784681 2021-06-08 11:41:27,509 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:41:27,509 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:41:28,736 | test | DEBUG | pool-5-thread-4 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:41:28,736 | test | DEBUG | pool-5-thread-4 | [task:print_ep_queue_size:3508] ep_queue_size for default: 784681 2021-06-08 11:41:28,739 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:41:28,739 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:41:29,960 | test | DEBUG | pool-5-thread-4 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:41:29,960 | test | DEBUG | pool-5-thread-4 | [task:print_ep_queue_size:3508] ep_queue_size for default: 784681 2021-06-08 11:41:29,961 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:41:29,961 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:41:31,178 | test | DEBUG | pool-5-thread-4 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:41:31,178 | test | DEBUG | pool-5-thread-4 | [task:print_ep_queue_size:3508] ep_queue_size for default: 784681 2021-06-08 11:41:31,180 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:41:31,180 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:41:32,403 | test | DEBUG | pool-5-thread-4 | [task:record_bucket_ops:3495] Ops rate for 'default': 3000.000000 2021-06-08 11:41:32,403 | test | DEBUG | pool-5-thread-4 | [task:print_ep_queue_size:3508] ep_queue_size for default: 891475 2021-06-08 11:41:32,404 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:41:32,404 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:41:33,638 | test | DEBUG | pool-5-thread-4 | [task:record_bucket_ops:3495] Ops rate for 'default': 3000.000000 2021-06-08 11:41:33,638 | test | DEBUG | pool-5-thread-4 | [task:print_ep_queue_size:3508] ep_queue_size for default: 891475 2021-06-08 11:41:33,640 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:41:33,640 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:41:34,862 | test | DEBUG | pool-5-thread-4 | [task:record_bucket_ops:3495] Ops rate for 'default': 3000.000000 2021-06-08 11:41:34,862 | test | DEBUG | pool-5-thread-4 | [task:print_ep_queue_size:3508] ep_queue_size for default: 891475 2021-06-08 11:41:34,864 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:41:34,864 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:41:36,088 | test | DEBUG | pool-5-thread-4 | [task:record_bucket_ops:3495] Ops rate for 'default': 3000.000000 2021-06-08 11:41:36,088 | test | DEBUG | pool-5-thread-4 | [task:print_ep_queue_size:3508] ep_queue_size for default: 891475 2021-06-08 11:41:36,089 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:41:36,089 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:41:37,305 | test | DEBUG | pool-5-thread-4 | [task:record_bucket_ops:3495] Ops rate for 'default': 3000.000000 2021-06-08 11:41:37,305 | test | DEBUG | pool-5-thread-4 | [task:print_ep_queue_size:3508] ep_queue_size for default: 891475 2021-06-08 11:41:37,306 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:41:37,306 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:41:38,523 | test | DEBUG | pool-5-thread-4 | [task:record_bucket_ops:3495] Ops rate for 'default': 3000.000000 2021-06-08 11:41:38,525 | test | DEBUG | pool-5-thread-4 | [task:print_ep_queue_size:3508] ep_queue_size for default: 891475 2021-06-08 11:41:38,526 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:41:38,526 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:41:39,740 | test | DEBUG | pool-5-thread-4 | [task:record_bucket_ops:3495] Ops rate for 'default': 3000.000000 2021-06-08 11:41:39,740 | test | DEBUG | pool-5-thread-4 | [task:print_ep_queue_size:3508] ep_queue_size for default: 891475 2021-06-08 11:41:39,742 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:41:39,742 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:41:41,063 | test | DEBUG | pool-5-thread-4 | [task:record_bucket_ops:3495] Ops rate for 'default': 3000.000000 2021-06-08 11:41:41,063 | test | DEBUG | pool-5-thread-4 | [task:print_ep_queue_size:3508] ep_queue_size for default: 891475 2021-06-08 11:41:41,066 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:41:41,068 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:41:42,397 | test | DEBUG | pool-5-thread-4 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:41:42,397 | test | DEBUG | pool-5-thread-4 | [task:print_ep_queue_size:3508] ep_queue_size for default: 891475 2021-06-08 11:41:42,400 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:41:42,400 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:41:43,684 | test | DEBUG | pool-5-thread-4 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:41:43,684 | test | DEBUG | pool-5-thread-4 | [task:print_ep_queue_size:3508] ep_queue_size for default: 1250761 2021-06-08 11:41:43,687 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:41:43,687 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:41:45,178 | test | DEBUG | pool-5-thread-4 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:41:45,178 | test | DEBUG | pool-5-thread-4 | [task:print_ep_queue_size:3508] ep_queue_size for default: 1250761 2021-06-08 11:41:45,181 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:41:45,181 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:41:46,602 | test | DEBUG | pool-5-thread-4 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:41:46,604 | test | DEBUG | pool-5-thread-4 | [task:print_ep_queue_size:3508] ep_queue_size for default: 1250761 2021-06-08 11:41:46,605 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:41:46,605 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:41:47,020 | infra | INFO | pool-5-thread-11 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623177545.08_create__5800000_5850000_1623177545.09' completed 2021-06-08 11:41:47,082 | infra | INFO | pool-5-thread-18 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623177545.08_create__5700000_5750000_1623177545.09' completed 2021-06-08 11:41:47,186 | infra | INFO | pool-5-thread-25 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623177545.08_create__5650000_5700000_1623177545.09' completed 2021-06-08 11:41:47,211 | infra | INFO | pool-5-thread-9 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623177545.08_create__5550000_5600000_1623177545.09' completed 2021-06-08 11:41:47,216 | infra | INFO | pool-5-thread-10 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623177545.08_create__5350000_5400000_1623177545.09' completed 2021-06-08 11:41:47,262 | infra | INFO | pool-5-thread-13 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623177545.08_create__5250000_5300000_1623177545.09' completed 2021-06-08 11:41:47,423 | infra | INFO | pool-5-thread-12 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623177545.08_create__5300000_5350000_1623177545.09' completed 2021-06-08 11:41:47,430 | infra | INFO | pool-5-thread-28 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623177545.08_create__5450000_5500000_1623177545.09' completed 2021-06-08 11:41:47,443 | infra | INFO | pool-5-thread-23 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623177545.08_create__5100000_5150000_1623177545.09' completed 2021-06-08 11:41:47,463 | infra | INFO | pool-5-thread-6 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623177545.08_create__5050000_5100000_1623177545.09' completed 2021-06-08 11:41:47,480 | infra | INFO | pool-5-thread-14 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623177545.08_create__5950000_6000000_1623177545.09' completed 2021-06-08 11:41:47,505 | infra | INFO | pool-5-thread-21 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623177545.08_create__5150000_5200000_1623177545.09' completed 2021-06-08 11:41:47,568 | infra | INFO | pool-5-thread-17 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623177545.08_create__5600000_5650000_1623177545.09' completed 2021-06-08 11:41:47,573 | infra | INFO | pool-5-thread-5 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623177545.08_create__5000000_5050000_1623177545.09' completed 2021-06-08 11:41:47,573 | infra | DEBUG | pool-5-thread-3 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623177545.08_create__5000000_5050000_1623177545.09 are 50000 2021-06-08 11:41:47,575 | infra | DEBUG | pool-5-thread-3 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623177545.08_create__5050000_5100000_1623177545.09 2021-06-08 11:41:47,575 | infra | DEBUG | pool-5-thread-3 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623177545.08_create__5050000_5100000_1623177545.09 are 50000 2021-06-08 11:41:47,575 | infra | DEBUG | pool-5-thread-3 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623177545.08_create__5100000_5150000_1623177545.09 2021-06-08 11:41:47,575 | infra | DEBUG | pool-5-thread-3 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623177545.08_create__5100000_5150000_1623177545.09 are 50000 2021-06-08 11:41:47,575 | infra | DEBUG | pool-5-thread-3 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623177545.08_create__5150000_5200000_1623177545.09 2021-06-08 11:41:47,575 | infra | DEBUG | pool-5-thread-3 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623177545.08_create__5150000_5200000_1623177545.09 are 50000 2021-06-08 11:41:47,575 | infra | DEBUG | pool-5-thread-3 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623177545.08_create__5200000_5250000_1623177545.09 2021-06-08 11:41:47,576 | infra | INFO | pool-5-thread-16 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623177545.08_create__5400000_5450000_1623177545.09' completed 2021-06-08 11:41:47,592 | infra | INFO | pool-5-thread-26 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623177545.08_create__5200000_5250000_1623177545.09' completed 2021-06-08 11:41:47,592 | infra | DEBUG | pool-5-thread-3 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623177545.08_create__5200000_5250000_1623177545.09 are 50000 2021-06-08 11:41:47,592 | infra | DEBUG | pool-5-thread-3 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623177545.08_create__5250000_5300000_1623177545.09 2021-06-08 11:41:47,592 | infra | DEBUG | pool-5-thread-3 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623177545.08_create__5250000_5300000_1623177545.09 are 50000 2021-06-08 11:41:47,592 | infra | DEBUG | pool-5-thread-3 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623177545.08_create__5300000_5350000_1623177545.09 2021-06-08 11:41:47,594 | infra | DEBUG | pool-5-thread-3 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623177545.08_create__5300000_5350000_1623177545.09 are 50000 2021-06-08 11:41:47,594 | infra | DEBUG | pool-5-thread-3 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623177545.08_create__5350000_5400000_1623177545.09 2021-06-08 11:41:47,594 | infra | DEBUG | pool-5-thread-3 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623177545.08_create__5350000_5400000_1623177545.09 are 50000 2021-06-08 11:41:47,594 | infra | DEBUG | pool-5-thread-3 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623177545.08_create__5400000_5450000_1623177545.09 2021-06-08 11:41:47,594 | infra | DEBUG | pool-5-thread-3 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623177545.08_create__5400000_5450000_1623177545.09 are 50000 2021-06-08 11:41:47,594 | infra | DEBUG | pool-5-thread-3 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623177545.08_create__5450000_5500000_1623177545.09 2021-06-08 11:41:47,594 | infra | DEBUG | pool-5-thread-3 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623177545.08_create__5450000_5500000_1623177545.09 are 50000 2021-06-08 11:41:47,595 | infra | DEBUG | pool-5-thread-3 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623177545.08_create__5500000_5550000_1623177545.09 2021-06-08 11:41:47,605 | infra | INFO | pool-5-thread-19 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623177545.08_create__5850000_5900000_1623177545.09' completed 2021-06-08 11:41:47,614 | infra | INFO | pool-5-thread-20 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623177545.08_create__5750000_5800000_1623177545.09' completed 2021-06-08 11:41:47,615 | infra | INFO | pool-5-thread-24 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623177545.08_create__5500000_5550000_1623177545.09' completed 2021-06-08 11:41:47,615 | infra | DEBUG | pool-5-thread-3 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623177545.08_create__5500000_5550000_1623177545.09 are 50000 2021-06-08 11:41:47,615 | infra | DEBUG | pool-5-thread-3 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623177545.08_create__5550000_5600000_1623177545.09 2021-06-08 11:41:47,615 | infra | DEBUG | pool-5-thread-3 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623177545.08_create__5550000_5600000_1623177545.09 are 50000 2021-06-08 11:41:47,615 | infra | DEBUG | pool-5-thread-3 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623177545.08_create__5600000_5650000_1623177545.09 2021-06-08 11:41:47,615 | infra | DEBUG | pool-5-thread-3 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623177545.08_create__5600000_5650000_1623177545.09 are 50000 2021-06-08 11:41:47,615 | infra | DEBUG | pool-5-thread-3 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623177545.08_create__5650000_5700000_1623177545.09 2021-06-08 11:41:47,615 | infra | DEBUG | pool-5-thread-3 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623177545.08_create__5650000_5700000_1623177545.09 are 50000 2021-06-08 11:41:47,615 | infra | DEBUG | pool-5-thread-3 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623177545.08_create__5700000_5750000_1623177545.09 2021-06-08 11:41:47,615 | infra | DEBUG | pool-5-thread-3 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623177545.08_create__5700000_5750000_1623177545.09 are 50000 2021-06-08 11:41:47,615 | infra | DEBUG | pool-5-thread-3 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623177545.08_create__5750000_5800000_1623177545.09 2021-06-08 11:41:47,615 | infra | DEBUG | pool-5-thread-3 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623177545.08_create__5750000_5800000_1623177545.09 are 50000 2021-06-08 11:41:47,615 | infra | DEBUG | pool-5-thread-3 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623177545.08_create__5800000_5850000_1623177545.09 2021-06-08 11:41:47,615 | infra | DEBUG | pool-5-thread-3 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623177545.08_create__5800000_5850000_1623177545.09 are 50000 2021-06-08 11:41:47,617 | infra | DEBUG | pool-5-thread-3 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623177545.08_create__5850000_5900000_1623177545.09 2021-06-08 11:41:47,617 | infra | DEBUG | pool-5-thread-3 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623177545.08_create__5850000_5900000_1623177545.09 are 50000 2021-06-08 11:41:47,617 | infra | DEBUG | pool-5-thread-3 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623177545.08_create__5900000_5950000_1623177545.09 2021-06-08 11:41:47,644 | infra | INFO | pool-5-thread-27 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623177545.08_create__5900000_5950000_1623177545.09' completed 2021-06-08 11:41:47,644 | infra | DEBUG | pool-5-thread-3 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623177545.08_create__5900000_5950000_1623177545.09 are 50000 2021-06-08 11:41:47,644 | infra | DEBUG | pool-5-thread-3 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623177545.08_create__5950000_6000000_1623177545.09 2021-06-08 11:41:47,644 | infra | DEBUG | pool-5-thread-3 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623177545.08_create__5950000_6000000_1623177545.09 are 50000 2021-06-08 11:41:47,644 | infra | DEBUG | pool-5-thread-3 | [task_manager:get_task_result:26] Getting task result for PrintBucketStats_default_1623177545.09 2021-06-08 11:41:47,921 | test | DEBUG | pool-5-thread-4 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:41:47,921 | test | DEBUG | pool-5-thread-4 | [task:print_ep_queue_size:3508] ep_queue_size for default: 1250761 2021-06-08 11:41:47,923 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:41:47,923 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:41:48,924 | test | INFO | pool-5-thread-4 | [table_view:display:72] Ops trend for bucket 'default' +---------+---------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +---------+---------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | *****..................*******X | 2302.400 | | 697.600 | ....********X | 697.600 | | 0.000 | *****************************************.....................................................................********...............................*******X | 12728.300 | | 0.000 | *****************************************.......................*******X | 3000.000 | | 0.000 | ****X | 0.000 | +---------+---------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------+ 2021-06-08 11:41:48,924 | infra | INFO | pool-5-thread-4 | [task:complete_task:102] Thread 'PrintBucketStats_default_1623177545.09' completed 2021-06-08 11:41:48,924 | infra | DEBUG | pool-5-thread-3 | [task:call:1663] ========= Tasks in loadgen pool======= 2021-06-08 11:41:48,924 | infra | WARNING | pool-5-thread-3 | [task_manager:print_tasks_in_pool:83] Task 'LoadDocsGen_default__default__default_create_5000000_6000000_1623177545.08' not completed 2021-06-08 11:41:48,924 | infra | DEBUG | pool-5-thread-3 | [task:call:1665] ====================================== 2021-06-08 11:41:48,924 | infra | DEBUG | pool-5-thread-3 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623177545.08_create__5000000_5050000_1623177545.09' complete. Loaded 50000 items 2021-06-08 11:41:48,924 | infra | DEBUG | pool-5-thread-3 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623177545.08_create__5050000_5100000_1623177545.09' complete. Loaded 50000 items 2021-06-08 11:41:48,926 | infra | DEBUG | pool-5-thread-3 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623177545.08_create__5100000_5150000_1623177545.09' complete. Loaded 50000 items 2021-06-08 11:41:48,926 | infra | DEBUG | pool-5-thread-3 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623177545.08_create__5150000_5200000_1623177545.09' complete. Loaded 50000 items 2021-06-08 11:41:48,926 | infra | DEBUG | pool-5-thread-3 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623177545.08_create__5200000_5250000_1623177545.09' complete. Loaded 50000 items 2021-06-08 11:41:48,926 | infra | DEBUG | pool-5-thread-3 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623177545.08_create__5250000_5300000_1623177545.09' complete. Loaded 50000 items 2021-06-08 11:41:48,926 | infra | DEBUG | pool-5-thread-3 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623177545.08_create__5300000_5350000_1623177545.09' complete. Loaded 50000 items 2021-06-08 11:41:48,926 | infra | DEBUG | pool-5-thread-3 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623177545.08_create__5350000_5400000_1623177545.09' complete. Loaded 50000 items 2021-06-08 11:41:48,926 | infra | DEBUG | pool-5-thread-3 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623177545.08_create__5400000_5450000_1623177545.09' complete. Loaded 50000 items 2021-06-08 11:41:48,926 | infra | DEBUG | pool-5-thread-3 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623177545.08_create__5450000_5500000_1623177545.09' complete. Loaded 50000 items 2021-06-08 11:41:48,926 | infra | DEBUG | pool-5-thread-3 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623177545.08_create__5500000_5550000_1623177545.09' complete. Loaded 50000 items 2021-06-08 11:41:48,927 | infra | DEBUG | pool-5-thread-3 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623177545.08_create__5550000_5600000_1623177545.09' complete. Loaded 50000 items 2021-06-08 11:41:48,927 | infra | DEBUG | pool-5-thread-3 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623177545.08_create__5600000_5650000_1623177545.09' complete. Loaded 50000 items 2021-06-08 11:41:48,927 | infra | DEBUG | pool-5-thread-3 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623177545.08_create__5650000_5700000_1623177545.09' complete. Loaded 50000 items 2021-06-08 11:41:48,927 | infra | DEBUG | pool-5-thread-3 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623177545.08_create__5700000_5750000_1623177545.09' complete. Loaded 50000 items 2021-06-08 11:41:48,927 | infra | DEBUG | pool-5-thread-3 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623177545.08_create__5750000_5800000_1623177545.09' complete. Loaded 50000 items 2021-06-08 11:41:48,927 | infra | DEBUG | pool-5-thread-3 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623177545.08_create__5800000_5850000_1623177545.09' complete. Loaded 50000 items 2021-06-08 11:41:48,927 | infra | DEBUG | pool-5-thread-3 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623177545.08_create__5850000_5900000_1623177545.09' complete. Loaded 50000 items 2021-06-08 11:41:48,927 | infra | DEBUG | pool-5-thread-3 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623177545.08_create__5900000_5950000_1623177545.09' complete. Loaded 50000 items 2021-06-08 11:41:48,927 | infra | DEBUG | pool-5-thread-3 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_5000000_6000000_1623177545.08_create__5950000_6000000_1623177545.09' complete. Loaded 50000 items 2021-06-08 11:41:48,927 | infra | INFO | pool-5-thread-3 | [task:complete_task:102] Thread 'LoadDocsGen_default__default__default_create_5000000_6000000_1623177545.08' completed 2021-06-08 11:41:48,930 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.125 with username: root, password: couchbase 2021-06-08 11:41:49,029 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 11:41:49,289 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: ls /etc/ 2021-06-08 11:41:49,378 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 11:41:49,378 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: ls /cygdrive/c/Windows 2021-06-08 11:41:49,400 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: uname -m 2021-06-08 11:41:49,417 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: cat /proc/cpuinfo 2021-06-08 11:41:49,434 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: df -Thl 2021-06-08 11:41:49,476 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: cat /proc/meminfo 2021-06-08 11:41:49,493 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: hostname 2021-06-08 11:41:49,507 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: hostname -d 2021-06-08 11:41:49,526 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:559] Starting StatsWaitTask for ep_data_write_failed on bucket default 2021-06-08 11:41:49,526 | infra | INFO | MainThread | [task_manager:add_new_task:23] Added new task: StatsWaitTask_default_ep_data_write_failed_1623177709.53 2021-06-08 11:41:49,528 | infra | INFO | pool-5-thread-15 | [task:start_task:87] Thread 'StatsWaitTask_default_ep_data_write_failed_1623177709.53' started 2021-06-08 11:41:49,529 | infra | DEBUG | pool-5-thread-15 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:41:49,530 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.155 with username: root, password: couchbase 2021-06-08 11:41:49,635 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 11:41:49,842 | test | DEBUG | pool-5-thread-15 | [task:_compare:2649] Comparing 74795 > 0 2021-06-08 11:41:49,842 | test | DEBUG | pool-5-thread-15 | [task:_get_all_stats_and_compare:2601] Ready: ep_data_write_failed 74795 > 0. Received: {'172.23.106.125': '74795'} for bucket 'default' 2021-06-08 11:41:49,844 | infra | INFO | pool-5-thread-15 | [task:complete_task:102] Thread 'StatsWaitTask_default_ep_data_write_failed_1623177709.53' completed 2021-06-08 11:41:49,884 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: ls /etc/ 2021-06-08 11:41:49,953 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 11:41:49,953 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: ls /cygdrive/c/Windows 2021-06-08 11:41:49,969 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: uname -m 2021-06-08 11:41:49,984 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: cat /proc/cpuinfo 2021-06-08 11:41:50,000 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: df -Thl 2021-06-08 11:41:50,019 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: cat /proc/meminfo 2021-06-08 11:41:50,032 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: hostname 2021-06-08 11:41:50,049 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: hostname -d 2021-06-08 11:41:50,068 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:559] Starting StatsWaitTask for ep_data_write_failed on bucket default 2021-06-08 11:41:50,068 | infra | INFO | MainThread | [task_manager:add_new_task:23] Added new task: StatsWaitTask_default_ep_data_write_failed_1623177710.07 2021-06-08 11:41:50,068 | infra | INFO | pool-5-thread-22 | [task:start_task:87] Thread 'StatsWaitTask_default_ep_data_write_failed_1623177710.07' started 2021-06-08 11:41:50,069 | infra | DEBUG | pool-5-thread-22 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:41:50,071 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.156 with username: root, password: couchbase 2021-06-08 11:41:50,161 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 11:41:50,328 | test | DEBUG | pool-5-thread-22 | [task:_compare:2649] Comparing 273201 > 0 2021-06-08 11:41:50,329 | test | DEBUG | pool-5-thread-22 | [task:_get_all_stats_and_compare:2601] Ready: ep_data_write_failed 273201 > 0. Received: {'172.23.106.155': '273201'} for bucket 'default' 2021-06-08 11:41:50,329 | infra | INFO | pool-5-thread-22 | [task:complete_task:102] Thread 'StatsWaitTask_default_ep_data_write_failed_1623177710.07' completed 2021-06-08 11:41:50,628 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: ls /etc/ 2021-06-08 11:41:50,691 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 11:41:50,693 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: ls /cygdrive/c/Windows 2021-06-08 11:41:50,706 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: uname -m 2021-06-08 11:41:50,720 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: cat /proc/cpuinfo 2021-06-08 11:41:50,733 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: df -Thl 2021-06-08 11:41:50,749 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: cat /proc/meminfo 2021-06-08 11:41:50,760 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: hostname 2021-06-08 11:41:50,773 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: hostname -d 2021-06-08 11:41:50,786 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:559] Starting StatsWaitTask for ep_data_write_failed on bucket default 2021-06-08 11:41:50,786 | infra | INFO | MainThread | [task_manager:add_new_task:23] Added new task: StatsWaitTask_default_ep_data_write_failed_1623177710.79 2021-06-08 11:41:50,786 | infra | INFO | pool-5-thread-8 | [task:start_task:87] Thread 'StatsWaitTask_default_ep_data_write_failed_1623177710.79' started 2021-06-08 11:41:50,788 | infra | DEBUG | pool-5-thread-8 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:41:50,789 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.122 with username: root, password: couchbase 2021-06-08 11:41:50,874 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 11:41:51,365 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: ls /etc/ 2021-06-08 11:41:51,411 | test | DEBUG | pool-5-thread-8 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 11:41:51,411 | infra | DEBUG | pool-5-thread-8 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.156': '0'} for bucket 'default' 2021-06-08 11:41:51,411 | infra | DEBUG | pool-5-thread-8 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 11:41:51,414 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 11:41:51,414 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 11:41:51,424 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 11:41:51,424 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: ls /cygdrive/c/Windows 2021-06-08 11:41:51,440 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: uname -m 2021-06-08 11:41:51,450 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: cat /proc/cpuinfo 2021-06-08 11:41:51,463 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: df -Thl 2021-06-08 11:41:51,479 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: cat /proc/meminfo 2021-06-08 11:41:51,492 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: hostname 2021-06-08 11:41:51,503 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: hostname -d 2021-06-08 11:41:51,517 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:559] Starting StatsWaitTask for ep_data_write_failed on bucket default 2021-06-08 11:41:51,519 | infra | INFO | MainThread | [task_manager:add_new_task:23] Added new task: StatsWaitTask_default_ep_data_write_failed_1623177711.52 2021-06-08 11:41:51,519 | infra | DEBUG | MainThread | [task_manager:get_task_result:26] Getting task result for StatsWaitTask_default_ep_data_write_failed_1623177709.53 2021-06-08 11:41:51,519 | infra | INFO | pool-5-thread-7 | [task:start_task:87] Thread 'StatsWaitTask_default_ep_data_write_failed_1623177711.52' started 2021-06-08 11:41:51,519 | infra | DEBUG | MainThread | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.125 2021-06-08 11:41:51,519 | infra | DEBUG | MainThread | [task_manager:get_task_result:26] Getting task result for StatsWaitTask_default_ep_data_write_failed_1623177710.07 2021-06-08 11:41:51,519 | infra | DEBUG | pool-5-thread-7 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:41:51,520 | infra | DEBUG | MainThread | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.155 2021-06-08 11:41:51,520 | infra | DEBUG | MainThread | [task_manager:get_task_result:26] Getting task result for StatsWaitTask_default_ep_data_write_failed_1623177710.79 2021-06-08 11:41:51,792 | test | DEBUG | pool-5-thread-7 | [task:_compare:2649] Comparing 74115 > 0 2021-06-08 11:41:51,792 | test | DEBUG | pool-5-thread-7 | [task:_get_all_stats_and_compare:2601] Ready: ep_data_write_failed 74115 > 0. Received: {'172.23.106.122': '74115'} for bucket 'default' 2021-06-08 11:41:51,792 | infra | INFO | pool-5-thread-7 | [task:complete_task:102] Thread 'StatsWaitTask_default_ep_data_write_failed_1623177711.52' completed 2021-06-08 11:41:56,414 | infra | DEBUG | pool-5-thread-8 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:41:56,618 | test | DEBUG | pool-5-thread-8 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 11:41:56,618 | infra | DEBUG | pool-5-thread-8 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.156': '0'} for bucket 'default' 2021-06-08 11:41:56,618 | infra | DEBUG | pool-5-thread-8 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 11:41:56,619 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 11:41:56,619 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 11:42:01,621 | infra | DEBUG | pool-5-thread-8 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:42:01,828 | test | DEBUG | pool-5-thread-8 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 11:42:01,828 | infra | DEBUG | pool-5-thread-8 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.156': '0'} for bucket 'default' 2021-06-08 11:42:01,828 | infra | DEBUG | pool-5-thread-8 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 11:42:01,829 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 11:42:01,831 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 11:42:06,831 | infra | DEBUG | pool-5-thread-8 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:42:07,033 | test | DEBUG | pool-5-thread-8 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 11:42:07,035 | infra | DEBUG | pool-5-thread-8 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.156': '0'} for bucket 'default' 2021-06-08 11:42:07,035 | infra | DEBUG | pool-5-thread-8 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 11:42:07,036 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 11:42:07,036 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 11:42:12,036 | infra | DEBUG | pool-5-thread-8 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:42:12,249 | test | DEBUG | pool-5-thread-8 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 11:42:12,250 | infra | DEBUG | pool-5-thread-8 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.156': '0'} for bucket 'default' 2021-06-08 11:42:12,250 | infra | DEBUG | pool-5-thread-8 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 11:42:12,250 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 11:42:12,252 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 11:42:17,252 | infra | DEBUG | pool-5-thread-8 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:42:17,460 | test | DEBUG | pool-5-thread-8 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 11:42:17,461 | infra | DEBUG | pool-5-thread-8 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.156': '0'} for bucket 'default' 2021-06-08 11:42:17,461 | infra | DEBUG | pool-5-thread-8 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 11:42:17,463 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 11:42:17,463 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 11:42:22,463 | infra | DEBUG | pool-5-thread-8 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:42:22,664 | test | DEBUG | pool-5-thread-8 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 11:42:22,664 | infra | DEBUG | pool-5-thread-8 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.156': '0'} for bucket 'default' 2021-06-08 11:42:22,664 | infra | DEBUG | pool-5-thread-8 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 11:42:22,667 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 11:42:22,667 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 11:42:27,667 | infra | DEBUG | pool-5-thread-8 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:42:27,874 | test | DEBUG | pool-5-thread-8 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 11:42:27,875 | infra | DEBUG | pool-5-thread-8 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.156': '0'} for bucket 'default' 2021-06-08 11:42:27,875 | infra | DEBUG | pool-5-thread-8 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 11:42:27,875 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 11:42:27,875 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 11:42:32,877 | infra | DEBUG | pool-5-thread-8 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:42:33,085 | test | DEBUG | pool-5-thread-8 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 11:42:33,085 | infra | DEBUG | pool-5-thread-8 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.156': '0'} for bucket 'default' 2021-06-08 11:42:33,085 | infra | DEBUG | pool-5-thread-8 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 11:42:33,086 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 11:42:33,086 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 11:42:38,088 | infra | DEBUG | pool-5-thread-8 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:42:38,292 | test | DEBUG | pool-5-thread-8 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 11:42:38,293 | infra | DEBUG | pool-5-thread-8 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.156': '0'} for bucket 'default' 2021-06-08 11:42:38,293 | infra | DEBUG | pool-5-thread-8 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 11:42:38,295 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 11:42:38,295 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 11:42:43,296 | infra | DEBUG | pool-5-thread-8 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:42:43,509 | test | DEBUG | pool-5-thread-8 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 11:42:43,509 | infra | DEBUG | pool-5-thread-8 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.156': '0'} for bucket 'default' 2021-06-08 11:42:43,509 | infra | DEBUG | pool-5-thread-8 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 11:42:43,512 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 11:42:43,512 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 11:42:48,513 | infra | DEBUG | pool-5-thread-8 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:42:48,730 | test | DEBUG | pool-5-thread-8 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 11:42:48,730 | infra | DEBUG | pool-5-thread-8 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.156': '0'} for bucket 'default' 2021-06-08 11:42:48,730 | infra | DEBUG | pool-5-thread-8 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 11:42:48,732 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 11:42:48,733 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 11:42:53,733 | infra | DEBUG | pool-5-thread-8 | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.156 2021-06-08 11:42:53,733 | infra | INFO | pool-5-thread-8 | [task:complete_task:102] Thread 'StatsWaitTask_default_ep_data_write_failed_1623177710.79' completed 2021-06-08 11:42:53,740 | infra | INFO | MainThread | [task_manager:add_new_task:23] Added new task: LoadDocsGen_default__default__default_create_6000000_7000000_1623177773.74 2021-06-08 11:42:53,740 | infra | INFO | pool-5-thread-30 | [task:start_task:87] Thread 'LoadDocsGen_default__default__default_create_6000000_7000000_1623177773.74' started 2021-06-08 11:42:53,740 | infra | DEBUG | MainThread | [task_manager:get_task_result:26] Getting task result for LoadDocsGen_default__default__default_create_6000000_7000000_1623177773.74 2021-06-08 11:42:53,769 | infra | INFO | pool-5-thread-30 | [task_manager:add_new_task:23] Added new task: PrintBucketStats_default_1623177773.76 2021-06-08 11:42:53,769 | infra | INFO | pool-5-thread-2 | [task:start_task:87] Thread 'PrintBucketStats_default_1623177773.76' started 2021-06-08 11:42:53,769 | infra | INFO | pool-5-thread-30 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623177773.74_create__6000000_6050000_1623177773.76 2021-06-08 11:42:53,769 | infra | INFO | pool-5-thread-1 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623177773.74_create__6000000_6050000_1623177773.76' started 2021-06-08 11:42:53,769 | infra | INFO | pool-5-thread-30 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623177773.74_create__6050000_6100000_1623177773.76 2021-06-08 11:42:53,769 | infra | INFO | pool-5-thread-29 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623177773.74_create__6050000_6100000_1623177773.76' started 2021-06-08 11:42:53,769 | infra | INFO | pool-5-thread-30 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623177773.74_create__6100000_6150000_1623177773.76 2021-06-08 11:42:53,769 | infra | INFO | pool-5-thread-11 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623177773.74_create__6100000_6150000_1623177773.76' started 2021-06-08 11:42:53,769 | infra | INFO | pool-5-thread-30 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623177773.74_create__6150000_6200000_1623177773.76 2021-06-08 11:42:53,769 | infra | INFO | pool-5-thread-30 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623177773.74_create__6200000_6250000_1623177773.76 2021-06-08 11:42:53,769 | infra | INFO | pool-5-thread-18 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623177773.74_create__6150000_6200000_1623177773.76' started 2021-06-08 11:42:53,769 | infra | INFO | pool-5-thread-25 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623177773.74_create__6200000_6250000_1623177773.76' started 2021-06-08 11:42:53,769 | infra | INFO | pool-5-thread-30 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623177773.74_create__6250000_6300000_1623177773.76 2021-06-08 11:42:53,769 | infra | INFO | pool-5-thread-9 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623177773.74_create__6250000_6300000_1623177773.76' started 2021-06-08 11:42:53,769 | infra | INFO | pool-5-thread-30 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623177773.74_create__6300000_6350000_1623177773.76 2021-06-08 11:42:53,769 | infra | INFO | pool-5-thread-10 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623177773.74_create__6300000_6350000_1623177773.76' started 2021-06-08 11:42:53,769 | infra | INFO | pool-5-thread-13 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623177773.74_create__6350000_6400000_1623177773.76' started 2021-06-08 11:42:53,769 | infra | INFO | pool-5-thread-30 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623177773.74_create__6350000_6400000_1623177773.76 2021-06-08 11:42:53,770 | infra | INFO | pool-5-thread-12 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623177773.74_create__6400000_6450000_1623177773.76' started 2021-06-08 11:42:53,773 | infra | INFO | pool-5-thread-30 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623177773.74_create__6400000_6450000_1623177773.76 2021-06-08 11:42:53,773 | infra | INFO | pool-5-thread-30 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623177773.74_create__6450000_6500000_1623177773.76 2021-06-08 11:42:53,773 | infra | INFO | pool-5-thread-28 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623177773.74_create__6450000_6500000_1623177773.76' started 2021-06-08 11:42:53,773 | infra | INFO | pool-5-thread-30 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623177773.74_create__6500000_6550000_1623177773.76 2021-06-08 11:42:53,773 | infra | INFO | pool-5-thread-23 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623177773.74_create__6500000_6550000_1623177773.76' started 2021-06-08 11:42:53,775 | infra | INFO | pool-5-thread-6 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623177773.74_create__6550000_6600000_1623177773.76' started 2021-06-08 11:42:53,775 | infra | INFO | pool-5-thread-30 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623177773.74_create__6550000_6600000_1623177773.76 2021-06-08 11:42:53,775 | infra | INFO | pool-5-thread-14 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623177773.74_create__6600000_6650000_1623177773.76' started 2021-06-08 11:42:53,775 | infra | INFO | pool-5-thread-30 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623177773.74_create__6600000_6650000_1623177773.76 2021-06-08 11:42:53,775 | infra | INFO | pool-5-thread-30 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623177773.74_create__6650000_6700000_1623177773.76 2021-06-08 11:42:53,775 | infra | INFO | pool-5-thread-21 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623177773.74_create__6650000_6700000_1623177773.76' started 2021-06-08 11:42:53,776 | infra | INFO | pool-5-thread-17 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623177773.74_create__6700000_6750000_1623177773.76' started 2021-06-08 11:42:53,776 | infra | INFO | pool-5-thread-30 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623177773.74_create__6700000_6750000_1623177773.76 2021-06-08 11:42:53,776 | infra | INFO | pool-5-thread-30 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623177773.74_create__6750000_6800000_1623177773.76 2021-06-08 11:42:53,778 | infra | INFO | pool-5-thread-30 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623177773.74_create__6800000_6850000_1623177773.76 2021-06-08 11:42:53,778 | infra | INFO | pool-5-thread-16 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623177773.74_create__6800000_6850000_1623177773.76' started 2021-06-08 11:42:53,778 | infra | INFO | pool-5-thread-30 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623177773.74_create__6850000_6900000_1623177773.76 2021-06-08 11:42:53,778 | infra | INFO | pool-5-thread-5 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623177773.74_create__6750000_6800000_1623177773.76' started 2021-06-08 11:42:53,778 | infra | INFO | pool-5-thread-26 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623177773.74_create__6850000_6900000_1623177773.76' started 2021-06-08 11:42:53,778 | infra | INFO | pool-5-thread-30 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623177773.74_create__6900000_6950000_1623177773.76 2021-06-08 11:42:53,778 | infra | INFO | pool-5-thread-19 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623177773.74_create__6900000_6950000_1623177773.76' started 2021-06-08 11:42:53,779 | infra | INFO | pool-5-thread-30 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623177773.74_create__6950000_7000000_1623177773.76 2021-06-08 11:42:53,779 | infra | INFO | pool-5-thread-20 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623177773.74_create__6950000_7000000_1623177773.76' started 2021-06-08 11:42:53,779 | infra | DEBUG | pool-5-thread-30 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623177773.74_create__6000000_6050000_1623177773.76 2021-06-08 11:42:54,071 | test | DEBUG | pool-5-thread-2 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:42:54,071 | test | DEBUG | pool-5-thread-2 | [task:print_ep_queue_size:3508] ep_queue_size for default: 3625683 2021-06-08 11:42:54,072 | infra | DEBUG | pool-5-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:42:54,072 | infra | DEBUG | pool-5-thread-2 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:42:55,319 | test | DEBUG | pool-5-thread-2 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:42:55,319 | test | DEBUG | pool-5-thread-2 | [task:print_ep_queue_size:3508] ep_queue_size for default: 3625683 2021-06-08 11:42:55,322 | infra | DEBUG | pool-5-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:42:55,322 | infra | DEBUG | pool-5-thread-2 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:42:56,559 | test | DEBUG | pool-5-thread-2 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:42:56,561 | test | DEBUG | pool-5-thread-2 | [task:print_ep_queue_size:3508] ep_queue_size for default: 3625683 2021-06-08 11:42:56,562 | infra | DEBUG | pool-5-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:42:56,562 | infra | DEBUG | pool-5-thread-2 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:42:57,841 | test | DEBUG | pool-5-thread-2 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:42:57,841 | test | DEBUG | pool-5-thread-2 | [task:print_ep_queue_size:3508] ep_queue_size for default: 3625683 2021-06-08 11:42:57,842 | infra | DEBUG | pool-5-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:42:57,842 | infra | DEBUG | pool-5-thread-2 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:42:59,171 | test | DEBUG | pool-5-thread-2 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:42:59,171 | test | DEBUG | pool-5-thread-2 | [task:print_ep_queue_size:3508] ep_queue_size for default: 3625683 2021-06-08 11:42:59,174 | infra | DEBUG | pool-5-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:42:59,174 | infra | DEBUG | pool-5-thread-2 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:43:00,427 | test | DEBUG | pool-5-thread-2 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:43:00,427 | test | DEBUG | pool-5-thread-2 | [task:print_ep_queue_size:3508] ep_queue_size for default: 3625683 2021-06-08 11:43:00,430 | infra | DEBUG | pool-5-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:43:00,430 | infra | DEBUG | pool-5-thread-2 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:43:01,805 | test | DEBUG | pool-5-thread-2 | [task:record_bucket_ops:3495] Ops rate for 'default': 19443.100000 2021-06-08 11:43:01,805 | test | DEBUG | pool-5-thread-2 | [task:print_ep_queue_size:3508] ep_queue_size for default: 4393097 2021-06-08 11:43:01,808 | infra | DEBUG | pool-5-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:43:01,808 | infra | DEBUG | pool-5-thread-2 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:43:03,167 | test | DEBUG | pool-5-thread-2 | [task:record_bucket_ops:3495] Ops rate for 'default': 19443.100000 2021-06-08 11:43:03,167 | test | DEBUG | pool-5-thread-2 | [task:print_ep_queue_size:3508] ep_queue_size for default: 4393097 2021-06-08 11:43:03,168 | infra | DEBUG | pool-5-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:43:03,168 | infra | DEBUG | pool-5-thread-2 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:43:04,492 | test | DEBUG | pool-5-thread-2 | [task:record_bucket_ops:3495] Ops rate for 'default': 19443.100000 2021-06-08 11:43:04,492 | test | DEBUG | pool-5-thread-2 | [task:print_ep_queue_size:3508] ep_queue_size for default: 4393097 2021-06-08 11:43:04,493 | infra | DEBUG | pool-5-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:43:04,493 | infra | DEBUG | pool-5-thread-2 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:43:05,984 | test | DEBUG | pool-5-thread-2 | [task:record_bucket_ops:3495] Ops rate for 'default': 19443.100000 2021-06-08 11:43:05,984 | test | DEBUG | pool-5-thread-2 | [task:print_ep_queue_size:3508] ep_queue_size for default: 4393097 2021-06-08 11:43:05,987 | infra | DEBUG | pool-5-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:43:05,987 | infra | DEBUG | pool-5-thread-2 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:43:07,408 | test | DEBUG | pool-5-thread-2 | [task:record_bucket_ops:3495] Ops rate for 'default': 19443.100000 2021-06-08 11:43:07,408 | test | DEBUG | pool-5-thread-2 | [task:print_ep_queue_size:3508] ep_queue_size for default: 4393097 2021-06-08 11:43:07,411 | infra | DEBUG | pool-5-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:43:07,411 | infra | DEBUG | pool-5-thread-2 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:43:08,959 | test | DEBUG | pool-5-thread-2 | [task:record_bucket_ops:3495] Ops rate for 'default': 19443.100000 2021-06-08 11:43:08,959 | test | DEBUG | pool-5-thread-2 | [task:print_ep_queue_size:3508] ep_queue_size for default: 4393097 2021-06-08 11:43:08,961 | infra | DEBUG | pool-5-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:43:08,961 | infra | DEBUG | pool-5-thread-2 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:43:10,236 | test | DEBUG | pool-5-thread-2 | [task:record_bucket_ops:3495] Ops rate for 'default': 19443.100000 2021-06-08 11:43:10,237 | test | DEBUG | pool-5-thread-2 | [task:print_ep_queue_size:3508] ep_queue_size for default: 4393097 2021-06-08 11:43:10,242 | infra | DEBUG | pool-5-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:43:10,242 | infra | DEBUG | pool-5-thread-2 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:43:11,526 | test | DEBUG | pool-5-thread-2 | [task:record_bucket_ops:3495] Ops rate for 'default': 54036.000000 2021-06-08 11:43:11,528 | test | DEBUG | pool-5-thread-2 | [task:print_ep_queue_size:3508] ep_queue_size for default: 6031216 2021-06-08 11:43:11,529 | infra | DEBUG | pool-5-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:43:11,529 | infra | DEBUG | pool-5-thread-2 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:43:12,875 | test | DEBUG | pool-5-thread-2 | [task:record_bucket_ops:3495] Ops rate for 'default': 67036.700000 2021-06-08 11:43:12,875 | test | DEBUG | pool-5-thread-2 | [task:print_ep_queue_size:3508] ep_queue_size for default: 6031216 2021-06-08 11:43:12,881 | infra | DEBUG | pool-5-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:43:12,881 | infra | DEBUG | pool-5-thread-2 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:43:14,184 | test | DEBUG | pool-5-thread-2 | [task:record_bucket_ops:3495] Ops rate for 'default': 67036.700000 2021-06-08 11:43:14,184 | test | DEBUG | pool-5-thread-2 | [task:print_ep_queue_size:3508] ep_queue_size for default: 6580207 2021-06-08 11:43:14,187 | infra | DEBUG | pool-5-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:43:14,190 | infra | DEBUG | pool-5-thread-2 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:43:15,381 | infra | INFO | pool-5-thread-14 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623177773.74_create__6600000_6650000_1623177773.76' completed 2021-06-08 11:43:15,443 | infra | INFO | pool-5-thread-17 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623177773.74_create__6700000_6750000_1623177773.76' completed 2021-06-08 11:43:15,471 | test | DEBUG | pool-5-thread-2 | [task:record_bucket_ops:3495] Ops rate for 'default': 67036.700000 2021-06-08 11:43:15,471 | test | DEBUG | pool-5-thread-2 | [task:print_ep_queue_size:3508] ep_queue_size for default: 6580207 2021-06-08 11:43:15,474 | infra | DEBUG | pool-5-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:43:15,474 | infra | DEBUG | pool-5-thread-2 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:43:15,572 | infra | INFO | pool-5-thread-1 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623177773.74_create__6000000_6050000_1623177773.76' completed 2021-06-08 11:43:15,572 | infra | DEBUG | pool-5-thread-30 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623177773.74_create__6000000_6050000_1623177773.76 are 50000 2021-06-08 11:43:15,572 | infra | DEBUG | pool-5-thread-30 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623177773.74_create__6050000_6100000_1623177773.76 2021-06-08 11:43:15,578 | infra | INFO | pool-5-thread-26 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623177773.74_create__6850000_6900000_1623177773.76' completed 2021-06-08 11:43:15,713 | infra | INFO | pool-5-thread-19 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623177773.74_create__6900000_6950000_1623177773.76' completed 2021-06-08 11:43:15,785 | infra | INFO | pool-5-thread-9 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623177773.74_create__6250000_6300000_1623177773.76' completed 2021-06-08 11:43:15,788 | infra | INFO | pool-5-thread-28 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623177773.74_create__6450000_6500000_1623177773.76' completed 2021-06-08 11:43:16,519 | infra | INFO | pool-5-thread-10 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623177773.74_create__6300000_6350000_1623177773.76' completed 2021-06-08 11:43:16,717 | test | DEBUG | pool-5-thread-2 | [task:record_bucket_ops:3495] Ops rate for 'default': 67036.700000 2021-06-08 11:43:16,717 | test | DEBUG | pool-5-thread-2 | [task:print_ep_queue_size:3508] ep_queue_size for default: 6580207 2021-06-08 11:43:16,720 | infra | DEBUG | pool-5-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:43:16,720 | infra | DEBUG | pool-5-thread-2 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:43:16,855 | infra | INFO | pool-5-thread-18 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623177773.74_create__6150000_6200000_1623177773.76' completed 2021-06-08 11:43:16,882 | infra | INFO | pool-5-thread-29 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623177773.74_create__6050000_6100000_1623177773.76' completed 2021-06-08 11:43:16,882 | infra | DEBUG | pool-5-thread-30 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623177773.74_create__6050000_6100000_1623177773.76 are 50000 2021-06-08 11:43:16,884 | infra | DEBUG | pool-5-thread-30 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623177773.74_create__6100000_6150000_1623177773.76 2021-06-08 11:43:16,891 | infra | INFO | pool-5-thread-6 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623177773.74_create__6550000_6600000_1623177773.76' completed 2021-06-08 11:43:16,895 | infra | INFO | pool-5-thread-11 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623177773.74_create__6100000_6150000_1623177773.76' completed 2021-06-08 11:43:16,897 | infra | DEBUG | pool-5-thread-30 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623177773.74_create__6100000_6150000_1623177773.76 are 50000 2021-06-08 11:43:16,897 | infra | DEBUG | pool-5-thread-30 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623177773.74_create__6150000_6200000_1623177773.76 2021-06-08 11:43:16,897 | infra | DEBUG | pool-5-thread-30 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623177773.74_create__6150000_6200000_1623177773.76 are 50000 2021-06-08 11:43:16,897 | infra | DEBUG | pool-5-thread-30 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623177773.74_create__6200000_6250000_1623177773.76 2021-06-08 11:43:16,911 | infra | INFO | pool-5-thread-25 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623177773.74_create__6200000_6250000_1623177773.76' completed 2021-06-08 11:43:16,911 | infra | DEBUG | pool-5-thread-30 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623177773.74_create__6200000_6250000_1623177773.76 are 50000 2021-06-08 11:43:16,911 | infra | DEBUG | pool-5-thread-30 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623177773.74_create__6250000_6300000_1623177773.76 2021-06-08 11:43:16,911 | infra | DEBUG | pool-5-thread-30 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623177773.74_create__6250000_6300000_1623177773.76 are 50000 2021-06-08 11:43:16,911 | infra | DEBUG | pool-5-thread-30 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623177773.74_create__6300000_6350000_1623177773.76 2021-06-08 11:43:16,911 | infra | DEBUG | pool-5-thread-30 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623177773.74_create__6300000_6350000_1623177773.76 are 50000 2021-06-08 11:43:16,911 | infra | DEBUG | pool-5-thread-30 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623177773.74_create__6350000_6400000_1623177773.76 2021-06-08 11:43:16,921 | infra | INFO | pool-5-thread-20 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623177773.74_create__6950000_7000000_1623177773.76' completed 2021-06-08 11:43:17,240 | infra | INFO | pool-5-thread-13 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623177773.74_create__6350000_6400000_1623177773.76' completed 2021-06-08 11:43:17,242 | infra | DEBUG | pool-5-thread-30 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623177773.74_create__6350000_6400000_1623177773.76 are 50000 2021-06-08 11:43:17,242 | infra | DEBUG | pool-5-thread-30 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623177773.74_create__6400000_6450000_1623177773.76 2021-06-08 11:43:17,269 | infra | INFO | pool-5-thread-5 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623177773.74_create__6750000_6800000_1623177773.76' completed 2021-06-08 11:43:17,289 | infra | INFO | pool-5-thread-12 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623177773.74_create__6400000_6450000_1623177773.76' completed 2021-06-08 11:43:17,289 | infra | DEBUG | pool-5-thread-30 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623177773.74_create__6400000_6450000_1623177773.76 are 50000 2021-06-08 11:43:17,289 | infra | DEBUG | pool-5-thread-30 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623177773.74_create__6450000_6500000_1623177773.76 2021-06-08 11:43:17,289 | infra | DEBUG | pool-5-thread-30 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623177773.74_create__6450000_6500000_1623177773.76 are 50000 2021-06-08 11:43:17,289 | infra | DEBUG | pool-5-thread-30 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623177773.74_create__6500000_6550000_1623177773.76 2021-06-08 11:43:17,290 | infra | INFO | pool-5-thread-23 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623177773.74_create__6500000_6550000_1623177773.76' completed 2021-06-08 11:43:17,292 | infra | DEBUG | pool-5-thread-30 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623177773.74_create__6500000_6550000_1623177773.76 are 50000 2021-06-08 11:43:17,292 | infra | DEBUG | pool-5-thread-30 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623177773.74_create__6550000_6600000_1623177773.76 2021-06-08 11:43:17,292 | infra | DEBUG | pool-5-thread-30 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623177773.74_create__6550000_6600000_1623177773.76 are 50000 2021-06-08 11:43:17,292 | infra | DEBUG | pool-5-thread-30 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623177773.74_create__6600000_6650000_1623177773.76 2021-06-08 11:43:17,292 | infra | DEBUG | pool-5-thread-30 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623177773.74_create__6600000_6650000_1623177773.76 are 50000 2021-06-08 11:43:17,292 | infra | DEBUG | pool-5-thread-30 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623177773.74_create__6650000_6700000_1623177773.76 2021-06-08 11:43:17,293 | infra | INFO | pool-5-thread-21 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623177773.74_create__6650000_6700000_1623177773.76' completed 2021-06-08 11:43:17,293 | infra | DEBUG | pool-5-thread-30 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623177773.74_create__6650000_6700000_1623177773.76 are 50000 2021-06-08 11:43:17,295 | infra | DEBUG | pool-5-thread-30 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623177773.74_create__6700000_6750000_1623177773.76 2021-06-08 11:43:17,295 | infra | DEBUG | pool-5-thread-30 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623177773.74_create__6700000_6750000_1623177773.76 are 50000 2021-06-08 11:43:17,295 | infra | DEBUG | pool-5-thread-30 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623177773.74_create__6750000_6800000_1623177773.76 2021-06-08 11:43:17,295 | infra | DEBUG | pool-5-thread-30 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623177773.74_create__6750000_6800000_1623177773.76 are 50000 2021-06-08 11:43:17,295 | infra | DEBUG | pool-5-thread-30 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623177773.74_create__6800000_6850000_1623177773.76 2021-06-08 11:43:17,305 | infra | INFO | pool-5-thread-16 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623177773.74_create__6800000_6850000_1623177773.76' completed 2021-06-08 11:43:17,305 | infra | DEBUG | pool-5-thread-30 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623177773.74_create__6800000_6850000_1623177773.76 are 50000 2021-06-08 11:43:17,305 | infra | DEBUG | pool-5-thread-30 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623177773.74_create__6850000_6900000_1623177773.76 2021-06-08 11:43:17,305 | infra | DEBUG | pool-5-thread-30 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623177773.74_create__6850000_6900000_1623177773.76 are 50000 2021-06-08 11:43:17,305 | infra | DEBUG | pool-5-thread-30 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623177773.74_create__6900000_6950000_1623177773.76 2021-06-08 11:43:17,305 | infra | DEBUG | pool-5-thread-30 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623177773.74_create__6900000_6950000_1623177773.76 are 50000 2021-06-08 11:43:17,305 | infra | DEBUG | pool-5-thread-30 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623177773.74_create__6950000_7000000_1623177773.76 2021-06-08 11:43:17,305 | infra | DEBUG | pool-5-thread-30 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623177773.74_create__6950000_7000000_1623177773.76 are 50000 2021-06-08 11:43:17,305 | infra | DEBUG | pool-5-thread-30 | [task_manager:get_task_result:26] Getting task result for PrintBucketStats_default_1623177773.76 2021-06-08 11:43:17,721 | test | INFO | pool-5-thread-2 | [table_view:display:72] Ops trend for bucket 'default' +-------+------------------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-------+------------------------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | ******.............................*******...................................................*....................***X | 67036.700 | +-------+------------------------------------------------------------------------------------------------------------------------+-----------+ 2021-06-08 11:43:17,721 | infra | INFO | pool-5-thread-2 | [task:complete_task:102] Thread 'PrintBucketStats_default_1623177773.76' completed 2021-06-08 11:43:17,721 | infra | DEBUG | pool-5-thread-30 | [task:call:1663] ========= Tasks in loadgen pool======= 2021-06-08 11:43:17,723 | infra | WARNING | pool-5-thread-30 | [task_manager:print_tasks_in_pool:83] Task 'LoadDocsGen_default__default__default_create_6000000_7000000_1623177773.74' not completed 2021-06-08 11:43:17,723 | infra | DEBUG | pool-5-thread-30 | [task:call:1665] ====================================== 2021-06-08 11:43:17,723 | infra | DEBUG | pool-5-thread-30 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623177773.74_create__6000000_6050000_1623177773.76' complete. Loaded 50000 items 2021-06-08 11:43:17,723 | infra | DEBUG | pool-5-thread-30 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623177773.74_create__6050000_6100000_1623177773.76' complete. Loaded 50000 items 2021-06-08 11:43:17,723 | infra | DEBUG | pool-5-thread-30 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623177773.74_create__6100000_6150000_1623177773.76' complete. Loaded 50000 items 2021-06-08 11:43:17,723 | infra | DEBUG | pool-5-thread-30 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623177773.74_create__6150000_6200000_1623177773.76' complete. Loaded 50000 items 2021-06-08 11:43:17,723 | infra | DEBUG | pool-5-thread-30 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623177773.74_create__6200000_6250000_1623177773.76' complete. Loaded 50000 items 2021-06-08 11:43:17,723 | infra | DEBUG | pool-5-thread-30 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623177773.74_create__6250000_6300000_1623177773.76' complete. Loaded 50000 items 2021-06-08 11:43:17,723 | infra | DEBUG | pool-5-thread-30 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623177773.74_create__6300000_6350000_1623177773.76' complete. Loaded 50000 items 2021-06-08 11:43:17,723 | infra | DEBUG | pool-5-thread-30 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623177773.74_create__6350000_6400000_1623177773.76' complete. Loaded 50000 items 2021-06-08 11:43:17,723 | infra | DEBUG | pool-5-thread-30 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623177773.74_create__6400000_6450000_1623177773.76' complete. Loaded 50000 items 2021-06-08 11:43:17,723 | infra | DEBUG | pool-5-thread-30 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623177773.74_create__6450000_6500000_1623177773.76' complete. Loaded 50000 items 2021-06-08 11:43:17,723 | infra | DEBUG | pool-5-thread-30 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623177773.74_create__6500000_6550000_1623177773.76' complete. Loaded 50000 items 2021-06-08 11:43:17,723 | infra | DEBUG | pool-5-thread-30 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623177773.74_create__6550000_6600000_1623177773.76' complete. Loaded 50000 items 2021-06-08 11:43:17,723 | infra | DEBUG | pool-5-thread-30 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623177773.74_create__6600000_6650000_1623177773.76' complete. Loaded 50000 items 2021-06-08 11:43:17,723 | infra | DEBUG | pool-5-thread-30 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623177773.74_create__6650000_6700000_1623177773.76' complete. Loaded 50000 items 2021-06-08 11:43:17,723 | infra | DEBUG | pool-5-thread-30 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623177773.74_create__6700000_6750000_1623177773.76' complete. Loaded 50000 items 2021-06-08 11:43:17,724 | infra | DEBUG | pool-5-thread-30 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623177773.74_create__6750000_6800000_1623177773.76' complete. Loaded 50000 items 2021-06-08 11:43:17,724 | infra | DEBUG | pool-5-thread-30 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623177773.74_create__6800000_6850000_1623177773.76' complete. Loaded 50000 items 2021-06-08 11:43:17,724 | infra | DEBUG | pool-5-thread-30 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623177773.74_create__6850000_6900000_1623177773.76' complete. Loaded 50000 items 2021-06-08 11:43:17,724 | infra | DEBUG | pool-5-thread-30 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623177773.74_create__6900000_6950000_1623177773.76' complete. Loaded 50000 items 2021-06-08 11:43:17,724 | infra | DEBUG | pool-5-thread-30 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_6000000_7000000_1623177773.74_create__6950000_7000000_1623177773.76' complete. Loaded 50000 items 2021-06-08 11:43:17,724 | infra | INFO | pool-5-thread-30 | [task:complete_task:102] Thread 'LoadDocsGen_default__default__default_create_6000000_7000000_1623177773.74' completed 2021-06-08 11:43:17,727 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.125 with username: root, password: couchbase 2021-06-08 11:43:17,815 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 11:43:18,033 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: ls /etc/ 2021-06-08 11:43:18,095 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 11:43:18,095 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: ls /cygdrive/c/Windows 2021-06-08 11:43:18,109 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: uname -m 2021-06-08 11:43:18,124 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: cat /proc/cpuinfo 2021-06-08 11:43:18,137 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: df -Thl 2021-06-08 11:43:18,154 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: cat /proc/meminfo 2021-06-08 11:43:18,168 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: hostname 2021-06-08 11:43:18,180 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: hostname -d 2021-06-08 11:43:18,194 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:559] Starting StatsWaitTask for ep_data_write_failed on bucket default 2021-06-08 11:43:18,194 | infra | INFO | MainThread | [task_manager:add_new_task:23] Added new task: StatsWaitTask_default_ep_data_write_failed_1623177798.19 2021-06-08 11:43:18,194 | infra | INFO | pool-5-thread-24 | [task:start_task:87] Thread 'StatsWaitTask_default_ep_data_write_failed_1623177798.19' started 2021-06-08 11:43:18,196 | infra | DEBUG | pool-5-thread-24 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:43:18,197 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.155 with username: root, password: couchbase 2021-06-08 11:43:18,289 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 11:43:18,408 | test | DEBUG | pool-5-thread-24 | [task:_compare:2649] Comparing 93895 > 0 2021-06-08 11:43:18,408 | test | DEBUG | pool-5-thread-24 | [task:_get_all_stats_and_compare:2601] Ready: ep_data_write_failed 93895 > 0. Received: {'172.23.106.125': '93895'} for bucket 'default' 2021-06-08 11:43:18,408 | infra | INFO | pool-5-thread-24 | [task:complete_task:102] Thread 'StatsWaitTask_default_ep_data_write_failed_1623177798.19' completed 2021-06-08 11:43:18,509 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: ls /etc/ 2021-06-08 11:43:18,571 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 11:43:18,571 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: ls /cygdrive/c/Windows 2021-06-08 11:43:18,585 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: uname -m 2021-06-08 11:43:18,598 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: cat /proc/cpuinfo 2021-06-08 11:43:18,609 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: df -Thl 2021-06-08 11:43:18,627 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: cat /proc/meminfo 2021-06-08 11:43:18,640 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: hostname 2021-06-08 11:43:18,654 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: hostname -d 2021-06-08 11:43:18,665 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:559] Starting StatsWaitTask for ep_data_write_failed on bucket default 2021-06-08 11:43:18,667 | infra | INFO | MainThread | [task_manager:add_new_task:23] Added new task: StatsWaitTask_default_ep_data_write_failed_1623177798.67 2021-06-08 11:43:18,667 | infra | INFO | pool-5-thread-27 | [task:start_task:87] Thread 'StatsWaitTask_default_ep_data_write_failed_1623177798.67' started 2021-06-08 11:43:18,667 | infra | DEBUG | pool-5-thread-27 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:43:18,670 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.156 with username: root, password: couchbase 2021-06-08 11:43:18,759 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 11:43:18,884 | test | DEBUG | pool-5-thread-27 | [task:_compare:2649] Comparing 290855 > 0 2021-06-08 11:43:18,884 | test | DEBUG | pool-5-thread-27 | [task:_get_all_stats_and_compare:2601] Ready: ep_data_write_failed 290855 > 0. Received: {'172.23.106.155': '290855'} for bucket 'default' 2021-06-08 11:43:18,884 | infra | INFO | pool-5-thread-27 | [task:complete_task:102] Thread 'StatsWaitTask_default_ep_data_write_failed_1623177798.67' completed 2021-06-08 11:43:18,966 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: ls /etc/ 2021-06-08 11:43:19,019 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 11:43:19,019 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: ls /cygdrive/c/Windows 2021-06-08 11:43:19,032 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: uname -m 2021-06-08 11:43:19,043 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: cat /proc/cpuinfo 2021-06-08 11:43:19,055 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: df -Thl 2021-06-08 11:43:19,069 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: cat /proc/meminfo 2021-06-08 11:43:19,082 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: hostname 2021-06-08 11:43:19,092 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: hostname -d 2021-06-08 11:43:19,105 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:559] Starting StatsWaitTask for ep_data_write_failed on bucket default 2021-06-08 11:43:19,105 | infra | INFO | MainThread | [task_manager:add_new_task:23] Added new task: StatsWaitTask_default_ep_data_write_failed_1623177799.11 2021-06-08 11:43:19,105 | infra | INFO | pool-5-thread-4 | [task:start_task:87] Thread 'StatsWaitTask_default_ep_data_write_failed_1623177799.11' started 2021-06-08 11:43:19,105 | infra | DEBUG | pool-5-thread-4 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:43:19,108 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.122 with username: root, password: couchbase 2021-06-08 11:43:19,178 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 11:43:19,395 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: ls /etc/ 2021-06-08 11:43:19,454 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 11:43:19,454 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: ls /cygdrive/c/Windows 2021-06-08 11:43:19,469 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: uname -m 2021-06-08 11:43:19,482 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: cat /proc/cpuinfo 2021-06-08 11:43:19,494 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: df -Thl 2021-06-08 11:43:19,510 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: cat /proc/meminfo 2021-06-08 11:43:19,523 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: hostname 2021-06-08 11:43:19,536 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: hostname -d 2021-06-08 11:43:19,552 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:559] Starting StatsWaitTask for ep_data_write_failed on bucket default 2021-06-08 11:43:19,552 | infra | INFO | MainThread | [task_manager:add_new_task:23] Added new task: StatsWaitTask_default_ep_data_write_failed_1623177799.55 2021-06-08 11:43:19,552 | infra | INFO | pool-5-thread-3 | [task:start_task:87] Thread 'StatsWaitTask_default_ep_data_write_failed_1623177799.55' started 2021-06-08 11:43:19,552 | infra | DEBUG | MainThread | [task_manager:get_task_result:26] Getting task result for StatsWaitTask_default_ep_data_write_failed_1623177798.19 2021-06-08 11:43:19,553 | infra | DEBUG | MainThread | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.125 2021-06-08 11:43:19,553 | infra | DEBUG | pool-5-thread-3 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:43:19,553 | infra | DEBUG | MainThread | [task_manager:get_task_result:26] Getting task result for StatsWaitTask_default_ep_data_write_failed_1623177798.67 2021-06-08 11:43:19,555 | infra | DEBUG | MainThread | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.155 2021-06-08 11:43:19,555 | infra | DEBUG | MainThread | [task_manager:get_task_result:26] Getting task result for StatsWaitTask_default_ep_data_write_failed_1623177799.11 2021-06-08 11:43:19,767 | test | DEBUG | pool-5-thread-3 | [task:_compare:2649] Comparing 92689 > 0 2021-06-08 11:43:19,769 | test | DEBUG | pool-5-thread-3 | [task:_get_all_stats_and_compare:2601] Ready: ep_data_write_failed 92689 > 0. Received: {'172.23.106.122': '92689'} for bucket 'default' 2021-06-08 11:43:19,769 | infra | INFO | pool-5-thread-3 | [task:complete_task:102] Thread 'StatsWaitTask_default_ep_data_write_failed_1623177799.55' completed 2021-06-08 11:43:20,996 | test | DEBUG | pool-5-thread-4 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 11:43:20,996 | infra | DEBUG | pool-5-thread-4 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.156': '0'} for bucket 'default' 2021-06-08 11:43:20,996 | infra | DEBUG | pool-5-thread-4 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 11:43:20,999 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 11:43:20,999 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 11:43:25,999 | infra | DEBUG | pool-5-thread-4 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:43:26,203 | test | DEBUG | pool-5-thread-4 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 11:43:26,203 | infra | DEBUG | pool-5-thread-4 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.156': '0'} for bucket 'default' 2021-06-08 11:43:26,203 | infra | DEBUG | pool-5-thread-4 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 11:43:26,204 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 11:43:26,206 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 11:43:31,206 | infra | DEBUG | pool-5-thread-4 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:44:01,348 | test | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 11:44:01,348 | test | INFO | pool-5-thread-4 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1010, in main() File "/opt/couchbase/lib/python/cbstats", line 1007, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 49, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 79, in g mc.sasl_auth_plain(username, password) File "/opt/couchbase/lib/python/mc_bin_client.py", line 488, in sasl_auth_plain return self.sasl_auth_start('PLAIN', '\0'.join([foruser, user, password])) File "/opt/couchbase/lib/python/mc_bin_client.py", line 484, in sasl_auth_start return self._doCmd(memcacheConstants.CMD_SASL_AUTH, mech, data) File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 281, in _handleKeyedResponse cmd, errcode, opaque, cas, keylen, extralen, rv = self._recvMsg() File "/opt/couchbase/lib/python/mc_bin_client.py", line 250, in _recvMsg data = self._socketRecv(MIN_RECV_PACKET - len(response)) File "/opt/couchbase/lib/python/mc_bin_client.py", line 245, in _socketRecv raise TimeoutError(30) mc_bin_client.TimeoutError: Error: Operation timed out (30 seconds) Please check list of arguments (e.g., IP address, port number) passed or the connectivity to a server to be connected 2021-06-08 11:44:06,348 | infra | DEBUG | pool-5-thread-4 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:44:19,832 | test | DEBUG | pool-5-thread-4 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 11:44:19,832 | infra | DEBUG | pool-5-thread-4 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.156': '0'} for bucket 'default' 2021-06-08 11:44:19,832 | infra | DEBUG | pool-5-thread-4 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 11:44:19,835 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 11:44:19,835 | infra | DEBUG | pool-5-thread-4 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 11:44:24,835 | infra | DEBUG | pool-5-thread-4 | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.156 2021-06-08 11:44:24,836 | infra | INFO | pool-5-thread-4 | [task:complete_task:102] Thread 'StatsWaitTask_default_ep_data_write_failed_1623177799.11' completed 2021-06-08 11:44:24,842 | infra | INFO | MainThread | [task_manager:add_new_task:23] Added new task: LoadDocsGen_default__default__default_create_7000000_8000000_1623177864.84 2021-06-08 11:44:24,842 | infra | INFO | pool-5-thread-15 | [task:start_task:87] Thread 'LoadDocsGen_default__default__default_create_7000000_8000000_1623177864.84' started 2021-06-08 11:44:24,842 | infra | DEBUG | MainThread | [task_manager:get_task_result:26] Getting task result for LoadDocsGen_default__default__default_create_7000000_8000000_1623177864.84 2021-06-08 11:44:24,867 | infra | INFO | pool-5-thread-15 | [task_manager:add_new_task:23] Added new task: PrintBucketStats_default_1623177864.86 2021-06-08 11:44:24,867 | infra | INFO | pool-5-thread-22 | [task:start_task:87] Thread 'PrintBucketStats_default_1623177864.86' started 2021-06-08 11:44:24,867 | infra | INFO | pool-5-thread-15 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623177864.84_create__7000000_7050000_1623177864.86 2021-06-08 11:44:24,867 | infra | INFO | pool-5-thread-7 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623177864.84_create__7000000_7050000_1623177864.86' started 2021-06-08 11:44:24,867 | infra | INFO | pool-5-thread-15 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623177864.84_create__7050000_7100000_1623177864.86 2021-06-08 11:44:24,867 | infra | INFO | pool-5-thread-8 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623177864.84_create__7050000_7100000_1623177864.86' started 2021-06-08 11:44:24,868 | infra | INFO | pool-5-thread-15 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623177864.84_create__7100000_7150000_1623177864.86 2021-06-08 11:44:24,868 | infra | INFO | pool-5-thread-14 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623177864.84_create__7100000_7150000_1623177864.86' started 2021-06-08 11:44:24,868 | infra | INFO | pool-5-thread-15 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623177864.84_create__7150000_7200000_1623177864.86 2021-06-08 11:44:24,868 | infra | INFO | pool-5-thread-17 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623177864.84_create__7150000_7200000_1623177864.86' started 2021-06-08 11:44:24,868 | infra | INFO | pool-5-thread-15 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623177864.84_create__7200000_7250000_1623177864.86 2021-06-08 11:44:24,868 | infra | INFO | pool-5-thread-1 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623177864.84_create__7200000_7250000_1623177864.86' started 2021-06-08 11:44:24,868 | infra | INFO | pool-5-thread-15 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623177864.84_create__7250000_7300000_1623177864.86 2021-06-08 11:44:24,868 | infra | INFO | pool-5-thread-26 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623177864.84_create__7250000_7300000_1623177864.86' started 2021-06-08 11:44:24,868 | infra | INFO | pool-5-thread-19 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623177864.84_create__7300000_7350000_1623177864.86' started 2021-06-08 11:44:24,868 | infra | INFO | pool-5-thread-15 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623177864.84_create__7300000_7350000_1623177864.86 2021-06-08 11:44:24,869 | infra | INFO | pool-5-thread-15 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623177864.84_create__7350000_7400000_1623177864.86 2021-06-08 11:44:24,869 | infra | INFO | pool-5-thread-28 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623177864.84_create__7400000_7450000_1623177864.86' started 2021-06-08 11:44:24,869 | infra | INFO | pool-5-thread-9 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623177864.84_create__7350000_7400000_1623177864.86' started 2021-06-08 11:44:24,869 | infra | INFO | pool-5-thread-15 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623177864.84_create__7400000_7450000_1623177864.86 2021-06-08 11:44:24,869 | infra | INFO | pool-5-thread-15 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623177864.84_create__7450000_7500000_1623177864.86 2021-06-08 11:44:24,869 | infra | INFO | pool-5-thread-10 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623177864.84_create__7450000_7500000_1623177864.86' started 2021-06-08 11:44:24,869 | infra | INFO | pool-5-thread-15 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623177864.84_create__7500000_7550000_1623177864.86 2021-06-08 11:44:24,871 | infra | INFO | pool-5-thread-18 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623177864.84_create__7500000_7550000_1623177864.86' started 2021-06-08 11:44:24,871 | infra | INFO | pool-5-thread-15 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623177864.84_create__7550000_7600000_1623177864.86 2021-06-08 11:44:24,871 | infra | INFO | pool-5-thread-29 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623177864.84_create__7550000_7600000_1623177864.86' started 2021-06-08 11:44:24,871 | infra | INFO | pool-5-thread-15 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623177864.84_create__7600000_7650000_1623177864.86 2021-06-08 11:44:24,871 | infra | INFO | pool-5-thread-6 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623177864.84_create__7600000_7650000_1623177864.86' started 2021-06-08 11:44:24,871 | infra | INFO | pool-5-thread-15 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623177864.84_create__7650000_7700000_1623177864.86 2021-06-08 11:44:24,871 | infra | INFO | pool-5-thread-11 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623177864.84_create__7650000_7700000_1623177864.86' started 2021-06-08 11:44:24,871 | infra | INFO | pool-5-thread-15 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623177864.84_create__7700000_7750000_1623177864.86 2021-06-08 11:44:24,871 | infra | INFO | pool-5-thread-25 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623177864.84_create__7700000_7750000_1623177864.86' started 2021-06-08 11:44:24,871 | infra | INFO | pool-5-thread-20 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623177864.84_create__7750000_7800000_1623177864.86' started 2021-06-08 11:44:24,871 | infra | INFO | pool-5-thread-15 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623177864.84_create__7750000_7800000_1623177864.86 2021-06-08 11:44:24,872 | infra | INFO | pool-5-thread-15 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623177864.84_create__7800000_7850000_1623177864.86 2021-06-08 11:44:24,872 | infra | INFO | pool-5-thread-13 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623177864.84_create__7800000_7850000_1623177864.86' started 2021-06-08 11:44:24,874 | infra | INFO | pool-5-thread-15 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623177864.84_create__7850000_7900000_1623177864.86 2021-06-08 11:44:24,874 | infra | INFO | pool-5-thread-5 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623177864.84_create__7850000_7900000_1623177864.86' started 2021-06-08 11:44:24,874 | infra | INFO | pool-5-thread-15 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623177864.84_create__7900000_7950000_1623177864.86 2021-06-08 11:44:24,874 | infra | INFO | pool-5-thread-15 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623177864.84_create__7950000_8000000_1623177864.86 2021-06-08 11:44:24,874 | infra | INFO | pool-5-thread-23 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623177864.84_create__7950000_8000000_1623177864.86' started 2021-06-08 11:44:24,874 | infra | INFO | pool-5-thread-12 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623177864.84_create__7900000_7950000_1623177864.86' started 2021-06-08 11:44:24,878 | infra | DEBUG | pool-5-thread-15 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623177864.84_create__7000000_7050000_1623177864.86 2021-06-08 11:44:25,124 | test | DEBUG | pool-5-thread-22 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:44:25,124 | test | DEBUG | pool-5-thread-22 | [task:print_ep_queue_size:3508] ep_queue_size for default: 6652458 2021-06-08 11:44:25,125 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:44:25,267 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:44:26,515 | test | DEBUG | pool-5-thread-22 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:44:26,515 | test | DEBUG | pool-5-thread-22 | [task:print_ep_queue_size:3508] ep_queue_size for default: 6652458 2021-06-08 11:44:26,516 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:44:26,516 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:44:27,749 | test | DEBUG | pool-5-thread-22 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:44:27,749 | test | DEBUG | pool-5-thread-22 | [task:print_ep_queue_size:3508] ep_queue_size for default: 6652458 2021-06-08 11:44:27,750 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:44:27,750 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:44:29,040 | test | DEBUG | pool-5-thread-22 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:44:29,040 | test | DEBUG | pool-5-thread-22 | [task:print_ep_queue_size:3508] ep_queue_size for default: 6652458 2021-06-08 11:44:29,043 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:44:29,043 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:44:30,286 | test | DEBUG | pool-5-thread-22 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:44:30,286 | test | DEBUG | pool-5-thread-22 | [task:print_ep_queue_size:3508] ep_queue_size for default: 6652458 2021-06-08 11:44:30,289 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:44:30,289 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:44:31,533 | test | DEBUG | pool-5-thread-22 | [task:record_bucket_ops:3495] Ops rate for 'default': 6448.900000 2021-06-08 11:44:31,535 | test | DEBUG | pool-5-thread-22 | [task:print_ep_queue_size:3508] ep_queue_size for default: 6685033 2021-06-08 11:44:31,536 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:44:31,536 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:44:32,779 | test | DEBUG | pool-5-thread-22 | [task:record_bucket_ops:3495] Ops rate for 'default': 6448.900000 2021-06-08 11:44:32,779 | test | DEBUG | pool-5-thread-22 | [task:print_ep_queue_size:3508] ep_queue_size for default: 6685033 2021-06-08 11:44:32,780 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:44:32,780 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:44:34,015 | test | DEBUG | pool-5-thread-22 | [task:record_bucket_ops:3495] Ops rate for 'default': 6448.900000 2021-06-08 11:44:34,015 | test | DEBUG | pool-5-thread-22 | [task:print_ep_queue_size:3508] ep_queue_size for default: 6685033 2021-06-08 11:44:34,016 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:44:34,016 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:44:35,250 | test | DEBUG | pool-5-thread-22 | [task:record_bucket_ops:3495] Ops rate for 'default': 6448.900000 2021-06-08 11:44:35,250 | test | DEBUG | pool-5-thread-22 | [task:print_ep_queue_size:3508] ep_queue_size for default: 6685033 2021-06-08 11:44:35,250 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:44:35,250 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:44:36,486 | test | DEBUG | pool-5-thread-22 | [task:record_bucket_ops:3495] Ops rate for 'default': 6448.900000 2021-06-08 11:44:36,486 | test | DEBUG | pool-5-thread-22 | [task:print_ep_queue_size:3508] ep_queue_size for default: 6685033 2021-06-08 11:44:36,489 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:44:36,489 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:44:37,730 | test | DEBUG | pool-5-thread-22 | [task:record_bucket_ops:3495] Ops rate for 'default': 6448.900000 2021-06-08 11:44:37,730 | test | DEBUG | pool-5-thread-22 | [task:print_ep_queue_size:3508] ep_queue_size for default: 6685033 2021-06-08 11:44:37,732 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:44:37,732 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:44:38,961 | test | DEBUG | pool-5-thread-22 | [task:record_bucket_ops:3495] Ops rate for 'default': 6448.900000 2021-06-08 11:44:38,963 | test | DEBUG | pool-5-thread-22 | [task:print_ep_queue_size:3508] ep_queue_size for default: 6685033 2021-06-08 11:44:38,963 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:44:38,963 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:44:40,244 | test | DEBUG | pool-5-thread-22 | [task:record_bucket_ops:3495] Ops rate for 'default': 8527.800000 2021-06-08 11:44:40,246 | test | DEBUG | pool-5-thread-22 | [task:print_ep_queue_size:3508] ep_queue_size for default: 6685033 2021-06-08 11:44:40,247 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:44:40,247 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:44:41,482 | test | DEBUG | pool-5-thread-22 | [task:record_bucket_ops:3495] Ops rate for 'default': 2529.600000 2021-06-08 11:44:41,484 | test | DEBUG | pool-5-thread-22 | [task:print_ep_queue_size:3508] ep_queue_size for default: 6704855 2021-06-08 11:44:41,486 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:44:41,486 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:44:42,826 | test | DEBUG | pool-5-thread-22 | [task:record_bucket_ops:3495] Ops rate for 'default': 2529.600000 2021-06-08 11:44:42,828 | test | DEBUG | pool-5-thread-22 | [task:print_ep_queue_size:3508] ep_queue_size for default: 6704855 2021-06-08 11:44:42,829 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:44:42,831 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:44:44,145 | test | DEBUG | pool-5-thread-22 | [task:record_bucket_ops:3495] Ops rate for 'default': 2529.600000 2021-06-08 11:44:44,147 | test | DEBUG | pool-5-thread-22 | [task:print_ep_queue_size:3508] ep_queue_size for default: 6704855 2021-06-08 11:44:44,151 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:44:44,151 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:44:45,398 | test | DEBUG | pool-5-thread-22 | [task:record_bucket_ops:3495] Ops rate for 'default': 2529.600000 2021-06-08 11:44:45,398 | test | DEBUG | pool-5-thread-22 | [task:print_ep_queue_size:3508] ep_queue_size for default: 6704855 2021-06-08 11:44:45,400 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:44:45,400 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:44:46,674 | test | DEBUG | pool-5-thread-22 | [task:record_bucket_ops:3495] Ops rate for 'default': 2529.600000 2021-06-08 11:44:46,674 | test | DEBUG | pool-5-thread-22 | [task:print_ep_queue_size:3508] ep_queue_size for default: 6704855 2021-06-08 11:44:46,677 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:44:46,677 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:44:48,022 | test | DEBUG | pool-5-thread-22 | [task:record_bucket_ops:3495] Ops rate for 'default': 2529.600000 2021-06-08 11:44:48,023 | test | DEBUG | pool-5-thread-22 | [task:print_ep_queue_size:3508] ep_queue_size for default: 6704855 2021-06-08 11:44:48,025 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:44:48,026 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:44:50,977 | test | DEBUG | pool-5-thread-22 | [task:record_bucket_ops:3495] Ops rate for 'default': 2529.600000 2021-06-08 11:44:50,979 | test | DEBUG | pool-5-thread-22 | [task:print_ep_queue_size:3508] ep_queue_size for default: 6704855 2021-06-08 11:44:50,982 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:44:50,982 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:44:52,247 | test | DEBUG | pool-5-thread-22 | [task:record_bucket_ops:3495] Ops rate for 'default': 18728.600000 2021-06-08 11:44:52,247 | test | DEBUG | pool-5-thread-22 | [task:print_ep_queue_size:3508] ep_queue_size for default: 5208907 2021-06-08 11:44:52,250 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:44:52,250 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:44:53,510 | test | DEBUG | pool-5-thread-22 | [task:record_bucket_ops:3495] Ops rate for 'default': 18728.600000 2021-06-08 11:44:53,510 | test | DEBUG | pool-5-thread-22 | [task:print_ep_queue_size:3508] ep_queue_size for default: 5208907 2021-06-08 11:44:53,513 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:44:53,513 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:44:54,749 | test | DEBUG | pool-5-thread-22 | [task:record_bucket_ops:3495] Ops rate for 'default': 18728.600000 2021-06-08 11:44:54,749 | test | DEBUG | pool-5-thread-22 | [task:print_ep_queue_size:3508] ep_queue_size for default: 5208907 2021-06-08 11:44:54,750 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:44:54,750 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:44:55,982 | test | DEBUG | pool-5-thread-22 | [task:record_bucket_ops:3495] Ops rate for 'default': 18728.600000 2021-06-08 11:44:55,982 | test | DEBUG | pool-5-thread-22 | [task:print_ep_queue_size:3508] ep_queue_size for default: 5208907 2021-06-08 11:44:55,983 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:44:55,983 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:44:57,224 | test | DEBUG | pool-5-thread-22 | [task:record_bucket_ops:3495] Ops rate for 'default': 18728.600000 2021-06-08 11:44:57,224 | test | DEBUG | pool-5-thread-22 | [task:print_ep_queue_size:3508] ep_queue_size for default: 5208907 2021-06-08 11:44:57,226 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:44:57,226 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:44:58,459 | test | DEBUG | pool-5-thread-22 | [task:record_bucket_ops:3495] Ops rate for 'default': 18728.600000 2021-06-08 11:44:58,459 | test | DEBUG | pool-5-thread-22 | [task:print_ep_queue_size:3508] ep_queue_size for default: 5208907 2021-06-08 11:44:58,460 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:44:58,460 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:44:59,700 | test | DEBUG | pool-5-thread-22 | [task:record_bucket_ops:3495] Ops rate for 'default': 18728.600000 2021-06-08 11:44:59,700 | test | DEBUG | pool-5-thread-22 | [task:print_ep_queue_size:3508] ep_queue_size for default: 5208907 2021-06-08 11:44:59,700 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:44:59,700 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:45:00,938 | test | DEBUG | pool-5-thread-22 | [task:record_bucket_ops:3495] Ops rate for 'default': 18728.600000 2021-06-08 11:45:00,938 | test | DEBUG | pool-5-thread-22 | [task:print_ep_queue_size:3508] ep_queue_size for default: 5208907 2021-06-08 11:45:00,940 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:45:00,940 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:45:02,163 | test | DEBUG | pool-5-thread-22 | [task:record_bucket_ops:3495] Ops rate for 'default': 5078.700000 2021-06-08 11:45:02,163 | test | DEBUG | pool-5-thread-22 | [task:print_ep_queue_size:3508] ep_queue_size for default: 5208907 2021-06-08 11:45:02,164 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:45:02,164 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:45:03,404 | test | DEBUG | pool-5-thread-22 | [task:record_bucket_ops:3495] Ops rate for 'default': 5078.700000 2021-06-08 11:45:03,404 | test | DEBUG | pool-5-thread-22 | [task:print_ep_queue_size:3508] ep_queue_size for default: 5208907 2021-06-08 11:45:03,407 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:45:03,407 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:45:04,915 | test | DEBUG | pool-5-thread-22 | [task:record_bucket_ops:3495] Ops rate for 'default': 5078.700000 2021-06-08 11:45:04,915 | test | DEBUG | pool-5-thread-22 | [task:print_ep_queue_size:3508] ep_queue_size for default: 5208907 2021-06-08 11:45:04,921 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:45:04,921 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:45:06,253 | test | DEBUG | pool-5-thread-22 | [task:record_bucket_ops:3495] Ops rate for 'default': 5078.700000 2021-06-08 11:45:06,253 | test | DEBUG | pool-5-thread-22 | [task:print_ep_queue_size:3508] ep_queue_size for default: 5208907 2021-06-08 11:45:06,256 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:45:06,256 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:45:07,515 | test | DEBUG | pool-5-thread-22 | [task:record_bucket_ops:3495] Ops rate for 'default': 5078.700000 2021-06-08 11:45:07,515 | test | DEBUG | pool-5-thread-22 | [task:print_ep_queue_size:3508] ep_queue_size for default: 5208907 2021-06-08 11:45:07,516 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:45:07,517 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:45:08,780 | test | DEBUG | pool-5-thread-22 | [task:record_bucket_ops:3495] Ops rate for 'default': 5078.700000 2021-06-08 11:45:08,780 | test | DEBUG | pool-5-thread-22 | [task:print_ep_queue_size:3508] ep_queue_size for default: 5208907 2021-06-08 11:45:08,782 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:45:08,783 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:45:10,154 | test | DEBUG | pool-5-thread-22 | [task:record_bucket_ops:3495] Ops rate for 'default': 5078.700000 2021-06-08 11:45:10,154 | test | DEBUG | pool-5-thread-22 | [task:print_ep_queue_size:3508] ep_queue_size for default: 5208907 2021-06-08 11:45:10,158 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:45:10,160 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:45:11,614 | test | DEBUG | pool-5-thread-22 | [task:record_bucket_ops:3495] Ops rate for 'default': 12128.900000 2021-06-08 11:45:11,614 | test | DEBUG | pool-5-thread-22 | [task:print_ep_queue_size:3508] ep_queue_size for default: 5208907 2021-06-08 11:45:11,617 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:45:11,617 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:45:13,121 | test | DEBUG | pool-5-thread-22 | [task:record_bucket_ops:3495] Ops rate for 'default': 12128.900000 2021-06-08 11:45:13,121 | test | DEBUG | pool-5-thread-22 | [task:print_ep_queue_size:3508] ep_queue_size for default: 5208907 2021-06-08 11:45:13,125 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:45:13,125 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:45:14,711 | test | DEBUG | pool-5-thread-22 | [task:record_bucket_ops:3495] Ops rate for 'default': 12128.900000 2021-06-08 11:45:14,713 | test | DEBUG | pool-5-thread-22 | [task:print_ep_queue_size:3508] ep_queue_size for default: 5208907 2021-06-08 11:45:14,717 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:45:14,717 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:45:15,950 | test | DEBUG | pool-5-thread-22 | [task:record_bucket_ops:3495] Ops rate for 'default': 12128.900000 2021-06-08 11:45:15,950 | test | DEBUG | pool-5-thread-22 | [task:print_ep_queue_size:3508] ep_queue_size for default: 5208907 2021-06-08 11:45:15,951 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:45:15,951 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:45:17,191 | test | DEBUG | pool-5-thread-22 | [task:record_bucket_ops:3495] Ops rate for 'default': 12128.900000 2021-06-08 11:45:17,191 | test | DEBUG | pool-5-thread-22 | [task:print_ep_queue_size:3508] ep_queue_size for default: 5208907 2021-06-08 11:45:17,193 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:45:17,194 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:45:18,438 | test | DEBUG | pool-5-thread-22 | [task:record_bucket_ops:3495] Ops rate for 'default': 12128.900000 2021-06-08 11:45:18,438 | test | DEBUG | pool-5-thread-22 | [task:print_ep_queue_size:3508] ep_queue_size for default: 5208907 2021-06-08 11:45:18,440 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:45:18,440 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:45:19,677 | test | DEBUG | pool-5-thread-22 | [task:record_bucket_ops:3495] Ops rate for 'default': 12128.900000 2021-06-08 11:45:19,677 | test | DEBUG | pool-5-thread-22 | [task:print_ep_queue_size:3508] ep_queue_size for default: 5208907 2021-06-08 11:45:19,677 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:45:19,678 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:45:20,918 | test | DEBUG | pool-5-thread-22 | [task:record_bucket_ops:3495] Ops rate for 'default': 12128.900000 2021-06-08 11:45:20,920 | test | DEBUG | pool-5-thread-22 | [task:print_ep_queue_size:3508] ep_queue_size for default: 5208907 2021-06-08 11:45:20,921 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:45:20,921 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:45:22,221 | test | DEBUG | pool-5-thread-22 | [task:record_bucket_ops:3495] Ops rate for 'default': 11100.000000 2021-06-08 11:45:22,223 | test | DEBUG | pool-5-thread-22 | [task:print_ep_queue_size:3508] ep_queue_size for default: 5208907 2021-06-08 11:45:22,224 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:45:22,224 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:45:23,694 | test | DEBUG | pool-5-thread-22 | [task:record_bucket_ops:3495] Ops rate for 'default': 13946.869188 2021-06-08 11:45:23,694 | test | DEBUG | pool-5-thread-22 | [task:print_ep_queue_size:3508] ep_queue_size for default: 6849322 2021-06-08 11:45:23,697 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:45:23,697 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:45:25,144 | test | DEBUG | pool-5-thread-22 | [task:record_bucket_ops:3495] Ops rate for 'default': 13946.869188 2021-06-08 11:45:25,144 | test | DEBUG | pool-5-thread-22 | [task:print_ep_queue_size:3508] ep_queue_size for default: 6849322 2021-06-08 11:45:25,147 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:45:25,147 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:45:26,519 | test | DEBUG | pool-5-thread-22 | [task:record_bucket_ops:3495] Ops rate for 'default': 13946.869188 2021-06-08 11:45:26,519 | test | DEBUG | pool-5-thread-22 | [task:print_ep_queue_size:3508] ep_queue_size for default: 6849322 2021-06-08 11:45:26,523 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:45:26,523 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:45:28,013 | test | DEBUG | pool-5-thread-22 | [task:record_bucket_ops:3495] Ops rate for 'default': 13946.869188 2021-06-08 11:45:28,013 | test | DEBUG | pool-5-thread-22 | [task:print_ep_queue_size:3508] ep_queue_size for default: 6849322 2021-06-08 11:45:28,016 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:45:28,016 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:45:29,388 | test | DEBUG | pool-5-thread-22 | [task:record_bucket_ops:3495] Ops rate for 'default': 13946.869188 2021-06-08 11:45:29,388 | test | DEBUG | pool-5-thread-22 | [task:print_ep_queue_size:3508] ep_queue_size for default: 6849322 2021-06-08 11:45:29,391 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:45:29,391 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:45:30,869 | test | DEBUG | pool-5-thread-22 | [task:record_bucket_ops:3495] Ops rate for 'default': 13946.869188 2021-06-08 11:45:30,869 | test | DEBUG | pool-5-thread-22 | [task:print_ep_queue_size:3508] ep_queue_size for default: 6849322 2021-06-08 11:45:30,872 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:45:30,872 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:45:32,269 | test | DEBUG | pool-5-thread-22 | [task:record_bucket_ops:3495] Ops rate for 'default': 17152.286933 2021-06-08 11:45:32,269 | test | DEBUG | pool-5-thread-22 | [task:print_ep_queue_size:3508] ep_queue_size for default: 6849322 2021-06-08 11:45:32,270 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:45:32,270 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:45:33,512 | test | DEBUG | pool-5-thread-22 | [task:record_bucket_ops:3495] Ops rate for 'default': 17152.286933 2021-06-08 11:45:33,512 | test | DEBUG | pool-5-thread-22 | [task:print_ep_queue_size:3508] ep_queue_size for default: 6849322 2021-06-08 11:45:33,513 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:45:33,513 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:45:34,736 | test | DEBUG | pool-5-thread-22 | [task:record_bucket_ops:3495] Ops rate for 'default': 17152.286933 2021-06-08 11:45:34,736 | test | DEBUG | pool-5-thread-22 | [task:print_ep_queue_size:3508] ep_queue_size for default: 6849322 2021-06-08 11:45:34,737 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:45:34,737 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:45:35,967 | test | DEBUG | pool-5-thread-22 | [task:record_bucket_ops:3495] Ops rate for 'default': 17152.286933 2021-06-08 11:45:35,969 | test | DEBUG | pool-5-thread-22 | [task:print_ep_queue_size:3508] ep_queue_size for default: 6849322 2021-06-08 11:45:35,970 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:45:35,970 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:45:37,203 | test | DEBUG | pool-5-thread-22 | [task:record_bucket_ops:3495] Ops rate for 'default': 17152.286933 2021-06-08 11:45:37,203 | test | DEBUG | pool-5-thread-22 | [task:print_ep_queue_size:3508] ep_queue_size for default: 6849322 2021-06-08 11:45:37,204 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:45:37,204 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:45:38,436 | test | DEBUG | pool-5-thread-22 | [task:record_bucket_ops:3495] Ops rate for 'default': 17152.286933 2021-06-08 11:45:38,436 | test | DEBUG | pool-5-thread-22 | [task:print_ep_queue_size:3508] ep_queue_size for default: 6849322 2021-06-08 11:45:38,437 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:45:38,437 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:45:39,680 | test | DEBUG | pool-5-thread-22 | [task:record_bucket_ops:3495] Ops rate for 'default': 17152.286933 2021-06-08 11:45:39,680 | test | DEBUG | pool-5-thread-22 | [task:print_ep_queue_size:3508] ep_queue_size for default: 6849322 2021-06-08 11:45:39,680 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:45:39,680 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:45:40,914 | test | DEBUG | pool-5-thread-22 | [task:record_bucket_ops:3495] Ops rate for 'default': 17152.286933 2021-06-08 11:45:40,914 | test | DEBUG | pool-5-thread-22 | [task:print_ep_queue_size:3508] ep_queue_size for default: 5208907 2021-06-08 11:45:40,915 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:45:40,915 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:45:42,137 | test | DEBUG | pool-5-thread-22 | [task:record_bucket_ops:3495] Ops rate for 'default': 5043.684834 2021-06-08 11:45:42,138 | test | DEBUG | pool-5-thread-22 | [task:print_ep_queue_size:3508] ep_queue_size for default: 5208907 2021-06-08 11:45:42,138 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:45:42,138 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:45:43,365 | test | DEBUG | pool-5-thread-22 | [task:record_bucket_ops:3495] Ops rate for 'default': 5043.684834 2021-06-08 11:45:43,367 | test | DEBUG | pool-5-thread-22 | [task:print_ep_queue_size:3508] ep_queue_size for default: 5208907 2021-06-08 11:45:43,368 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:45:43,368 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:45:44,598 | test | DEBUG | pool-5-thread-22 | [task:record_bucket_ops:3495] Ops rate for 'default': 5043.684834 2021-06-08 11:45:44,598 | test | DEBUG | pool-5-thread-22 | [task:print_ep_queue_size:3508] ep_queue_size for default: 5208907 2021-06-08 11:45:44,599 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:45:44,599 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:45:45,839 | test | DEBUG | pool-5-thread-22 | [task:record_bucket_ops:3495] Ops rate for 'default': 5043.684834 2021-06-08 11:45:45,839 | test | DEBUG | pool-5-thread-22 | [task:print_ep_queue_size:3508] ep_queue_size for default: 5208907 2021-06-08 11:45:45,842 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:45:45,842 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:45:47,075 | test | DEBUG | pool-5-thread-22 | [task:record_bucket_ops:3495] Ops rate for 'default': 5043.684834 2021-06-08 11:45:47,075 | test | DEBUG | pool-5-thread-22 | [task:print_ep_queue_size:3508] ep_queue_size for default: 5208907 2021-06-08 11:45:47,078 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:45:47,078 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:45:48,332 | test | DEBUG | pool-5-thread-22 | [task:record_bucket_ops:3495] Ops rate for 'default': 5043.684834 2021-06-08 11:45:48,332 | test | DEBUG | pool-5-thread-22 | [task:print_ep_queue_size:3508] ep_queue_size for default: 5208907 2021-06-08 11:45:48,335 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:45:48,335 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:45:49,572 | test | DEBUG | pool-5-thread-22 | [task:record_bucket_ops:3495] Ops rate for 'default': 5043.684834 2021-06-08 11:45:49,572 | test | DEBUG | pool-5-thread-22 | [task:print_ep_queue_size:3508] ep_queue_size for default: 5208907 2021-06-08 11:45:49,573 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:45:49,573 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:45:50,825 | test | DEBUG | pool-5-thread-22 | [task:record_bucket_ops:3495] Ops rate for 'default': 5043.684834 2021-06-08 11:45:50,825 | test | DEBUG | pool-5-thread-22 | [task:print_ep_queue_size:3508] ep_queue_size for default: 5208907 2021-06-08 11:45:50,826 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:45:50,828 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:45:52,058 | test | DEBUG | pool-5-thread-22 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:45:52,059 | test | DEBUG | pool-5-thread-22 | [task:print_ep_queue_size:3508] ep_queue_size for default: 5208907 2021-06-08 11:45:52,061 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:45:52,061 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:45:53,295 | test | DEBUG | pool-5-thread-22 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:45:53,295 | test | DEBUG | pool-5-thread-22 | [task:print_ep_queue_size:3508] ep_queue_size for default: 5208907 2021-06-08 11:45:53,296 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:45:53,296 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:45:54,528 | test | DEBUG | pool-5-thread-22 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:45:54,528 | test | DEBUG | pool-5-thread-22 | [task:print_ep_queue_size:3508] ep_queue_size for default: 5208907 2021-06-08 11:45:54,529 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:45:54,529 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:45:55,759 | test | DEBUG | pool-5-thread-22 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:45:55,760 | test | DEBUG | pool-5-thread-22 | [task:print_ep_queue_size:3508] ep_queue_size for default: 5208907 2021-06-08 11:45:55,762 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:45:55,762 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:45:56,999 | test | DEBUG | pool-5-thread-22 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:45:57,000 | test | DEBUG | pool-5-thread-22 | [task:print_ep_queue_size:3508] ep_queue_size for default: 5208907 2021-06-08 11:45:57,000 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:45:57,000 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:45:58,232 | test | DEBUG | pool-5-thread-22 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:45:58,232 | test | DEBUG | pool-5-thread-22 | [task:print_ep_queue_size:3508] ep_queue_size for default: 5208907 2021-06-08 11:45:58,233 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:45:58,233 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:45:59,463 | test | DEBUG | pool-5-thread-22 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:45:59,463 | test | DEBUG | pool-5-thread-22 | [task:print_ep_queue_size:3508] ep_queue_size for default: 5208907 2021-06-08 11:45:59,463 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:45:59,464 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:46:00,707 | test | DEBUG | pool-5-thread-22 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:46:00,707 | test | DEBUG | pool-5-thread-22 | [task:print_ep_queue_size:3508] ep_queue_size for default: 5208907 2021-06-08 11:46:00,707 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:46:00,707 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:46:01,928 | test | DEBUG | pool-5-thread-22 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:46:01,928 | test | DEBUG | pool-5-thread-22 | [task:print_ep_queue_size:3508] ep_queue_size for default: 5208907 2021-06-08 11:46:01,930 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:46:01,930 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:46:03,167 | test | DEBUG | pool-5-thread-22 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:46:03,167 | test | DEBUG | pool-5-thread-22 | [task:print_ep_queue_size:3508] ep_queue_size for default: 5208907 2021-06-08 11:46:03,168 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:46:03,168 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:46:04,388 | test | DEBUG | pool-5-thread-22 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:46:04,388 | test | DEBUG | pool-5-thread-22 | [task:print_ep_queue_size:3508] ep_queue_size for default: 5208907 2021-06-08 11:46:04,391 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:46:04,391 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:46:05,634 | test | DEBUG | pool-5-thread-22 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:46:05,634 | test | DEBUG | pool-5-thread-22 | [task:print_ep_queue_size:3508] ep_queue_size for default: 5208907 2021-06-08 11:46:05,635 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:46:05,635 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:46:06,858 | test | DEBUG | pool-5-thread-22 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:46:06,858 | test | DEBUG | pool-5-thread-22 | [task:print_ep_queue_size:3508] ep_queue_size for default: 5208907 2021-06-08 11:46:06,859 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:46:06,859 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:46:08,085 | test | DEBUG | pool-5-thread-22 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:46:08,085 | test | DEBUG | pool-5-thread-22 | [task:print_ep_queue_size:3508] ep_queue_size for default: 5208907 2021-06-08 11:46:08,088 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:46:08,088 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:46:09,315 | test | DEBUG | pool-5-thread-22 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:46:09,315 | test | DEBUG | pool-5-thread-22 | [task:print_ep_queue_size:3508] ep_queue_size for default: 5208907 2021-06-08 11:46:09,316 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:46:09,316 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:46:10,555 | test | DEBUG | pool-5-thread-22 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:46:10,555 | test | DEBUG | pool-5-thread-22 | [task:print_ep_queue_size:3508] ep_queue_size for default: 5208907 2021-06-08 11:46:10,556 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:46:10,556 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:46:11,783 | test | DEBUG | pool-5-thread-22 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:46:11,783 | test | DEBUG | pool-5-thread-22 | [task:print_ep_queue_size:3508] ep_queue_size for default: 5208907 2021-06-08 11:46:11,785 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:46:11,785 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:46:13,013 | test | DEBUG | pool-5-thread-22 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:46:13,013 | test | DEBUG | pool-5-thread-22 | [task:print_ep_queue_size:3508] ep_queue_size for default: 5208907 2021-06-08 11:46:13,015 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:46:13,015 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:46:14,246 | test | DEBUG | pool-5-thread-22 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:46:14,246 | test | DEBUG | pool-5-thread-22 | [task:print_ep_queue_size:3508] ep_queue_size for default: 5208907 2021-06-08 11:46:14,247 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:46:14,247 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:46:15,480 | test | DEBUG | pool-5-thread-22 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:46:15,482 | test | DEBUG | pool-5-thread-22 | [task:print_ep_queue_size:3508] ep_queue_size for default: 5208907 2021-06-08 11:46:15,483 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:46:15,483 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:46:16,709 | test | DEBUG | pool-5-thread-22 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:46:16,709 | test | DEBUG | pool-5-thread-22 | [task:print_ep_queue_size:3508] ep_queue_size for default: 5208907 2021-06-08 11:46:16,710 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:46:16,710 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:46:17,940 | test | DEBUG | pool-5-thread-22 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:46:17,940 | test | DEBUG | pool-5-thread-22 | [task:print_ep_queue_size:3508] ep_queue_size for default: 5208907 2021-06-08 11:46:17,943 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:46:17,943 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:46:19,176 | test | DEBUG | pool-5-thread-22 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:46:19,176 | test | DEBUG | pool-5-thread-22 | [task:print_ep_queue_size:3508] ep_queue_size for default: 5208907 2021-06-08 11:46:19,177 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:46:19,177 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:46:20,407 | test | DEBUG | pool-5-thread-22 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:46:20,407 | test | DEBUG | pool-5-thread-22 | [task:print_ep_queue_size:3508] ep_queue_size for default: 5208907 2021-06-08 11:46:20,407 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:46:20,407 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:46:21,632 | test | DEBUG | pool-5-thread-22 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:46:21,632 | test | DEBUG | pool-5-thread-22 | [task:print_ep_queue_size:3508] ep_queue_size for default: 5208907 2021-06-08 11:46:21,634 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:46:21,634 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:46:22,865 | test | DEBUG | pool-5-thread-22 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:46:22,865 | test | DEBUG | pool-5-thread-22 | [task:print_ep_queue_size:3508] ep_queue_size for default: 5208907 2021-06-08 11:46:22,868 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:46:22,868 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:46:24,092 | test | DEBUG | pool-5-thread-22 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:46:24,092 | test | DEBUG | pool-5-thread-22 | [task:print_ep_queue_size:3508] ep_queue_size for default: 5208907 2021-06-08 11:46:24,094 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:46:24,094 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:46:25,334 | test | DEBUG | pool-5-thread-22 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:46:25,334 | test | DEBUG | pool-5-thread-22 | [task:print_ep_queue_size:3508] ep_queue_size for default: 5208907 2021-06-08 11:46:25,335 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:46:25,335 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:46:26,562 | test | DEBUG | pool-5-thread-22 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:46:26,562 | test | DEBUG | pool-5-thread-22 | [task:print_ep_queue_size:3508] ep_queue_size for default: 5208907 2021-06-08 11:46:26,565 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:46:26,565 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:46:27,799 | test | DEBUG | pool-5-thread-22 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:46:27,799 | test | DEBUG | pool-5-thread-22 | [task:print_ep_queue_size:3508] ep_queue_size for default: 5208907 2021-06-08 11:46:27,801 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:46:27,802 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:46:29,023 | test | DEBUG | pool-5-thread-22 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:46:29,025 | test | DEBUG | pool-5-thread-22 | [task:print_ep_queue_size:3508] ep_queue_size for default: 5208907 2021-06-08 11:46:29,026 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:46:29,026 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:46:30,259 | test | DEBUG | pool-5-thread-22 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:46:30,260 | test | DEBUG | pool-5-thread-22 | [task:print_ep_queue_size:3508] ep_queue_size for default: 5208907 2021-06-08 11:46:30,262 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:46:30,262 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:46:31,506 | test | DEBUG | pool-5-thread-22 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:46:31,506 | test | DEBUG | pool-5-thread-22 | [task:print_ep_queue_size:3508] ep_queue_size for default: 5208907 2021-06-08 11:46:31,506 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:46:31,506 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:46:32,421 | infra | INFO | pool-5-thread-29 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623177864.84_create__7550000_7600000_1623177864.86' completed 2021-06-08 11:46:32,569 | infra | INFO | pool-5-thread-17 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623177864.84_create__7150000_7200000_1623177864.86' completed 2021-06-08 11:46:32,786 | test | DEBUG | pool-5-thread-22 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:46:32,786 | test | DEBUG | pool-5-thread-22 | [task:print_ep_queue_size:3508] ep_queue_size for default: 5208907 2021-06-08 11:46:32,788 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:46:32,789 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:46:32,980 | infra | INFO | pool-5-thread-1 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623177864.84_create__7200000_7250000_1623177864.86' completed 2021-06-08 11:46:33,043 | infra | INFO | pool-5-thread-25 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623177864.84_create__7700000_7750000_1623177864.86' completed 2021-06-08 11:46:34,012 | test | DEBUG | pool-5-thread-22 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:46:34,012 | test | DEBUG | pool-5-thread-22 | [task:print_ep_queue_size:3508] ep_queue_size for default: 5208907 2021-06-08 11:46:34,013 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:46:34,013 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:46:35,269 | test | DEBUG | pool-5-thread-22 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:46:35,269 | test | DEBUG | pool-5-thread-22 | [task:print_ep_queue_size:3508] ep_queue_size for default: 5208907 2021-06-08 11:46:35,270 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:46:35,272 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:46:35,676 | infra | INFO | pool-5-thread-7 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623177864.84_create__7000000_7050000_1623177864.86' completed 2021-06-08 11:46:35,677 | infra | DEBUG | pool-5-thread-15 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623177864.84_create__7000000_7050000_1623177864.86 are 50000 2021-06-08 11:46:35,677 | infra | DEBUG | pool-5-thread-15 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623177864.84_create__7050000_7100000_1623177864.86 2021-06-08 11:46:35,723 | infra | INFO | pool-5-thread-9 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623177864.84_create__7350000_7400000_1623177864.86' completed 2021-06-08 11:46:35,757 | infra | INFO | pool-5-thread-28 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623177864.84_create__7400000_7450000_1623177864.86' completed 2021-06-08 11:46:35,789 | infra | INFO | pool-5-thread-6 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623177864.84_create__7600000_7650000_1623177864.86' completed 2021-06-08 11:46:35,872 | infra | INFO | pool-5-thread-18 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623177864.84_create__7500000_7550000_1623177864.86' completed 2021-06-08 11:46:35,894 | infra | INFO | pool-5-thread-10 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623177864.84_create__7450000_7500000_1623177864.86' completed 2021-06-08 11:46:36,276 | infra | INFO | pool-5-thread-12 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623177864.84_create__7900000_7950000_1623177864.86' completed 2021-06-08 11:46:36,467 | infra | INFO | pool-5-thread-11 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623177864.84_create__7650000_7700000_1623177864.86' completed 2021-06-08 11:46:36,473 | infra | INFO | pool-5-thread-5 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623177864.84_create__7850000_7900000_1623177864.86' completed 2021-06-08 11:46:36,502 | infra | INFO | pool-5-thread-23 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623177864.84_create__7950000_8000000_1623177864.86' completed 2021-06-08 11:46:36,505 | infra | INFO | pool-5-thread-20 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623177864.84_create__7750000_7800000_1623177864.86' completed 2021-06-08 11:46:36,509 | test | DEBUG | pool-5-thread-22 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:46:36,509 | test | DEBUG | pool-5-thread-22 | [task:print_ep_queue_size:3508] ep_queue_size for default: 5208907 2021-06-08 11:46:36,513 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:46:36,513 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:46:36,717 | infra | INFO | pool-5-thread-8 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623177864.84_create__7050000_7100000_1623177864.86' completed 2021-06-08 11:46:36,719 | infra | DEBUG | pool-5-thread-15 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623177864.84_create__7050000_7100000_1623177864.86 are 50000 2021-06-08 11:46:36,719 | infra | DEBUG | pool-5-thread-15 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623177864.84_create__7100000_7150000_1623177864.86 2021-06-08 11:46:36,796 | infra | INFO | pool-5-thread-19 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623177864.84_create__7300000_7350000_1623177864.86' completed 2021-06-08 11:46:36,822 | infra | INFO | pool-5-thread-13 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623177864.84_create__7800000_7850000_1623177864.86' completed 2021-06-08 11:46:36,950 | infra | INFO | pool-5-thread-14 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623177864.84_create__7100000_7150000_1623177864.86' completed 2021-06-08 11:46:36,951 | infra | DEBUG | pool-5-thread-15 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623177864.84_create__7100000_7150000_1623177864.86 are 50000 2021-06-08 11:46:36,951 | infra | INFO | pool-5-thread-26 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623177864.84_create__7250000_7300000_1623177864.86' completed 2021-06-08 11:46:36,951 | infra | DEBUG | pool-5-thread-15 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623177864.84_create__7150000_7200000_1623177864.86 2021-06-08 11:46:36,951 | infra | DEBUG | pool-5-thread-15 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623177864.84_create__7150000_7200000_1623177864.86 are 50000 2021-06-08 11:46:36,951 | infra | DEBUG | pool-5-thread-15 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623177864.84_create__7200000_7250000_1623177864.86 2021-06-08 11:46:36,951 | infra | DEBUG | pool-5-thread-15 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623177864.84_create__7200000_7250000_1623177864.86 are 50000 2021-06-08 11:46:36,953 | infra | DEBUG | pool-5-thread-15 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623177864.84_create__7250000_7300000_1623177864.86 2021-06-08 11:46:36,953 | infra | DEBUG | pool-5-thread-15 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623177864.84_create__7250000_7300000_1623177864.86 are 50000 2021-06-08 11:46:36,953 | infra | DEBUG | pool-5-thread-15 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623177864.84_create__7300000_7350000_1623177864.86 2021-06-08 11:46:36,953 | infra | DEBUG | pool-5-thread-15 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623177864.84_create__7300000_7350000_1623177864.86 are 50000 2021-06-08 11:46:36,953 | infra | DEBUG | pool-5-thread-15 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623177864.84_create__7350000_7400000_1623177864.86 2021-06-08 11:46:36,953 | infra | DEBUG | pool-5-thread-15 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623177864.84_create__7350000_7400000_1623177864.86 are 50000 2021-06-08 11:46:36,953 | infra | DEBUG | pool-5-thread-15 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623177864.84_create__7400000_7450000_1623177864.86 2021-06-08 11:46:36,953 | infra | DEBUG | pool-5-thread-15 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623177864.84_create__7400000_7450000_1623177864.86 are 50000 2021-06-08 11:46:36,953 | infra | DEBUG | pool-5-thread-15 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623177864.84_create__7450000_7500000_1623177864.86 2021-06-08 11:46:36,953 | infra | DEBUG | pool-5-thread-15 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623177864.84_create__7450000_7500000_1623177864.86 are 50000 2021-06-08 11:46:36,953 | infra | DEBUG | pool-5-thread-15 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623177864.84_create__7500000_7550000_1623177864.86 2021-06-08 11:46:36,953 | infra | DEBUG | pool-5-thread-15 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623177864.84_create__7500000_7550000_1623177864.86 are 50000 2021-06-08 11:46:36,953 | infra | DEBUG | pool-5-thread-15 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623177864.84_create__7550000_7600000_1623177864.86 2021-06-08 11:46:36,953 | infra | DEBUG | pool-5-thread-15 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623177864.84_create__7550000_7600000_1623177864.86 are 50000 2021-06-08 11:46:36,953 | infra | DEBUG | pool-5-thread-15 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623177864.84_create__7600000_7650000_1623177864.86 2021-06-08 11:46:36,953 | infra | DEBUG | pool-5-thread-15 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623177864.84_create__7600000_7650000_1623177864.86 are 50000 2021-06-08 11:46:36,953 | infra | DEBUG | pool-5-thread-15 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623177864.84_create__7650000_7700000_1623177864.86 2021-06-08 11:46:36,953 | infra | DEBUG | pool-5-thread-15 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623177864.84_create__7650000_7700000_1623177864.86 are 50000 2021-06-08 11:46:36,953 | infra | DEBUG | pool-5-thread-15 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623177864.84_create__7700000_7750000_1623177864.86 2021-06-08 11:46:36,954 | infra | DEBUG | pool-5-thread-15 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623177864.84_create__7700000_7750000_1623177864.86 are 50000 2021-06-08 11:46:36,954 | infra | DEBUG | pool-5-thread-15 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623177864.84_create__7750000_7800000_1623177864.86 2021-06-08 11:46:36,954 | infra | DEBUG | pool-5-thread-15 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623177864.84_create__7750000_7800000_1623177864.86 are 50000 2021-06-08 11:46:36,954 | infra | DEBUG | pool-5-thread-15 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623177864.84_create__7800000_7850000_1623177864.86 2021-06-08 11:46:36,954 | infra | DEBUG | pool-5-thread-15 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623177864.84_create__7800000_7850000_1623177864.86 are 50000 2021-06-08 11:46:36,954 | infra | DEBUG | pool-5-thread-15 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623177864.84_create__7850000_7900000_1623177864.86 2021-06-08 11:46:36,954 | infra | DEBUG | pool-5-thread-15 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623177864.84_create__7850000_7900000_1623177864.86 are 50000 2021-06-08 11:46:36,954 | infra | DEBUG | pool-5-thread-15 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623177864.84_create__7900000_7950000_1623177864.86 2021-06-08 11:46:36,954 | infra | DEBUG | pool-5-thread-15 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623177864.84_create__7900000_7950000_1623177864.86 are 50000 2021-06-08 11:46:36,954 | infra | DEBUG | pool-5-thread-15 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623177864.84_create__7950000_8000000_1623177864.86 2021-06-08 11:46:36,956 | infra | DEBUG | pool-5-thread-15 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623177864.84_create__7950000_8000000_1623177864.86 are 50000 2021-06-08 11:46:36,956 | infra | DEBUG | pool-5-thread-15 | [task_manager:get_task_result:26] Getting task result for PrintBucketStats_default_1623177864.86 2021-06-08 11:46:37,515 | test | INFO | pool-5-thread-22 | [table_view:display:72] Ops trend for bucket 'default' +-----------+--------------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-----------+--------------------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | *****..................................*******...........X | 8527.800 | | 2529.600 | ............*******.......................................................................................*******X | 18728.600 | | 5078.700 | ..........................*******.....................................*******X | 12128.900 | | 11100.000 | ..........................................................*...............******.................*******X | 17152.287 | | 5043.685 | .........................*******X | 5043.685 | | 0.000 | ************************************X | 0.000 | +-----------+--------------------------------------------------------------------------------------------------------------------+-----------+ 2021-06-08 11:46:37,515 | infra | INFO | pool-5-thread-22 | [task:complete_task:102] Thread 'PrintBucketStats_default_1623177864.86' completed 2021-06-08 11:46:37,516 | infra | DEBUG | pool-5-thread-15 | [task:call:1663] ========= Tasks in loadgen pool======= 2021-06-08 11:46:37,516 | infra | WARNING | pool-5-thread-15 | [task_manager:print_tasks_in_pool:83] Task 'LoadDocsGen_default__default__default_create_7000000_8000000_1623177864.84' not completed 2021-06-08 11:46:37,516 | infra | DEBUG | pool-5-thread-15 | [task:call:1665] ====================================== 2021-06-08 11:46:37,516 | infra | DEBUG | pool-5-thread-15 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623177864.84_create__7000000_7050000_1623177864.86' complete. Loaded 50000 items 2021-06-08 11:46:37,516 | infra | DEBUG | pool-5-thread-15 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623177864.84_create__7050000_7100000_1623177864.86' complete. Loaded 50000 items 2021-06-08 11:46:37,516 | infra | DEBUG | pool-5-thread-15 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623177864.84_create__7100000_7150000_1623177864.86' complete. Loaded 50000 items 2021-06-08 11:46:37,516 | infra | DEBUG | pool-5-thread-15 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623177864.84_create__7150000_7200000_1623177864.86' complete. Loaded 50000 items 2021-06-08 11:46:37,516 | infra | DEBUG | pool-5-thread-15 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623177864.84_create__7200000_7250000_1623177864.86' complete. Loaded 50000 items 2021-06-08 11:46:37,516 | infra | DEBUG | pool-5-thread-15 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623177864.84_create__7250000_7300000_1623177864.86' complete. Loaded 50000 items 2021-06-08 11:46:37,517 | infra | DEBUG | pool-5-thread-15 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623177864.84_create__7300000_7350000_1623177864.86' complete. Loaded 50000 items 2021-06-08 11:46:37,517 | infra | DEBUG | pool-5-thread-15 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623177864.84_create__7350000_7400000_1623177864.86' complete. Loaded 50000 items 2021-06-08 11:46:37,517 | infra | DEBUG | pool-5-thread-15 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623177864.84_create__7400000_7450000_1623177864.86' complete. Loaded 50000 items 2021-06-08 11:46:37,517 | infra | DEBUG | pool-5-thread-15 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623177864.84_create__7450000_7500000_1623177864.86' complete. Loaded 50000 items 2021-06-08 11:46:37,517 | infra | DEBUG | pool-5-thread-15 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623177864.84_create__7500000_7550000_1623177864.86' complete. Loaded 50000 items 2021-06-08 11:46:37,517 | infra | DEBUG | pool-5-thread-15 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623177864.84_create__7550000_7600000_1623177864.86' complete. Loaded 50000 items 2021-06-08 11:46:37,517 | infra | DEBUG | pool-5-thread-15 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623177864.84_create__7600000_7650000_1623177864.86' complete. Loaded 50000 items 2021-06-08 11:46:37,519 | infra | DEBUG | pool-5-thread-15 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623177864.84_create__7650000_7700000_1623177864.86' complete. Loaded 50000 items 2021-06-08 11:46:37,519 | infra | DEBUG | pool-5-thread-15 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623177864.84_create__7700000_7750000_1623177864.86' complete. Loaded 50000 items 2021-06-08 11:46:37,519 | infra | DEBUG | pool-5-thread-15 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623177864.84_create__7750000_7800000_1623177864.86' complete. Loaded 50000 items 2021-06-08 11:46:37,519 | infra | DEBUG | pool-5-thread-15 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623177864.84_create__7800000_7850000_1623177864.86' complete. Loaded 50000 items 2021-06-08 11:46:37,519 | infra | DEBUG | pool-5-thread-15 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623177864.84_create__7850000_7900000_1623177864.86' complete. Loaded 50000 items 2021-06-08 11:46:37,519 | infra | DEBUG | pool-5-thread-15 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623177864.84_create__7900000_7950000_1623177864.86' complete. Loaded 50000 items 2021-06-08 11:46:37,519 | infra | DEBUG | pool-5-thread-15 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_7000000_8000000_1623177864.84_create__7950000_8000000_1623177864.86' complete. Loaded 50000 items 2021-06-08 11:46:37,519 | infra | INFO | pool-5-thread-15 | [task:complete_task:102] Thread 'LoadDocsGen_default__default__default_create_7000000_8000000_1623177864.84' completed 2021-06-08 11:46:37,522 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.125 with username: root, password: couchbase 2021-06-08 11:46:37,621 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 11:46:37,855 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: ls /etc/ 2021-06-08 11:46:37,915 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 11:46:37,917 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: ls /cygdrive/c/Windows 2021-06-08 11:46:37,930 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: uname -m 2021-06-08 11:46:37,943 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: cat /proc/cpuinfo 2021-06-08 11:46:37,957 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: df -Thl 2021-06-08 11:46:37,973 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: cat /proc/meminfo 2021-06-08 11:46:37,986 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: hostname 2021-06-08 11:46:38,000 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: hostname -d 2021-06-08 11:46:38,013 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:559] Starting StatsWaitTask for ep_data_write_failed on bucket default 2021-06-08 11:46:38,013 | infra | INFO | MainThread | [task_manager:add_new_task:23] Added new task: StatsWaitTask_default_ep_data_write_failed_1623177998.01 2021-06-08 11:46:38,013 | infra | INFO | pool-5-thread-21 | [task:start_task:87] Thread 'StatsWaitTask_default_ep_data_write_failed_1623177998.01' started 2021-06-08 11:46:38,016 | infra | DEBUG | pool-5-thread-21 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:46:38,017 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.155 with username: root, password: couchbase 2021-06-08 11:46:38,098 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 11:46:38,233 | test | DEBUG | pool-5-thread-21 | [task:_compare:2649] Comparing 117789 > 0 2021-06-08 11:46:38,233 | test | DEBUG | pool-5-thread-21 | [task:_get_all_stats_and_compare:2601] Ready: ep_data_write_failed 117789 > 0. Received: {'172.23.106.125': '117789'} for bucket 'default' 2021-06-08 11:46:38,233 | infra | INFO | pool-5-thread-21 | [task:complete_task:102] Thread 'StatsWaitTask_default_ep_data_write_failed_1623177998.01' completed 2021-06-08 11:46:38,316 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: ls /etc/ 2021-06-08 11:46:38,381 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 11:46:38,381 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: ls /cygdrive/c/Windows 2021-06-08 11:46:38,395 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: uname -m 2021-06-08 11:46:38,407 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: cat /proc/cpuinfo 2021-06-08 11:46:38,421 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: df -Thl 2021-06-08 11:46:38,437 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: cat /proc/meminfo 2021-06-08 11:46:38,450 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: hostname 2021-06-08 11:46:38,464 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: hostname -d 2021-06-08 11:46:38,480 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:559] Starting StatsWaitTask for ep_data_write_failed on bucket default 2021-06-08 11:46:38,480 | infra | INFO | MainThread | [task_manager:add_new_task:23] Added new task: StatsWaitTask_default_ep_data_write_failed_1623177998.48 2021-06-08 11:46:38,480 | infra | INFO | pool-5-thread-16 | [task:start_task:87] Thread 'StatsWaitTask_default_ep_data_write_failed_1623177998.48' started 2021-06-08 11:46:38,480 | infra | DEBUG | pool-5-thread-16 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:46:38,483 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.156 with username: root, password: couchbase 2021-06-08 11:46:38,572 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 11:46:38,703 | test | DEBUG | pool-5-thread-16 | [task:_compare:2649] Comparing 313940 > 0 2021-06-08 11:46:38,703 | test | DEBUG | pool-5-thread-16 | [task:_get_all_stats_and_compare:2601] Ready: ep_data_write_failed 313940 > 0. Received: {'172.23.106.155': '313940'} for bucket 'default' 2021-06-08 11:46:38,703 | infra | INFO | pool-5-thread-16 | [task:complete_task:102] Thread 'StatsWaitTask_default_ep_data_write_failed_1623177998.48' completed 2021-06-08 11:46:38,786 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: ls /etc/ 2021-06-08 11:46:38,842 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 11:46:38,842 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: ls /cygdrive/c/Windows 2021-06-08 11:46:38,855 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: uname -m 2021-06-08 11:46:38,868 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: cat /proc/cpuinfo 2021-06-08 11:46:38,881 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: df -Thl 2021-06-08 11:46:38,898 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: cat /proc/meminfo 2021-06-08 11:46:38,910 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: hostname 2021-06-08 11:46:38,921 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: hostname -d 2021-06-08 11:46:38,934 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:559] Starting StatsWaitTask for ep_data_write_failed on bucket default 2021-06-08 11:46:38,934 | infra | INFO | MainThread | [task_manager:add_new_task:23] Added new task: StatsWaitTask_default_ep_data_write_failed_1623177998.93 2021-06-08 11:46:38,934 | infra | INFO | pool-5-thread-2 | [task:start_task:87] Thread 'StatsWaitTask_default_ep_data_write_failed_1623177998.93' started 2021-06-08 11:46:38,936 | infra | DEBUG | pool-5-thread-2 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:46:38,937 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.122 with username: root, password: couchbase 2021-06-08 11:46:39,030 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 11:46:39,250 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: ls /etc/ 2021-06-08 11:46:39,309 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 11:46:39,309 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: ls /cygdrive/c/Windows 2021-06-08 11:46:39,323 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: uname -m 2021-06-08 11:46:39,338 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: cat /proc/cpuinfo 2021-06-08 11:46:39,351 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: df -Thl 2021-06-08 11:46:39,369 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: cat /proc/meminfo 2021-06-08 11:46:39,384 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: hostname 2021-06-08 11:46:39,397 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: hostname -d 2021-06-08 11:46:39,411 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:559] Starting StatsWaitTask for ep_data_write_failed on bucket default 2021-06-08 11:46:39,411 | infra | INFO | MainThread | [task_manager:add_new_task:23] Added new task: StatsWaitTask_default_ep_data_write_failed_1623177999.41 2021-06-08 11:46:39,411 | infra | DEBUG | MainThread | [task_manager:get_task_result:26] Getting task result for StatsWaitTask_default_ep_data_write_failed_1623177998.01 2021-06-08 11:46:39,411 | infra | INFO | pool-5-thread-30 | [task:start_task:87] Thread 'StatsWaitTask_default_ep_data_write_failed_1623177999.41' started 2021-06-08 11:46:39,411 | infra | DEBUG | MainThread | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.125 2021-06-08 11:46:39,411 | infra | DEBUG | MainThread | [task_manager:get_task_result:26] Getting task result for StatsWaitTask_default_ep_data_write_failed_1623177998.48 2021-06-08 11:46:39,413 | infra | DEBUG | pool-5-thread-30 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:46:39,414 | infra | DEBUG | MainThread | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.155 2021-06-08 11:46:39,414 | infra | DEBUG | MainThread | [task_manager:get_task_result:26] Getting task result for StatsWaitTask_default_ep_data_write_failed_1623177998.93 2021-06-08 11:46:39,628 | test | DEBUG | pool-5-thread-30 | [task:_compare:2649] Comparing 116775 > 0 2021-06-08 11:46:39,628 | test | DEBUG | pool-5-thread-30 | [task:_get_all_stats_and_compare:2601] Ready: ep_data_write_failed 116775 > 0. Received: {'172.23.106.122': '116775'} for bucket 'default' 2021-06-08 11:46:39,630 | infra | INFO | pool-5-thread-30 | [task:complete_task:102] Thread 'StatsWaitTask_default_ep_data_write_failed_1623177999.41' completed 2021-06-08 11:46:44,686 | test | DEBUG | pool-5-thread-2 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 11:46:44,686 | infra | DEBUG | pool-5-thread-2 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.156': '0'} for bucket 'default' 2021-06-08 11:46:44,687 | infra | DEBUG | pool-5-thread-2 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 11:46:44,687 | infra | DEBUG | pool-5-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 11:46:44,688 | infra | DEBUG | pool-5-thread-2 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 11:46:49,688 | infra | DEBUG | pool-5-thread-2 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:46:54,240 | test | DEBUG | pool-5-thread-2 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 11:46:54,240 | infra | DEBUG | pool-5-thread-2 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.156': '0'} for bucket 'default' 2021-06-08 11:46:54,240 | infra | DEBUG | pool-5-thread-2 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 11:46:54,243 | infra | DEBUG | pool-5-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 11:46:54,243 | infra | DEBUG | pool-5-thread-2 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 11:46:59,243 | infra | DEBUG | pool-5-thread-2 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:46:59,697 | test | DEBUG | pool-5-thread-2 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 11:46:59,697 | infra | DEBUG | pool-5-thread-2 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.156': '0'} for bucket 'default' 2021-06-08 11:46:59,697 | infra | DEBUG | pool-5-thread-2 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 11:46:59,700 | infra | DEBUG | pool-5-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 11:46:59,700 | infra | DEBUG | pool-5-thread-2 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 11:47:04,700 | infra | DEBUG | pool-5-thread-2 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:47:47,283 | test | DEBUG | pool-5-thread-2 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 11:47:47,283 | infra | DEBUG | pool-5-thread-2 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.156': '0'} for bucket 'default' 2021-06-08 11:47:47,283 | infra | DEBUG | pool-5-thread-2 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 11:47:47,286 | infra | DEBUG | pool-5-thread-2 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 11:47:47,286 | infra | DEBUG | pool-5-thread-2 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 11:47:52,286 | infra | DEBUG | pool-5-thread-2 | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.156 2021-06-08 11:47:52,288 | infra | INFO | pool-5-thread-2 | [task:complete_task:102] Thread 'StatsWaitTask_default_ep_data_write_failed_1623177998.93' completed 2021-06-08 11:47:52,292 | infra | INFO | MainThread | [task_manager:add_new_task:23] Added new task: LoadDocsGen_default__default__default_create_8000000_9000000_1623178072.29 2021-06-08 11:47:52,292 | infra | INFO | pool-5-thread-24 | [task:start_task:87] Thread 'LoadDocsGen_default__default__default_create_8000000_9000000_1623178072.29' started 2021-06-08 11:47:52,293 | infra | DEBUG | MainThread | [task_manager:get_task_result:26] Getting task result for LoadDocsGen_default__default__default_create_8000000_9000000_1623178072.29 2021-06-08 11:47:52,319 | infra | INFO | pool-5-thread-24 | [task_manager:add_new_task:23] Added new task: PrintBucketStats_default_1623178072.31 2021-06-08 11:47:52,319 | infra | INFO | pool-5-thread-27 | [task:start_task:87] Thread 'PrintBucketStats_default_1623178072.31' started 2021-06-08 11:47:52,319 | infra | INFO | pool-5-thread-24 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623178072.29_create__8000000_8050000_1623178072.31 2021-06-08 11:47:52,319 | infra | INFO | pool-5-thread-24 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623178072.29_create__8050000_8100000_1623178072.31 2021-06-08 11:47:52,319 | infra | INFO | pool-5-thread-4 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623178072.29_create__8050000_8100000_1623178072.31' started 2021-06-08 11:47:52,319 | infra | INFO | pool-5-thread-29 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623178072.29_create__8100000_8150000_1623178072.31' started 2021-06-08 11:47:52,319 | infra | INFO | pool-5-thread-3 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623178072.29_create__8000000_8050000_1623178072.31' started 2021-06-08 11:47:52,319 | infra | INFO | pool-5-thread-24 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623178072.29_create__8100000_8150000_1623178072.31 2021-06-08 11:47:52,321 | infra | INFO | pool-5-thread-24 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623178072.29_create__8150000_8200000_1623178072.31 2021-06-08 11:47:52,321 | infra | INFO | pool-5-thread-17 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623178072.29_create__8150000_8200000_1623178072.31' started 2021-06-08 11:47:52,321 | infra | INFO | pool-5-thread-24 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623178072.29_create__8200000_8250000_1623178072.31 2021-06-08 11:47:52,322 | infra | INFO | pool-5-thread-24 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623178072.29_create__8250000_8300000_1623178072.31 2021-06-08 11:47:52,322 | infra | INFO | pool-5-thread-24 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623178072.29_create__8300000_8350000_1623178072.31 2021-06-08 11:47:52,322 | infra | INFO | pool-5-thread-24 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623178072.29_create__8350000_8400000_1623178072.31 2021-06-08 11:47:52,322 | infra | INFO | pool-5-thread-24 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623178072.29_create__8400000_8450000_1623178072.31 2021-06-08 11:47:52,322 | infra | INFO | pool-5-thread-1 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623178072.29_create__8200000_8250000_1623178072.31' started 2021-06-08 11:47:52,322 | infra | INFO | pool-5-thread-7 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623178072.29_create__8300000_8350000_1623178072.31' started 2021-06-08 11:47:52,322 | infra | INFO | pool-5-thread-24 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623178072.29_create__8450000_8500000_1623178072.31 2021-06-08 11:47:52,322 | infra | INFO | pool-5-thread-28 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623178072.29_create__8400000_8450000_1623178072.31' started 2021-06-08 11:47:52,322 | infra | INFO | pool-5-thread-25 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623178072.29_create__8250000_8300000_1623178072.31' started 2021-06-08 11:47:52,322 | infra | INFO | pool-5-thread-9 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623178072.29_create__8350000_8400000_1623178072.31' started 2021-06-08 11:47:52,322 | infra | INFO | pool-5-thread-6 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623178072.29_create__8450000_8500000_1623178072.31' started 2021-06-08 11:47:52,323 | infra | INFO | pool-5-thread-24 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623178072.29_create__8500000_8550000_1623178072.31 2021-06-08 11:47:52,323 | infra | INFO | pool-5-thread-18 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623178072.29_create__8500000_8550000_1623178072.31' started 2021-06-08 11:47:52,323 | infra | INFO | pool-5-thread-24 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623178072.29_create__8550000_8600000_1623178072.31 2021-06-08 11:47:52,323 | infra | INFO | pool-5-thread-10 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623178072.29_create__8550000_8600000_1623178072.31' started 2021-06-08 11:47:52,325 | infra | INFO | pool-5-thread-12 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623178072.29_create__8600000_8650000_1623178072.31' started 2021-06-08 11:47:52,325 | infra | INFO | pool-5-thread-24 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623178072.29_create__8600000_8650000_1623178072.31 2021-06-08 11:47:52,325 | infra | INFO | pool-5-thread-11 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623178072.29_create__8650000_8700000_1623178072.31' started 2021-06-08 11:47:52,325 | infra | INFO | pool-5-thread-24 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623178072.29_create__8650000_8700000_1623178072.31 2021-06-08 11:47:52,326 | infra | INFO | pool-5-thread-24 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623178072.29_create__8700000_8750000_1623178072.31 2021-06-08 11:47:52,328 | infra | INFO | pool-5-thread-5 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623178072.29_create__8700000_8750000_1623178072.31' started 2021-06-08 11:47:52,328 | infra | INFO | pool-5-thread-24 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623178072.29_create__8750000_8800000_1623178072.31 2021-06-08 11:47:52,328 | infra | INFO | pool-5-thread-23 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623178072.29_create__8750000_8800000_1623178072.31' started 2021-06-08 11:47:52,328 | infra | INFO | pool-5-thread-24 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623178072.29_create__8800000_8850000_1623178072.31 2021-06-08 11:47:52,328 | infra | INFO | pool-5-thread-24 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623178072.29_create__8850000_8900000_1623178072.31 2021-06-08 11:47:52,328 | infra | INFO | pool-5-thread-24 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623178072.29_create__8900000_8950000_1623178072.31 2021-06-08 11:47:52,328 | infra | INFO | pool-5-thread-24 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623178072.29_create__8950000_9000000_1623178072.31 2021-06-08 11:47:52,328 | infra | DEBUG | pool-5-thread-24 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623178072.29_create__8000000_8050000_1623178072.31 2021-06-08 11:47:52,328 | infra | INFO | pool-5-thread-20 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623178072.29_create__8800000_8850000_1623178072.31' started 2021-06-08 11:47:52,328 | infra | INFO | pool-5-thread-19 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623178072.29_create__8900000_8950000_1623178072.31' started 2021-06-08 11:47:52,328 | infra | INFO | pool-5-thread-13 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623178072.29_create__8950000_9000000_1623178072.31' started 2021-06-08 11:47:52,328 | infra | INFO | pool-5-thread-8 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623178072.29_create__8850000_8900000_1623178072.31' started 2021-06-08 11:47:52,582 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:47:52,582 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 5208907 2021-06-08 11:47:52,585 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:47:52,585 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:47:53,805 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:47:53,805 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 5208907 2021-06-08 11:47:53,805 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:47:53,805 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:47:55,032 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:47:55,033 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 5208907 2021-06-08 11:47:55,035 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:47:55,036 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:47:56,269 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:47:56,269 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 5208907 2021-06-08 11:47:56,272 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:47:56,272 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:47:57,545 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:47:57,545 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 5208907 2021-06-08 11:47:57,548 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:47:57,548 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:47:58,778 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:47:58,779 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 5208907 2021-06-08 11:47:58,779 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:47:58,779 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:48:00,003 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:48:00,003 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 5208907 2021-06-08 11:48:00,005 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:48:00,006 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:48:01,230 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 10199.800000 2021-06-08 11:48:01,230 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 5208907 2021-06-08 11:48:01,232 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:48:01,233 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:48:02,453 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 10199.800000 2021-06-08 11:48:02,453 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 5208907 2021-06-08 11:48:02,454 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:48:02,456 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:48:03,674 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 10199.800000 2021-06-08 11:48:03,676 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 5208907 2021-06-08 11:48:03,677 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:48:03,677 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:48:04,907 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 10199.800000 2021-06-08 11:48:04,907 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 5208907 2021-06-08 11:48:04,908 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:48:04,910 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:48:06,127 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 10199.800000 2021-06-08 11:48:06,127 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 5208907 2021-06-08 11:48:06,128 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:48:06,128 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:48:07,357 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 10199.800000 2021-06-08 11:48:07,357 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 5208907 2021-06-08 11:48:07,358 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:48:07,358 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:48:08,586 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 10199.800000 2021-06-08 11:48:08,586 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 5208907 2021-06-08 11:48:08,588 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:48:08,588 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:48:09,811 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 10199.800000 2021-06-08 11:48:09,811 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 5208907 2021-06-08 11:48:09,812 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:48:09,812 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:48:11,038 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 10199.800000 2021-06-08 11:48:11,038 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 5208907 2021-06-08 11:48:11,039 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:48:11,039 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:48:12,280 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:48:12,282 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 5208907 2021-06-08 11:48:12,282 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:48:12,283 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:48:13,513 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:48:13,513 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 5208907 2021-06-08 11:48:13,515 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:48:13,515 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:48:14,736 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:48:14,736 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 5208907 2021-06-08 11:48:14,736 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:48:14,736 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:48:15,967 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:48:15,967 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 5208907 2021-06-08 11:48:15,969 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:48:15,970 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:48:17,210 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:48:17,210 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 5208907 2021-06-08 11:48:17,211 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:48:17,211 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:48:18,441 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:48:18,443 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 5208907 2021-06-08 11:48:18,444 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:48:18,444 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:48:19,670 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:48:19,670 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 5208907 2021-06-08 11:48:19,671 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:48:19,671 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:48:20,897 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:48:20,898 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 5208907 2021-06-08 11:48:20,898 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:48:20,898 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:48:22,127 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:48:22,127 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 5208907 2021-06-08 11:48:22,128 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:48:22,128 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:48:23,357 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:48:23,357 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 5208907 2021-06-08 11:48:23,358 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:48:23,358 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:48:24,599 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:48:24,601 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 5208907 2021-06-08 11:48:24,601 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:48:24,601 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:48:25,835 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:48:25,835 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 5208907 2021-06-08 11:48:25,838 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:48:25,838 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:48:27,062 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:48:27,062 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 5208907 2021-06-08 11:48:27,063 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:48:27,063 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:48:28,292 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:48:28,292 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 5208907 2021-06-08 11:48:28,293 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:48:28,293 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:48:29,520 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:48:29,520 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 5208907 2021-06-08 11:48:29,523 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:48:29,523 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:48:30,763 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:48:30,765 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 5208907 2021-06-08 11:48:30,766 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:48:30,766 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:48:31,987 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:48:31,987 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 5208907 2021-06-08 11:48:31,990 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:48:31,990 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:48:33,236 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:48:33,236 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 5208907 2021-06-08 11:48:33,237 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:48:33,237 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:48:34,467 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:48:34,467 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 5208907 2021-06-08 11:48:34,470 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:48:34,470 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:48:35,694 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:48:35,696 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 5208907 2021-06-08 11:48:35,697 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:48:35,697 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:48:36,927 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:48:36,927 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 5208907 2021-06-08 11:48:36,928 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:48:36,928 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:48:38,154 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:48:38,155 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 5208907 2021-06-08 11:48:38,157 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:48:38,157 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:48:39,430 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:48:39,430 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 5208907 2021-06-08 11:48:39,433 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:48:39,433 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:48:43,585 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 4893.589237 2021-06-08 11:48:43,585 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 5208907 2021-06-08 11:48:43,588 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:48:43,588 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:48:44,822 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 4893.589237 2021-06-08 11:48:44,822 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 5208907 2021-06-08 11:48:44,823 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:48:44,825 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:48:46,046 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 4893.589237 2021-06-08 11:48:46,046 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 5208907 2021-06-08 11:48:46,049 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:48:46,049 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:48:47,305 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 4893.589237 2021-06-08 11:48:47,305 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 5208907 2021-06-08 11:48:47,309 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:48:47,309 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:48:48,536 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 4893.589237 2021-06-08 11:48:48,536 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 5208907 2021-06-08 11:48:48,536 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:48:48,538 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:48:49,769 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 4893.589237 2021-06-08 11:48:49,769 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 5208907 2021-06-08 11:48:49,772 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:48:49,772 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:48:51,002 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 4893.589237 2021-06-08 11:48:51,003 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 5208907 2021-06-08 11:48:51,003 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:48:51,003 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:48:52,237 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 4856.700000 2021-06-08 11:48:52,237 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 5209259 2021-06-08 11:48:52,240 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:48:52,240 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:48:53,489 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 4856.700000 2021-06-08 11:48:53,489 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 5209259 2021-06-08 11:48:53,490 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:48:53,490 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:48:54,719 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 4856.700000 2021-06-08 11:48:54,719 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 5209259 2021-06-08 11:48:54,720 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:48:54,720 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:48:56,154 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 4856.700000 2021-06-08 11:48:56,154 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 5209259 2021-06-08 11:48:56,155 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:48:56,155 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:48:57,426 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 4856.700000 2021-06-08 11:48:57,426 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 5209259 2021-06-08 11:48:57,428 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:48:57,428 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:48:59,589 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 4856.700000 2021-06-08 11:48:59,589 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 5209259 2021-06-08 11:48:59,592 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:48:59,592 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:49:01,207 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 4856.700000 2021-06-08 11:49:01,207 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 5209259 2021-06-08 11:49:01,210 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:49:01,210 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:49:02,825 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 8221.400000 2021-06-08 11:49:02,825 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 5605775 2021-06-08 11:49:02,829 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:49:02,829 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:49:04,250 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 8221.400000 2021-06-08 11:49:04,250 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 5605775 2021-06-08 11:49:04,253 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:49:04,253 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:49:05,709 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 8221.400000 2021-06-08 11:49:05,710 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 5605775 2021-06-08 11:49:05,711 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:49:05,711 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:49:06,982 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 8221.400000 2021-06-08 11:49:06,982 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 5605775 2021-06-08 11:49:06,983 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:49:06,983 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:49:08,207 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 8221.400000 2021-06-08 11:49:08,207 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 5605775 2021-06-08 11:49:08,210 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:49:08,210 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:49:09,437 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 8221.400000 2021-06-08 11:49:09,437 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 5605775 2021-06-08 11:49:09,437 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:49:09,438 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:49:10,668 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 8221.400000 2021-06-08 11:49:10,668 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 5605775 2021-06-08 11:49:10,670 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:49:10,671 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:49:11,895 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 30472.313943 2021-06-08 11:49:11,897 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 6241135 2021-06-08 11:49:11,898 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:49:11,898 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:49:13,125 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 30472.313943 2021-06-08 11:49:13,125 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 6241135 2021-06-08 11:49:13,128 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:49:13,128 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:49:14,358 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 30472.313943 2021-06-08 11:49:14,358 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 6241135 2021-06-08 11:49:14,358 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:49:14,358 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:49:15,592 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 30472.313943 2021-06-08 11:49:15,592 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 6241135 2021-06-08 11:49:15,594 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:49:15,594 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:49:16,825 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 30472.313943 2021-06-08 11:49:16,825 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 6241135 2021-06-08 11:49:16,828 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:49:16,828 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:49:18,055 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 30472.313943 2021-06-08 11:49:18,056 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 6241135 2021-06-08 11:49:18,058 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:49:18,058 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:49:19,293 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 30472.313943 2021-06-08 11:49:19,293 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 6241135 2021-06-08 11:49:19,295 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:49:19,296 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:49:20,522 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 30472.313943 2021-06-08 11:49:20,522 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 6241135 2021-06-08 11:49:20,523 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:49:20,523 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:49:21,749 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:49:21,749 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 6241135 2021-06-08 11:49:21,750 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:49:21,750 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:49:22,992 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:49:22,992 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 6241135 2021-06-08 11:49:22,993 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:49:22,993 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:49:24,216 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:49:24,216 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 6241135 2021-06-08 11:49:24,217 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:49:24,217 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:49:25,443 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:49:25,443 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 6241135 2021-06-08 11:49:25,444 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:49:25,444 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:49:26,674 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:49:26,676 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 6241135 2021-06-08 11:49:26,677 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:49:26,677 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:49:27,900 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:49:27,900 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 6241135 2021-06-08 11:49:27,901 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:49:27,901 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:49:29,125 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:49:29,127 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 6241135 2021-06-08 11:49:29,128 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:49:29,128 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:49:30,355 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:49:30,355 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 6241135 2021-06-08 11:49:30,355 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:49:30,355 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:49:31,588 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:49:31,588 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 6241135 2021-06-08 11:49:31,589 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:49:31,589 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:49:32,812 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:49:32,813 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 6241135 2021-06-08 11:49:32,815 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:49:32,815 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:49:34,035 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:49:34,036 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 6241135 2021-06-08 11:49:34,036 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:49:34,036 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:49:35,405 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:49:35,405 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 6241135 2021-06-08 11:49:35,408 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:49:35,408 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:49:36,888 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:49:36,888 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 6241135 2021-06-08 11:49:36,890 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:49:36,891 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:49:38,394 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:49:38,394 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 6241135 2021-06-08 11:49:38,398 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:49:38,398 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:49:38,703 | infra | INFO | pool-5-thread-17 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623178072.29_create__8150000_8200000_1623178072.31' completed 2021-06-08 11:49:38,729 | infra | INFO | pool-5-thread-13 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623178072.29_create__8950000_9000000_1623178072.31' completed 2021-06-08 11:49:38,815 | infra | INFO | pool-5-thread-6 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623178072.29_create__8450000_8500000_1623178072.31' completed 2021-06-08 11:49:38,838 | infra | INFO | pool-5-thread-9 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623178072.29_create__8350000_8400000_1623178072.31' completed 2021-06-08 11:49:38,891 | infra | INFO | pool-5-thread-10 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623178072.29_create__8550000_8600000_1623178072.31' completed 2021-06-08 11:49:38,959 | infra | INFO | pool-5-thread-20 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623178072.29_create__8800000_8850000_1623178072.31' completed 2021-06-08 11:49:38,976 | infra | INFO | pool-5-thread-5 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623178072.29_create__8700000_8750000_1623178072.31' completed 2021-06-08 11:49:39,063 | infra | INFO | pool-5-thread-4 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623178072.29_create__8050000_8100000_1623178072.31' completed 2021-06-08 11:49:39,157 | infra | INFO | pool-5-thread-7 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623178072.29_create__8300000_8350000_1623178072.31' completed 2021-06-08 11:49:39,459 | infra | INFO | pool-5-thread-29 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623178072.29_create__8100000_8150000_1623178072.31' completed 2021-06-08 11:49:39,987 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:49:39,989 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 6241135 2021-06-08 11:49:39,990 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:49:39,990 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:49:40,154 | infra | INFO | pool-5-thread-23 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623178072.29_create__8750000_8800000_1623178072.31' completed 2021-06-08 11:49:40,194 | infra | INFO | pool-5-thread-11 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623178072.29_create__8650000_8700000_1623178072.31' completed 2021-06-08 11:49:40,240 | infra | INFO | pool-5-thread-3 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623178072.29_create__8000000_8050000_1623178072.31' completed 2021-06-08 11:49:40,240 | infra | DEBUG | pool-5-thread-24 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623178072.29_create__8000000_8050000_1623178072.31 are 50000 2021-06-08 11:49:40,240 | infra | DEBUG | pool-5-thread-24 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623178072.29_create__8050000_8100000_1623178072.31 2021-06-08 11:49:40,240 | infra | DEBUG | pool-5-thread-24 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623178072.29_create__8050000_8100000_1623178072.31 are 50000 2021-06-08 11:49:40,240 | infra | DEBUG | pool-5-thread-24 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623178072.29_create__8100000_8150000_1623178072.31 2021-06-08 11:49:40,240 | infra | DEBUG | pool-5-thread-24 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623178072.29_create__8100000_8150000_1623178072.31 are 50000 2021-06-08 11:49:40,242 | infra | DEBUG | pool-5-thread-24 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623178072.29_create__8150000_8200000_1623178072.31 2021-06-08 11:49:40,242 | infra | DEBUG | pool-5-thread-24 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623178072.29_create__8150000_8200000_1623178072.31 are 50000 2021-06-08 11:49:40,242 | infra | DEBUG | pool-5-thread-24 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623178072.29_create__8200000_8250000_1623178072.31 2021-06-08 11:49:40,299 | infra | INFO | pool-5-thread-25 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623178072.29_create__8250000_8300000_1623178072.31' completed 2021-06-08 11:49:40,302 | infra | INFO | pool-5-thread-18 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623178072.29_create__8500000_8550000_1623178072.31' completed 2021-06-08 11:49:40,313 | infra | INFO | pool-5-thread-12 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623178072.29_create__8600000_8650000_1623178072.31' completed 2021-06-08 11:49:40,384 | infra | INFO | pool-5-thread-1 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623178072.29_create__8200000_8250000_1623178072.31' completed 2021-06-08 11:49:40,384 | infra | DEBUG | pool-5-thread-24 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623178072.29_create__8200000_8250000_1623178072.31 are 50000 2021-06-08 11:49:40,384 | infra | DEBUG | pool-5-thread-24 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623178072.29_create__8250000_8300000_1623178072.31 2021-06-08 11:49:40,384 | infra | INFO | pool-5-thread-28 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623178072.29_create__8400000_8450000_1623178072.31' completed 2021-06-08 11:49:40,384 | infra | DEBUG | pool-5-thread-24 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623178072.29_create__8250000_8300000_1623178072.31 are 50000 2021-06-08 11:49:40,384 | infra | DEBUG | pool-5-thread-24 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623178072.29_create__8300000_8350000_1623178072.31 2021-06-08 11:49:40,384 | infra | DEBUG | pool-5-thread-24 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623178072.29_create__8300000_8350000_1623178072.31 are 50000 2021-06-08 11:49:40,384 | infra | DEBUG | pool-5-thread-24 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623178072.29_create__8350000_8400000_1623178072.31 2021-06-08 11:49:40,385 | infra | DEBUG | pool-5-thread-24 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623178072.29_create__8350000_8400000_1623178072.31 are 50000 2021-06-08 11:49:40,385 | infra | DEBUG | pool-5-thread-24 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623178072.29_create__8400000_8450000_1623178072.31 2021-06-08 11:49:40,385 | infra | DEBUG | pool-5-thread-24 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623178072.29_create__8400000_8450000_1623178072.31 are 50000 2021-06-08 11:49:40,385 | infra | DEBUG | pool-5-thread-24 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623178072.29_create__8450000_8500000_1623178072.31 2021-06-08 11:49:40,385 | infra | DEBUG | pool-5-thread-24 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623178072.29_create__8450000_8500000_1623178072.31 are 50000 2021-06-08 11:49:40,385 | infra | DEBUG | pool-5-thread-24 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623178072.29_create__8500000_8550000_1623178072.31 2021-06-08 11:49:40,385 | infra | DEBUG | pool-5-thread-24 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623178072.29_create__8500000_8550000_1623178072.31 are 50000 2021-06-08 11:49:40,385 | infra | DEBUG | pool-5-thread-24 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623178072.29_create__8550000_8600000_1623178072.31 2021-06-08 11:49:40,385 | infra | DEBUG | pool-5-thread-24 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623178072.29_create__8550000_8600000_1623178072.31 are 50000 2021-06-08 11:49:40,387 | infra | DEBUG | pool-5-thread-24 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623178072.29_create__8600000_8650000_1623178072.31 2021-06-08 11:49:40,387 | infra | DEBUG | pool-5-thread-24 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623178072.29_create__8600000_8650000_1623178072.31 are 50000 2021-06-08 11:49:40,387 | infra | DEBUG | pool-5-thread-24 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623178072.29_create__8650000_8700000_1623178072.31 2021-06-08 11:49:40,387 | infra | DEBUG | pool-5-thread-24 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623178072.29_create__8650000_8700000_1623178072.31 are 50000 2021-06-08 11:49:40,387 | infra | DEBUG | pool-5-thread-24 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623178072.29_create__8700000_8750000_1623178072.31 2021-06-08 11:49:40,387 | infra | DEBUG | pool-5-thread-24 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623178072.29_create__8700000_8750000_1623178072.31 are 50000 2021-06-08 11:49:40,387 | infra | DEBUG | pool-5-thread-24 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623178072.29_create__8750000_8800000_1623178072.31 2021-06-08 11:49:40,387 | infra | DEBUG | pool-5-thread-24 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623178072.29_create__8750000_8800000_1623178072.31 are 50000 2021-06-08 11:49:40,387 | infra | DEBUG | pool-5-thread-24 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623178072.29_create__8800000_8850000_1623178072.31 2021-06-08 11:49:40,388 | infra | DEBUG | pool-5-thread-24 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623178072.29_create__8800000_8850000_1623178072.31 are 50000 2021-06-08 11:49:40,388 | infra | DEBUG | pool-5-thread-24 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623178072.29_create__8850000_8900000_1623178072.31 2021-06-08 11:49:40,482 | infra | INFO | pool-5-thread-8 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623178072.29_create__8850000_8900000_1623178072.31' completed 2021-06-08 11:49:40,482 | infra | DEBUG | pool-5-thread-24 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623178072.29_create__8850000_8900000_1623178072.31 are 50000 2021-06-08 11:49:40,482 | infra | DEBUG | pool-5-thread-24 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623178072.29_create__8900000_8950000_1623178072.31 2021-06-08 11:49:40,483 | infra | INFO | pool-5-thread-19 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623178072.29_create__8900000_8950000_1623178072.31' completed 2021-06-08 11:49:40,483 | infra | DEBUG | pool-5-thread-24 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623178072.29_create__8900000_8950000_1623178072.31 are 50000 2021-06-08 11:49:40,483 | infra | DEBUG | pool-5-thread-24 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623178072.29_create__8950000_9000000_1623178072.31 2021-06-08 11:49:40,483 | infra | DEBUG | pool-5-thread-24 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623178072.29_create__8950000_9000000_1623178072.31 are 50000 2021-06-08 11:49:40,483 | infra | DEBUG | pool-5-thread-24 | [task_manager:get_task_result:26] Getting task result for PrintBucketStats_default_1623178072.31 2021-06-08 11:49:40,992 | test | INFO | pool-5-thread-27 | [table_view:display:72] Ops trend for bucket 'default' +----------+---------------------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +----------+---------------------------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | *******.................................********X | 10199.800 | | 0.000 | ***********************................******X | 4893.589 | | 4856.700 | ..............*******...........*******..........................................................................*******X | 30472.314 | | 0.000 | **************X | 0.000 | +----------+---------------------------------------------------------------------------------------------------------------------------+-----------+ 2021-06-08 11:49:40,992 | infra | INFO | pool-5-thread-27 | [task:complete_task:102] Thread 'PrintBucketStats_default_1623178072.31' completed 2021-06-08 11:49:40,992 | infra | DEBUG | pool-5-thread-24 | [task:call:1663] ========= Tasks in loadgen pool======= 2021-06-08 11:49:40,992 | infra | WARNING | pool-5-thread-24 | [task_manager:print_tasks_in_pool:83] Task 'LoadDocsGen_default__default__default_create_8000000_9000000_1623178072.29' not completed 2021-06-08 11:49:40,992 | infra | DEBUG | pool-5-thread-24 | [task:call:1665] ====================================== 2021-06-08 11:49:40,992 | infra | DEBUG | pool-5-thread-24 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623178072.29_create__8000000_8050000_1623178072.31' complete. Loaded 50000 items 2021-06-08 11:49:40,993 | infra | DEBUG | pool-5-thread-24 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623178072.29_create__8050000_8100000_1623178072.31' complete. Loaded 50000 items 2021-06-08 11:49:40,993 | infra | DEBUG | pool-5-thread-24 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623178072.29_create__8100000_8150000_1623178072.31' complete. Loaded 50000 items 2021-06-08 11:49:40,993 | infra | DEBUG | pool-5-thread-24 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623178072.29_create__8150000_8200000_1623178072.31' complete. Loaded 50000 items 2021-06-08 11:49:40,993 | infra | DEBUG | pool-5-thread-24 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623178072.29_create__8200000_8250000_1623178072.31' complete. Loaded 50000 items 2021-06-08 11:49:40,993 | infra | DEBUG | pool-5-thread-24 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623178072.29_create__8250000_8300000_1623178072.31' complete. Loaded 50000 items 2021-06-08 11:49:40,993 | infra | DEBUG | pool-5-thread-24 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623178072.29_create__8300000_8350000_1623178072.31' complete. Loaded 50000 items 2021-06-08 11:49:40,993 | infra | DEBUG | pool-5-thread-24 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623178072.29_create__8350000_8400000_1623178072.31' complete. Loaded 50000 items 2021-06-08 11:49:40,993 | infra | DEBUG | pool-5-thread-24 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623178072.29_create__8400000_8450000_1623178072.31' complete. Loaded 50000 items 2021-06-08 11:49:40,993 | infra | DEBUG | pool-5-thread-24 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623178072.29_create__8450000_8500000_1623178072.31' complete. Loaded 50000 items 2021-06-08 11:49:40,993 | infra | DEBUG | pool-5-thread-24 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623178072.29_create__8500000_8550000_1623178072.31' complete. Loaded 50000 items 2021-06-08 11:49:40,993 | infra | DEBUG | pool-5-thread-24 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623178072.29_create__8550000_8600000_1623178072.31' complete. Loaded 50000 items 2021-06-08 11:49:40,993 | infra | DEBUG | pool-5-thread-24 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623178072.29_create__8600000_8650000_1623178072.31' complete. Loaded 50000 items 2021-06-08 11:49:40,993 | infra | DEBUG | pool-5-thread-24 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623178072.29_create__8650000_8700000_1623178072.31' complete. Loaded 50000 items 2021-06-08 11:49:40,993 | infra | DEBUG | pool-5-thread-24 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623178072.29_create__8700000_8750000_1623178072.31' complete. Loaded 50000 items 2021-06-08 11:49:40,993 | infra | DEBUG | pool-5-thread-24 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623178072.29_create__8750000_8800000_1623178072.31' complete. Loaded 50000 items 2021-06-08 11:49:40,993 | infra | DEBUG | pool-5-thread-24 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623178072.29_create__8800000_8850000_1623178072.31' complete. Loaded 50000 items 2021-06-08 11:49:40,993 | infra | DEBUG | pool-5-thread-24 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623178072.29_create__8850000_8900000_1623178072.31' complete. Loaded 50000 items 2021-06-08 11:49:40,993 | infra | DEBUG | pool-5-thread-24 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623178072.29_create__8900000_8950000_1623178072.31' complete. Loaded 50000 items 2021-06-08 11:49:40,993 | infra | DEBUG | pool-5-thread-24 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_8000000_9000000_1623178072.29_create__8950000_9000000_1623178072.31' complete. Loaded 50000 items 2021-06-08 11:49:40,994 | infra | INFO | pool-5-thread-24 | [task:complete_task:102] Thread 'LoadDocsGen_default__default__default_create_8000000_9000000_1623178072.29' completed 2021-06-08 11:49:40,996 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.125 with username: root, password: couchbase 2021-06-08 11:49:41,095 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 11:49:41,318 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: ls /etc/ 2021-06-08 11:49:41,378 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 11:49:41,378 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: ls /cygdrive/c/Windows 2021-06-08 11:49:41,391 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: uname -m 2021-06-08 11:49:41,404 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: cat /proc/cpuinfo 2021-06-08 11:49:41,417 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: df -Thl 2021-06-08 11:49:41,433 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: cat /proc/meminfo 2021-06-08 11:49:41,444 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: hostname 2021-06-08 11:49:41,459 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: hostname -d 2021-06-08 11:49:41,473 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:559] Starting StatsWaitTask for ep_data_write_failed on bucket default 2021-06-08 11:49:41,473 | infra | INFO | MainThread | [task_manager:add_new_task:23] Added new task: StatsWaitTask_default_ep_data_write_failed_1623178181.47 2021-06-08 11:49:41,473 | infra | INFO | pool-5-thread-14 | [task:start_task:87] Thread 'StatsWaitTask_default_ep_data_write_failed_1623178181.47' started 2021-06-08 11:49:41,473 | infra | DEBUG | pool-5-thread-14 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:49:41,476 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.155 with username: root, password: couchbase 2021-06-08 11:49:41,548 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 11:49:41,700 | test | DEBUG | pool-5-thread-14 | [task:_compare:2649] Comparing 137703 > 0 2021-06-08 11:49:41,700 | test | DEBUG | pool-5-thread-14 | [task:_get_all_stats_and_compare:2601] Ready: ep_data_write_failed 137703 > 0. Received: {'172.23.106.125': '137703'} for bucket 'default' 2021-06-08 11:49:41,700 | infra | INFO | pool-5-thread-14 | [task:complete_task:102] Thread 'StatsWaitTask_default_ep_data_write_failed_1623178181.47' completed 2021-06-08 11:49:41,772 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: ls /etc/ 2021-06-08 11:49:41,834 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 11:49:41,834 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: ls /cygdrive/c/Windows 2021-06-08 11:49:41,848 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: uname -m 2021-06-08 11:49:41,859 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: cat /proc/cpuinfo 2021-06-08 11:49:41,872 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: df -Thl 2021-06-08 11:49:41,888 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: cat /proc/meminfo 2021-06-08 11:49:41,901 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: hostname 2021-06-08 11:49:41,914 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: hostname -d 2021-06-08 11:49:41,927 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:559] Starting StatsWaitTask for ep_data_write_failed on bucket default 2021-06-08 11:49:41,927 | infra | INFO | MainThread | [task_manager:add_new_task:23] Added new task: StatsWaitTask_default_ep_data_write_failed_1623178181.93 2021-06-08 11:49:41,927 | infra | INFO | pool-5-thread-26 | [task:start_task:87] Thread 'StatsWaitTask_default_ep_data_write_failed_1623178181.93' started 2021-06-08 11:49:41,930 | infra | DEBUG | pool-5-thread-26 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:49:41,931 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.156 with username: root, password: couchbase 2021-06-08 11:49:42,023 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 11:49:42,151 | test | DEBUG | pool-5-thread-26 | [task:_compare:2649] Comparing 334059 > 0 2021-06-08 11:49:42,151 | test | DEBUG | pool-5-thread-26 | [task:_get_all_stats_and_compare:2601] Ready: ep_data_write_failed 334059 > 0. Received: {'172.23.106.155': '334059'} for bucket 'default' 2021-06-08 11:49:42,151 | infra | INFO | pool-5-thread-26 | [task:complete_task:102] Thread 'StatsWaitTask_default_ep_data_write_failed_1623178181.93' completed 2021-06-08 11:49:42,240 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: ls /etc/ 2021-06-08 11:49:42,296 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 11:49:42,296 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: ls /cygdrive/c/Windows 2021-06-08 11:49:42,309 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: uname -m 2021-06-08 11:49:42,323 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: cat /proc/cpuinfo 2021-06-08 11:49:42,336 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: df -Thl 2021-06-08 11:49:42,351 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: cat /proc/meminfo 2021-06-08 11:49:42,364 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: hostname 2021-06-08 11:49:42,377 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: hostname -d 2021-06-08 11:49:42,391 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:559] Starting StatsWaitTask for ep_data_write_failed on bucket default 2021-06-08 11:49:42,391 | infra | INFO | MainThread | [task_manager:add_new_task:23] Added new task: StatsWaitTask_default_ep_data_write_failed_1623178182.39 2021-06-08 11:49:42,391 | infra | INFO | pool-5-thread-22 | [task:start_task:87] Thread 'StatsWaitTask_default_ep_data_write_failed_1623178182.39' started 2021-06-08 11:49:42,391 | infra | DEBUG | pool-5-thread-22 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:49:42,394 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.122 with username: root, password: couchbase 2021-06-08 11:49:42,482 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 11:49:42,703 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: ls /etc/ 2021-06-08 11:49:42,766 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 11:49:42,766 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: ls /cygdrive/c/Windows 2021-06-08 11:49:42,779 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: uname -m 2021-06-08 11:49:42,792 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: cat /proc/cpuinfo 2021-06-08 11:49:42,805 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: df -Thl 2021-06-08 11:49:42,822 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: cat /proc/meminfo 2021-06-08 11:49:42,836 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: hostname 2021-06-08 11:49:42,851 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: hostname -d 2021-06-08 11:49:42,868 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:559] Starting StatsWaitTask for ep_data_write_failed on bucket default 2021-06-08 11:49:42,868 | infra | INFO | MainThread | [task_manager:add_new_task:23] Added new task: StatsWaitTask_default_ep_data_write_failed_1623178182.87 2021-06-08 11:49:42,868 | infra | INFO | pool-5-thread-15 | [task:start_task:87] Thread 'StatsWaitTask_default_ep_data_write_failed_1623178182.87' started 2021-06-08 11:49:42,868 | infra | DEBUG | MainThread | [task_manager:get_task_result:26] Getting task result for StatsWaitTask_default_ep_data_write_failed_1623178181.47 2021-06-08 11:49:42,868 | infra | DEBUG | MainThread | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.125 2021-06-08 11:49:42,868 | infra | DEBUG | pool-5-thread-15 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:49:42,868 | infra | DEBUG | MainThread | [task_manager:get_task_result:26] Getting task result for StatsWaitTask_default_ep_data_write_failed_1623178181.93 2021-06-08 11:49:42,871 | infra | DEBUG | MainThread | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.155 2021-06-08 11:49:42,871 | infra | DEBUG | MainThread | [task_manager:get_task_result:26] Getting task result for StatsWaitTask_default_ep_data_write_failed_1623178182.39 2021-06-08 11:49:43,098 | test | DEBUG | pool-5-thread-15 | [task:_compare:2649] Comparing 136469 > 0 2021-06-08 11:49:43,099 | test | DEBUG | pool-5-thread-15 | [task:_get_all_stats_and_compare:2601] Ready: ep_data_write_failed 136469 > 0. Received: {'172.23.106.122': '136469'} for bucket 'default' 2021-06-08 11:49:43,099 | infra | INFO | pool-5-thread-15 | [task:complete_task:102] Thread 'StatsWaitTask_default_ep_data_write_failed_1623178182.87' completed 2021-06-08 11:49:45,864 | test | DEBUG | pool-5-thread-22 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 11:49:45,865 | infra | DEBUG | pool-5-thread-22 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.156': '0'} for bucket 'default' 2021-06-08 11:49:45,865 | infra | DEBUG | pool-5-thread-22 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 11:49:45,865 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 11:49:45,867 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 11:49:50,867 | infra | DEBUG | pool-5-thread-22 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:49:58,773 | test | DEBUG | pool-5-thread-22 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 11:49:58,773 | infra | DEBUG | pool-5-thread-22 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.156': '0'} for bucket 'default' 2021-06-08 11:49:58,773 | infra | DEBUG | pool-5-thread-22 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 11:49:58,775 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 11:49:58,775 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 11:50:03,775 | infra | DEBUG | pool-5-thread-22 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:50:14,305 | test | DEBUG | pool-5-thread-22 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 11:50:14,305 | infra | DEBUG | pool-5-thread-22 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.156': '0'} for bucket 'default' 2021-06-08 11:50:14,305 | infra | DEBUG | pool-5-thread-22 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 11:50:14,306 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 11:50:14,306 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 11:50:19,308 | infra | DEBUG | pool-5-thread-22 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:50:21,747 | test | DEBUG | pool-5-thread-22 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 11:50:21,747 | infra | DEBUG | pool-5-thread-22 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.156': '0'} for bucket 'default' 2021-06-08 11:50:21,749 | infra | DEBUG | pool-5-thread-22 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 11:50:21,750 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 11:50:21,752 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 11:50:26,752 | infra | DEBUG | pool-5-thread-22 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:50:30,901 | test | DEBUG | pool-5-thread-22 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 11:50:30,901 | infra | DEBUG | pool-5-thread-22 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.156': '0'} for bucket 'default' 2021-06-08 11:50:30,901 | infra | DEBUG | pool-5-thread-22 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 11:50:30,903 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 11:50:30,903 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 11:50:35,903 | infra | DEBUG | pool-5-thread-22 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:50:47,931 | test | DEBUG | pool-5-thread-22 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 11:50:47,933 | infra | DEBUG | pool-5-thread-22 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.156': '0'} for bucket 'default' 2021-06-08 11:50:47,933 | infra | DEBUG | pool-5-thread-22 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 11:50:47,934 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 11:50:47,934 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 11:50:52,934 | infra | DEBUG | pool-5-thread-22 | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.156 2021-06-08 11:50:52,936 | infra | INFO | pool-5-thread-22 | [task:complete_task:102] Thread 'StatsWaitTask_default_ep_data_write_failed_1623178182.39' completed 2021-06-08 11:50:52,941 | infra | INFO | MainThread | [task_manager:add_new_task:23] Added new task: LoadDocsGen_default__default__default_create_9000000_10000000_1623178252.94 2021-06-08 11:50:52,941 | infra | INFO | pool-5-thread-21 | [task:start_task:87] Thread 'LoadDocsGen_default__default__default_create_9000000_10000000_1623178252.94' started 2021-06-08 11:50:52,941 | infra | DEBUG | MainThread | [task_manager:get_task_result:26] Getting task result for LoadDocsGen_default__default__default_create_9000000_10000000_1623178252.94 2021-06-08 11:50:52,980 | infra | INFO | pool-5-thread-21 | [task_manager:add_new_task:23] Added new task: PrintBucketStats_default_1623178252.97 2021-06-08 11:50:52,980 | infra | INFO | pool-5-thread-16 | [task:start_task:87] Thread 'PrintBucketStats_default_1623178252.97' started 2021-06-08 11:50:52,980 | infra | INFO | pool-5-thread-21 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_9000000_10000000_1623178252.94_create__9000000_9050000_1623178252.97 2021-06-08 11:50:52,980 | infra | INFO | pool-5-thread-30 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_9000000_10000000_1623178252.94_create__9000000_9050000_1623178252.97' started 2021-06-08 11:50:52,980 | infra | INFO | pool-5-thread-21 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_9000000_10000000_1623178252.94_create__9050000_9100000_1623178252.97 2021-06-08 11:50:52,980 | infra | INFO | pool-5-thread-2 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_9000000_10000000_1623178252.94_create__9050000_9100000_1623178252.97' started 2021-06-08 11:50:52,982 | infra | INFO | pool-5-thread-21 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_9000000_10000000_1623178252.94_create__9100000_9150000_1623178252.97 2021-06-08 11:50:52,982 | infra | INFO | pool-5-thread-17 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_9000000_10000000_1623178252.94_create__9100000_9150000_1623178252.97' started 2021-06-08 11:50:52,982 | infra | INFO | pool-5-thread-21 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_9000000_10000000_1623178252.94_create__9150000_9200000_1623178252.97 2021-06-08 11:50:52,982 | infra | INFO | pool-5-thread-13 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_9000000_10000000_1623178252.94_create__9150000_9200000_1623178252.97' started 2021-06-08 11:50:52,982 | infra | INFO | pool-5-thread-21 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_9000000_10000000_1623178252.94_create__9200000_9250000_1623178252.97 2021-06-08 11:50:52,982 | infra | INFO | pool-5-thread-6 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_9000000_10000000_1623178252.94_create__9200000_9250000_1623178252.97' started 2021-06-08 11:50:52,982 | infra | INFO | pool-5-thread-21 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_9000000_10000000_1623178252.94_create__9250000_9300000_1623178252.97 2021-06-08 11:50:52,982 | infra | INFO | pool-5-thread-9 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_9000000_10000000_1623178252.94_create__9250000_9300000_1623178252.97' started 2021-06-08 11:50:52,983 | infra | INFO | pool-5-thread-21 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_9000000_10000000_1623178252.94_create__9300000_9350000_1623178252.97 2021-06-08 11:50:52,983 | infra | INFO | pool-5-thread-10 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_9000000_10000000_1623178252.94_create__9300000_9350000_1623178252.97' started 2021-06-08 11:50:52,983 | infra | INFO | pool-5-thread-21 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_9000000_10000000_1623178252.94_create__9350000_9400000_1623178252.97 2021-06-08 11:50:52,983 | infra | INFO | pool-5-thread-20 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_9000000_10000000_1623178252.94_create__9350000_9400000_1623178252.97' started 2021-06-08 11:50:52,983 | infra | INFO | pool-5-thread-21 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_9000000_10000000_1623178252.94_create__9400000_9450000_1623178252.97 2021-06-08 11:50:52,983 | infra | INFO | pool-5-thread-5 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_9000000_10000000_1623178252.94_create__9400000_9450000_1623178252.97' started 2021-06-08 11:50:52,983 | infra | INFO | pool-5-thread-4 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_9000000_10000000_1623178252.94_create__9450000_9500000_1623178252.97' started 2021-06-08 11:50:52,983 | infra | INFO | pool-5-thread-21 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_9000000_10000000_1623178252.94_create__9450000_9500000_1623178252.97 2021-06-08 11:50:52,983 | infra | INFO | pool-5-thread-21 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_9000000_10000000_1623178252.94_create__9500000_9550000_1623178252.97 2021-06-08 11:50:52,983 | infra | INFO | pool-5-thread-29 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_9000000_10000000_1623178252.94_create__9550000_9600000_1623178252.97' started 2021-06-08 11:50:52,983 | infra | INFO | pool-5-thread-7 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_9000000_10000000_1623178252.94_create__9500000_9550000_1623178252.97' started 2021-06-08 11:50:52,983 | infra | INFO | pool-5-thread-21 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_9000000_10000000_1623178252.94_create__9550000_9600000_1623178252.97 2021-06-08 11:50:52,984 | infra | INFO | pool-5-thread-21 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_9000000_10000000_1623178252.94_create__9600000_9650000_1623178252.97 2021-06-08 11:50:52,984 | infra | INFO | pool-5-thread-23 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_9000000_10000000_1623178252.94_create__9600000_9650000_1623178252.97' started 2021-06-08 11:50:52,984 | infra | INFO | pool-5-thread-21 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_9000000_10000000_1623178252.94_create__9650000_9700000_1623178252.97 2021-06-08 11:50:52,984 | infra | INFO | pool-5-thread-11 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_9000000_10000000_1623178252.94_create__9650000_9700000_1623178252.97' started 2021-06-08 11:50:52,984 | infra | INFO | pool-5-thread-21 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_9000000_10000000_1623178252.94_create__9700000_9750000_1623178252.97 2021-06-08 11:50:52,984 | infra | INFO | pool-5-thread-3 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_9000000_10000000_1623178252.94_create__9700000_9750000_1623178252.97' started 2021-06-08 11:50:52,984 | infra | INFO | pool-5-thread-21 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_9000000_10000000_1623178252.94_create__9750000_9800000_1623178252.97 2021-06-08 11:50:52,984 | infra | INFO | pool-5-thread-21 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_9000000_10000000_1623178252.94_create__9800000_9850000_1623178252.97 2021-06-08 11:50:52,984 | infra | INFO | pool-5-thread-18 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_9000000_10000000_1623178252.94_create__9800000_9850000_1623178252.97' started 2021-06-08 11:50:52,984 | infra | INFO | pool-5-thread-25 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_9000000_10000000_1623178252.94_create__9750000_9800000_1623178252.97' started 2021-06-08 11:50:52,984 | infra | INFO | pool-5-thread-21 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_9000000_10000000_1623178252.94_create__9850000_9900000_1623178252.97 2021-06-08 11:50:52,984 | infra | INFO | pool-5-thread-12 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_9000000_10000000_1623178252.94_create__9850000_9900000_1623178252.97' started 2021-06-08 11:50:52,992 | infra | INFO | pool-5-thread-21 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_9000000_10000000_1623178252.94_create__9900000_9950000_1623178252.97 2021-06-08 11:50:52,992 | infra | INFO | pool-5-thread-1 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_9000000_10000000_1623178252.94_create__9900000_9950000_1623178252.97' started 2021-06-08 11:50:52,992 | infra | INFO | pool-5-thread-21 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_9000000_10000000_1623178252.94_create__9950000_10000000_1623178252.97 2021-06-08 11:50:52,992 | infra | INFO | pool-5-thread-28 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_9000000_10000000_1623178252.94_create__9950000_10000000_1623178252.97' started 2021-06-08 11:50:52,992 | infra | DEBUG | pool-5-thread-21 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_9000000_10000000_1623178252.94_create__9000000_9050000_1623178252.97 2021-06-08 11:50:53,230 | test | DEBUG | pool-5-thread-16 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:50:53,230 | test | DEBUG | pool-5-thread-16 | [task:print_ep_queue_size:3508] ep_queue_size for default: 1606392 2021-06-08 11:50:53,233 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:50:53,233 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:50:54,461 | test | DEBUG | pool-5-thread-16 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:50:54,463 | test | DEBUG | pool-5-thread-16 | [task:print_ep_queue_size:3508] ep_queue_size for default: 1606392 2021-06-08 11:50:54,464 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:50:54,464 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:50:55,684 | test | DEBUG | pool-5-thread-16 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:50:55,686 | test | DEBUG | pool-5-thread-16 | [task:print_ep_queue_size:3508] ep_queue_size for default: 1606392 2021-06-08 11:50:55,687 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:50:55,687 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:50:56,903 | test | DEBUG | pool-5-thread-16 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:50:56,903 | test | DEBUG | pool-5-thread-16 | [task:print_ep_queue_size:3508] ep_queue_size for default: 1606392 2021-06-08 11:50:56,904 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:50:56,905 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:50:58,200 | test | DEBUG | pool-5-thread-16 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:50:58,200 | test | DEBUG | pool-5-thread-16 | [task:print_ep_queue_size:3508] ep_queue_size for default: 1606392 2021-06-08 11:50:58,206 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:50:58,207 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:50:59,489 | test | DEBUG | pool-5-thread-16 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:50:59,489 | test | DEBUG | pool-5-thread-16 | [task:print_ep_queue_size:3508] ep_queue_size for default: 1606392 2021-06-08 11:50:59,490 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:50:59,490 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:51:00,700 | test | DEBUG | pool-5-thread-16 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:51:00,700 | test | DEBUG | pool-5-thread-16 | [task:print_ep_queue_size:3508] ep_queue_size for default: 1606392 2021-06-08 11:51:00,703 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:51:00,703 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:51:01,911 | test | DEBUG | pool-5-thread-16 | [task:record_bucket_ops:3495] Ops rate for 'default': 4085.000000 2021-06-08 11:51:01,911 | test | DEBUG | pool-5-thread-16 | [task:print_ep_queue_size:3508] ep_queue_size for default: 1606392 2021-06-08 11:51:01,911 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:51:01,911 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:51:03,128 | test | DEBUG | pool-5-thread-16 | [task:record_bucket_ops:3495] Ops rate for 'default': 4085.000000 2021-06-08 11:51:03,128 | test | DEBUG | pool-5-thread-16 | [task:print_ep_queue_size:3508] ep_queue_size for default: 1606392 2021-06-08 11:51:03,131 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:51:03,131 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:51:04,341 | test | DEBUG | pool-5-thread-16 | [task:record_bucket_ops:3495] Ops rate for 'default': 4085.000000 2021-06-08 11:51:04,341 | test | DEBUG | pool-5-thread-16 | [task:print_ep_queue_size:3508] ep_queue_size for default: 1606392 2021-06-08 11:51:04,342 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:51:04,342 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:51:05,549 | test | DEBUG | pool-5-thread-16 | [task:record_bucket_ops:3495] Ops rate for 'default': 4085.000000 2021-06-08 11:51:05,551 | test | DEBUG | pool-5-thread-16 | [task:print_ep_queue_size:3508] ep_queue_size for default: 1606392 2021-06-08 11:51:05,552 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:51:05,552 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:51:06,770 | test | DEBUG | pool-5-thread-16 | [task:record_bucket_ops:3495] Ops rate for 'default': 4085.000000 2021-06-08 11:51:06,770 | test | DEBUG | pool-5-thread-16 | [task:print_ep_queue_size:3508] ep_queue_size for default: 1606392 2021-06-08 11:51:06,773 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:51:06,773 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:51:07,982 | test | DEBUG | pool-5-thread-16 | [task:record_bucket_ops:3495] Ops rate for 'default': 4085.000000 2021-06-08 11:51:07,982 | test | DEBUG | pool-5-thread-16 | [task:print_ep_queue_size:3508] ep_queue_size for default: 1606392 2021-06-08 11:51:07,983 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:51:07,983 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:51:09,197 | test | DEBUG | pool-5-thread-16 | [task:record_bucket_ops:3495] Ops rate for 'default': 4085.000000 2021-06-08 11:51:09,197 | test | DEBUG | pool-5-thread-16 | [task:print_ep_queue_size:3508] ep_queue_size for default: 1606392 2021-06-08 11:51:09,197 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:51:09,197 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:51:10,414 | test | DEBUG | pool-5-thread-16 | [task:record_bucket_ops:3495] Ops rate for 'default': 4085.000000 2021-06-08 11:51:10,415 | test | DEBUG | pool-5-thread-16 | [task:print_ep_queue_size:3508] ep_queue_size for default: 1606392 2021-06-08 11:51:10,417 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:51:10,417 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:51:11,618 | test | DEBUG | pool-5-thread-16 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:51:11,619 | test | DEBUG | pool-5-thread-16 | [task:print_ep_queue_size:3508] ep_queue_size for default: 0 2021-06-08 11:51:11,621 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:51:11,621 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:51:12,828 | test | DEBUG | pool-5-thread-16 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:51:12,828 | test | DEBUG | pool-5-thread-16 | [task:print_ep_queue_size:3508] ep_queue_size for default: 0 2021-06-08 11:51:12,831 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:51:12,831 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:51:14,078 | test | DEBUG | pool-5-thread-16 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:51:14,078 | test | DEBUG | pool-5-thread-16 | [task:print_ep_queue_size:3508] ep_queue_size for default: 0 2021-06-08 11:51:14,079 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:51:14,079 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:51:15,305 | test | DEBUG | pool-5-thread-16 | [task:record_bucket_ops:3495] Ops rate for 'default': 2044.450000 2021-06-08 11:51:15,305 | test | DEBUG | pool-5-thread-16 | [task:print_ep_queue_size:3508] ep_queue_size for default: 1564107 2021-06-08 11:51:15,308 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:51:15,308 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:51:18,032 | test | DEBUG | pool-5-thread-16 | [task:record_bucket_ops:3495] Ops rate for 'default': 2044.450000 2021-06-08 11:51:18,032 | test | DEBUG | pool-5-thread-16 | [task:print_ep_queue_size:3508] ep_queue_size for default: 1564107 2021-06-08 11:51:18,038 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:51:18,038 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:51:19,263 | test | DEBUG | pool-5-thread-16 | [task:record_bucket_ops:3495] Ops rate for 'default': 2044.450000 2021-06-08 11:51:19,263 | test | DEBUG | pool-5-thread-16 | [task:print_ep_queue_size:3508] ep_queue_size for default: 1564107 2021-06-08 11:51:19,266 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:51:19,266 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:51:20,493 | test | DEBUG | pool-5-thread-16 | [task:record_bucket_ops:3495] Ops rate for 'default': 2044.450000 2021-06-08 11:51:20,493 | test | DEBUG | pool-5-thread-16 | [task:print_ep_queue_size:3508] ep_queue_size for default: 1564107 2021-06-08 11:51:20,496 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:51:20,496 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:51:22,447 | test | DEBUG | pool-5-thread-16 | [task:record_bucket_ops:3495] Ops rate for 'default': 4143.650000 2021-06-08 11:51:22,447 | test | DEBUG | pool-5-thread-16 | [task:print_ep_queue_size:3508] ep_queue_size for default: 1564107 2021-06-08 11:51:22,450 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:51:22,450 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:51:24,101 | test | DEBUG | pool-5-thread-16 | [task:record_bucket_ops:3495] Ops rate for 'default': 5276.300000 2021-06-08 11:51:24,101 | test | DEBUG | pool-5-thread-16 | [task:print_ep_queue_size:3508] ep_queue_size for default: 1547366 2021-06-08 11:51:24,104 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:51:24,104 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:51:25,305 | test | DEBUG | pool-5-thread-16 | [task:record_bucket_ops:3495] Ops rate for 'default': 5276.300000 2021-06-08 11:51:25,306 | test | DEBUG | pool-5-thread-16 | [task:print_ep_queue_size:3508] ep_queue_size for default: 1547366 2021-06-08 11:51:25,308 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:51:25,308 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:51:26,516 | test | DEBUG | pool-5-thread-16 | [task:record_bucket_ops:3495] Ops rate for 'default': 5276.300000 2021-06-08 11:51:26,516 | test | DEBUG | pool-5-thread-16 | [task:print_ep_queue_size:3508] ep_queue_size for default: 1547366 2021-06-08 11:51:26,517 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:51:26,519 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:51:27,928 | test | DEBUG | pool-5-thread-16 | [task:record_bucket_ops:3495] Ops rate for 'default': 5276.300000 2021-06-08 11:51:27,928 | test | DEBUG | pool-5-thread-16 | [task:print_ep_queue_size:3508] ep_queue_size for default: 1547366 2021-06-08 11:51:27,933 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:51:27,933 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:51:29,517 | test | DEBUG | pool-5-thread-16 | [task:record_bucket_ops:3495] Ops rate for 'default': 5276.300000 2021-06-08 11:51:29,519 | test | DEBUG | pool-5-thread-16 | [task:print_ep_queue_size:3508] ep_queue_size for default: 1547366 2021-06-08 11:51:29,520 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:51:29,520 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:51:31,082 | test | DEBUG | pool-5-thread-16 | [task:record_bucket_ops:3495] Ops rate for 'default': 5276.300000 2021-06-08 11:51:31,082 | test | DEBUG | pool-5-thread-16 | [task:print_ep_queue_size:3508] ep_queue_size for default: 1547366 2021-06-08 11:51:31,085 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:51:31,085 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:51:32,325 | test | DEBUG | pool-5-thread-16 | [task:record_bucket_ops:3495] Ops rate for 'default': 7074.700000 2021-06-08 11:51:32,325 | test | DEBUG | pool-5-thread-16 | [task:print_ep_queue_size:3508] ep_queue_size for default: 1547366 2021-06-08 11:51:32,328 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:51:32,328 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:51:33,555 | test | DEBUG | pool-5-thread-16 | [task:record_bucket_ops:3495] Ops rate for 'default': 7074.700000 2021-06-08 11:51:33,555 | test | DEBUG | pool-5-thread-16 | [task:print_ep_queue_size:3508] ep_queue_size for default: 1547366 2021-06-08 11:51:33,558 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:51:33,558 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:51:34,769 | test | DEBUG | pool-5-thread-16 | [task:record_bucket_ops:3495] Ops rate for 'default': 7074.700000 2021-06-08 11:51:34,769 | test | DEBUG | pool-5-thread-16 | [task:print_ep_queue_size:3508] ep_queue_size for default: 1547366 2021-06-08 11:51:34,772 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:51:34,773 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:51:36,056 | test | DEBUG | pool-5-thread-16 | [task:record_bucket_ops:3495] Ops rate for 'default': 7074.700000 2021-06-08 11:51:36,058 | test | DEBUG | pool-5-thread-16 | [task:print_ep_queue_size:3508] ep_queue_size for default: 1547366 2021-06-08 11:51:36,062 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:51:36,062 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:51:37,272 | test | DEBUG | pool-5-thread-16 | [task:record_bucket_ops:3495] Ops rate for 'default': 7074.700000 2021-06-08 11:51:37,273 | test | DEBUG | pool-5-thread-16 | [task:print_ep_queue_size:3508] ep_queue_size for default: 1547366 2021-06-08 11:51:37,275 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:51:37,275 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:51:38,484 | test | DEBUG | pool-5-thread-16 | [task:record_bucket_ops:3495] Ops rate for 'default': 7074.700000 2021-06-08 11:51:38,486 | test | DEBUG | pool-5-thread-16 | [task:print_ep_queue_size:3508] ep_queue_size for default: 1547366 2021-06-08 11:51:38,486 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:51:38,486 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:51:39,690 | test | DEBUG | pool-5-thread-16 | [task:record_bucket_ops:3495] Ops rate for 'default': 7074.700000 2021-06-08 11:51:39,690 | test | DEBUG | pool-5-thread-16 | [task:print_ep_queue_size:3508] ep_queue_size for default: 1547366 2021-06-08 11:51:39,691 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:51:39,691 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:51:40,956 | test | DEBUG | pool-5-thread-16 | [task:record_bucket_ops:3495] Ops rate for 'default': 10452.500000 2021-06-08 11:51:40,956 | test | DEBUG | pool-5-thread-16 | [task:print_ep_queue_size:3508] ep_queue_size for default: 1517593 2021-06-08 11:51:40,963 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:51:40,963 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:51:42,178 | test | DEBUG | pool-5-thread-16 | [task:record_bucket_ops:3495] Ops rate for 'default': 10456.700000 2021-06-08 11:51:42,178 | test | DEBUG | pool-5-thread-16 | [task:print_ep_queue_size:3508] ep_queue_size for default: 1517593 2021-06-08 11:51:42,181 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:51:42,181 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:51:43,676 | test | DEBUG | pool-5-thread-16 | [task:record_bucket_ops:3495] Ops rate for 'default': 10456.700000 2021-06-08 11:51:43,676 | test | DEBUG | pool-5-thread-16 | [task:print_ep_queue_size:3508] ep_queue_size for default: 1517593 2021-06-08 11:51:43,678 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:51:43,680 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:51:44,898 | test | DEBUG | pool-5-thread-16 | [task:record_bucket_ops:3495] Ops rate for 'default': 5104.100000 2021-06-08 11:51:44,900 | test | DEBUG | pool-5-thread-16 | [task:print_ep_queue_size:3508] ep_queue_size for default: 1454325 2021-06-08 11:51:44,901 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:51:44,901 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:51:47,868 | test | DEBUG | pool-5-thread-16 | [task:record_bucket_ops:3495] Ops rate for 'default': 5104.100000 2021-06-08 11:51:47,868 | test | DEBUG | pool-5-thread-16 | [task:print_ep_queue_size:3508] ep_queue_size for default: 1454325 2021-06-08 11:51:47,871 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:51:47,871 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:51:49,085 | test | DEBUG | pool-5-thread-16 | [task:record_bucket_ops:3495] Ops rate for 'default': 5104.100000 2021-06-08 11:51:49,085 | test | DEBUG | pool-5-thread-16 | [task:print_ep_queue_size:3508] ep_queue_size for default: 1454325 2021-06-08 11:51:49,088 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:51:49,088 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:51:50,394 | test | DEBUG | pool-5-thread-16 | [task:record_bucket_ops:3495] Ops rate for 'default': 5104.100000 2021-06-08 11:51:50,394 | test | DEBUG | pool-5-thread-16 | [task:print_ep_queue_size:3508] ep_queue_size for default: 1454325 2021-06-08 11:51:50,398 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:51:50,398 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:51:51,986 | test | DEBUG | pool-5-thread-16 | [task:record_bucket_ops:3495] Ops rate for 'default': 4451.700000 2021-06-08 11:51:51,986 | test | DEBUG | pool-5-thread-16 | [task:print_ep_queue_size:3508] ep_queue_size for default: 1454325 2021-06-08 11:51:51,990 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:51:51,990 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:51:53,198 | test | DEBUG | pool-5-thread-16 | [task:record_bucket_ops:3495] Ops rate for 'default': 4451.700000 2021-06-08 11:51:53,200 | test | DEBUG | pool-5-thread-16 | [task:print_ep_queue_size:3508] ep_queue_size for default: 1454325 2021-06-08 11:51:53,200 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:51:53,200 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:51:54,410 | test | DEBUG | pool-5-thread-16 | [task:record_bucket_ops:3495] Ops rate for 'default': 4451.700000 2021-06-08 11:51:54,410 | test | DEBUG | pool-5-thread-16 | [task:print_ep_queue_size:3508] ep_queue_size for default: 1454325 2021-06-08 11:51:54,411 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:51:54,411 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:51:55,625 | test | DEBUG | pool-5-thread-16 | [task:record_bucket_ops:3495] Ops rate for 'default': 4451.700000 2021-06-08 11:51:55,625 | test | DEBUG | pool-5-thread-16 | [task:print_ep_queue_size:3508] ep_queue_size for default: 1454325 2021-06-08 11:51:55,627 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:51:55,627 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:51:56,832 | test | DEBUG | pool-5-thread-16 | [task:record_bucket_ops:3495] Ops rate for 'default': 4451.700000 2021-06-08 11:51:56,832 | test | DEBUG | pool-5-thread-16 | [task:print_ep_queue_size:3508] ep_queue_size for default: 1454325 2021-06-08 11:51:56,835 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:51:56,835 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:51:58,197 | test | DEBUG | pool-5-thread-16 | [task:record_bucket_ops:3495] Ops rate for 'default': 6287.100000 2021-06-08 11:51:58,197 | test | DEBUG | pool-5-thread-16 | [task:print_ep_queue_size:3508] ep_queue_size for default: 1375527 2021-06-08 11:51:58,200 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:51:58,200 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:51:59,453 | test | DEBUG | pool-5-thread-16 | [task:record_bucket_ops:3495] Ops rate for 'default': 6287.100000 2021-06-08 11:51:59,453 | test | DEBUG | pool-5-thread-16 | [task:print_ep_queue_size:3508] ep_queue_size for default: 1375527 2021-06-08 11:51:59,457 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:51:59,457 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:52:00,674 | test | DEBUG | pool-5-thread-16 | [task:record_bucket_ops:3495] Ops rate for 'default': 6287.100000 2021-06-08 11:52:00,674 | test | DEBUG | pool-5-thread-16 | [task:print_ep_queue_size:3508] ep_queue_size for default: 1375527 2021-06-08 11:52:00,677 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:52:00,677 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:52:02,081 | test | DEBUG | pool-5-thread-16 | [task:record_bucket_ops:3495] Ops rate for 'default': 5052.594042 2021-06-08 11:52:02,081 | test | DEBUG | pool-5-thread-16 | [task:print_ep_queue_size:3508] ep_queue_size for default: 1375527 2021-06-08 11:52:02,084 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:52:02,084 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:52:03,917 | test | DEBUG | pool-5-thread-16 | [task:record_bucket_ops:3495] Ops rate for 'default': 5052.594042 2021-06-08 11:52:03,917 | test | DEBUG | pool-5-thread-16 | [task:print_ep_queue_size:3508] ep_queue_size for default: 1375527 2021-06-08 11:52:03,921 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:52:03,921 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:52:05,388 | test | DEBUG | pool-5-thread-16 | [task:record_bucket_ops:3495] Ops rate for 'default': 5052.594042 2021-06-08 11:52:05,388 | test | DEBUG | pool-5-thread-16 | [task:print_ep_queue_size:3508] ep_queue_size for default: 1375527 2021-06-08 11:52:05,391 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:52:05,391 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:52:06,602 | test | DEBUG | pool-5-thread-16 | [task:record_bucket_ops:3495] Ops rate for 'default': 5052.594042 2021-06-08 11:52:06,602 | test | DEBUG | pool-5-thread-16 | [task:print_ep_queue_size:3508] ep_queue_size for default: 1375527 2021-06-08 11:52:06,605 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:52:06,605 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:52:07,878 | test | DEBUG | pool-5-thread-16 | [task:record_bucket_ops:3495] Ops rate for 'default': 5052.594042 2021-06-08 11:52:07,878 | test | DEBUG | pool-5-thread-16 | [task:print_ep_queue_size:3508] ep_queue_size for default: 1375527 2021-06-08 11:52:07,881 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:52:07,881 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None Could not publish Event because the queue is full. RequestNotRetriedEvent{severity=INFO, category=com.couchbase.request, duration=PT0S, createdAt=9552764044943117, description=Request InsertRequest not retried per RetryStrategy (Reason: KV_TEMPORARY_FAILURE), context=RequestContext{completed=false, coreId=0x607200bd0000003d, idempotent=false, lastChannelId=607200BD0000003D/00000000CDA4202B, lastDispatchedFrom=172.23.107.117:59256, lastDispatchedTo=172.23.106.156:11210, requestId=46360973, requestType=InsertRequest, retried=0, service={bucket=default, collection=_default, documentId=test_docs-9447228, opaque=0x2c5e708, scope=_default, type=kv}, timeoutMs=60000, timings={dispatchMicros=9259, encodingMicros=6, totalDispatchMicros=9259, totalServerMicros=0, serverMicros=0}}, cause=null} Could not publish Event because the queue is full. RequestNotRetriedEvent{severity=INFO, category=com.couchbase.request, duration=PT0S, createdAt=9552764045584877, description=Request InsertRequest not retried per RetryStrategy (Reason: ENDPOINT_NOT_AVAILABLE), context=RequestContext{completed=false, coreId=0x607200bd0000003d, idempotent=false, lastDispatchedTo=172.23.106.125, requestId=46362829, requestType=InsertRequest, retried=0, service={bucket=default, collection=_default, documentId=test_docs-9099126, opaque=0x2c5ee48, scope=_default, type=kv}, timeoutMs=60000, timings={encodingMicros=6}}, cause=null} Could not publish Event because the queue is full. RequestNotRetriedEvent{severity=INFO, category=com.couchbase.request, duration=PT0S, createdAt=9552764045290442, description=Request InsertRequest not retried per RetryStrategy (Reason: KV_TEMPORARY_FAILURE), context=RequestContext{completed=false, coreId=0x607200bd0000003d, idempotent=false, lastChannelId=607200BD0000003D/000000003870A282, lastDispatchedFrom=172.23.107.117:52372, lastDispatchedTo=172.23.106.155:11210, requestId=46362697, requestType=InsertRequest, retried=0, service={bucket=default, collection=_default, documentId=test_docs-9447449, opaque=0x2c5edc4, scope=_default, type=kv}, timeoutMs=60000, timings={dispatchMicros=574, encodingMicros=7, totalDispatchMicros=574, totalServerMicros=0, serverMicros=0}}, cause=null} Could not publish Event because the queue is full. RequestNotRetriedEvent{severity=INFO, category=com.couchbase.request, duration=PT0S, createdAt=9552764045293099, description=Request InsertRequest not retried per RetryStrategy (Reason: ENDPOINT_NOT_AVAILABLE), context=RequestContext{completed=false, coreId=0x607200bd0000003d, idempotent=false, lastDispatchedTo=172.23.106.125, requestId=46362791, requestType=InsertRequest, retried=0, service={bucket=default, collection=_default, documentId=test_docs-9295852, opaque=0x2c5ee22, scope=_default, type=kv}, timeoutMs=60000, timings={encodingMicros=7}}, cause=null} Could not publish Event because the queue is full. RequestNotRetriedEvent{severity=INFO, category=com.couchbase.request, duration=PT0S, createdAt=9552764044963344, description=Request InsertRequest not retried per RetryStrategy (Reason: ENDPOINT_NOT_AVAILABLE), context=RequestContext{completed=false, coreId=0x607200bd0000003d, idempotent=false, lastDispatchedTo=172.23.106.122, requestId=46362761, requestType=InsertRequest, retried=0, service={bucket=default, collection=_default, documentId=test_docs-9597747, opaque=0x2c5ee04, scope=_default, type=kv}, timeoutMs=60000, timings={encodingMicros=6}}, cause=null} Could not publish Event because the queue is full. RequestNotRetriedEvent{severity=INFO, category=com.couchbase.request, duration=PT0S, createdAt=9552764046141411, description=Request InsertRequest not retried per RetryStrategy (Reason: ENDPOINT_NOT_AVAILABLE), context=RequestContext{completed=false, coreId=0x607200bd0000003d, idempotent=false, lastDispatchedTo=172.23.106.122, requestId=46362880, requestType=InsertRequest, retried=0, service={bucket=default, collection=_default, documentId=test_docs-9099136, opaque=0x2c5ee7b, scope=_default, type=kv}, timeoutMs=60000, timings={encodingMicros=12}}, cause=null} Could not publish Event because the queue is full. RequestNotRetriedEvent{severity=INFO, category=com.couchbase.request, duration=PT0S, createdAt=9552764046101222, description=Request InsertRequest not retried per RetryStrategy (Reason: KV_TEMPORARY_FAILURE), context=RequestContext{completed=false, coreId=0x607200bd0000003d, idempotent=false, lastChannelId=607200BD0000003D/00000000D244D464, lastDispatchedFrom=172.23.107.117:52322, lastDispatchedTo=172.23.106.155:11210, requestId=46361936, requestType=InsertRequest, retried=0, service={bucket=default, collection=_default, documentId=test_docs-9295730, opaque=0x2c5eacb, scope=_default, type=kv}, timeoutMs=60000, timings={dispatchMicros=5555, encodingMicros=7, totalDispatchMicros=5555, totalServerMicros=0, serverMicros=0}}, cause=null} Could not publish Event because the queue is full. RequestNotRetriedEvent{severity=INFO, category=com.couchbase.request, duration=PT0S, createdAt=9552764046098942, description=Request InsertRequest not retried per RetryStrategy (Reason: KV_TEMPORARY_FAILURE), context=RequestContext{completed=false, coreId=0x607200bd0000003d, idempotent=false, lastChannelId=607200BD0000003D/000000000024E57C, lastDispatchedFrom=172.23.107.117:59264, lastDispatchedTo=172.23.106.156:11210, requestId=46362835, requestType=InsertRequest, retried=0, service={bucket=default, collection=_default, documentId=test_docs-9494713, opaque=0x2c5ee4e, scope=_default, type=kv}, timeoutMs=60000, timings={dispatchMicros=409, encodingMicros=5, totalDispatchMicros=409, totalServerMicros=0, serverMicros=0}}, cause=null} Could not publish Event because the queue is full. RequestNotRetriedEvent{severity=INFO, category=com.couchbase.request, duration=PT0S, createdAt=9552764046412697, description=Request InsertRequest not retried per RetryStrategy (Reason: ENDPOINT_NOT_AVAILABLE), context=RequestContext{completed=false, coreId=0x607200bd0000003d, idempotent=false, lastDispatchedTo=172.23.106.125, requestId=46362914, requestType=InsertRequest, retried=0, service={bucket=default, collection=_default, documentId=test_docs-9494733, opaque=0x2c5ee9d, scope=_default, type=kv}, timeoutMs=60000, timings={encodingMicros=6}}, cause=null} Could not publish Event because the queue is full. EndpointConnectionFailedEvent{severity=WARN, category=com.couchbase.endpoint, duration=PT0.000624443S, createdAt=9552764046417964, description=Connect attempt 7 failed because of AnnotatedConnectException: finishConnect(..) failed: Connection refused: /172.23.106.125:11210, context=EndpointContext{channelId=607200BD0000003C/000000003DC6B0AD, circuitBreaker=DISABLED, coreId=0x607200bd0000003c, local=172.23.107.117:52902, remote=172.23.106.125:11210, type=KV}, cause=com.couchbase.client.core.deps.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: /172.23.106.125:11210} Could not publish Event because the queue is full. RequestNotRetriedEvent{severity=INFO, category=com.couchbase.request, duration=PT0S, createdAt=9552764046527285, description=Request InsertRequest not retried per RetryStrategy (Reason: KV_TEMPORARY_FAILURE), context=RequestContext{completed=false, coreId=0x607200bd0000003d, idempotent=false, lastChannelId=607200BD0000003D/00000000A6B51CE7, lastDispatchedFrom=172.23.107.117:52186, lastDispatchedTo=172.23.106.155:11210, requestId=46358181, requestType=InsertRequest, retried=0, service={bucket=default, collection=_default, documentId=test_docs-9793708, opaque=0x2c5dc20, scope=_default, type=kv}, timeoutMs=60000, timings={dispatchMicros=369129, encodingMicros=6, totalDispatchMicros=369129, totalServerMicros=0, serverMicros=0}}, cause=null} Could not publish Event because the queue is full. RequestNotRetriedEvent{severity=INFO, category=com.couchbase.request, duration=PT0S, createdAt=9552764046733064, description=Request InsertRequest not retried per RetryStrategy (Reason: ENDPOINT_NOT_AVAILABLE), context=RequestContext{completed=false, coreId=0x607200bd0000003d, idempotent=false, lastDispatchedTo=172.23.106.122, requestId=46362961, requestType=InsertRequest, retried=0, service={bucket=default, collection=_default, documentId=test_docs-9947155, opaque=0x2c5eecc, scope=_default, type=kv}, timeoutMs=60000, timings={encodingMicros=6}}, cause=null} Could not publish Event because the queue is full. RequestNotRetriedEvent{severity=INFO, category=com.couchbase.request, duration=PT0S, createdAt=9552764046814898, description=Request InsertRequest not retried per RetryStrategy (Reason: ENDPOINT_NOT_AVAILABLE), context=RequestContext{completed=false, coreId=0x607200bd0000003d, idempotent=false, lastDispatchedTo=172.23.106.125, requestId=46362971, requestType=InsertRequest, retried=0, service={bucket=default, collection=_default, documentId=test_docs-9597758, opaque=0x2c5eed6, scope=_default, type=kv}, timeoutMs=60000, timings={encodingMicros=6}}, cause=null} Could not publish Event because the queue is full. RequestNotRetriedEvent{severity=INFO, category=com.couchbase.request, duration=PT0S, createdAt=9552764045622641, description=Request InsertRequest not retried per RetryStrategy (Reason: KV_TEMPORARY_FAILURE), context=RequestContext{completed=false, coreId=0x607200bd0000003d, idempotent=false, lastChannelId=607200BD0000003D/00000000CDA4202B, lastDispatchedFrom=172.23.107.117:59256, lastDispatchedTo=172.23.106.156:11210, requestId=46361091, requestType=InsertRequest, retried=0, service={bucket=default, collection=_default, documentId=test_docs-9098879, opaque=0x2c5e77e, scope=_default, type=kv}, timeoutMs=60000, timings={dispatchMicros=9799, encodingMicros=6, totalDispatchMicros=9799, totalServerMicros=0, serverMicros=0}}, cause=null} Could not publish Event because the queue is full. RequestNotRetriedEvent{severity=INFO, category=com.couchbase.request, duration=PT0S, createdAt=9552764045549058, description=Request InsertRequest not retried per RetryStrategy (Reason: KV_TEMPORARY_FAILURE), context=RequestContext{completed=false, coreId=0x607200bd0000003d, idempotent=false, lastChannelId=607200BD0000003D/00000000504BDBA6, lastDispatchedFrom=172.23.107.117:52014, lastDispatchedTo=172.23.106.155:11210, requestId=46352991, requestType=InsertRequest, retried=0, service={bucket=default, collection=_default, documentId=test_docs-9997602, opaque=0x2c5c7da, scope=_default, type=kv}, timeoutMs=60000, timings={dispatchMicros=391640, encodingMicros=7, totalDispatchMicros=391640, totalServerMicros=0, serverMicros=0}}, cause=null} Could not publish Event because the queue is full. RequestNotRetriedEvent{severity=INFO, category=com.couchbase.request, duration=PT0S, createdAt=9552764046816078, description=Request InsertRequest not retried per RetryStrategy (Reason: KV_TEMPORARY_FAILURE), context=RequestContext{completed=false, coreId=0x607200bd0000003d, idempotent=false, lastChannelId=607200BD0000003D/00000000B21A2DEC, lastDispatchedFrom=172.23.107.117:59272, lastDispatchedTo=172.23.106.156:11210, requestId=46362868, requestType=InsertRequest, retried=0, service={bucket=default, collection=_default, documentId=test_docs-9494722, opaque=0x2c5ee6f, scope=_default, type=kv}, timeoutMs=60000, timings={dispatchMicros=817, encodingMicros=6, totalDispatchMicros=817, totalServerMicros=0, serverMicros=0}}, cause=null} Could not publish Event because the queue is full. RequestNotRetriedEvent{severity=INFO, category=com.couchbase.request, duration=PT0S, createdAt=9552764047197485, description=Request InsertRequest not retried per RetryStrategy (Reason: KV_TEMPORARY_FAILURE), context=RequestContext{completed=false, coreId=0x607200bd0000003d, idempotent=false, lastChannelId=607200BD0000003D/00000000829B3C86, lastDispatchedFrom=172.23.107.117:52074, lastDispatchedTo=172.23.106.155:11210, requestId=46350605, requestType=InsertRequest, retried=0, service={bucket=default, collection=_default, documentId=test_docs-9745455, opaque=0x2c5be88, scope=_default, type=kv}, timeoutMs=60000, timings={dispatchMicros=406337, encodingMicros=6, totalDispatchMicros=406337, totalServerMicros=0, serverMicros=0}}, cause=null} Could not publish Event because the queue is full. RequestNotRetriedEvent{severity=INFO, category=com.couchbase.request, duration=PT0S, createdAt=9552764047256048, description=Request InsertRequest not retried per RetryStrategy (Reason: KV_TEMPORARY_FAILURE), context=RequestContext{completed=false, coreId=0x607200bd0000003d, idempotent=false, lastChannelId=607200BD0000003D/00000000B21A2DEC, lastDispatchedFrom=172.23.107.117:59272, lastDispatchedTo=172.23.106.156:11210, requestId=46362889, requestType=InsertRequest, retried=0, service={bucket=default, collection=_default, documentId=test_docs-9494728, opaque=0x2c5ee84, scope=_default, type=kv}, timeoutMs=60000, timings={dispatchMicros=917, encodingMicros=5, totalDispatchMicros=917, totalServerMicros=0, serverMicros=0}}, cause=null} Could not publish Event because the queue is full. RequestNotRetriedEvent{severity=INFO, category=com.couchbase.request, duration=PT0S, createdAt=9552764047275675, description=Request InsertRequest not retried per RetryStrategy (Reason: KV_TEMPORARY_FAILURE), context=RequestContext{completed=false, coreId=0x607200bd0000003d, idempotent=false, lastChannelId=607200BD0000003D/00000000CDA4202B, lastDispatchedFrom=172.23.107.117:59256, lastDispatchedTo=172.23.106.156:11210, requestId=46361200, requestType=InsertRequest, retried=0, service={bucket=default, collection=_default, documentId=test_docs-9098894, opaque=0x2c5e7eb, scope=_default, type=kv}, timeoutMs=60000, timings={dispatchMicros=8958, encodingMicros=5, totalDispatchMicros=8958, totalServerMicros=0, serverMicros=0}}, cause=null} Could not publish Event because the queue is full. RequestNotRetriedEvent{severity=INFO, category=com.couchbase.request, duration=PT0S, createdAt=9552764047305986, description=Request InsertRequest not retried per RetryStrategy (Reason: ENDPOINT_NOT_AVAILABLE), context=RequestContext{completed=false, coreId=0x607200bd0000003d, idempotent=false, lastDispatchedTo=172.23.106.122, requestId=46363039, requestType=InsertRequest, retried=0, service={bucket=default, collection=_default, documentId=test_docs-9494752, opaque=0x2c5ef1a, scope=_default, type=kv}, timeoutMs=60000, timings={encodingMicros=6}}, cause=null} Could not publish Event because the queue is full. RequestNotRetriedEvent{severity=INFO, category=com.couchbase.request, duration=PT0S, createdAt=9552764047375895, description=Request InsertRequest not retried per RetryStrategy (Reason: KV_TEMPORARY_FAILURE), context=RequestContext{completed=false, coreId=0x607200bd0000003d, idempotent=false, lastChannelId=607200BD0000003D/00000000CDA4202B, lastDispatchedFrom=172.23.107.117:59256, lastDispatchedTo=172.23.106.156:11210, requestId=46361214, requestType=InsertRequest, retried=0, service={bucket=default, collection=_default, documentId=test_docs-9447256, opaque=0x2c5e7f9, scope=_default, type=kv}, timeoutMs=60000, timings={dispatchMicros=9017, encodingMicros=8, totalDispatchMicros=9017, totalServerMicros=0, serverMicros=0}}, cause=null} Could not publish Event because the queue is full. RequestNotRetriedEvent{severity=INFO, category=com.couchbase.request, duration=PT0S, createdAt=9552764047569042, description=Request InsertRequest not retried per RetryStrategy (Reason: ENDPOINT_NOT_AVAILABLE), context=RequestContext{completed=false, coreId=0x607200bd0000003d, idempotent=false, lastDispatchedTo=172.23.106.125, requestId=46363063, requestType=InsertRequest, retried=0, service={bucket=default, collection=_default, documentId=test_docs-9597775, opaque=0x2c5ef32, scope=_default, type=kv}, timeoutMs=60000, timings={encodingMicros=6}}, cause=null} Could not publish Event because the queue is full. RequestNotRetriedEvent{severity=INFO, category=com.couchbase.request, duration=PT0S, createdAt=9552764047213826, description=Request InsertRequest not retried per RetryStrategy (Reason: ENDPOINT_NOT_AVAILABLE), context=RequestContext{completed=false, coreId=0x607200bd0000003d, idempotent=false, lastDispatchedTo=172.23.106.122, requestId=46363026, requestType=InsertRequest, retried=0, service={bucket=default, collection=_default, documentId=test_docs-9947164, opaque=0x2c5ef0d, scope=_default, type=kv}, timeoutMs=60000, timings={encodingMicros=6}}, cause=null} Could not publish Event because the queue is full. RequestNotRetriedEvent{severity=INFO, category=com.couchbase.request, duration=PT0S, createdAt=9552764047568010, description=Request InsertRequest not retried per RetryStrategy (Reason: KV_TEMPORARY_FAILURE), context=RequestContext{completed=false, coreId=0x607200bd0000003d, idempotent=false, lastChannelId=607200BD0000003D/00000000E6C08B2D, lastDispatchedFrom=172.23.107.117:52146, lastDispatchedTo=172.23.106.155:11210, requestId=46362041, requestType=InsertRequest, retried=0, service={bucket=default, collection=_default, documentId=test_docs-9295744, opaque=0x2c5eb34, scope=_default, type=kv}, timeoutMs=60000, timings={dispatchMicros=6790, encodingMicros=6, totalDispatchMicros=6790, totalServerMicros=0, serverMicros=0}}, cause=null} Could not publish Event because the queue is full. RequestNotRetriedEvent{severity=INFO, category=com.couchbase.request, duration=PT0S, createdAt=9552764047565789, description=Request InsertRequest not retried per RetryStrategy (Reason: KV_TEMPORARY_FAILURE), context=RequestContext{completed=false, coreId=0x607200bd0000003d, idempotent=false, lastChannelId=607200BD0000003D/00000000829B3C86, lastDispatchedFrom=172.23.107.117:52074, lastDispatchedTo=172.23.106.155:11210, requestId=46351272, requestType=InsertRequest, retried=0, service={bucket=default, collection=_default, documentId=test_docs-9946184, opaque=0x2c5c123, scope=_default, type=kv}, timeoutMs=60000, timings={dispatchMicros=383002, encodingMicros=6, totalDispatchMicros=383002, totalServerMicros=0, serverMicros=0}}, cause=null} Could not publish Event because the queue is full. RequestNotRetriedEvent{severity=INFO, category=com.couchbase.request, duration=PT0S, createdAt=9552764047310804, description=Request InsertRequest not retried per RetryStrategy (Reason: ENDPOINT_NOT_AVAILABLE), context=RequestContext{completed=false, coreId=0x607200bd0000003d, idempotent=false, lastDispatchedTo=172.23.106.122, requestId=46363040, requestType=InsertRequest, retried=0, service={bucket=default, collection=_default, documentId=test_docs-9099160, opaque=0x2c5ef1b, scope=_default, type=kv}, timeoutMs=60000, timings={encodingMicros=7}}, cause=null} Could not publish Event because the queue is full. RequestNotRetriedEvent{severity=INFO, category=com.couchbase.request, duration=PT0S, createdAt=9552764048573650, description=Request InsertRequest not retried per RetryStrategy (Reason: ENDPOINT_NOT_AVAILABLE), context=RequestContext{completed=false, coreId=0x607200bd0000003d, idempotent=false, lastDispatchedTo=172.23.106.122, requestId=46363154, requestType=InsertRequest, retried=0, service={bucket=default, collection=_default, documentId=test_docs-9099165, opaque=0x2c5ef8d, scope=_default, type=kv}, timeoutMs=60000, timings={encodingMicros=6}}, cause=null} Could not publish Event because the queue is full. RequestNotRetriedEvent{severity=INFO, category=com.couchbase.request, duration=PT0S, createdAt=9552764048672871, description=Request InsertRequest not retried per RetryStrategy (Reason: KV_TEMPORARY_FAILURE), context=RequestContext{completed=false, coreId=0x607200bd0000003d, idempotent=false, lastChannelId=607200BD0000003D/000000003870A282, lastDispatchedFrom=172.23.107.117:52372, lastDispatchedTo=172.23.106.155:11210, requestId=46363083, requestType=InsertRequest, retried=0, service={bucket=default, collection=_default, documentId=test_docs-9494764, opaque=0x2c5ef46, scope=_default, type=kv}, timeoutMs=60000, timings={dispatchMicros=684, encodingMicros=6, totalDispatchMicros=684, totalServerMicros=0, serverMicros=0}}, cause=null} Could not publish Event because the queue is full. EndpointStateChangedEvent{severity=DEBUG, category=com.couchbase.endpoint, duration=PT0S, createdAt=9552764048654498, description=Endpoint changed state from DISCONNECTED to CONNECTING, context=EndpointContext{channelId=607200BD0000002C/000000001DACF033, circuitBreaker=DISABLED, coreId=0x607200bd0000002c, local=172.23.107.117:52432, remote=172.23.106.125:11210, type=KV}, cause=null} Could not publish Event because the queue is full. RequestNotRetriedEvent{severity=INFO, category=com.couchbase.request, duration=PT0S, createdAt=9552764048829849, description=Request InsertRequest not retried per RetryStrategy (Reason: KV_TEMPORARY_FAILURE), context=RequestContext{completed=false, coreId=0x607200bd0000003d, idempotent=false, lastChannelId=607200BD0000003D/00000000504BDBA6, lastDispatchedFrom=172.23.107.117:52014, lastDispatchedTo=172.23.106.155:11210, requestId=46356218, requestType=InsertRequest, retried=0, service={bucket=default, collection=_default, documentId=test_docs-9045364, opaque=0x2c5d475, scope=_default, type=kv}, timeoutMs=60000, timings={dispatchMicros=376660, encodingMicros=6, totalDispatchMicros=376660, totalServerMicros=0, serverMicros=0}}, cause=null} Could not publish Event because the queue is full. RequestNotRetriedEvent{severity=INFO, category=com.couchbase.request, duration=PT0S, createdAt=9552764048971010, description=Request InsertRequest not retried per RetryStrategy (Reason: ENDPOINT_NOT_AVAILABLE), context=RequestContext{completed=false, coreId=0x607200bd0000003d, idempotent=false, lastDispatchedTo=172.23.106.122, requestId=46363204, requestType=InsertRequest, retried=0, service={bucket=default, collection=_default, documentId=test_docs-9494792, opaque=0x2c5efbf, scope=_default, type=kv}, timeoutMs=60000, timings={encodingMicros=7}}, cause=null} Could not publish Event because the queue is full. RequestNotRetriedEvent{severity=INFO, category=com.couchbase.request, duration=PT0S, createdAt=9552764049055606, description=Request InsertRequest not retried per RetryStrategy (Reason: ENDPOINT_NOT_AVAILABLE), context=RequestContext{completed=false, coreId=0x607200bd0000003d, idempotent=false, lastDispatchedTo=172.23.106.125, requestId=46363207, requestType=InsertRequest, retried=0, service={bucket=default, collection=_default, documentId=test_docs-9099175, opaque=0x2c5efc2, scope=_default, type=kv}, timeoutMs=60000, timings={encodingMicros=54}}, cause=null} Could not publish Event because the queue is full. RequestNotRetriedEvent{severity=INFO, category=com.couchbase.request, duration=PT0S, createdAt=9552764049057387, description=Request InsertRequest not retried per RetryStrategy (Reason: KV_TEMPORARY_FAILURE), context=RequestContext{completed=false, coreId=0x607200bd0000003d, idempotent=false, lastChannelId=607200BD0000003D/000000002B282457, lastDispatchedFrom=172.23.107.117:59270, lastDispatchedTo=172.23.106.156:11210, requestId=46363138, requestType=InsertRequest, retried=0, service={bucket=default, collection=_default, documentId=test_docs-9099161, opaque=0x2c5ef7d, scope=_default, type=kv}, timeoutMs=60000, timings={dispatchMicros=549, encodingMicros=13, totalDispatchMicros=549, totalServerMicros=0, serverMicros=0}}, cause=null} Could not publish Event because the queue is full. EndpointStateChangedEvent{severity=DEBUG, category=com.couchbase.endpoint, duration=PT0S, createdAt=9552764049111747, description=Endpoint changed state from DISCONNECTED to CONNECTING, context=EndpointContext{channelId=607200BD00000036/0000000073712F3C, circuitBreaker=DISABLED, coreId=0x607200bd00000036, local=172.23.107.117:60150, remote=172.23.106.122:11210, type=KV}, cause=null} Could not publish Event because the queue is full. RequestNotRetriedEvent{severity=INFO, category=com.couchbase.request, duration=PT0S, createdAt=9552764048573022, description=Request InsertRequest not retried per RetryStrategy (Reason: KV_TEMPORARY_FAILURE), context=RequestContext{completed=false, coreId=0x607200bd0000003d, idempotent=false, lastChannelId=607200BD0000003D/00000000829B3C86, lastDispatchedFrom=172.23.107.117:52074, lastDispatchedTo=172.23.106.155:11210, requestId=46351712, requestType=InsertRequest, retried=0, service={bucket=default, collection=_default, documentId=test_docs-9793052, opaque=0x2c5c2db, scope=_default, type=kv}, timeoutMs=60000, timings={dispatchMicros=383342, encodingMicros=6, totalDispatchMicros=383342, totalServerMicros=0, serverMicros=0}}, cause=null} Could not publish Event because the queue is full. RequestNotRetriedEvent{severity=INFO, category=com.couchbase.request, duration=PT0S, createdAt=9552764047287995, description=Request InsertRequest not retried per RetryStrategy (Reason: ENDPOINT_NOT_AVAILABLE), context=RequestContext{completed=false, coreId=0x607200bd0000003d, idempotent=false, lastDispatchedTo=172.23.106.122, requestId=46363035, requestType=InsertRequest, retried=0, service={bucket=default, collection=_default, documentId=test_docs-9447507, opaque=0x2c5ef16, scope=_default, type=kv}, timeoutMs=60000, timings={encodingMicros=6}}, cause=null} Could not publish Event because the queue is full. RequestNotRetriedEvent{severity=INFO, category=com.couchbase.request, duration=PT0S, createdAt=9552764049178149, description=Request InsertRequest not retried per RetryStrategy (Reason: ENDPOINT_NOT_AVAILABLE), context=RequestContext{completed=false, coreId=0x607200bd0000003d, idempotent=false, lastDispatchedTo=172.23.106.122, requestId=46363222, requestType=InsertRequest, retried=0, service={bucket=default, collection=_default, documentId=test_docs-9099176, opaque=0x2c5efd1, scope=_default, type=kv}, timeoutMs=60000, timings={encodingMicros=7}}, cause=null} Could not publish Event because the queue is full. EndpointStateChangedEvent{severity=DEBUG, category=com.couchbase.endpoint, duration=PT0S, createdAt=9552764049183956, description=Endpoint changed state from DISCONNECTED to CONNECTING, context=EndpointContext{channelId=607200BD00000035/00000000028187B4, circuitBreaker=DISABLED, coreId=0x607200bd00000035, local=172.23.107.117:59828, remote=172.23.106.122:11210, type=KV}, cause=null} Could not publish Event because the queue is full. RequestNotRetriedEvent{severity=INFO, category=com.couchbase.request, duration=PT0S, createdAt=9552764049174557, description=Request InsertRequest not retried per RetryStrategy (Reason: ENDPOINT_NOT_AVAILABLE), context=RequestContext{completed=false, coreId=0x607200bd0000003d, idempotent=false, lastDispatchedTo=172.23.106.125, requestId=46363221, requestType=InsertRequest, retried=0, service={bucket=default, collection=_default, documentId=test_docs-9597813, opaque=0x2c5efd0, scope=_default, type=kv}, timeoutMs=60000, timings={encodingMicros=6}}, cause=null} Could not publish Event because the queue is full. EndpointStateChangedEvent{severity=DEBUG, category=com.couchbase.endpoint, duration=PT0S, createdAt=9552764049532274, description=Endpoint changed state from DISCONNECTED to CONNECTING, context=EndpointContext{channelId=607200BD0000003D/0000000050C14013, circuitBreaker=DISABLED, coreId=0x607200bd0000003d, local=172.23.107.117:60144, remote=172.23.106.122:11210, type=KV}, cause=null} Could not publish Event because the queue is full. RequestNotRetriedEvent{severity=INFO, category=com.couchbase.request, duration=PT0S, createdAt=9552764049637997, description=Request InsertRequest not retried per RetryStrategy (Reason: KV_TEMPORARY_FAILURE), context=RequestContext{completed=false, coreId=0x607200bd0000003d, idempotent=false, lastChannelId=607200BD0000003D/000000000024E57C, lastDispatchedFrom=172.23.107.117:59264, lastDispatchedTo=172.23.106.156:11210, requestId=46363179, requestType=InsertRequest, retried=0, service={bucket=default, collection=_default, documentId=test_docs-9295913, opaque=0x2c5efa6, scope=_default, type=kv}, timeoutMs=60000, timings={dispatchMicros=812, encodingMicros=6, totalDispatchMicros=812, totalServerMicros=0, serverMicros=0}}, cause=null} Could not publish Event because the queue is full. RequestNotRetriedEvent{severity=INFO, category=com.couchbase.request, duration=PT0S, createdAt=9552764049122255, description=Request InsertRequest not retried per RetryStrategy (Reason: ENDPOINT_NOT_AVAILABLE), context=RequestContext{completed=false, coreId=0x607200bd0000003d, idempotent=false, lastDispatchedTo=172.23.106.125, requestId=46363213, requestType=InsertRequest, retried=0, service={bucket=default, collection=_default, documentId=test_docs-9947185, opaque=0x2c5efc8, scope=_default, type=kv}, timeoutMs=60000, timings={encodingMicros=7}}, cause=null} Could not publish Event because the queue is full. EndpointStateChangedEvent{severity=DEBUG, category=com.couchbase.endpoint, duration=PT0S, createdAt=9552764048821403, description=Endpoint changed state from DISCONNECTED to CONNECTING, context=EndpointContext{channelId=607200BD0000002A/000000000BF39841, circuitBreaker=DISABLED, coreId=0x607200bd0000002a, local=172.23.107.117:59766, remote=172.23.106.122:11210, type=KV}, cause=null} Could not publish Event because the queue is full. RequestNotRetriedEvent{severity=INFO, category=com.couchbase.request, duration=PT0S, createdAt=9552764048588856, description=Request InsertRequest not retried per RetryStrategy (Reason: KV_TEMPORARY_FAILURE), context=RequestContext{completed=false, coreId=0x607200bd0000003d, idempotent=false, lastChannelId=607200BD0000003D/00000000CE353CB0, lastDispatchedFrom=172.23.107.117:59244, lastDispatchedTo=172.23.106.156:11210, requestId=46362143, requestType=InsertRequest, retried=0, service={bucket=default, collection=_default, documentId=test_docs-9643168, opaque=0x2c5eb9a, scope=_default, type=kv}, timeoutMs=60000, timings={dispatchMicros=4289, encodingMicros=7, totalDispatchMicros=4289, totalServerMicros=0, serverMicros=0}}, cause=null} Could not publish Event because the queue is full. RequestNotRetriedEvent{severity=INFO, category=com.couchbase.request, duration=PT0S, createdAt=9552764049635868, description=Request InsertRequest not retried per RetryStrategy (Reason: ENDPOINT_NOT_AVAILABLE), context=RequestContext{completed=false, coreId=0x607200bd0000003d, idempotent=false, lastDispatchedTo=172.23.106.125, requestId=46363261, requestType=InsertRequest, retried=0, service={bucket=default, collection=_default, documentId=test_docs-9099181, opaque=0x2c5eff8, scope=_default, type=kv}, timeoutMs=60000, timings={encodingMicros=5}}, cause=null} Could not publish Event because the queue is full. RequestNotRetriedEvent{severity=INFO, category=com.couchbase.request, duration=PT0S, createdAt=9552764049619240, description=Request InsertRequest not retried per RetryStrategy (Reason: KV_TEMPORARY_FAILURE), context=RequestContext{completed=false, coreId=0x607200bd0000003d, idempotent=false, lastChannelId=607200BD0000003D/000000004DD45382, lastDispatchedFrom=172.23.107.117:52034, lastDispatchedTo=172.23.106.155:11210, requestId=46363229, requestType=InsertRequest, retried=0, service={bucket=default, collection=_default, documentId=test_docs-9295924, opaque=0x2c5efd8, scope=_default, type=kv}, timeoutMs=60000, timings={dispatchMicros=362, encodingMicros=6, totalDispatchMicros=362, totalServerMicros=0, serverMicros=0}}, cause=null} Could not publish Event because the queue is full. RequestNotRetriedEvent{severity=INFO, category=com.couchbase.request, duration=PT0S, createdAt=9552764049590776, description=Request InsertRequest not retried per RetryStrategy (Reason: ENDPOINT_NOT_AVAILABLE), context=RequestContext{completed=false, coreId=0x607200bd0000003d, idempotent=false, lastDispatchedTo=172.23.106.122, requestId=46363256, requestType=InsertRequest, retried=0, service={bucket=default, collection=_default, documentId=test_docs-9295930, opaque=0x2c5eff3, scope=_default, type=kv}, timeoutMs=60000, timings={encodingMicros=7}}, cause=null} Could not publish Event because the queue is full. RequestNotRetriedEvent{severity=INFO, category=com.couchbase.request, duration=PT0S, createdAt=9552764049623625, description=Request InsertRequest not retried per RetryStrategy (Reason: KV_TEMPORARY_FAILURE), context=RequestContext{completed=false, coreId=0x607200bd0000003d, idempotent=false, lastChannelId=607200BD0000003D/000000002B282457, lastDispatchedFrom=172.23.107.117:59270, lastDispatchedTo=172.23.106.156:11210, requestId=46363226, requestType=InsertRequest, retried=0, service={bucket=default, collection=_default, documentId=test_docs-9643291, opaque=0x2c5efd5, scope=_default, type=kv}, timeoutMs=60000, timings={dispatchMicros=329, encodingMicros=6, totalDispatchMicros=329, totalServerMicros=0, serverMicros=0}}, cause=null} Could not publish Event because the queue is full. RequestNotRetriedEvent{severity=INFO, category=com.couchbase.request, duration=PT0S, createdAt=9552764050577220, description=Request InsertRequest not retried per RetryStrategy (Reason: ENDPOINT_NOT_AVAILABLE), context=RequestContext{completed=false, coreId=0x607200bd0000003d, idempotent=false, lastDispatchedTo=172.23.106.125, requestId=46363361, requestType=InsertRequest, retried=0, service={bucket=default, collection=_default, documentId=test_docs-9295933, opaque=0x2c5f05c, scope=_default, type=kv}, timeoutMs=60000, timings={encodingMicros=6}}, cause=null} Could not publish Event because the queue is full. EndpointStateChangedEvent{severity=DEBUG, category=com.couchbase.endpoint, duration=PT0S, createdAt=9552764049558831, description=Endpoint changed state from DISCONNECTED to CONNECTING, context=EndpointContext{channelId=607200BD0000002F/0000000005CB0F89, circuitBreaker=DISABLED, coreId=0x607200bd0000002f, local=172.23.107.117:60252, remote=172.23.106.122:11210, type=KV}, cause=null} Could not publish Event because the queue is full. RequestNotRetriedEvent{severity=INFO, category=com.couchbase.request, duration=PT0S, createdAt=9552764050665457, description=Request InsertRequest not retried per RetryStrategy (Reason: ENDPOINT_NOT_AVAILABLE), context=RequestContext{completed=false, coreId=0x607200bd0000003d, idempotent=false, lastDispatchedTo=172.23.106.125, requestId=46363372, requestType=InsertRequest, retried=0, service={bucket=default, collection=_default, documentId=test_docs-9643310, opaque=0x2c5f067, scope=_default, type=kv}, timeoutMs=60000, timings={encodingMicros=6}}, cause=null} Could not publish Event because the queue is full. EndpointStateChangedEvent{severity=DEBUG, category=com.couchbase.endpoint, duration=PT0S, createdAt=9552764050691434, description=Endpoint changed state from CONNECTING to DISCONNECTED, context=EndpointContext{channelId=607200BD00000037/0000000010B4D1C7, circuitBreaker=DISABLED, coreId=0x607200bd00000037, local=172.23.107.117:60554, remote=172.23.106.122:11210, type=KV}, cause=null} Could not publish Event because the queue is full. RequestNotRetriedEvent{severity=INFO, category=com.couchbase.request, duration=PT0S, createdAt=9552764050758417, description=Request InsertRequest not retried per RetryStrategy (Reason: ENDPOINT_NOT_AVAILABLE), context=RequestContext{completed=false, coreId=0x607200bd0000003d, idempotent=false, lastDispatchedTo=172.23.106.125, requestId=46363382, requestType=InsertRequest, retried=0, service={bucket=default, collection=_default, documentId=test_docs-9447544, opaque=0x2c5f071, scope=_default, type=kv}, timeoutMs=60000, timings={encodingMicros=6}}, cause=null} Could not publish Event because the queue is full. RequestNotRetriedEvent{severity=INFO, category=com.couchbase.request, duration=PT0S, createdAt=9552764050789483, description=Request InsertRequest not retried per RetryStrategy (Reason: KV_TEMPORARY_FAILURE), context=RequestContext{completed=false, coreId=0x607200bd0000003d, idempotent=false, lastChannelId=607200BD0000003D/000000002DBB08BB, lastDispatchedFrom=172.23.107.117:59230, lastDispatchedTo=172.23.106.156:11210, requestId=46353627, requestType=InsertRequest, retried=0, service={bucket=default, collection=_default, documentId=test_docs-9745791, opaque=0x2c5ca56, scope=_default, type=kv}, timeoutMs=60000, timings={dispatchMicros=395710, encodingMicros=5, totalDispatchMicros=395710, totalServerMicros=0, serverMicros=0}}, cause=null} Could not publish Event because the queue is full. RequestNotRetriedEvent{severity=INFO, category=com.couchbase.request, duration=PT0S, createdAt=9552764050980703, description=Request InsertRequest not retried per RetryStrategy (Reason: KV_TEMPORARY_FAILURE), context=RequestContext{completed=false, coreId=0x607200bd0000003d, idempotent=false, lastChannelId=607200BD0000003D/00000000829B3C86, lastDispatchedFrom=172.23.107.117:52074, lastDispatchedTo=172.23.106.155:11210, requestId=46354346, requestType=InsertRequest, retried=0, service={bucket=default, collection=_default, documentId=test_docs-9446611, opaque=0x2c5cd25, scope=_default, type=kv}, timeoutMs=60000, timings={dispatchMicros=380731, encodingMicros=6, totalDispatchMicros=380731, totalServerMicros=0, serverMicros=0}}, cause=null} Could not publish Event because the queue is full. RequestNotRetriedEvent{severity=INFO, category=com.couchbase.request, duration=PT0S, createdAt=9552764051170231, description=Request InsertRequest not retried per RetryStrategy (Reason: KV_TEMPORARY_FAILURE), context=RequestContext{completed=false, coreId=0x607200bd0000003d, idempotent=false, lastChannelId=607200BD0000003D/000000008EBF2300, lastDispatchedFrom=172.23.107.117:52236, lastDispatchedTo=172.23.106.155:11210, requestId=46362233, requestType=InsertRequest, retried=0, service={bucket=default, collection=_default, documentId=test_docs-9447385, opaque=0x2c5ebf4, scope=_default, type=kv}, timeoutMs=60000, timings={dispatchMicros=2973, encodingMicros=6, totalDispatchMicros=2973, totalServerMicros=0, serverMicros=0}}, cause=null} Could not publish Event because the queue is full. RequestNotRetriedEvent{severity=INFO, category=com.couchbase.request, duration=PT0S, createdAt=9552764051301892, description=Request InsertRequest not retried per RetryStrategy (Reason: KV_TEMPORARY_FAILURE), context=RequestContext{completed=false, coreId=0x607200bd0000003d, idempotent=false, lastChannelId=607200BD0000003D/00000000829B3C86, lastDispatchedFrom=172.23.107.117:52074, lastDispatchedTo=172.23.106.155:11210, requestId=46354556, requestType=InsertRequest, retried=0, service={bucket=default, collection=_default, documentId=test_docs-9997735, opaque=0x2c5cdf7, scope=_default, type=kv}, timeoutMs=60000, timings={dispatchMicros=380685, encodingMicros=7, totalDispatchMicros=380685, totalServerMicros=0, serverMicros=0}}, cause=null} Could not publish Event because the queue is full. RequestNotRetriedEvent{severity=INFO, category=com.couchbase.request, duration=PT0S, createdAt=9552764051388438, description=Request InsertRequest not retried per RetryStrategy (Reason: KV_TEMPORARY_FAILURE), context=RequestContext{completed=false, coreId=0x607200bd0000003d, idempotent=false, lastChannelId=607200BD0000003D/000000002DBB08BB, lastDispatchedFrom=172.23.107.117:59230, lastDispatchedTo=172.23.106.156:11210, requestId=46354106, requestType=InsertRequest, retried=0, service={bucket=default, collection=_default, documentId=test_docs-9997697, opaque=0x2c5cc36, scope=_default, type=kv}, timeoutMs=60000, timings={dispatchMicros=381856, encodingMicros=14, totalDispatchMicros=381856, totalServerMicros=0, serverMicros=0}}, cause=null} Could not publish Event because the queue is full. RequestNotRetriedEvent{severity=INFO, category=com.couchbase.request, duration=PT0S, createdAt=9552764051603362, description=Request InsertRequest not retried per RetryStrategy (Reason: ENDPOINT_NOT_AVAILABLE), context=RequestContext{completed=false, coreId=0x607200bd0000003d, idempotent=false, lastDispatchedTo=172.23.106.122, requestId=46363461, requestType=InsertRequest, retried=0, service={bucket=default, collection=_default, documentId=test_docs-9947218, opaque=0x2c5f0c0, scope=_default, type=kv}, timeoutMs=60000, timings={encodingMicros=7}}, cause=null} Could not publish Event because the queue is full. RequestNotRetriedEvent{severity=INFO, category=com.couchbase.request, duration=PT0S, createdAt=9552764051633855, description=Request InsertRequest not retried per RetryStrategy (Reason: KV_TEMPORARY_FAILURE), context=RequestContext{completed=false, coreId=0x607200bd0000003d, idempotent=false, lastChannelId=607200BD0000003D/000000004DD45382, lastDispatchedFrom=172.23.107.117:52034, lastDispatchedTo=172.23.106.155:11210, requestId=46363417, requestType=InsertRequest, retried=0, service={bucket=default, collection=_default, documentId=test_docs-9597852, opaque=0x2c5f094, scope=_default, type=kv}, timeoutMs=60000, timings={dispatchMicros=426, encodingMicros=6, totalDispatchMicros=426, totalServerMicros=0, serverMicros=0}}, cause=null} Could not publish Event because the queue is full. RequestNotRetriedEvent{severity=INFO, category=com.couchbase.request, duration=PT0S, createdAt=9552764050610902, description=Request InsertRequest not retried per RetryStrategy (Reason: KV_TEMPORARY_FAILURE), context=RequestContext{completed=false, coreId=0x607200bd0000003d, idempotent=false, lastChannelId=607200BD0000003D/00000000CE353CB0, lastDispatchedFrom=172.23.107.117:59244, lastDispatchedTo=172.23.106.156:11210, requestId=46362262, requestType=InsertRequest, retried=0, service={bucket=default, collection=_default, documentId=test_docs-9643185, opaque=0x2c5ec11, scope=_default, type=kv}, timeoutMs=60000, timings={dispatchMicros=6266, encodingMicros=64, totalDispatchMicros=6266, totalServerMicros=0, serverMicros=0}}, cause=null} Could not publish Event because the queue is full. RequestNotRetriedEvent{severity=INFO, category=com.couchbase.request, duration=PT0S, createdAt=9552764050577758, description=Request InsertRequest not retried per RetryStrategy (Reason: ENDPOINT_NOT_AVAILABLE), context=RequestContext{completed=false, coreId=0x607200bd0000003d, idempotent=false, lastDispatchedTo=172.23.106.122, requestId=46363360, requestType=InsertRequest, retried=0, service={bucket=default, collection=_default, documentId=test_docs-9494831, opaque=0x2c5f05b, scope=_default, type=kv}, timeoutMs=60000, timings={encodingMicros=7}}, cause=null} Could not publish Event because the queue is full. RequestNotRetriedEvent{severity=INFO, category=com.couchbase.request, duration=PT0S, createdAt=9552764050591627, description=Request InsertRequest not retried per RetryStrategy (Reason: ENDPOINT_NOT_AVAILABLE), context=RequestContext{completed=false, coreId=0x607200bd0000003d, idempotent=false, lastDispatchedTo=172.23.106.122, requestId=46363364, requestType=InsertRequest, retried=0, service={bucket=default, collection=_default, documentId=test_docs-9099187, opaque=0x2c5f05f, scope=_default, type=kv}, timeoutMs=60000, timings={encodingMicros=5}}, cause=null} Could not publish Event because the queue is full. RequestNotRetriedEvent{severity=INFO, category=com.couchbase.request, duration=PT0S, createdAt=9552764051312021, description=Request InsertRequest not retried per RetryStrategy (Reason: KV_TEMPORARY_FAILURE), context=RequestContext{completed=false, coreId=0x607200bd0000003d, idempotent=false, lastChannelId=607200BD0000003D/000000008EBF2300, lastDispatchedFrom=172.23.107.117:52236, lastDispatchedTo=172.23.106.155:11210, requestId=46362795, requestType=InsertRequest, retried=0, service={bucket=default, collection=_default, documentId=test_docs-9643257, opaque=0x2c5ee26, scope=_default, type=kv}, timeoutMs=60000, timings={dispatchMicros=2818, encodingMicros=7, totalDispatchMicros=2818, totalServerMicros=0, serverMicros=0}}, cause=null} Could not publish Event because the queue is full. EndpointConnectionFailedEvent{severity=WARN, category=com.couchbase.endpoint, duration=PT0.00039352S, createdAt=9552764050787730, description=Connect attempt 7 failed because of AnnotatedConnectException: finishConnect(..) failed: Connection refused: /172.23.106.122:11210, context=EndpointContext{channelId=607200BD00000037/0000000010B4D1C7, circuitBreaker=DISABLED, coreId=0x607200bd00000037, local=172.23.107.117:60554, remote=172.23.106.122:11210, type=KV}, cause=com.couchbase.client.core.deps.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: /172.23.106.122:11210} Could not publish Event because the queue is full. RequestNotRetriedEvent{severity=INFO, category=com.couchbase.request, duration=PT0S, createdAt=9552764052201090, description=Request InsertRequest not retried per RetryStrategy (Reason: ENDPOINT_NOT_AVAILABLE), context=RequestContext{completed=false, coreId=0x607200bd0000003d, idempotent=false, lastDispatchedTo=172.23.106.122, requestId=46363513, requestType=InsertRequest, retried=0, service={bucket=default, collection=_default, documentId=test_docs-9494834, opaque=0x2c5f0f4, scope=_default, type=kv}, timeoutMs=60000, timings={encodingMicros=6}}, cause=null} Could not publish Event because the queue is full. RequestNotRetriedEvent{severity=INFO, category=com.couchbase.request, duration=PT0S, createdAt=9552764052199562, description=Request InsertRequest not retried per RetryStrategy (Reason: ENDPOINT_NOT_AVAILABLE), context=RequestContext{completed=false, coreId=0x607200bd0000003d, idempotent=false, lastDispatchedTo=172.23.106.125, requestId=46363511, requestType=InsertRequest, retried=0, service={bucket=default, collection=_default, documentId=test_docs-9295951, opaque=0x2c5f0f2, scope=_default, type=kv}, timeoutMs=60000, timings={encodingMicros=6}}, cause=null} Could not publish Event because the queue is full. RequestNotRetriedEvent{severity=INFO, category=com.couchbase.request, duration=PT0S, createdAt=9552764052353167, description=Request InsertRequest not retried per RetryStrategy (Reason: ENDPOINT_NOT_AVAILABLE), context=RequestContext{completed=false, coreId=0x607200bd0000003d, idempotent=false, lastDispatchedTo=172.23.106.125, requestId=46363529, requestType=InsertRequest, retried=0, service={bucket=default, collection=_default, documentId=test_docs-9447575, opaque=0x2c5f104, scope=_default, type=kv}, timeoutMs=60000, timings={encodingMicros=6}}, cause=null} Could not publish Event because the queue is full. RequestNotRetriedEvent{severity=INFO, category=com.couchbase.request, duration=PT0S, createdAt=9552764052391728, description=Request InsertRequest not retried per RetryStrategy (Reason: KV_TEMPORARY_FAILURE), context=RequestContext{completed=false, coreId=0x607200bd0000003d, idempotent=false, lastChannelId=607200BD0000003D/00000000CDA4202B, lastDispatchedFrom=172.23.107.117:59256, lastDispatchedTo=172.23.106.156:11210, requestId=46361810, requestType=InsertRequest, retried=0, service={bucket=default, collection=_default, documentId=test_docs-9494558, opaque=0x2c5ea4d, scope=_default, type=kv}, timeoutMs=60000, timings={dispatchMicros=4573, encodingMicros=6, totalDispatchMicros=4573, totalServerMicros=0, serverMicros=0}}, cause=null} Could not publish Event because the queue is full. RequestNotRetriedEvent{severity=INFO, category=com.couchbase.request, duration=PT0S, createdAt=9552764052379271, description=Request InsertRequest not retried per RetryStrategy (Reason: ENDPOINT_NOT_AVAILABLE), context=RequestContext{completed=false, coreId=0x607200bd0000003d, idempotent=false, lastDispatchedTo=172.23.106.122, requestId=46363534, requestType=InsertRequest, retried=0, service={bucket=default, collection=_default, documentId=test_docs-9597883, opaque=0x2c5f109, scope=_default, type=kv}, timeoutMs=60000, timings={encodingMicros=8}}, cause=null} Could not publish Event because the queue is full. RequestNotRetriedEvent{severity=INFO, category=com.couchbase.request, duration=PT0S, createdAt=9552764052520761, description=Request InsertRequest not retried per RetryStrategy (Reason: KV_TEMPORARY_FAILURE), context=RequestContext{completed=false, coreId=0x607200bd0000003d, idempotent=false, lastChannelId=607200BD0000003D/00000000CA5ED266, lastDispatchedFrom=172.23.107.117:59268, lastDispatchedTo=172.23.106.156:11210, requestId=46363487, requestType=InsertRequest, retried=0, service={bucket=default, collection=_default, documentId=test_docs-9597873, opaque=0x2c5f0da, scope=_default, type=kv}, timeoutMs=60000, timings={dispatchMicros=414, encodingMicros=6, totalDispatchMicros=414, totalServerMicros=0, serverMicros=0}}, cause=null} Could not publish Event because the queue is full. RequestNotRetriedEvent{severity=INFO, category=com.couchbase.request, duration=PT0S, createdAt=9552764052615846, description=Request InsertRequest not retried per RetryStrategy (Reason: ENDPOINT_NOT_AVAILABLE), context=RequestContext{completed=false, coreId=0x607200bd0000003d, idempotent=false, lastDispatchedTo=172.23.106.122, requestId=46363568, requestType=InsertRequest, retried=0, service={bucket=default, collection=_default, documentId=test_docs-9295958, opaque=0x2c5f12b, scope=_default, type=kv}, timeoutMs=60000, timings={encodingMicros=6}}, cause=null} Could not publish Event because the queue is full. RequestNotRetriedEvent{severity=INFO, category=com.couchbase.request, duration=PT0S, createdAt=9552764052617797, description=Request InsertRequest not retried per RetryStrategy (Reason: KV_TEMPORARY_FAILURE), context=RequestContext{completed=false, coreId=0x607200bd0000003d, idempotent=false, lastChannelId=607200BD0000003D/00000000059D8BAF, lastDispatchedFrom=172.23.107.117:59940, lastDispatchedTo=172.23.106.155:11210, requestId=46363488, requestType=InsertRequest, retried=0, service={bucket=default, collection=_default, documentId=test_docs-9947224, opaque=0x2c5f0db, scope=_default, type=kv}, timeoutMs=60000, timings={dispatchMicros=499, encodingMicros=29, totalDispatchMicros=499, totalServerMicros=0, serverMicros=0}}, cause=null} Could not publish Event because the queue is full. RequestNotRetriedEvent{severity=INFO, category=com.couchbase.request, duration=PT0S, createdAt=9552764052770827, description=Request InsertRequest not retried per RetryStrategy (Reason: KV_TEMPORARY_FAILURE), context=RequestContext{completed=false, coreId=0x607200bd0000003d, idempotent=false, lastChannelId=607200BD0000003D/00000000CDA4202B, lastDispatchedFrom=172.23.107.117:59256, lastDispatchedTo=172.23.106.156:11210, requestId=46362011, requestType=InsertRequest, retried=0, service={bucket=default, collection=_default, documentId=test_docs-9447355, opaque=0x2c5eb16, scope=_default, type=kv}, timeoutMs=60000, timings={dispatchMicros=4711, encodingMicros=7, totalDispatchMicros=4711, totalServerMicros=0, serverMicros=0}}, cause=null} Could not publish Event because the queue is full. RequestNotRetriedEvent{severity=INFO, category=com.couchbase.request, duration=PT0S, createdAt=9552764052795918, description=Request InsertRequest not retried per RetryStrategy (Reason: ENDPOINT_NOT_AVAILABLE), context=RequestContext{completed=false, coreId=0x607200bd0000003d, idempotent=false, lastDispatchedTo=172.23.106.122, requestId=46363589, requestType=InsertRequest, retried=0, service={bucket=default, collection=_default, documentId=test_docs-9643331, opaque=0x2c5f140, scope=_default, type=kv}, timeoutMs=60000, timings={encodingMicros=7}}, cause=null} Could not publish Event because the queue is full. RequestNotRetriedEvent{severity=INFO, category=com.couchbase.request, duration=PT0S, createdAt=9552764052787049, description=Request InsertRequest not retried per RetryStrategy (Reason: KV_TEMPORARY_FAILURE), context=RequestContext{completed=false, coreId=0x607200bd0000003d, idempotent=false, lastChannelId=607200BD0000003D/00000000829B3C86, lastDispatchedFrom=172.23.107.117:52074, lastDispatchedTo=172.23.106.155:11210, requestId=46356896, requestType=InsertRequest, retried=0, service={bucket=default, collection=_default, documentId=test_docs-9295152, opaque=0x2c5d71b, scope=_default, type=kv}, timeoutMs=60000, timings={dispatchMicros=378096, encodingMicros=5, totalDispatchMicros=378096, totalServerMicros=0, serverMicros=0}}, cause=null} Could not publish Event because the queue is full. EndpointStateChangedEvent{severity=DEBUG, category=com.couchbase.endpoint, duration=PT0S, createdAt=9552764052887756, description=Endpoint changed state from CONNECTING to DISCONNECTED, context=EndpointContext{channelId=607200BD0000003D/0000000019163CC8, circuitBreaker=DISABLED, coreId=0x607200bd0000003d, local=172.23.107.117:60450, remote=172.23.106.122:11210, type=KV}, cause=null} Could not publish Event because the queue is full. RequestNotRetriedEvent{severity=INFO, category=com.couchbase.request, duration=PT0S, createdAt=9552764052914700, description=Request InsertRequest not retried per RetryStrategy (Reason: ENDPOINT_NOT_AVAILABLE), context=RequestContext{completed=false, coreId=0x607200bd0000003d, idempotent=false, lastDispatchedTo=172.23.106.125, requestId=46363608, requestType=InsertRequest, retried=0, service={bucket=default, collection=_default, documentId=test_docs-9099209, opaque=0x2c5f153, scope=_default, type=kv}, timeoutMs=60000, timings={encodingMicros=6}}, cause=null} Could not publish Event because the queue is full. EndpointConnectionFailedEvent{severity=WARN, category=com.couchbase.endpoint, duration=PT0.001553382S, createdAt=9552764052971093, description=Connect attempt 6 failed because of AnnotatedConnectException: finishConnect(..) failed: Connection refused: /172.23.106.122:11210, context=EndpointContext{channelId=607200BD0000003D/0000000019163CC8, circuitBreaker=DISABLED, coreId=0x607200bd0000003d, local=172.23.107.117:60450, remote=172.23.106.122:11210, type=KV}, cause=com.couchbase.client.core.deps.io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: /172.23.106.122:11210} Could not publish Event because the queue is full. RequestNotRetriedEvent{severity=INFO, category=com.couchbase.request, duration=PT0S, createdAt=9552764052977072, description=Request InsertRequest not retried per RetryStrategy (Reason: ENDPOINT_NOT_AVAILABLE), context=RequestContext{completed=false, coreId=0x607200bd0000003d, idempotent=false, lastDispatchedTo=172.23.106.125, requestId=46363618, requestType=InsertRequest, retried=0, service={bucket=default, collection=_default, documentId=test_docs-9447581, opaque=0x2c5f15d, scope=_default, type=kv}, timeoutMs=60000, timings={encodingMicros=10}}, cause=null} Could not publish Event because the queue is full. RequestNotRetriedEvent{severity=INFO, category=com.couchbase.request, duration=PT0S, createdAt=9552764053050192, description=Request InsertRequest not retried per RetryStrategy (Reason: KV_TEMPORARY_FAILURE), context=RequestContext{completed=false, coreId=0x607200bd0000003d, idempotent=false, lastChannelId=607200BD0000003D/00000000829B3C86, lastDispatchedFrom=172.23.107.117:52074, lastDispatchedTo=172.23.106.155:11210, requestId=46357192, requestType=InsertRequest, retried=0, service={bucket=default, collection=_default, documentId=test_docs-9642508, opaque=0x2c5d843, scope=_default, type=kv}, timeoutMs=60000, timings={dispatchMicros=377537, encodingMicros=7, totalDispatchMicros=377537, totalServerMicros=0, serverMicros=0}}, cause=null} Could not publish Event because the queue is full. RequestNotRetriedEvent{severity=INFO, category=com.couchbase.request, duration=PT0S, createdAt=9552764053061010, description=Request InsertRequest not retried per RetryStrategy (Reason: KV_TEMPORARY_FAILURE), context=RequestContext{completed=false, coreId=0x607200bd0000003d, idempotent=false, lastChannelId=607200BD0000003D/00000000CDA4202B, lastDispatchedFrom=172.23.107.117:59256, lastDispatchedTo=172.23.106.156:11210, requestId=46363006, requestType=InsertRequest, retried=0, service={bucket=default, collection=_default, documentId=test_docs-9597761, opaque=0x2c5eef9, scope=_default, type=kv}, timeoutMs=60000, timings={dispatchMicros=4393, encodingMicros=7, totalDispatchMicros=4393, totalServerMicros=0, serverMicros=0}}, cause=null} Could not publish Event because the queue is full. RequestNotRetriedEvent{severity=INFO, category=com.couchbase.request, duration=PT0S, createdAt=9552764053050192, description=Request InsertRequest not retried per RetryStrategy (Reason: KV_TEMPORARY_FAILURE), context=RequestContext{completed=false, coreId=0x607200bd0000003d, idempotent=false, lastChannelId=607200BD0000003D/0000000060AD4514, lastDispatchedFrom=172.23.107.117:59242, lastDispatchedTo=172.23.106.156:11210, requestId=46362303, requestType=InsertRequest, retried=0, service={bucket=default, collection=_default, documentId=test_docs-9295779, opaque=0x2c5ec3a, scope=_default, type=kv}, timeoutMs=60000, timings={dispatchMicros=8659, encodingMicros=5, totalDispatchMicros=8659, totalServerMicros=0, serverMicros=0}}, cause=null} Could not publish Event because the queue is full. RequestNotRetriedEvent{severity=INFO, category=com.couchbase.request, duration=PT0S, createdAt=9552764053731419, description=Request InsertRequest not retried per RetryStrategy (Reason: ENDPOINT_NOT_AVAILABLE), context=RequestContext{completed=false, coreId=0x607200bd0000003d, idempotent=false, lastDispatchedTo=172.23.106.122, requestId=46363687, requestType=InsertRequest, retried=0, service={bucket=default, collection=_default, documentId=test_docs-9099222, opaque=0x2c5f1a2, scope=_default, type=kv}, timeoutMs=60000, timings={encodingMicros=6}}, cause=null} Could not publish Event because the queue is full. RequestNotRetriedEvent{severity=INFO, category=com.couchbase.request, duration=PT0S, createdAt=9552764053033995, description=Request InsertRequest not retried per RetryStrategy (Reason: ENDPOINT_NOT_AVAILABLE), context=RequestContext{completed=false, coreId=0x607200bd0000003d, idempotent=false, lastDispatchedTo=172.23.106.125, requestId=46363622, requestType=InsertRequest, retried=0, service={bucket=default, collection=_default, documentId=test_docs-9947251, opaque=0x2c5f161, scope=_default, type=kv}, timeoutMs=60000, timings={encodingMicros=6}}, cause=null} Could not publish Event because the queue is full. RequestNotRetriedEvent{severity=INFO, category=com.couchbase.request, duration=PT0S, createdAt=9552764054130688, description=Request InsertRequest not retried per RetryStrategy (Reason: ENDPOINT_NOT_AVAILABLE), context=RequestContext{completed=false, coreId=0x607200bd0000003d, idempotent=false, lastDispatchedTo=172.23.106.125, requestId=46363723, requestType=InsertRequest, retried=0, service={bucket=default, collection=_default, documentId=test_docs-9494883, opaque=0x2c5f1c6, scope=_default, type=kv}, timeoutMs=60000, timings={encodingMicros=6}}, cause=null} Could not publish Event because the queue is full. RequestNotRetriedEvent{severity=INFO, category=com.couchbase.request, duration=PT0S, createdAt=9552764053732014, description=Request InsertRequest not retried per RetryStrategy (Reason: KV_TEMPORARY_FAILURE), context=RequestContext{completed=false, coreId=0x607200bd0000003d, idempotent=false, lastChannelId=607200BD0000003D/000000008E311097, lastDispatchedFrom=172.23.107.117:59252, lastDispatchedTo=172.23.106.156:11210, requestId=46363560, requestType=InsertRequest, retried=0, service={bucket=default, collection=_default, documentId=test_docs-9947240, opaque=0x2c5f123, scope=_default, type=kv}, timeoutMs=60000, timings={dispatchMicros=1071, encodingMicros=6, totalDispatchMicros=1071, totalServerMicros=0, serverMicros=0}}, cause=null} Could not publish Event because the queue is full. RequestNotRetriedEvent{severity=INFO, category=com.couchbase.request, duration=PT0S, createdAt=9552764054127996, description=Request InsertRequest not retried per RetryStrategy (Reason: KV_TEMPORARY_FAILURE), context=RequestContext{completed=false, coreId=0x607200bd0000003d, idempotent=false, lastChannelId=607200BD0000003D/0000000058C3552E, lastDispatchedFrom=172.23.107.117:59254, lastDispatchedTo=172.23.106.156:11210, requestId=46361809, requestType=InsertRequest, retried=0, service={bucket=default, collection=_default, documentId=test_docs-9098975, opaque=0x2c5ea4c, scope=_default, type=kv}, timeoutMs=60000, timings={dispatchMicros=6320, encodingMicros=6, totalDispatchMicros=6320, totalServerMicros=0, serverMicros=0}}, cause=null} Could not publish Event because the queue is full. RequestNotRetriedEvent{severity=INFO, category=com.couchbase.request, duration=PT0S, createdAt=9552764054132702, description=Request InsertRequest not retried per RetryStrategy (Reason: ENDPOINT_NOT_AVAILABLE), context=RequestContext{completed=false, coreId=0x607200bd0000003d, idempotent=false, lastDispatchedTo=172.23.106.125, requestId=46363724, requestType=InsertRequest, retried=0, service={bucket=default, collection=_default, documentId=test_docs-9643355, opaque=0x2c5f1c7, scope=_default, type=kv}, timeoutMs=60000, timings={encodingMicros=6}}, cause=null} 2021-06-08 11:52:09,338 | test | DEBUG | pool-5-thread-16 | [task:record_bucket_ops:3495] Ops rate for 'default': 5052.594042 2021-06-08 11:52:09,338 | test | DEBUG | pool-5-thread-16 | [task:print_ep_queue_size:3508] ep_queue_size for default: 1375527 2021-06-08 11:52:09,341 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:52:09,341 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:52:10,548 | test | DEBUG | pool-5-thread-16 | [task:record_bucket_ops:3495] Ops rate for 'default': 5052.594042 2021-06-08 11:52:10,548 | test | DEBUG | pool-5-thread-16 | [task:print_ep_queue_size:3508] ep_queue_size for default: 1375527 2021-06-08 11:52:10,549 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:52:10,549 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:52:11,763 | test | DEBUG | pool-5-thread-16 | [task:record_bucket_ops:3495] Ops rate for 'default': 7123.034134 2021-06-08 11:52:11,763 | test | DEBUG | pool-5-thread-16 | [task:print_ep_queue_size:3508] ep_queue_size for default: 0 2021-06-08 11:52:11,765 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:52:11,765 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:52:12,966 | test | DEBUG | pool-5-thread-16 | [task:record_bucket_ops:3495] Ops rate for 'default': 7123.034134 2021-06-08 11:52:12,966 | test | DEBUG | pool-5-thread-16 | [task:print_ep_queue_size:3508] ep_queue_size for default: 0 2021-06-08 11:52:12,967 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:52:12,967 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:52:14,177 | test | DEBUG | pool-5-thread-16 | [task:record_bucket_ops:3495] Ops rate for 'default': 7123.034134 2021-06-08 11:52:14,177 | test | DEBUG | pool-5-thread-16 | [task:print_ep_queue_size:3508] ep_queue_size for default: 0 2021-06-08 11:52:14,178 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:52:14,178 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:52:15,381 | test | DEBUG | pool-5-thread-16 | [task:record_bucket_ops:3495] Ops rate for 'default': 7123.034134 2021-06-08 11:52:15,381 | test | DEBUG | pool-5-thread-16 | [task:print_ep_queue_size:3508] ep_queue_size for default: 0 2021-06-08 11:52:15,382 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:52:15,384 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:52:16,701 | test | DEBUG | pool-5-thread-16 | [task:record_bucket_ops:3495] Ops rate for 'default': 6702.129935 2021-06-08 11:52:16,701 | test | DEBUG | pool-5-thread-16 | [task:print_ep_queue_size:3508] ep_queue_size for default: 0 2021-06-08 11:52:16,711 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:52:16,711 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:52:17,970 | test | DEBUG | pool-5-thread-16 | [task:record_bucket_ops:3495] Ops rate for 'default': 6702.129935 2021-06-08 11:52:17,971 | test | DEBUG | pool-5-thread-16 | [task:print_ep_queue_size:3508] ep_queue_size for default: 1267964 2021-06-08 11:52:17,973 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:52:17,973 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:52:18,220 | infra | INFO | pool-5-thread-2 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_9000000_10000000_1623178252.94_create__9050000_9100000_1623178252.97' completed 2021-06-08 11:52:18,362 | infra | INFO | pool-5-thread-11 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_9000000_10000000_1623178252.94_create__9650000_9700000_1623178252.97' completed 2021-06-08 11:52:21,408 | test | DEBUG | pool-5-thread-16 | [task:record_bucket_ops:3495] Ops rate for 'default': 3601.995801 2021-06-08 11:52:21,408 | test | DEBUG | pool-5-thread-16 | [task:print_ep_queue_size:3508] ep_queue_size for default: 1267964 2021-06-08 11:52:21,411 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:52:21,411 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:52:22,618 | test | DEBUG | pool-5-thread-16 | [task:record_bucket_ops:3495] Ops rate for 'default': 3601.995801 2021-06-08 11:52:22,618 | test | DEBUG | pool-5-thread-16 | [task:print_ep_queue_size:3508] ep_queue_size for default: 1267964 2021-06-08 11:52:22,619 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:52:22,619 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:52:22,667 | infra | INFO | pool-5-thread-28 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_9000000_10000000_1623178252.94_create__9950000_10000000_1623178252.97' completed 2021-06-08 11:52:22,739 | infra | INFO | pool-5-thread-7 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_9000000_10000000_1623178252.94_create__9500000_9550000_1623178252.97' completed 2021-06-08 11:52:22,744 | infra | INFO | pool-5-thread-1 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_9000000_10000000_1623178252.94_create__9900000_9950000_1623178252.97' completed 2021-06-08 11:52:22,822 | infra | INFO | pool-5-thread-29 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_9000000_10000000_1623178252.94_create__9550000_9600000_1623178252.97' completed 2021-06-08 11:52:22,831 | infra | INFO | pool-5-thread-5 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_9000000_10000000_1623178252.94_create__9400000_9450000_1623178252.97' completed 2021-06-08 11:52:22,877 | infra | INFO | pool-5-thread-10 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_9000000_10000000_1623178252.94_create__9300000_9350000_1623178252.97' completed 2021-06-08 11:52:22,892 | infra | INFO | pool-5-thread-13 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_9000000_10000000_1623178252.94_create__9150000_9200000_1623178252.97' completed 2021-06-08 11:52:23,325 | infra | INFO | pool-5-thread-9 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_9000000_10000000_1623178252.94_create__9250000_9300000_1623178252.97' completed 2021-06-08 11:52:23,569 | infra | INFO | pool-5-thread-18 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_9000000_10000000_1623178252.94_create__9800000_9850000_1623178252.97' completed 2021-06-08 11:52:23,569 | infra | INFO | pool-5-thread-30 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_9000000_10000000_1623178252.94_create__9000000_9050000_1623178252.97' completed 2021-06-08 11:52:23,569 | infra | DEBUG | pool-5-thread-21 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_9000000_10000000_1623178252.94_create__9000000_9050000_1623178252.97 are 50000 2021-06-08 11:52:23,571 | infra | DEBUG | pool-5-thread-21 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_9000000_10000000_1623178252.94_create__9050000_9100000_1623178252.97 2021-06-08 11:52:23,571 | infra | DEBUG | pool-5-thread-21 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_9000000_10000000_1623178252.94_create__9050000_9100000_1623178252.97 are 50000 2021-06-08 11:52:23,571 | infra | DEBUG | pool-5-thread-21 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_9000000_10000000_1623178252.94_create__9100000_9150000_1623178252.97 2021-06-08 11:52:23,904 | test | DEBUG | pool-5-thread-16 | [task:record_bucket_ops:3495] Ops rate for 'default': 3601.995801 2021-06-08 11:52:23,904 | test | DEBUG | pool-5-thread-16 | [task:print_ep_queue_size:3508] ep_queue_size for default: 1267964 2021-06-08 11:52:23,907 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:52:23,907 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:52:24,161 | infra | INFO | pool-5-thread-4 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_9000000_10000000_1623178252.94_create__9450000_9500000_1623178252.97' completed 2021-06-08 11:52:24,163 | infra | INFO | pool-5-thread-3 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_9000000_10000000_1623178252.94_create__9700000_9750000_1623178252.97' completed 2021-06-08 11:52:24,177 | infra | INFO | pool-5-thread-20 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_9000000_10000000_1623178252.94_create__9350000_9400000_1623178252.97' completed 2021-06-08 11:52:24,217 | infra | INFO | pool-5-thread-12 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_9000000_10000000_1623178252.94_create__9850000_9900000_1623178252.97' completed 2021-06-08 11:52:24,219 | infra | INFO | pool-5-thread-6 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_9000000_10000000_1623178252.94_create__9200000_9250000_1623178252.97' completed 2021-06-08 11:52:24,220 | infra | INFO | pool-5-thread-17 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_9000000_10000000_1623178252.94_create__9100000_9150000_1623178252.97' completed 2021-06-08 11:52:24,220 | infra | DEBUG | pool-5-thread-21 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_9000000_10000000_1623178252.94_create__9100000_9150000_1623178252.97 are 50000 2021-06-08 11:52:24,220 | infra | DEBUG | pool-5-thread-21 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_9000000_10000000_1623178252.94_create__9150000_9200000_1623178252.97 2021-06-08 11:52:24,220 | infra | DEBUG | pool-5-thread-21 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_9000000_10000000_1623178252.94_create__9150000_9200000_1623178252.97 are 50000 2021-06-08 11:52:24,220 | infra | DEBUG | pool-5-thread-21 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_9000000_10000000_1623178252.94_create__9200000_9250000_1623178252.97 2021-06-08 11:52:24,220 | infra | DEBUG | pool-5-thread-21 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_9000000_10000000_1623178252.94_create__9200000_9250000_1623178252.97 are 50000 2021-06-08 11:52:24,220 | infra | DEBUG | pool-5-thread-21 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_9000000_10000000_1623178252.94_create__9250000_9300000_1623178252.97 2021-06-08 11:52:24,221 | infra | DEBUG | pool-5-thread-21 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_9000000_10000000_1623178252.94_create__9250000_9300000_1623178252.97 are 50000 2021-06-08 11:52:24,221 | infra | DEBUG | pool-5-thread-21 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_9000000_10000000_1623178252.94_create__9300000_9350000_1623178252.97 2021-06-08 11:52:24,221 | infra | DEBUG | pool-5-thread-21 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_9000000_10000000_1623178252.94_create__9300000_9350000_1623178252.97 are 50000 2021-06-08 11:52:24,221 | infra | DEBUG | pool-5-thread-21 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_9000000_10000000_1623178252.94_create__9350000_9400000_1623178252.97 2021-06-08 11:52:24,221 | infra | DEBUG | pool-5-thread-21 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_9000000_10000000_1623178252.94_create__9350000_9400000_1623178252.97 are 50000 2021-06-08 11:52:24,221 | infra | DEBUG | pool-5-thread-21 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_9000000_10000000_1623178252.94_create__9400000_9450000_1623178252.97 2021-06-08 11:52:24,221 | infra | DEBUG | pool-5-thread-21 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_9000000_10000000_1623178252.94_create__9400000_9450000_1623178252.97 are 50000 2021-06-08 11:52:24,221 | infra | DEBUG | pool-5-thread-21 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_9000000_10000000_1623178252.94_create__9450000_9500000_1623178252.97 2021-06-08 11:52:24,221 | infra | DEBUG | pool-5-thread-21 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_9000000_10000000_1623178252.94_create__9450000_9500000_1623178252.97 are 50000 2021-06-08 11:52:24,223 | infra | DEBUG | pool-5-thread-21 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_9000000_10000000_1623178252.94_create__9500000_9550000_1623178252.97 2021-06-08 11:52:24,223 | infra | DEBUG | pool-5-thread-21 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_9000000_10000000_1623178252.94_create__9500000_9550000_1623178252.97 are 50000 2021-06-08 11:52:24,223 | infra | DEBUG | pool-5-thread-21 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_9000000_10000000_1623178252.94_create__9550000_9600000_1623178252.97 2021-06-08 11:52:24,223 | infra | DEBUG | pool-5-thread-21 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_9000000_10000000_1623178252.94_create__9550000_9600000_1623178252.97 are 50000 2021-06-08 11:52:24,223 | infra | DEBUG | pool-5-thread-21 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_9000000_10000000_1623178252.94_create__9600000_9650000_1623178252.97 2021-06-08 11:52:24,378 | infra | INFO | pool-5-thread-23 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_9000000_10000000_1623178252.94_create__9600000_9650000_1623178252.97' completed 2021-06-08 11:52:24,378 | infra | DEBUG | pool-5-thread-21 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_9000000_10000000_1623178252.94_create__9600000_9650000_1623178252.97 are 50000 2021-06-08 11:52:24,378 | infra | DEBUG | pool-5-thread-21 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_9000000_10000000_1623178252.94_create__9650000_9700000_1623178252.97 2021-06-08 11:52:24,378 | infra | DEBUG | pool-5-thread-21 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_9000000_10000000_1623178252.94_create__9650000_9700000_1623178252.97 are 50000 2021-06-08 11:52:24,378 | infra | DEBUG | pool-5-thread-21 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_9000000_10000000_1623178252.94_create__9700000_9750000_1623178252.97 2021-06-08 11:52:24,378 | infra | DEBUG | pool-5-thread-21 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_9000000_10000000_1623178252.94_create__9700000_9750000_1623178252.97 are 50000 2021-06-08 11:52:24,378 | infra | DEBUG | pool-5-thread-21 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_9000000_10000000_1623178252.94_create__9750000_9800000_1623178252.97 2021-06-08 11:52:24,387 | infra | INFO | pool-5-thread-25 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_9000000_10000000_1623178252.94_create__9750000_9800000_1623178252.97' completed 2021-06-08 11:52:24,388 | infra | DEBUG | pool-5-thread-21 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_9000000_10000000_1623178252.94_create__9750000_9800000_1623178252.97 are 50000 2021-06-08 11:52:24,388 | infra | DEBUG | pool-5-thread-21 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_9000000_10000000_1623178252.94_create__9800000_9850000_1623178252.97 2021-06-08 11:52:24,388 | infra | DEBUG | pool-5-thread-21 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_9000000_10000000_1623178252.94_create__9800000_9850000_1623178252.97 are 50000 2021-06-08 11:52:24,388 | infra | DEBUG | pool-5-thread-21 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_9000000_10000000_1623178252.94_create__9850000_9900000_1623178252.97 2021-06-08 11:52:24,388 | infra | DEBUG | pool-5-thread-21 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_9000000_10000000_1623178252.94_create__9850000_9900000_1623178252.97 are 50000 2021-06-08 11:52:24,388 | infra | DEBUG | pool-5-thread-21 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_9000000_10000000_1623178252.94_create__9900000_9950000_1623178252.97 2021-06-08 11:52:24,388 | infra | DEBUG | pool-5-thread-21 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_9000000_10000000_1623178252.94_create__9900000_9950000_1623178252.97 are 50000 2021-06-08 11:52:24,388 | infra | DEBUG | pool-5-thread-21 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_9000000_10000000_1623178252.94_create__9950000_10000000_1623178252.97 2021-06-08 11:52:24,388 | infra | DEBUG | pool-5-thread-21 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_9000000_10000000_1623178252.94_create__9950000_10000000_1623178252.97 are 50000 2021-06-08 11:52:24,388 | infra | DEBUG | pool-5-thread-21 | [task_manager:get_task_result:26] Getting task result for PrintBucketStats_default_1623178252.97 2021-06-08 11:52:24,907 | test | INFO | pool-5-thread-16 | [table_view:display:72] Ops trend for bucket 'default' +----------+------------------------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +----------+------------------------------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | *******.......................................*******X | 4085.000 | | 0.000 | ***...................****....................*...........******.................*******................................*.*X | 10456.700 | | 5104.100 | ...............................................***X | 5104.100 | | 4451.700 | .........................................*****..................**X | 6287.100 | | 5052.594 | ...............................................*******....................***X | 7123.034 | | 6702.130 | ...............................................................*X | 6702.130 | | 3601.996 | .................................**X | 3601.996 | +----------+------------------------------------------------------------------------------------------------------------------------------+-----------+ 2021-06-08 11:52:24,907 | infra | INFO | pool-5-thread-16 | [task:complete_task:102] Thread 'PrintBucketStats_default_1623178252.97' completed 2021-06-08 11:52:24,908 | infra | DEBUG | pool-5-thread-21 | [task:call:1663] ========= Tasks in loadgen pool======= 2021-06-08 11:52:24,908 | infra | WARNING | pool-5-thread-21 | [task_manager:print_tasks_in_pool:83] Task 'LoadDocsGen_default__default__default_create_9000000_10000000_1623178252.94' not completed 2021-06-08 11:52:24,908 | infra | DEBUG | pool-5-thread-21 | [task:call:1665] ====================================== 2021-06-08 11:52:24,908 | infra | DEBUG | pool-5-thread-21 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_9000000_10000000_1623178252.94_create__9000000_9050000_1623178252.97' complete. Loaded 50000 items 2021-06-08 11:52:24,908 | infra | DEBUG | pool-5-thread-21 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_9000000_10000000_1623178252.94_create__9050000_9100000_1623178252.97' complete. Loaded 50000 items 2021-06-08 11:52:24,908 | infra | DEBUG | pool-5-thread-21 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_9000000_10000000_1623178252.94_create__9100000_9150000_1623178252.97' complete. Loaded 50000 items 2021-06-08 11:52:24,908 | infra | DEBUG | pool-5-thread-21 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_9000000_10000000_1623178252.94_create__9150000_9200000_1623178252.97' complete. Loaded 50000 items 2021-06-08 11:52:24,908 | infra | DEBUG | pool-5-thread-21 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_9000000_10000000_1623178252.94_create__9200000_9250000_1623178252.97' complete. Loaded 50000 items 2021-06-08 11:52:24,910 | infra | DEBUG | pool-5-thread-21 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_9000000_10000000_1623178252.94_create__9250000_9300000_1623178252.97' complete. Loaded 50000 items 2021-06-08 11:52:24,910 | infra | DEBUG | pool-5-thread-21 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_9000000_10000000_1623178252.94_create__9300000_9350000_1623178252.97' complete. Loaded 50000 items 2021-06-08 11:52:24,910 | infra | DEBUG | pool-5-thread-21 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_9000000_10000000_1623178252.94_create__9350000_9400000_1623178252.97' complete. Loaded 50000 items 2021-06-08 11:52:24,910 | infra | DEBUG | pool-5-thread-21 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_9000000_10000000_1623178252.94_create__9400000_9450000_1623178252.97' complete. Loaded 50000 items 2021-06-08 11:52:24,910 | infra | DEBUG | pool-5-thread-21 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_9000000_10000000_1623178252.94_create__9450000_9500000_1623178252.97' complete. Loaded 50000 items 2021-06-08 11:52:24,910 | infra | DEBUG | pool-5-thread-21 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_9000000_10000000_1623178252.94_create__9500000_9550000_1623178252.97' complete. Loaded 50000 items 2021-06-08 11:52:24,910 | infra | DEBUG | pool-5-thread-21 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_9000000_10000000_1623178252.94_create__9550000_9600000_1623178252.97' complete. Loaded 50000 items 2021-06-08 11:52:24,910 | infra | DEBUG | pool-5-thread-21 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_9000000_10000000_1623178252.94_create__9600000_9650000_1623178252.97' complete. Loaded 50000 items 2021-06-08 11:52:24,910 | infra | DEBUG | pool-5-thread-21 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_9000000_10000000_1623178252.94_create__9650000_9700000_1623178252.97' complete. Loaded 50000 items 2021-06-08 11:52:24,911 | infra | DEBUG | pool-5-thread-21 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_9000000_10000000_1623178252.94_create__9700000_9750000_1623178252.97' complete. Loaded 50000 items 2021-06-08 11:52:24,911 | infra | DEBUG | pool-5-thread-21 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_9000000_10000000_1623178252.94_create__9750000_9800000_1623178252.97' complete. Loaded 50000 items 2021-06-08 11:52:24,911 | infra | DEBUG | pool-5-thread-21 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_9000000_10000000_1623178252.94_create__9800000_9850000_1623178252.97' complete. Loaded 50000 items 2021-06-08 11:52:24,911 | infra | DEBUG | pool-5-thread-21 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_9000000_10000000_1623178252.94_create__9850000_9900000_1623178252.97' complete. Loaded 50000 items 2021-06-08 11:52:24,911 | infra | DEBUG | pool-5-thread-21 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_9000000_10000000_1623178252.94_create__9900000_9950000_1623178252.97' complete. Loaded 50000 items 2021-06-08 11:52:24,911 | infra | DEBUG | pool-5-thread-21 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_9000000_10000000_1623178252.94_create__9950000_10000000_1623178252.97' complete. Loaded 50000 items 2021-06-08 11:52:24,911 | infra | INFO | pool-5-thread-21 | [task:complete_task:102] Thread 'LoadDocsGen_default__default__default_create_9000000_10000000_1623178252.94' completed 2021-06-08 11:52:24,914 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.125 with username: root, password: couchbase 2021-06-08 11:52:25,000 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 11:52:25,210 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: ls /etc/ 2021-06-08 11:52:25,265 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 11:52:25,265 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: ls /cygdrive/c/Windows 2021-06-08 11:52:25,278 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: uname -m 2021-06-08 11:52:25,289 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: cat /proc/cpuinfo 2021-06-08 11:52:25,305 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: df -Thl 2021-06-08 11:52:25,322 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: cat /proc/meminfo 2021-06-08 11:52:25,334 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: hostname 2021-06-08 11:52:25,345 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: hostname -d 2021-06-08 11:52:25,358 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:559] Starting StatsWaitTask for ep_data_write_failed on bucket default 2021-06-08 11:52:25,358 | infra | INFO | MainThread | [task_manager:add_new_task:23] Added new task: StatsWaitTask_default_ep_data_write_failed_1623178345.36 2021-06-08 11:52:25,358 | infra | INFO | pool-5-thread-8 | [task:start_task:87] Thread 'StatsWaitTask_default_ep_data_write_failed_1623178345.36' started 2021-06-08 11:52:25,359 | infra | DEBUG | pool-5-thread-8 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:52:25,361 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.155 with username: root, password: couchbase 2021-06-08 11:52:25,434 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 11:52:25,482 | test | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 11:52:25,483 | test | INFO | pool-5-thread-8 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1010, in main() File "/opt/couchbase/lib/python/cbstats", line 1007, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 49, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-06-08 11:52:25,647 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: ls /etc/ 2021-06-08 11:52:25,700 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 11:52:25,700 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: ls /cygdrive/c/Windows 2021-06-08 11:52:25,713 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: uname -m 2021-06-08 11:52:25,723 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: cat /proc/cpuinfo 2021-06-08 11:52:25,736 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: df -Thl 2021-06-08 11:52:25,750 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: cat /proc/meminfo 2021-06-08 11:52:25,762 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: hostname 2021-06-08 11:52:25,773 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: hostname -d 2021-06-08 11:52:25,785 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:559] Starting StatsWaitTask for ep_data_write_failed on bucket default 2021-06-08 11:52:25,785 | infra | INFO | MainThread | [task_manager:add_new_task:23] Added new task: StatsWaitTask_default_ep_data_write_failed_1623178345.79 2021-06-08 11:52:25,786 | infra | INFO | pool-5-thread-19 | [task:start_task:87] Thread 'StatsWaitTask_default_ep_data_write_failed_1623178345.79' started 2021-06-08 11:52:25,788 | infra | DEBUG | pool-5-thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:52:25,789 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.156 with username: root, password: couchbase 2021-06-08 11:52:25,858 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 11:52:26,009 | test | DEBUG | pool-5-thread-19 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 11:52:26,009 | infra | DEBUG | pool-5-thread-19 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.155': '0'} for bucket 'default' 2021-06-08 11:52:26,009 | infra | DEBUG | pool-5-thread-19 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 11:52:26,013 | infra | DEBUG | pool-5-thread-19 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 11:52:26,013 | infra | DEBUG | pool-5-thread-19 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 11:52:26,062 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: ls /etc/ 2021-06-08 11:52:26,114 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 11:52:26,114 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: ls /cygdrive/c/Windows 2021-06-08 11:52:26,125 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: uname -m 2021-06-08 11:52:26,137 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: cat /proc/cpuinfo 2021-06-08 11:52:26,148 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: df -Thl 2021-06-08 11:52:26,161 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: cat /proc/meminfo 2021-06-08 11:52:26,173 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: hostname 2021-06-08 11:52:26,184 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: hostname -d 2021-06-08 11:52:26,196 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:559] Starting StatsWaitTask for ep_data_write_failed on bucket default 2021-06-08 11:52:26,197 | infra | INFO | MainThread | [task_manager:add_new_task:23] Added new task: StatsWaitTask_default_ep_data_write_failed_1623178346.2 2021-06-08 11:52:26,197 | infra | INFO | pool-5-thread-27 | [task:start_task:87] Thread 'StatsWaitTask_default_ep_data_write_failed_1623178346.2' started 2021-06-08 11:52:26,197 | infra | DEBUG | pool-5-thread-27 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:52:26,200 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.122 with username: root, password: couchbase 2021-06-08 11:52:26,279 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 11:52:26,411 | test | DEBUG | pool-5-thread-27 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 11:52:26,411 | infra | DEBUG | pool-5-thread-27 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.156': '0'} for bucket 'default' 2021-06-08 11:52:26,411 | infra | DEBUG | pool-5-thread-27 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 11:52:26,413 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 11:52:26,413 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 11:52:26,492 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: ls /etc/ 2021-06-08 11:52:26,545 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 11:52:26,546 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: ls /cygdrive/c/Windows 2021-06-08 11:52:26,559 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: uname -m 2021-06-08 11:52:26,571 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: cat /proc/cpuinfo 2021-06-08 11:52:26,582 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: df -Thl 2021-06-08 11:52:26,596 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: cat /proc/meminfo 2021-06-08 11:52:26,608 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: hostname 2021-06-08 11:52:26,621 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: hostname -d 2021-06-08 11:52:26,634 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:559] Starting StatsWaitTask for ep_data_write_failed on bucket default 2021-06-08 11:52:26,634 | infra | INFO | MainThread | [task_manager:add_new_task:23] Added new task: StatsWaitTask_default_ep_data_write_failed_1623178346.63 2021-06-08 11:52:26,634 | infra | DEBUG | MainThread | [task_manager:get_task_result:26] Getting task result for StatsWaitTask_default_ep_data_write_failed_1623178345.36 2021-06-08 11:52:26,634 | infra | INFO | pool-5-thread-24 | [task:start_task:87] Thread 'StatsWaitTask_default_ep_data_write_failed_1623178346.63' started 2021-06-08 11:52:26,635 | infra | DEBUG | pool-5-thread-24 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:52:26,753 | test | DEBUG | pool-5-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 11:52:26,753 | test | INFO | pool-5-thread-24 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 11:52:30,483 | infra | DEBUG | pool-5-thread-8 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:52:30,605 | test | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 11:52:30,605 | test | INFO | pool-5-thread-8 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 11:52:31,013 | infra | DEBUG | pool-5-thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:52:31,209 | test | DEBUG | pool-5-thread-19 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 11:52:31,209 | infra | DEBUG | pool-5-thread-19 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.155': '0'} for bucket 'default' 2021-06-08 11:52:31,210 | infra | DEBUG | pool-5-thread-19 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 11:52:31,210 | infra | DEBUG | pool-5-thread-19 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 11:52:31,210 | infra | DEBUG | pool-5-thread-19 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 11:52:31,414 | infra | DEBUG | pool-5-thread-27 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:52:31,755 | infra | DEBUG | pool-5-thread-24 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:52:31,871 | test | DEBUG | pool-5-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 11:52:31,871 | test | INFO | pool-5-thread-24 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 11:52:35,605 | infra | DEBUG | pool-5-thread-8 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:52:35,744 | test | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 11:52:35,746 | test | INFO | pool-5-thread-8 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1010, in main() File "/opt/couchbase/lib/python/cbstats", line 1007, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 49, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-06-08 11:52:36,211 | infra | DEBUG | pool-5-thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:52:36,411 | test | DEBUG | pool-5-thread-19 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 11:52:36,411 | infra | DEBUG | pool-5-thread-19 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.155': '0'} for bucket 'default' 2021-06-08 11:52:36,411 | infra | DEBUG | pool-5-thread-19 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 11:52:36,413 | infra | DEBUG | pool-5-thread-19 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 11:52:36,413 | infra | DEBUG | pool-5-thread-19 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 11:52:36,871 | infra | DEBUG | pool-5-thread-24 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:52:36,987 | test | DEBUG | pool-5-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 11:52:36,987 | test | INFO | pool-5-thread-24 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 11:52:40,746 | infra | DEBUG | pool-5-thread-8 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:52:40,862 | test | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 11:52:40,864 | test | INFO | pool-5-thread-8 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 11:52:41,414 | infra | DEBUG | pool-5-thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:52:41,611 | test | DEBUG | pool-5-thread-19 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 11:52:41,611 | infra | DEBUG | pool-5-thread-19 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.155': '0'} for bucket 'default' 2021-06-08 11:52:41,611 | infra | DEBUG | pool-5-thread-19 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 11:52:41,614 | infra | DEBUG | pool-5-thread-19 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 11:52:41,614 | infra | DEBUG | pool-5-thread-19 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 11:52:41,989 | infra | DEBUG | pool-5-thread-24 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:52:42,105 | test | DEBUG | pool-5-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 11:52:42,105 | test | INFO | pool-5-thread-24 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 11:52:45,864 | infra | DEBUG | pool-5-thread-8 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:52:45,983 | test | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 11:52:45,983 | test | INFO | pool-5-thread-8 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1010, in main() File "/opt/couchbase/lib/python/cbstats", line 1007, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 49, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-06-08 11:52:46,615 | infra | DEBUG | pool-5-thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:52:46,815 | test | DEBUG | pool-5-thread-19 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 11:52:46,815 | infra | DEBUG | pool-5-thread-19 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.155': '0'} for bucket 'default' 2021-06-08 11:52:46,815 | infra | DEBUG | pool-5-thread-19 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 11:52:46,816 | infra | DEBUG | pool-5-thread-19 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 11:52:46,818 | infra | DEBUG | pool-5-thread-19 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 11:52:47,105 | infra | DEBUG | pool-5-thread-24 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:52:47,224 | test | DEBUG | pool-5-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 11:52:47,224 | test | INFO | pool-5-thread-24 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 11:52:50,668 | test | DEBUG | pool-5-thread-27 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 11:52:50,668 | infra | DEBUG | pool-5-thread-27 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.156': '0'} for bucket 'default' 2021-06-08 11:52:50,668 | infra | DEBUG | pool-5-thread-27 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 11:52:50,671 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 11:52:50,671 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 11:52:50,983 | infra | DEBUG | pool-5-thread-8 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:52:51,099 | test | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 11:52:51,099 | test | INFO | pool-5-thread-8 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 11:52:51,818 | infra | DEBUG | pool-5-thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:52:52,013 | test | DEBUG | pool-5-thread-19 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 11:52:52,013 | infra | DEBUG | pool-5-thread-19 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.155': '0'} for bucket 'default' 2021-06-08 11:52:52,013 | infra | DEBUG | pool-5-thread-19 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 11:52:52,015 | infra | DEBUG | pool-5-thread-19 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 11:52:52,015 | infra | DEBUG | pool-5-thread-19 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 11:52:52,226 | infra | DEBUG | pool-5-thread-24 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:52:52,344 | test | DEBUG | pool-5-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 11:52:52,344 | test | INFO | pool-5-thread-24 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 11:52:55,671 | infra | DEBUG | pool-5-thread-27 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:52:56,101 | infra | DEBUG | pool-5-thread-8 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:52:56,377 | test | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 11:52:56,377 | test | INFO | pool-5-thread-8 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1010, in main() File "/opt/couchbase/lib/python/cbstats", line 1007, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 49, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-06-08 11:52:57,016 | infra | DEBUG | pool-5-thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:52:57,213 | test | DEBUG | pool-5-thread-19 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 11:52:57,213 | infra | DEBUG | pool-5-thread-19 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.155': '0'} for bucket 'default' 2021-06-08 11:52:57,213 | infra | DEBUG | pool-5-thread-19 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 11:52:57,214 | infra | DEBUG | pool-5-thread-19 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 11:52:57,214 | infra | DEBUG | pool-5-thread-19 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 11:52:57,345 | infra | DEBUG | pool-5-thread-24 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:52:57,463 | test | DEBUG | pool-5-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 11:52:57,463 | test | INFO | pool-5-thread-24 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 11:53:01,378 | infra | DEBUG | pool-5-thread-8 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:53:01,494 | test | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 11:53:01,494 | test | INFO | pool-5-thread-8 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 11:53:02,216 | infra | DEBUG | pool-5-thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:53:02,410 | test | DEBUG | pool-5-thread-19 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 11:53:02,410 | infra | DEBUG | pool-5-thread-19 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.155': '0'} for bucket 'default' 2021-06-08 11:53:02,410 | infra | DEBUG | pool-5-thread-19 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 11:53:02,411 | infra | DEBUG | pool-5-thread-19 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 11:53:02,411 | infra | DEBUG | pool-5-thread-19 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 11:53:02,463 | infra | DEBUG | pool-5-thread-24 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:53:02,581 | test | DEBUG | pool-5-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 11:53:02,582 | test | INFO | pool-5-thread-24 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 11:53:06,494 | infra | DEBUG | pool-5-thread-8 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:53:06,612 | test | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 11:53:06,612 | test | INFO | pool-5-thread-8 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 11:53:07,413 | infra | DEBUG | pool-5-thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:53:07,582 | infra | DEBUG | pool-5-thread-24 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:53:07,611 | test | DEBUG | pool-5-thread-19 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 11:53:07,612 | infra | DEBUG | pool-5-thread-19 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.155': '0'} for bucket 'default' 2021-06-08 11:53:07,612 | infra | DEBUG | pool-5-thread-19 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 11:53:07,615 | infra | DEBUG | pool-5-thread-19 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 11:53:07,615 | infra | DEBUG | pool-5-thread-19 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 11:53:07,700 | test | DEBUG | pool-5-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 11:53:07,700 | test | INFO | pool-5-thread-24 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 11:53:11,614 | infra | DEBUG | pool-5-thread-8 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:53:11,733 | test | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 11:53:11,734 | test | INFO | pool-5-thread-8 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 11:53:12,615 | infra | DEBUG | pool-5-thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:53:12,700 | infra | DEBUG | pool-5-thread-24 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:53:12,816 | test | DEBUG | pool-5-thread-19 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 11:53:12,818 | infra | DEBUG | pool-5-thread-19 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.155': '0'} for bucket 'default' 2021-06-08 11:53:12,818 | infra | DEBUG | pool-5-thread-19 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 11:53:12,819 | infra | DEBUG | pool-5-thread-19 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 11:53:12,819 | infra | DEBUG | pool-5-thread-19 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 11:53:12,821 | test | DEBUG | pool-5-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 11:53:12,821 | test | INFO | pool-5-thread-24 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 11:53:16,734 | test | DEBUG | pool-5-thread-8 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 11:53:16,734 | infra | DEBUG | pool-5-thread-8 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {} for bucket 'default' 2021-06-08 11:53:16,734 | infra | DEBUG | pool-5-thread-8 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 11:53:16,736 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 11:53:16,737 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 11:53:17,821 | infra | DEBUG | pool-5-thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:53:17,822 | test | DEBUG | pool-5-thread-24 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 11:53:17,822 | infra | DEBUG | pool-5-thread-24 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {} for bucket 'default' 2021-06-08 11:53:17,822 | infra | DEBUG | pool-5-thread-24 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 11:53:17,823 | infra | DEBUG | pool-5-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 11:53:17,823 | infra | DEBUG | pool-5-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 11:53:18,017 | test | DEBUG | pool-5-thread-19 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 11:53:18,017 | infra | DEBUG | pool-5-thread-19 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.155': '0'} for bucket 'default' 2021-06-08 11:53:18,019 | infra | DEBUG | pool-5-thread-19 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 11:53:18,020 | infra | DEBUG | pool-5-thread-19 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 11:53:18,020 | infra | DEBUG | pool-5-thread-19 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 11:53:21,737 | infra | DEBUG | pool-5-thread-8 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:53:21,855 | test | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 11:53:21,855 | test | INFO | pool-5-thread-8 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 11:53:22,823 | infra | DEBUG | pool-5-thread-24 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:53:22,948 | test | DEBUG | pool-5-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 11:53:22,950 | test | INFO | pool-5-thread-24 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1010, in main() File "/opt/couchbase/lib/python/cbstats", line 1007, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 49, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-06-08 11:53:23,020 | infra | DEBUG | pool-5-thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:53:23,216 | test | DEBUG | pool-5-thread-19 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 11:53:23,216 | infra | DEBUG | pool-5-thread-19 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.155': '0'} for bucket 'default' 2021-06-08 11:53:23,216 | infra | DEBUG | pool-5-thread-19 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 11:53:23,219 | infra | DEBUG | pool-5-thread-19 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 11:53:23,219 | infra | DEBUG | pool-5-thread-19 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 11:53:25,802 | test | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 11:53:25,802 | test | INFO | pool-5-thread-27 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 11:53:26,878 | infra | DEBUG | pool-5-thread-8 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:53:26,994 | test | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 11:53:26,994 | test | INFO | pool-5-thread-8 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 11:53:27,950 | infra | DEBUG | pool-5-thread-24 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:53:28,066 | test | DEBUG | pool-5-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 11:53:28,068 | test | INFO | pool-5-thread-24 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 11:53:28,219 | infra | DEBUG | pool-5-thread-19 | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.155 2021-06-08 11:53:28,220 | infra | INFO | pool-5-thread-19 | [task:complete_task:102] Thread 'StatsWaitTask_default_ep_data_write_failed_1623178345.79' completed 2021-06-08 11:53:30,803 | infra | DEBUG | pool-5-thread-27 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:53:31,996 | infra | DEBUG | pool-5-thread-8 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:53:32,111 | test | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 11:53:32,111 | test | INFO | pool-5-thread-8 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 11:53:33,068 | infra | DEBUG | pool-5-thread-24 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:53:33,194 | test | DEBUG | pool-5-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 11:53:33,194 | test | INFO | pool-5-thread-24 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1010, in main() File "/opt/couchbase/lib/python/cbstats", line 1007, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 49, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-06-08 11:53:37,112 | infra | DEBUG | pool-5-thread-8 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:53:37,229 | test | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 11:53:37,229 | test | INFO | pool-5-thread-8 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 11:53:38,194 | infra | DEBUG | pool-5-thread-24 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:53:38,311 | test | DEBUG | pool-5-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 11:53:38,311 | test | INFO | pool-5-thread-24 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 11:53:42,230 | infra | DEBUG | pool-5-thread-8 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:53:42,346 | test | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 11:53:42,346 | test | INFO | pool-5-thread-8 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 11:53:43,312 | infra | DEBUG | pool-5-thread-24 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:53:43,436 | test | DEBUG | pool-5-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 11:53:43,436 | test | INFO | pool-5-thread-24 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1010, in main() File "/opt/couchbase/lib/python/cbstats", line 1007, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 49, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-06-08 11:53:47,348 | infra | DEBUG | pool-5-thread-8 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:53:47,464 | test | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 11:53:47,466 | test | INFO | pool-5-thread-8 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 11:53:48,436 | infra | DEBUG | pool-5-thread-24 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:53:48,559 | test | DEBUG | pool-5-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 11:53:48,559 | test | INFO | pool-5-thread-24 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 11:53:52,466 | infra | DEBUG | pool-5-thread-8 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:53:52,584 | test | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 11:53:52,585 | test | INFO | pool-5-thread-8 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 11:53:53,559 | infra | DEBUG | pool-5-thread-24 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:53:53,684 | test | DEBUG | pool-5-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 11:53:53,684 | test | INFO | pool-5-thread-24 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1010, in main() File "/opt/couchbase/lib/python/cbstats", line 1007, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 49, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-06-08 11:53:57,365 | test | DEBUG | pool-5-thread-27 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 11:53:57,365 | infra | DEBUG | pool-5-thread-27 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.156': '0'} for bucket 'default' 2021-06-08 11:53:57,365 | infra | DEBUG | pool-5-thread-27 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 11:53:57,367 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 11:53:57,368 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 11:53:57,585 | infra | DEBUG | pool-5-thread-8 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:53:57,703 | test | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 11:53:57,703 | test | INFO | pool-5-thread-8 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 11:53:58,684 | infra | DEBUG | pool-5-thread-24 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:53:58,802 | test | DEBUG | pool-5-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 11:53:58,802 | test | INFO | pool-5-thread-24 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 11:54:02,368 | infra | DEBUG | pool-5-thread-27 | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.156 2021-06-08 11:54:02,368 | infra | INFO | pool-5-thread-27 | [task:complete_task:102] Thread 'StatsWaitTask_default_ep_data_write_failed_1623178346.2' completed 2021-06-08 11:54:02,704 | infra | DEBUG | pool-5-thread-8 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:54:02,822 | test | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 11:54:02,822 | test | INFO | pool-5-thread-8 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 11:54:03,803 | infra | DEBUG | pool-5-thread-24 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:54:03,996 | test | DEBUG | pool-5-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 11:54:03,996 | test | INFO | pool-5-thread-24 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1010, in main() File "/opt/couchbase/lib/python/cbstats", line 1007, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 49, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 79, in g mc.sasl_auth_plain(username, password) File "/opt/couchbase/lib/python/mc_bin_client.py", line 488, in sasl_auth_plain return self.sasl_auth_start('PLAIN', '\0'.join([foruser, user, password])) File "/opt/couchbase/lib/python/mc_bin_client.py", line 484, in sasl_auth_start return self._doCmd(memcacheConstants.CMD_SASL_AUTH, mech, data) File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 281, in _handleKeyedResponse cmd, errcode, opaque, cas, keylen, extralen, rv = self._recvMsg() File "/opt/couchbase/lib/python/mc_bin_client.py", line 250, in _recvMsg data = self._socketRecv(MIN_RECV_PACKET - len(response)) File "/opt/couchbase/lib/python/mc_bin_client.py", line 244, in _socketRecv return self.s.recv(amount) ConnectionResetError: [Errno 104] Connection reset by peer 2021-06-08 11:54:07,822 | infra | DEBUG | pool-5-thread-8 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:54:07,940 | test | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 11:54:07,940 | test | INFO | pool-5-thread-8 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 11:54:08,997 | infra | DEBUG | pool-5-thread-24 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:54:09,115 | test | DEBUG | pool-5-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 11:54:09,115 | test | INFO | pool-5-thread-24 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 11:54:12,940 | test | DEBUG | pool-5-thread-8 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 11:54:12,940 | infra | DEBUG | pool-5-thread-8 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {} for bucket 'default' 2021-06-08 11:54:12,940 | infra | DEBUG | pool-5-thread-8 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 11:54:12,941 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 11:54:12,941 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 11:54:14,115 | test | DEBUG | pool-5-thread-24 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 11:54:14,117 | infra | DEBUG | pool-5-thread-24 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {} for bucket 'default' 2021-06-08 11:54:14,117 | infra | DEBUG | pool-5-thread-24 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 11:54:14,118 | infra | DEBUG | pool-5-thread-24 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 11:54:14,118 | infra | DEBUG | pool-5-thread-24 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 11:54:17,943 | infra | DEBUG | pool-5-thread-8 | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.125 2021-06-08 11:54:17,944 | infra | INFO | pool-5-thread-8 | [task:complete_task:102] Thread 'StatsWaitTask_default_ep_data_write_failed_1623178345.36' completed 2021-06-08 11:54:17,948 | infra | INFO | MainThread | [task_manager:add_new_task:23] Added new task: LoadDocsGen_default__default__default_create_10000000_11000000_1623178457.95 2021-06-08 11:54:17,948 | infra | INFO | pool-5-thread-14 | [task:start_task:87] Thread 'LoadDocsGen_default__default__default_create_10000000_11000000_1623178457.95' started 2021-06-08 11:54:17,948 | infra | DEBUG | MainThread | [task_manager:get_task_result:26] Getting task result for LoadDocsGen_default__default__default_create_10000000_11000000_1623178457.95 2021-06-08 11:54:17,999 | infra | INFO | pool-5-thread-14 | [task_manager:add_new_task:23] Added new task: PrintBucketStats_default_1623178457.99 2021-06-08 11:54:18,000 | infra | INFO | pool-5-thread-26 | [task:start_task:87] Thread 'PrintBucketStats_default_1623178457.99' started 2021-06-08 11:54:18,000 | infra | INFO | pool-5-thread-14 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_10000000_11000000_1623178457.95_create__10000000_10050000_1623178457.99 2021-06-08 11:54:18,000 | infra | INFO | pool-5-thread-15 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_10000000_11000000_1623178457.95_create__10000000_10050000_1623178457.99' started 2021-06-08 11:54:18,000 | infra | INFO | pool-5-thread-14 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_10000000_11000000_1623178457.95_create__10050000_10100000_1623178457.99 2021-06-08 11:54:18,000 | infra | INFO | pool-5-thread-22 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_10000000_11000000_1623178457.95_create__10050000_10100000_1623178457.99' started 2021-06-08 11:54:18,000 | infra | INFO | pool-5-thread-14 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_10000000_11000000_1623178457.95_create__10100000_10150000_1623178457.99 2021-06-08 11:54:18,000 | infra | INFO | pool-5-thread-2 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_10000000_11000000_1623178457.95_create__10100000_10150000_1623178457.99' started 2021-06-08 11:54:18,000 | infra | INFO | pool-5-thread-14 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_10000000_11000000_1623178457.95_create__10150000_10200000_1623178457.99 2021-06-08 11:54:18,000 | infra | INFO | pool-5-thread-11 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_10000000_11000000_1623178457.95_create__10150000_10200000_1623178457.99' started 2021-06-08 11:54:18,000 | infra | INFO | pool-5-thread-14 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_10000000_11000000_1623178457.95_create__10200000_10250000_1623178457.99 2021-06-08 11:54:18,000 | infra | INFO | pool-5-thread-28 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_10000000_11000000_1623178457.95_create__10200000_10250000_1623178457.99' started 2021-06-08 11:54:18,000 | infra | INFO | pool-5-thread-7 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_10000000_11000000_1623178457.95_create__10250000_10300000_1623178457.99' started 2021-06-08 11:54:18,000 | infra | INFO | pool-5-thread-14 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_10000000_11000000_1623178457.95_create__10250000_10300000_1623178457.99 2021-06-08 11:54:18,000 | infra | INFO | pool-5-thread-14 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_10000000_11000000_1623178457.95_create__10300000_10350000_1623178457.99 2021-06-08 11:54:18,000 | infra | INFO | pool-5-thread-1 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_10000000_11000000_1623178457.95_create__10300000_10350000_1623178457.99' started 2021-06-08 11:54:18,000 | infra | INFO | pool-5-thread-14 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_10000000_11000000_1623178457.95_create__10350000_10400000_1623178457.99 2021-06-08 11:54:18,000 | infra | INFO | pool-5-thread-29 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_10000000_11000000_1623178457.95_create__10350000_10400000_1623178457.99' started 2021-06-08 11:54:18,002 | infra | INFO | pool-5-thread-5 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_10000000_11000000_1623178457.95_create__10400000_10450000_1623178457.99' started 2021-06-08 11:54:18,000 | infra | INFO | pool-5-thread-14 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_10000000_11000000_1623178457.95_create__10400000_10450000_1623178457.99 2021-06-08 11:54:18,003 | infra | INFO | pool-5-thread-14 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_10000000_11000000_1623178457.95_create__10450000_10500000_1623178457.99 2021-06-08 11:54:18,003 | infra | INFO | pool-5-thread-10 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_10000000_11000000_1623178457.95_create__10450000_10500000_1623178457.99' started 2021-06-08 11:54:18,003 | infra | INFO | pool-5-thread-14 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_10000000_11000000_1623178457.95_create__10500000_10550000_1623178457.99 2021-06-08 11:54:18,003 | infra | INFO | pool-5-thread-13 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_10000000_11000000_1623178457.95_create__10500000_10550000_1623178457.99' started 2021-06-08 11:54:18,003 | infra | INFO | pool-5-thread-14 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_10000000_11000000_1623178457.95_create__10550000_10600000_1623178457.99 2021-06-08 11:54:18,003 | infra | INFO | pool-5-thread-9 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_10000000_11000000_1623178457.95_create__10550000_10600000_1623178457.99' started 2021-06-08 11:54:18,006 | infra | INFO | pool-5-thread-14 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_10000000_11000000_1623178457.95_create__10600000_10650000_1623178457.99 2021-06-08 11:54:18,006 | infra | INFO | pool-5-thread-18 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_10000000_11000000_1623178457.95_create__10600000_10650000_1623178457.99' started 2021-06-08 11:54:18,006 | infra | INFO | pool-5-thread-30 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_10000000_11000000_1623178457.95_create__10650000_10700000_1623178457.99' started 2021-06-08 11:54:18,006 | infra | INFO | pool-5-thread-14 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_10000000_11000000_1623178457.95_create__10650000_10700000_1623178457.99 2021-06-08 11:54:18,006 | infra | INFO | pool-5-thread-14 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_10000000_11000000_1623178457.95_create__10700000_10750000_1623178457.99 2021-06-08 11:54:18,006 | infra | INFO | pool-5-thread-4 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_10000000_11000000_1623178457.95_create__10700000_10750000_1623178457.99' started 2021-06-08 11:54:18,006 | infra | INFO | pool-5-thread-14 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_10000000_11000000_1623178457.95_create__10750000_10800000_1623178457.99 2021-06-08 11:54:18,006 | infra | INFO | pool-5-thread-3 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_10000000_11000000_1623178457.95_create__10750000_10800000_1623178457.99' started 2021-06-08 11:54:18,006 | infra | INFO | pool-5-thread-14 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_10000000_11000000_1623178457.95_create__10800000_10850000_1623178457.99 2021-06-08 11:54:18,006 | infra | INFO | pool-5-thread-20 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_10000000_11000000_1623178457.95_create__10800000_10850000_1623178457.99' started 2021-06-08 11:54:18,007 | infra | INFO | pool-5-thread-14 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_10000000_11000000_1623178457.95_create__10850000_10900000_1623178457.99 2021-06-08 11:54:18,007 | infra | INFO | pool-5-thread-12 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_10000000_11000000_1623178457.95_create__10850000_10900000_1623178457.99' started 2021-06-08 11:54:18,007 | infra | INFO | pool-5-thread-6 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_10000000_11000000_1623178457.95_create__10900000_10950000_1623178457.99' started 2021-06-08 11:54:18,007 | infra | INFO | pool-5-thread-14 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_10000000_11000000_1623178457.95_create__10900000_10950000_1623178457.99 2021-06-08 11:54:18,009 | infra | INFO | pool-5-thread-14 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_10000000_11000000_1623178457.95_create__10950000_11000000_1623178457.99 2021-06-08 11:54:18,009 | infra | INFO | pool-5-thread-17 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_10000000_11000000_1623178457.95_create__10950000_11000000_1623178457.99' started 2021-06-08 11:54:18,009 | infra | DEBUG | pool-5-thread-14 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_10000000_11000000_1623178457.95_create__10000000_10050000_1623178457.99 2021-06-08 11:54:18,219 | test | DEBUG | pool-5-thread-26 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:54:18,220 | test | DEBUG | pool-5-thread-26 | [task:print_ep_queue_size:3508] ep_queue_size for default: 0 2021-06-08 11:54:18,221 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:54:18,221 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:54:19,118 | infra | DEBUG | pool-5-thread-24 | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.122 2021-06-08 11:54:19,121 | infra | INFO | pool-5-thread-24 | [task:complete_task:102] Thread 'StatsWaitTask_default_ep_data_write_failed_1623178346.63' completed 2021-06-08 11:54:19,420 | test | DEBUG | pool-5-thread-26 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:54:19,420 | test | DEBUG | pool-5-thread-26 | [task:print_ep_queue_size:3508] ep_queue_size for default: 0 2021-06-08 11:54:19,421 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:54:19,421 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:54:20,686 | test | DEBUG | pool-5-thread-26 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:54:20,686 | test | DEBUG | pool-5-thread-26 | [task:print_ep_queue_size:3508] ep_queue_size for default: 0 2021-06-08 11:54:20,693 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:54:20,693 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:54:22,109 | test | DEBUG | pool-5-thread-26 | [task:record_bucket_ops:3495] Ops rate for 'default': 999.600000 2021-06-08 11:54:22,111 | test | DEBUG | pool-5-thread-26 | [task:print_ep_queue_size:3508] ep_queue_size for default: 0 2021-06-08 11:54:22,112 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:54:22,112 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:54:23,493 | test | DEBUG | pool-5-thread-26 | [task:record_bucket_ops:3495] Ops rate for 'default': 999.600000 2021-06-08 11:54:23,494 | test | DEBUG | pool-5-thread-26 | [task:print_ep_queue_size:3508] ep_queue_size for default: 0 2021-06-08 11:54:23,496 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:54:23,496 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:54:24,693 | test | DEBUG | pool-5-thread-26 | [task:record_bucket_ops:3495] Ops rate for 'default': 999.600000 2021-06-08 11:54:24,693 | test | DEBUG | pool-5-thread-26 | [task:print_ep_queue_size:3508] ep_queue_size for default: 0 2021-06-08 11:54:24,696 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:54:24,696 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:54:25,954 | test | DEBUG | pool-5-thread-26 | [task:record_bucket_ops:3495] Ops rate for 'default': 999.600000 2021-06-08 11:54:25,956 | test | DEBUG | pool-5-thread-26 | [task:print_ep_queue_size:3508] ep_queue_size for default: 0 2021-06-08 11:54:25,957 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:54:25,957 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:54:27,486 | test | DEBUG | pool-5-thread-26 | [task:record_bucket_ops:3495] Ops rate for 'default': 999.600000 2021-06-08 11:54:27,486 | test | DEBUG | pool-5-thread-26 | [task:print_ep_queue_size:3508] ep_queue_size for default: 0 2021-06-08 11:54:27,490 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:54:27,490 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:54:28,927 | test | DEBUG | pool-5-thread-26 | [task:record_bucket_ops:3495] Ops rate for 'default': 999.600000 2021-06-08 11:54:28,927 | test | DEBUG | pool-5-thread-26 | [task:print_ep_queue_size:3508] ep_queue_size for default: 0 2021-06-08 11:54:28,930 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:54:28,930 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:54:30,552 | test | DEBUG | pool-5-thread-26 | [task:record_bucket_ops:3495] Ops rate for 'default': 999.600000 2021-06-08 11:54:30,552 | test | DEBUG | pool-5-thread-26 | [task:print_ep_queue_size:3508] ep_queue_size for default: 0 2021-06-08 11:54:30,555 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:54:30,555 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:54:31,747 | test | DEBUG | pool-5-thread-26 | [task:record_bucket_ops:3495] Ops rate for 'default': 6283.800000 2021-06-08 11:54:31,747 | test | DEBUG | pool-5-thread-26 | [task:print_ep_queue_size:3508] ep_queue_size for default: 0 2021-06-08 11:54:31,749 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:54:31,749 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:54:32,946 | test | DEBUG | pool-5-thread-26 | [task:record_bucket_ops:3495] Ops rate for 'default': 6283.800000 2021-06-08 11:54:32,946 | test | DEBUG | pool-5-thread-26 | [task:print_ep_queue_size:3508] ep_queue_size for default: 0 2021-06-08 11:54:32,947 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:54:32,947 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:54:34,140 | test | DEBUG | pool-5-thread-26 | [task:record_bucket_ops:3495] Ops rate for 'default': 6283.800000 2021-06-08 11:54:34,140 | test | DEBUG | pool-5-thread-26 | [task:print_ep_queue_size:3508] ep_queue_size for default: 0 2021-06-08 11:54:34,141 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:54:34,141 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:54:35,342 | test | DEBUG | pool-5-thread-26 | [task:record_bucket_ops:3495] Ops rate for 'default': 6283.800000 2021-06-08 11:54:35,344 | test | DEBUG | pool-5-thread-26 | [task:print_ep_queue_size:3508] ep_queue_size for default: 0 2021-06-08 11:54:35,345 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:54:35,345 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:54:36,545 | test | DEBUG | pool-5-thread-26 | [task:record_bucket_ops:3495] Ops rate for 'default': 6283.800000 2021-06-08 11:54:36,546 | test | DEBUG | pool-5-thread-26 | [task:print_ep_queue_size:3508] ep_queue_size for default: 0 2021-06-08 11:54:36,546 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:54:36,546 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:54:37,742 | test | DEBUG | pool-5-thread-26 | [task:record_bucket_ops:3495] Ops rate for 'default': 6283.800000 2021-06-08 11:54:37,742 | test | DEBUG | pool-5-thread-26 | [task:print_ep_queue_size:3508] ep_queue_size for default: 0 2021-06-08 11:54:37,743 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:54:37,743 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:54:38,934 | test | DEBUG | pool-5-thread-26 | [task:record_bucket_ops:3495] Ops rate for 'default': 6283.800000 2021-06-08 11:54:38,934 | test | DEBUG | pool-5-thread-26 | [task:print_ep_queue_size:3508] ep_queue_size for default: 0 2021-06-08 11:54:38,936 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:54:38,936 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:54:40,125 | test | DEBUG | pool-5-thread-26 | [task:record_bucket_ops:3495] Ops rate for 'default': 6283.800000 2021-06-08 11:54:40,127 | test | DEBUG | pool-5-thread-26 | [task:print_ep_queue_size:3508] ep_queue_size for default: 0 2021-06-08 11:54:40,128 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:54:40,128 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:54:41,325 | test | DEBUG | pool-5-thread-26 | [task:record_bucket_ops:3495] Ops rate for 'default': 14.600000 2021-06-08 11:54:41,326 | test | DEBUG | pool-5-thread-26 | [task:print_ep_queue_size:3508] ep_queue_size for default: 0 2021-06-08 11:54:41,328 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:54:41,328 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:54:42,517 | test | DEBUG | pool-5-thread-26 | [task:record_bucket_ops:3495] Ops rate for 'default': 14.600000 2021-06-08 11:54:42,517 | test | DEBUG | pool-5-thread-26 | [task:print_ep_queue_size:3508] ep_queue_size for default: 0 2021-06-08 11:54:42,519 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:54:42,519 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:54:43,719 | test | DEBUG | pool-5-thread-26 | [task:record_bucket_ops:3495] Ops rate for 'default': 14.600000 2021-06-08 11:54:43,719 | test | DEBUG | pool-5-thread-26 | [task:print_ep_queue_size:3508] ep_queue_size for default: 0 2021-06-08 11:54:43,720 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:54:43,720 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:54:44,921 | test | DEBUG | pool-5-thread-26 | [task:record_bucket_ops:3495] Ops rate for 'default': 14.600000 2021-06-08 11:54:44,921 | test | DEBUG | pool-5-thread-26 | [task:print_ep_queue_size:3508] ep_queue_size for default: 0 2021-06-08 11:54:44,923 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:54:44,923 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:54:46,117 | test | DEBUG | pool-5-thread-26 | [task:record_bucket_ops:3495] Ops rate for 'default': 14.600000 2021-06-08 11:54:46,117 | test | DEBUG | pool-5-thread-26 | [task:print_ep_queue_size:3508] ep_queue_size for default: 0 2021-06-08 11:54:46,118 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:54:46,118 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:54:47,315 | test | DEBUG | pool-5-thread-26 | [task:record_bucket_ops:3495] Ops rate for 'default': 14.600000 2021-06-08 11:54:47,315 | test | DEBUG | pool-5-thread-26 | [task:print_ep_queue_size:3508] ep_queue_size for default: 0 2021-06-08 11:54:47,316 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:54:47,316 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:54:48,522 | test | DEBUG | pool-5-thread-26 | [task:record_bucket_ops:3495] Ops rate for 'default': 14.600000 2021-06-08 11:54:48,523 | test | DEBUG | pool-5-thread-26 | [task:print_ep_queue_size:3508] ep_queue_size for default: 0 2021-06-08 11:54:48,523 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:54:48,523 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:54:49,720 | test | DEBUG | pool-5-thread-26 | [task:record_bucket_ops:3495] Ops rate for 'default': 14.600000 2021-06-08 11:54:49,720 | test | DEBUG | pool-5-thread-26 | [task:print_ep_queue_size:3508] ep_queue_size for default: 0 2021-06-08 11:54:49,721 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:54:49,721 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:54:50,920 | test | DEBUG | pool-5-thread-26 | [task:record_bucket_ops:3495] Ops rate for 'default': 14.600000 2021-06-08 11:54:50,920 | test | DEBUG | pool-5-thread-26 | [task:print_ep_queue_size:3508] ep_queue_size for default: 0 2021-06-08 11:54:50,921 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:54:50,921 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:54:52,111 | test | DEBUG | pool-5-thread-26 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:54:52,111 | test | DEBUG | pool-5-thread-26 | [task:print_ep_queue_size:3508] ep_queue_size for default: 0 2021-06-08 11:54:52,112 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:54:52,112 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:54:53,316 | test | DEBUG | pool-5-thread-26 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:54:53,316 | test | DEBUG | pool-5-thread-26 | [task:print_ep_queue_size:3508] ep_queue_size for default: 0 2021-06-08 11:54:53,318 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:54:53,318 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:54:54,526 | test | DEBUG | pool-5-thread-26 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:54:54,526 | test | DEBUG | pool-5-thread-26 | [task:print_ep_queue_size:3508] ep_queue_size for default: 0 2021-06-08 11:54:54,528 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:54:54,528 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:54:55,730 | test | DEBUG | pool-5-thread-26 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:54:55,730 | test | DEBUG | pool-5-thread-26 | [task:print_ep_queue_size:3508] ep_queue_size for default: 0 2021-06-08 11:54:55,732 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:54:55,732 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:54:56,923 | test | DEBUG | pool-5-thread-26 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:54:56,923 | test | DEBUG | pool-5-thread-26 | [task:print_ep_queue_size:3508] ep_queue_size for default: 0 2021-06-08 11:54:56,924 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:54:56,924 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:54:58,196 | test | DEBUG | pool-5-thread-26 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:54:58,197 | test | DEBUG | pool-5-thread-26 | [task:print_ep_queue_size:3508] ep_queue_size for default: 0 2021-06-08 11:54:58,200 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:54:58,200 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:54:59,447 | test | DEBUG | pool-5-thread-26 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:54:59,447 | test | DEBUG | pool-5-thread-26 | [task:print_ep_queue_size:3508] ep_queue_size for default: 949204 2021-06-08 11:54:59,448 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:54:59,450 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:55:00,927 | test | DEBUG | pool-5-thread-26 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:55:00,927 | test | DEBUG | pool-5-thread-26 | [task:print_ep_queue_size:3508] ep_queue_size for default: 949204 2021-06-08 11:55:00,930 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:55:00,930 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:55:02,296 | test | DEBUG | pool-5-thread-26 | [task:record_bucket_ops:3495] Ops rate for 'default': 1239.800000 2021-06-08 11:55:02,298 | test | DEBUG | pool-5-thread-26 | [task:print_ep_queue_size:3508] ep_queue_size for default: 949204 2021-06-08 11:55:02,299 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:55:02,301 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:55:03,859 | test | DEBUG | pool-5-thread-26 | [task:record_bucket_ops:3495] Ops rate for 'default': 1239.800000 2021-06-08 11:55:03,861 | test | DEBUG | pool-5-thread-26 | [task:print_ep_queue_size:3508] ep_queue_size for default: 949204 2021-06-08 11:55:03,862 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:55:03,862 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:55:05,272 | test | DEBUG | pool-5-thread-26 | [task:record_bucket_ops:3495] Ops rate for 'default': 1239.800000 2021-06-08 11:55:05,272 | test | DEBUG | pool-5-thread-26 | [task:print_ep_queue_size:3508] ep_queue_size for default: 949204 2021-06-08 11:55:05,275 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:55:05,275 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:55:08,595 | test | DEBUG | pool-5-thread-26 | [task:record_bucket_ops:3495] Ops rate for 'default': 5792.131399 2021-06-08 11:55:08,595 | test | DEBUG | pool-5-thread-26 | [task:print_ep_queue_size:3508] ep_queue_size for default: 961812 2021-06-08 11:55:08,598 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:55:08,598 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:55:09,801 | test | DEBUG | pool-5-thread-26 | [task:record_bucket_ops:3495] Ops rate for 'default': 5792.131399 2021-06-08 11:55:09,801 | test | DEBUG | pool-5-thread-26 | [task:print_ep_queue_size:3508] ep_queue_size for default: 961812 2021-06-08 11:55:09,802 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:55:09,802 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:55:11,095 | test | DEBUG | pool-5-thread-26 | [task:record_bucket_ops:3495] Ops rate for 'default': 5792.131399 2021-06-08 11:55:11,098 | test | DEBUG | pool-5-thread-26 | [task:print_ep_queue_size:3508] ep_queue_size for default: 961812 2021-06-08 11:55:11,102 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:55:11,102 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:55:12,969 | test | DEBUG | pool-5-thread-26 | [task:record_bucket_ops:3495] Ops rate for 'default': 9944.231399 2021-06-08 11:55:12,970 | test | DEBUG | pool-5-thread-26 | [task:print_ep_queue_size:3508] ep_queue_size for default: 961812 2021-06-08 11:55:12,973 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:55:12,973 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:55:14,285 | test | DEBUG | pool-5-thread-26 | [task:record_bucket_ops:3495] Ops rate for 'default': 9944.231399 2021-06-08 11:55:14,285 | test | DEBUG | pool-5-thread-26 | [task:print_ep_queue_size:3508] ep_queue_size for default: 961812 2021-06-08 11:55:14,286 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:55:14,288 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:55:15,490 | test | DEBUG | pool-5-thread-26 | [task:record_bucket_ops:3495] Ops rate for 'default': 9944.231399 2021-06-08 11:55:15,490 | test | DEBUG | pool-5-thread-26 | [task:print_ep_queue_size:3508] ep_queue_size for default: 961812 2021-06-08 11:55:15,493 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:55:15,493 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:55:16,742 | test | DEBUG | pool-5-thread-26 | [task:record_bucket_ops:3495] Ops rate for 'default': 10529.800000 2021-06-08 11:55:16,743 | test | DEBUG | pool-5-thread-26 | [task:print_ep_queue_size:3508] ep_queue_size for default: 961812 2021-06-08 11:55:16,744 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:55:16,746 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:55:18,134 | test | DEBUG | pool-5-thread-26 | [task:record_bucket_ops:3495] Ops rate for 'default': 10529.800000 2021-06-08 11:55:18,134 | test | DEBUG | pool-5-thread-26 | [task:print_ep_queue_size:3508] ep_queue_size for default: 979061 2021-06-08 11:55:18,138 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:55:18,138 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:55:19,355 | test | DEBUG | pool-5-thread-26 | [task:record_bucket_ops:3495] Ops rate for 'default': 10529.800000 2021-06-08 11:55:19,355 | test | DEBUG | pool-5-thread-26 | [task:print_ep_queue_size:3508] ep_queue_size for default: 979061 2021-06-08 11:55:19,358 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:55:19,358 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:55:20,566 | test | DEBUG | pool-5-thread-26 | [task:record_bucket_ops:3495] Ops rate for 'default': 10529.800000 2021-06-08 11:55:20,566 | test | DEBUG | pool-5-thread-26 | [task:print_ep_queue_size:3508] ep_queue_size for default: 979061 2021-06-08 11:55:20,569 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:55:20,569 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:55:21,819 | test | DEBUG | pool-5-thread-26 | [task:record_bucket_ops:3495] Ops rate for 'default': 10905.000000 2021-06-08 11:55:21,819 | test | DEBUG | pool-5-thread-26 | [task:print_ep_queue_size:3508] ep_queue_size for default: 979061 2021-06-08 11:55:21,822 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:55:21,822 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:55:23,204 | test | DEBUG | pool-5-thread-26 | [task:record_bucket_ops:3495] Ops rate for 'default': 10905.000000 2021-06-08 11:55:23,204 | test | DEBUG | pool-5-thread-26 | [task:print_ep_queue_size:3508] ep_queue_size for default: 979061 2021-06-08 11:55:23,207 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:55:23,207 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:55:24,611 | test | DEBUG | pool-5-thread-26 | [task:record_bucket_ops:3495] Ops rate for 'default': 10905.000000 2021-06-08 11:55:24,611 | test | DEBUG | pool-5-thread-26 | [task:print_ep_queue_size:3508] ep_queue_size for default: 979061 2021-06-08 11:55:24,615 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:55:24,615 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:55:25,851 | test | DEBUG | pool-5-thread-26 | [task:record_bucket_ops:3495] Ops rate for 'default': 10905.000000 2021-06-08 11:55:25,851 | test | DEBUG | pool-5-thread-26 | [task:print_ep_queue_size:3508] ep_queue_size for default: 979061 2021-06-08 11:55:25,854 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:55:25,854 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:55:27,069 | test | DEBUG | pool-5-thread-26 | [task:record_bucket_ops:3495] Ops rate for 'default': 10905.000000 2021-06-08 11:55:27,069 | test | DEBUG | pool-5-thread-26 | [task:print_ep_queue_size:3508] ep_queue_size for default: 979061 2021-06-08 11:55:27,072 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:55:27,072 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:55:27,382 | infra | INFO | pool-5-thread-9 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_10000000_11000000_1623178457.95_create__10550000_10600000_1623178457.99' completed 2021-06-08 11:55:27,579 | infra | INFO | pool-5-thread-12 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_10000000_11000000_1623178457.95_create__10850000_10900000_1623178457.99' completed 2021-06-08 11:55:28,239 | infra | INFO | pool-5-thread-5 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_10000000_11000000_1623178457.95_create__10400000_10450000_1623178457.99' completed 2021-06-08 11:55:28,345 | test | DEBUG | pool-5-thread-26 | [task:record_bucket_ops:3495] Ops rate for 'default': 10905.000000 2021-06-08 11:55:28,346 | test | DEBUG | pool-5-thread-26 | [task:print_ep_queue_size:3508] ep_queue_size for default: 979061 2021-06-08 11:55:28,348 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:55:28,348 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:55:28,704 | infra | INFO | pool-5-thread-28 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_10000000_11000000_1623178457.95_create__10200000_10250000_1623178457.99' completed 2021-06-08 11:55:28,707 | infra | INFO | pool-5-thread-30 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_10000000_11000000_1623178457.95_create__10650000_10700000_1623178457.99' completed 2021-06-08 11:55:28,857 | infra | INFO | pool-5-thread-13 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_10000000_11000000_1623178457.95_create__10500000_10550000_1623178457.99' completed 2021-06-08 11:55:28,953 | infra | INFO | pool-5-thread-20 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_10000000_11000000_1623178457.95_create__10800000_10850000_1623178457.99' completed 2021-06-08 11:55:29,180 | infra | INFO | pool-5-thread-11 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_10000000_11000000_1623178457.95_create__10150000_10200000_1623178457.99' completed 2021-06-08 11:55:29,206 | infra | INFO | pool-5-thread-3 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_10000000_11000000_1623178457.95_create__10750000_10800000_1623178457.99' completed 2021-06-08 11:55:29,247 | infra | INFO | pool-5-thread-7 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_10000000_11000000_1623178457.95_create__10250000_10300000_1623178457.99' completed 2021-06-08 11:55:29,256 | infra | INFO | pool-5-thread-15 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_10000000_11000000_1623178457.95_create__10000000_10050000_1623178457.99' completed 2021-06-08 11:55:29,256 | infra | DEBUG | pool-5-thread-14 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_10000000_11000000_1623178457.95_create__10000000_10050000_1623178457.99 are 50000 2021-06-08 11:55:29,256 | infra | DEBUG | pool-5-thread-14 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_10000000_11000000_1623178457.95_create__10050000_10100000_1623178457.99 2021-06-08 11:55:29,270 | infra | INFO | pool-5-thread-2 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_10000000_11000000_1623178457.95_create__10100000_10150000_1623178457.99' completed 2021-06-08 11:55:29,273 | infra | INFO | pool-5-thread-6 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_10000000_11000000_1623178457.95_create__10900000_10950000_1623178457.99' completed 2021-06-08 11:55:29,276 | infra | INFO | pool-5-thread-29 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_10000000_11000000_1623178457.95_create__10350000_10400000_1623178457.99' completed 2021-06-08 11:55:29,286 | infra | INFO | pool-5-thread-22 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_10000000_11000000_1623178457.95_create__10050000_10100000_1623178457.99' completed 2021-06-08 11:55:29,286 | infra | DEBUG | pool-5-thread-14 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_10000000_11000000_1623178457.95_create__10050000_10100000_1623178457.99 are 50000 2021-06-08 11:55:29,288 | infra | DEBUG | pool-5-thread-14 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_10000000_11000000_1623178457.95_create__10100000_10150000_1623178457.99 2021-06-08 11:55:29,288 | infra | DEBUG | pool-5-thread-14 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_10000000_11000000_1623178457.95_create__10100000_10150000_1623178457.99 are 50000 2021-06-08 11:55:29,288 | infra | DEBUG | pool-5-thread-14 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_10000000_11000000_1623178457.95_create__10150000_10200000_1623178457.99 2021-06-08 11:55:29,288 | infra | DEBUG | pool-5-thread-14 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_10000000_11000000_1623178457.95_create__10150000_10200000_1623178457.99 are 50000 2021-06-08 11:55:29,288 | infra | DEBUG | pool-5-thread-14 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_10000000_11000000_1623178457.95_create__10200000_10250000_1623178457.99 2021-06-08 11:55:29,288 | infra | DEBUG | pool-5-thread-14 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_10000000_11000000_1623178457.95_create__10200000_10250000_1623178457.99 are 50000 2021-06-08 11:55:29,288 | infra | DEBUG | pool-5-thread-14 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_10000000_11000000_1623178457.95_create__10250000_10300000_1623178457.99 2021-06-08 11:55:29,288 | infra | DEBUG | pool-5-thread-14 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_10000000_11000000_1623178457.95_create__10250000_10300000_1623178457.99 are 50000 2021-06-08 11:55:29,289 | infra | DEBUG | pool-5-thread-14 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_10000000_11000000_1623178457.95_create__10300000_10350000_1623178457.99 2021-06-08 11:55:29,325 | infra | INFO | pool-5-thread-17 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_10000000_11000000_1623178457.95_create__10950000_11000000_1623178457.99' completed 2021-06-08 11:55:29,331 | infra | INFO | pool-5-thread-4 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_10000000_11000000_1623178457.95_create__10700000_10750000_1623178457.99' completed 2021-06-08 11:55:29,342 | infra | INFO | pool-5-thread-18 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_10000000_11000000_1623178457.95_create__10600000_10650000_1623178457.99' completed 2021-06-08 11:55:29,369 | infra | INFO | pool-5-thread-1 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_10000000_11000000_1623178457.95_create__10300000_10350000_1623178457.99' completed 2021-06-08 11:55:29,369 | infra | DEBUG | pool-5-thread-14 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_10000000_11000000_1623178457.95_create__10300000_10350000_1623178457.99 are 50000 2021-06-08 11:55:29,369 | infra | DEBUG | pool-5-thread-14 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_10000000_11000000_1623178457.95_create__10350000_10400000_1623178457.99 2021-06-08 11:55:29,369 | infra | DEBUG | pool-5-thread-14 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_10000000_11000000_1623178457.95_create__10350000_10400000_1623178457.99 are 50000 2021-06-08 11:55:29,369 | infra | DEBUG | pool-5-thread-14 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_10000000_11000000_1623178457.95_create__10400000_10450000_1623178457.99 2021-06-08 11:55:29,371 | infra | DEBUG | pool-5-thread-14 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_10000000_11000000_1623178457.95_create__10400000_10450000_1623178457.99 are 50000 2021-06-08 11:55:29,371 | infra | DEBUG | pool-5-thread-14 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_10000000_11000000_1623178457.95_create__10450000_10500000_1623178457.99 2021-06-08 11:55:29,437 | infra | INFO | pool-5-thread-10 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_10000000_11000000_1623178457.95_create__10450000_10500000_1623178457.99' completed 2021-06-08 11:55:29,437 | infra | DEBUG | pool-5-thread-14 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_10000000_11000000_1623178457.95_create__10450000_10500000_1623178457.99 are 50000 2021-06-08 11:55:29,437 | infra | DEBUG | pool-5-thread-14 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_10000000_11000000_1623178457.95_create__10500000_10550000_1623178457.99 2021-06-08 11:55:29,437 | infra | DEBUG | pool-5-thread-14 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_10000000_11000000_1623178457.95_create__10500000_10550000_1623178457.99 are 50000 2021-06-08 11:55:29,438 | infra | DEBUG | pool-5-thread-14 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_10000000_11000000_1623178457.95_create__10550000_10600000_1623178457.99 2021-06-08 11:55:29,438 | infra | DEBUG | pool-5-thread-14 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_10000000_11000000_1623178457.95_create__10550000_10600000_1623178457.99 are 50000 2021-06-08 11:55:29,438 | infra | DEBUG | pool-5-thread-14 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_10000000_11000000_1623178457.95_create__10600000_10650000_1623178457.99 2021-06-08 11:55:29,438 | infra | DEBUG | pool-5-thread-14 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_10000000_11000000_1623178457.95_create__10600000_10650000_1623178457.99 are 50000 2021-06-08 11:55:29,438 | infra | DEBUG | pool-5-thread-14 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_10000000_11000000_1623178457.95_create__10650000_10700000_1623178457.99 2021-06-08 11:55:29,438 | infra | DEBUG | pool-5-thread-14 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_10000000_11000000_1623178457.95_create__10650000_10700000_1623178457.99 are 50000 2021-06-08 11:55:29,438 | infra | DEBUG | pool-5-thread-14 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_10000000_11000000_1623178457.95_create__10700000_10750000_1623178457.99 2021-06-08 11:55:29,438 | infra | DEBUG | pool-5-thread-14 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_10000000_11000000_1623178457.95_create__10700000_10750000_1623178457.99 are 50000 2021-06-08 11:55:29,438 | infra | DEBUG | pool-5-thread-14 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_10000000_11000000_1623178457.95_create__10750000_10800000_1623178457.99 2021-06-08 11:55:29,438 | infra | DEBUG | pool-5-thread-14 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_10000000_11000000_1623178457.95_create__10750000_10800000_1623178457.99 are 50000 2021-06-08 11:55:29,440 | infra | DEBUG | pool-5-thread-14 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_10000000_11000000_1623178457.95_create__10800000_10850000_1623178457.99 2021-06-08 11:55:29,440 | infra | DEBUG | pool-5-thread-14 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_10000000_11000000_1623178457.95_create__10800000_10850000_1623178457.99 are 50000 2021-06-08 11:55:29,440 | infra | DEBUG | pool-5-thread-14 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_10000000_11000000_1623178457.95_create__10850000_10900000_1623178457.99 2021-06-08 11:55:29,440 | infra | DEBUG | pool-5-thread-14 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_10000000_11000000_1623178457.95_create__10850000_10900000_1623178457.99 are 50000 2021-06-08 11:55:29,440 | infra | DEBUG | pool-5-thread-14 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_10000000_11000000_1623178457.95_create__10900000_10950000_1623178457.99 2021-06-08 11:55:29,440 | infra | DEBUG | pool-5-thread-14 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_10000000_11000000_1623178457.95_create__10900000_10950000_1623178457.99 are 50000 2021-06-08 11:55:29,440 | infra | DEBUG | pool-5-thread-14 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_10000000_11000000_1623178457.95_create__10950000_11000000_1623178457.99 2021-06-08 11:55:29,440 | infra | DEBUG | pool-5-thread-14 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_10000000_11000000_1623178457.95_create__10950000_11000000_1623178457.99 are 50000 2021-06-08 11:55:29,440 | infra | DEBUG | pool-5-thread-14 | [task_manager:get_task_result:26] Getting task result for PrintBucketStats_default_1623178457.99 2021-06-08 11:55:29,546 | test | DEBUG | pool-5-thread-26 | [task:record_bucket_ops:3495] Ops rate for 'default': 10905.000000 2021-06-08 11:55:29,546 | test | DEBUG | pool-5-thread-26 | [task:print_ep_queue_size:3508] ep_queue_size for default: 979061 2021-06-08 11:55:29,548 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:55:29,548 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:55:30,549 | test | INFO | pool-5-thread-26 | [table_view:display:72] Ops trend for bucket 'default' +--------+----------------------------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +--------+----------------------------------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | ***.........*******................................................*******X | 6283.800 | | 14.600 | ********X | 14.600 | | 0.000 | ********...........***..........................................***......................................***.....****....******X | 10905.000 | +--------+----------------------------------------------------------------------------------------------------------------------------------+-----------+ 2021-06-08 11:55:30,549 | infra | INFO | pool-5-thread-26 | [task:complete_task:102] Thread 'PrintBucketStats_default_1623178457.99' completed 2021-06-08 11:55:30,549 | infra | DEBUG | pool-5-thread-14 | [task:call:1663] ========= Tasks in loadgen pool======= 2021-06-08 11:55:30,549 | infra | WARNING | pool-5-thread-14 | [task_manager:print_tasks_in_pool:83] Task 'LoadDocsGen_default__default__default_create_10000000_11000000_1623178457.95' not completed 2021-06-08 11:55:30,549 | infra | DEBUG | pool-5-thread-14 | [task:call:1665] ====================================== 2021-06-08 11:55:30,549 | infra | DEBUG | pool-5-thread-14 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_10000000_11000000_1623178457.95_create__10000000_10050000_1623178457.99' complete. Loaded 50000 items 2021-06-08 11:55:30,549 | infra | DEBUG | pool-5-thread-14 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_10000000_11000000_1623178457.95_create__10050000_10100000_1623178457.99' complete. Loaded 50000 items 2021-06-08 11:55:30,549 | infra | DEBUG | pool-5-thread-14 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_10000000_11000000_1623178457.95_create__10100000_10150000_1623178457.99' complete. Loaded 50000 items 2021-06-08 11:55:30,549 | infra | DEBUG | pool-5-thread-14 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_10000000_11000000_1623178457.95_create__10150000_10200000_1623178457.99' complete. Loaded 50000 items 2021-06-08 11:55:30,551 | infra | DEBUG | pool-5-thread-14 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_10000000_11000000_1623178457.95_create__10200000_10250000_1623178457.99' complete. Loaded 50000 items 2021-06-08 11:55:30,551 | infra | DEBUG | pool-5-thread-14 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_10000000_11000000_1623178457.95_create__10250000_10300000_1623178457.99' complete. Loaded 50000 items 2021-06-08 11:55:30,551 | infra | DEBUG | pool-5-thread-14 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_10000000_11000000_1623178457.95_create__10300000_10350000_1623178457.99' complete. Loaded 50000 items 2021-06-08 11:55:30,551 | infra | DEBUG | pool-5-thread-14 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_10000000_11000000_1623178457.95_create__10350000_10400000_1623178457.99' complete. Loaded 50000 items 2021-06-08 11:55:30,551 | infra | DEBUG | pool-5-thread-14 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_10000000_11000000_1623178457.95_create__10400000_10450000_1623178457.99' complete. Loaded 50000 items 2021-06-08 11:55:30,551 | infra | DEBUG | pool-5-thread-14 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_10000000_11000000_1623178457.95_create__10450000_10500000_1623178457.99' complete. Loaded 50000 items 2021-06-08 11:55:30,551 | infra | DEBUG | pool-5-thread-14 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_10000000_11000000_1623178457.95_create__10500000_10550000_1623178457.99' complete. Loaded 50000 items 2021-06-08 11:55:30,551 | infra | DEBUG | pool-5-thread-14 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_10000000_11000000_1623178457.95_create__10550000_10600000_1623178457.99' complete. Loaded 50000 items 2021-06-08 11:55:30,551 | infra | DEBUG | pool-5-thread-14 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_10000000_11000000_1623178457.95_create__10600000_10650000_1623178457.99' complete. Loaded 50000 items 2021-06-08 11:55:30,552 | infra | DEBUG | pool-5-thread-14 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_10000000_11000000_1623178457.95_create__10650000_10700000_1623178457.99' complete. Loaded 50000 items 2021-06-08 11:55:30,552 | infra | DEBUG | pool-5-thread-14 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_10000000_11000000_1623178457.95_create__10700000_10750000_1623178457.99' complete. Loaded 50000 items 2021-06-08 11:55:30,552 | infra | DEBUG | pool-5-thread-14 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_10000000_11000000_1623178457.95_create__10750000_10800000_1623178457.99' complete. Loaded 50000 items 2021-06-08 11:55:30,552 | infra | DEBUG | pool-5-thread-14 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_10000000_11000000_1623178457.95_create__10800000_10850000_1623178457.99' complete. Loaded 50000 items 2021-06-08 11:55:30,552 | infra | DEBUG | pool-5-thread-14 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_10000000_11000000_1623178457.95_create__10850000_10900000_1623178457.99' complete. Loaded 50000 items 2021-06-08 11:55:30,552 | infra | DEBUG | pool-5-thread-14 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_10000000_11000000_1623178457.95_create__10900000_10950000_1623178457.99' complete. Loaded 50000 items 2021-06-08 11:55:30,552 | infra | DEBUG | pool-5-thread-14 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_10000000_11000000_1623178457.95_create__10950000_11000000_1623178457.99' complete. Loaded 50000 items 2021-06-08 11:55:30,552 | infra | INFO | pool-5-thread-14 | [task:complete_task:102] Thread 'LoadDocsGen_default__default__default_create_10000000_11000000_1623178457.95' completed 2021-06-08 11:55:30,555 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.125 with username: root, password: couchbase 2021-06-08 11:55:30,635 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 11:55:30,851 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: ls /etc/ 2021-06-08 11:55:30,904 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 11:55:30,904 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: ls /cygdrive/c/Windows 2021-06-08 11:55:30,917 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: uname -m 2021-06-08 11:55:30,928 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: cat /proc/cpuinfo 2021-06-08 11:55:30,940 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: df -Thl 2021-06-08 11:55:30,956 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: cat /proc/meminfo 2021-06-08 11:55:30,967 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: hostname 2021-06-08 11:55:30,979 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: hostname -d 2021-06-08 11:55:30,992 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:559] Starting StatsWaitTask for ep_data_write_failed on bucket default 2021-06-08 11:55:30,992 | infra | INFO | MainThread | [task_manager:add_new_task:23] Added new task: StatsWaitTask_default_ep_data_write_failed_1623178530.99 2021-06-08 11:55:30,992 | infra | INFO | pool-5-thread-23 | [task:start_task:87] Thread 'StatsWaitTask_default_ep_data_write_failed_1623178530.99' started 2021-06-08 11:55:30,993 | infra | DEBUG | pool-5-thread-23 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:55:30,994 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.155 with username: root, password: couchbase 2021-06-08 11:55:31,088 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 11:55:31,109 | test | DEBUG | pool-5-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 11:55:31,109 | test | INFO | pool-5-thread-23 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 11:55:31,299 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: ls /etc/ 2021-06-08 11:55:31,352 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 11:55:31,352 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: ls /cygdrive/c/Windows 2021-06-08 11:55:31,365 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: uname -m 2021-06-08 11:55:31,377 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: cat /proc/cpuinfo 2021-06-08 11:55:31,388 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: df -Thl 2021-06-08 11:55:31,404 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: cat /proc/meminfo 2021-06-08 11:55:31,414 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: hostname 2021-06-08 11:55:31,427 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: hostname -d 2021-06-08 11:55:31,438 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:559] Starting StatsWaitTask for ep_data_write_failed on bucket default 2021-06-08 11:55:31,438 | infra | INFO | MainThread | [task_manager:add_new_task:23] Added new task: StatsWaitTask_default_ep_data_write_failed_1623178531.44 2021-06-08 11:55:31,440 | infra | INFO | pool-5-thread-25 | [task:start_task:87] Thread 'StatsWaitTask_default_ep_data_write_failed_1623178531.44' started 2021-06-08 11:55:31,440 | infra | DEBUG | pool-5-thread-25 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:55:31,443 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.156 with username: root, password: couchbase 2021-06-08 11:55:31,529 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 11:55:31,637 | test | DEBUG | pool-5-thread-25 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 11:55:31,638 | infra | DEBUG | pool-5-thread-25 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.155': '0'} for bucket 'default' 2021-06-08 11:55:31,638 | infra | DEBUG | pool-5-thread-25 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 11:55:31,640 | infra | DEBUG | pool-5-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 11:55:31,640 | infra | DEBUG | pool-5-thread-25 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 11:55:31,733 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: ls /etc/ 2021-06-08 11:55:31,789 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 11:55:31,789 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: ls /cygdrive/c/Windows 2021-06-08 11:55:31,802 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: uname -m 2021-06-08 11:55:31,815 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: cat /proc/cpuinfo 2021-06-08 11:55:31,826 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: df -Thl 2021-06-08 11:55:31,844 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: cat /proc/meminfo 2021-06-08 11:55:31,855 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: hostname 2021-06-08 11:55:31,868 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: hostname -d 2021-06-08 11:55:31,881 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:559] Starting StatsWaitTask for ep_data_write_failed on bucket default 2021-06-08 11:55:31,881 | infra | INFO | MainThread | [task_manager:add_new_task:23] Added new task: StatsWaitTask_default_ep_data_write_failed_1623178531.88 2021-06-08 11:55:31,881 | infra | INFO | pool-5-thread-16 | [task:start_task:87] Thread 'StatsWaitTask_default_ep_data_write_failed_1623178531.88' started 2021-06-08 11:55:31,882 | infra | DEBUG | pool-5-thread-16 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:55:31,884 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.122 with username: root, password: couchbase 2021-06-08 11:55:31,963 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 11:55:32,174 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: ls /etc/ 2021-06-08 11:55:32,233 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 11:55:32,233 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: ls /cygdrive/c/Windows 2021-06-08 11:55:32,250 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: uname -m 2021-06-08 11:55:32,266 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: cat /proc/cpuinfo 2021-06-08 11:55:32,280 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: df -Thl 2021-06-08 11:55:32,296 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: cat /proc/meminfo 2021-06-08 11:55:32,312 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: hostname 2021-06-08 11:55:32,323 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: hostname -d 2021-06-08 11:55:32,336 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:559] Starting StatsWaitTask for ep_data_write_failed on bucket default 2021-06-08 11:55:32,338 | infra | INFO | MainThread | [task_manager:add_new_task:23] Added new task: StatsWaitTask_default_ep_data_write_failed_1623178532.34 2021-06-08 11:55:32,338 | infra | INFO | pool-5-thread-21 | [task:start_task:87] Thread 'StatsWaitTask_default_ep_data_write_failed_1623178532.34' started 2021-06-08 11:55:32,338 | infra | DEBUG | MainThread | [task_manager:get_task_result:26] Getting task result for StatsWaitTask_default_ep_data_write_failed_1623178530.99 2021-06-08 11:55:32,338 | infra | DEBUG | pool-5-thread-21 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:55:32,459 | test | DEBUG | pool-5-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 11:55:32,459 | test | INFO | pool-5-thread-21 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1010, in main() File "/opt/couchbase/lib/python/cbstats", line 1007, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 49, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-06-08 11:55:36,109 | infra | DEBUG | pool-5-thread-23 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:55:36,227 | test | DEBUG | pool-5-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 11:55:36,227 | test | INFO | pool-5-thread-23 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 11:55:36,640 | infra | DEBUG | pool-5-thread-25 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:55:36,832 | test | DEBUG | pool-5-thread-25 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 11:55:36,832 | infra | DEBUG | pool-5-thread-25 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.155': '0'} for bucket 'default' 2021-06-08 11:55:36,834 | infra | DEBUG | pool-5-thread-25 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 11:55:36,835 | infra | DEBUG | pool-5-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 11:55:36,835 | infra | DEBUG | pool-5-thread-25 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 11:55:37,459 | infra | DEBUG | pool-5-thread-21 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:55:37,576 | test | DEBUG | pool-5-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 11:55:37,576 | test | INFO | pool-5-thread-21 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 11:55:41,229 | infra | DEBUG | pool-5-thread-23 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:55:41,348 | test | DEBUG | pool-5-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 11:55:41,348 | test | INFO | pool-5-thread-23 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 11:55:41,835 | infra | DEBUG | pool-5-thread-25 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:55:42,032 | test | DEBUG | pool-5-thread-25 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 11:55:42,032 | infra | DEBUG | pool-5-thread-25 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.155': '0'} for bucket 'default' 2021-06-08 11:55:42,032 | infra | DEBUG | pool-5-thread-25 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 11:55:42,035 | infra | DEBUG | pool-5-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 11:55:42,035 | infra | DEBUG | pool-5-thread-25 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 11:55:42,576 | infra | DEBUG | pool-5-thread-21 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:55:42,789 | test | DEBUG | pool-5-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 11:55:42,789 | test | INFO | pool-5-thread-21 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1010, in main() File "/opt/couchbase/lib/python/cbstats", line 1007, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 49, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 79, in g mc.sasl_auth_plain(username, password) File "/opt/couchbase/lib/python/mc_bin_client.py", line 488, in sasl_auth_plain return self.sasl_auth_start('PLAIN', '\0'.join([foruser, user, password])) File "/opt/couchbase/lib/python/mc_bin_client.py", line 484, in sasl_auth_start return self._doCmd(memcacheConstants.CMD_SASL_AUTH, mech, data) File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 281, in _handleKeyedResponse cmd, errcode, opaque, cas, keylen, extralen, rv = self._recvMsg() File "/opt/couchbase/lib/python/mc_bin_client.py", line 252, in _recvMsg raise EOFError("Got empty data (remote died?).") EOFError: Got empty data (remote died?). 2021-06-08 11:55:46,348 | infra | DEBUG | pool-5-thread-23 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:55:46,467 | test | DEBUG | pool-5-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 11:55:46,467 | test | INFO | pool-5-thread-23 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 11:55:47,035 | infra | DEBUG | pool-5-thread-25 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:55:47,230 | test | DEBUG | pool-5-thread-25 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 11:55:47,230 | infra | DEBUG | pool-5-thread-25 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.155': '0'} for bucket 'default' 2021-06-08 11:55:47,230 | infra | DEBUG | pool-5-thread-25 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 11:55:47,232 | infra | DEBUG | pool-5-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 11:55:47,232 | infra | DEBUG | pool-5-thread-25 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 11:55:47,789 | infra | DEBUG | pool-5-thread-21 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:55:47,908 | test | DEBUG | pool-5-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 11:55:47,908 | test | INFO | pool-5-thread-21 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 11:55:51,467 | infra | DEBUG | pool-5-thread-23 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:55:51,584 | test | DEBUG | pool-5-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 11:55:51,584 | test | INFO | pool-5-thread-23 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 11:55:52,233 | infra | DEBUG | pool-5-thread-25 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:55:52,437 | test | DEBUG | pool-5-thread-25 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 11:55:52,437 | infra | DEBUG | pool-5-thread-25 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.155': '0'} for bucket 'default' 2021-06-08 11:55:52,437 | infra | DEBUG | pool-5-thread-25 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 11:55:52,440 | infra | DEBUG | pool-5-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 11:55:52,440 | infra | DEBUG | pool-5-thread-25 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 11:55:52,908 | infra | DEBUG | pool-5-thread-21 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:55:53,029 | test | DEBUG | pool-5-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 11:55:53,029 | test | INFO | pool-5-thread-21 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 11:55:56,585 | infra | DEBUG | pool-5-thread-23 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:55:56,703 | test | DEBUG | pool-5-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 11:55:56,703 | test | INFO | pool-5-thread-23 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 11:55:57,440 | infra | DEBUG | pool-5-thread-25 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:55:57,634 | test | DEBUG | pool-5-thread-25 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 11:55:57,634 | infra | DEBUG | pool-5-thread-25 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.155': '0'} for bucket 'default' 2021-06-08 11:55:57,634 | infra | DEBUG | pool-5-thread-25 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 11:55:57,637 | infra | DEBUG | pool-5-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 11:55:57,637 | infra | DEBUG | pool-5-thread-25 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 11:55:58,029 | infra | DEBUG | pool-5-thread-21 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:55:58,144 | test | DEBUG | pool-5-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 11:55:58,144 | test | INFO | pool-5-thread-21 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 11:56:01,703 | infra | DEBUG | pool-5-thread-23 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:56:01,822 | test | DEBUG | pool-5-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 11:56:01,822 | test | INFO | pool-5-thread-23 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 11:56:02,035 | test | DEBUG | pool-5-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 11:56:02,036 | test | INFO | pool-5-thread-16 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 11:56:02,637 | infra | DEBUG | pool-5-thread-25 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:56:02,844 | test | DEBUG | pool-5-thread-25 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 11:56:02,844 | infra | DEBUG | pool-5-thread-25 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.155': '0'} for bucket 'default' 2021-06-08 11:56:02,844 | infra | DEBUG | pool-5-thread-25 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 11:56:02,846 | infra | DEBUG | pool-5-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 11:56:02,846 | infra | DEBUG | pool-5-thread-25 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 11:56:03,144 | infra | DEBUG | pool-5-thread-21 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:56:03,262 | test | DEBUG | pool-5-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 11:56:03,262 | test | INFO | pool-5-thread-21 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 11:56:06,822 | infra | DEBUG | pool-5-thread-23 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:56:06,937 | test | DEBUG | pool-5-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 11:56:06,937 | test | INFO | pool-5-thread-23 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 11:56:07,036 | infra | DEBUG | pool-5-thread-16 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:56:07,846 | infra | DEBUG | pool-5-thread-25 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:56:08,042 | test | DEBUG | pool-5-thread-25 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 11:56:08,042 | infra | DEBUG | pool-5-thread-25 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.155': '0'} for bucket 'default' 2021-06-08 11:56:08,042 | infra | DEBUG | pool-5-thread-25 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 11:56:08,045 | infra | DEBUG | pool-5-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 11:56:08,045 | infra | DEBUG | pool-5-thread-25 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 11:56:08,263 | infra | DEBUG | pool-5-thread-21 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:56:08,380 | test | DEBUG | pool-5-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 11:56:08,381 | test | INFO | pool-5-thread-21 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 11:56:09,753 | test | DEBUG | pool-5-thread-16 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 11:56:09,753 | infra | DEBUG | pool-5-thread-16 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.156': '0'} for bucket 'default' 2021-06-08 11:56:09,753 | infra | DEBUG | pool-5-thread-16 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 11:56:09,755 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 11:56:09,755 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 11:56:11,938 | infra | DEBUG | pool-5-thread-23 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:56:12,056 | test | DEBUG | pool-5-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 11:56:12,056 | test | INFO | pool-5-thread-23 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 11:56:13,045 | infra | DEBUG | pool-5-thread-25 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:56:13,240 | test | DEBUG | pool-5-thread-25 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 11:56:13,240 | infra | DEBUG | pool-5-thread-25 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.155': '0'} for bucket 'default' 2021-06-08 11:56:13,240 | infra | DEBUG | pool-5-thread-25 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 11:56:13,243 | infra | DEBUG | pool-5-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 11:56:13,243 | infra | DEBUG | pool-5-thread-25 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 11:56:13,381 | infra | DEBUG | pool-5-thread-21 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:56:13,500 | test | DEBUG | pool-5-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 11:56:13,500 | test | INFO | pool-5-thread-21 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 11:56:14,755 | infra | DEBUG | pool-5-thread-16 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:56:17,056 | infra | DEBUG | pool-5-thread-23 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:56:17,173 | test | DEBUG | pool-5-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 11:56:17,173 | test | INFO | pool-5-thread-23 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 11:56:18,243 | infra | DEBUG | pool-5-thread-25 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:56:18,444 | test | DEBUG | pool-5-thread-25 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 11:56:18,444 | infra | DEBUG | pool-5-thread-25 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.155': '0'} for bucket 'default' 2021-06-08 11:56:18,444 | infra | DEBUG | pool-5-thread-25 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 11:56:18,446 | infra | DEBUG | pool-5-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 11:56:18,446 | infra | DEBUG | pool-5-thread-25 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 11:56:18,500 | infra | DEBUG | pool-5-thread-21 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:56:18,617 | test | DEBUG | pool-5-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 11:56:18,617 | test | INFO | pool-5-thread-21 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 11:56:22,173 | test | DEBUG | pool-5-thread-23 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 11:56:22,174 | infra | DEBUG | pool-5-thread-23 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {} for bucket 'default' 2021-06-08 11:56:22,174 | infra | DEBUG | pool-5-thread-23 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 11:56:22,174 | infra | DEBUG | pool-5-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 11:56:22,176 | infra | DEBUG | pool-5-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 11:56:23,447 | infra | DEBUG | pool-5-thread-25 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:56:23,618 | test | DEBUG | pool-5-thread-21 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 11:56:23,618 | infra | DEBUG | pool-5-thread-21 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {} for bucket 'default' 2021-06-08 11:56:23,618 | infra | DEBUG | pool-5-thread-21 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 11:56:23,621 | infra | DEBUG | pool-5-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 11:56:23,621 | infra | DEBUG | pool-5-thread-21 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 11:56:23,642 | test | DEBUG | pool-5-thread-25 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 11:56:23,642 | infra | DEBUG | pool-5-thread-25 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.155': '0'} for bucket 'default' 2021-06-08 11:56:23,642 | infra | DEBUG | pool-5-thread-25 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 11:56:23,645 | infra | DEBUG | pool-5-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 11:56:23,645 | infra | DEBUG | pool-5-thread-25 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 11:56:24,053 | test | DEBUG | pool-5-thread-16 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 11:56:24,055 | infra | DEBUG | pool-5-thread-16 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.156': '0'} for bucket 'default' 2021-06-08 11:56:24,055 | infra | DEBUG | pool-5-thread-16 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 11:56:24,055 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 11:56:24,056 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 11:56:27,176 | infra | DEBUG | pool-5-thread-23 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:56:27,292 | test | DEBUG | pool-5-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 11:56:27,292 | test | INFO | pool-5-thread-23 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 11:56:28,621 | infra | DEBUG | pool-5-thread-21 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:56:28,645 | infra | DEBUG | pool-5-thread-25 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:56:28,739 | test | DEBUG | pool-5-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 11:56:28,740 | test | INFO | pool-5-thread-21 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 11:56:28,844 | test | DEBUG | pool-5-thread-25 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 11:56:28,844 | infra | DEBUG | pool-5-thread-25 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.155': '0'} for bucket 'default' 2021-06-08 11:56:28,844 | infra | DEBUG | pool-5-thread-25 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 11:56:28,845 | infra | DEBUG | pool-5-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 11:56:28,845 | infra | DEBUG | pool-5-thread-25 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 11:56:29,056 | infra | DEBUG | pool-5-thread-16 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:56:32,292 | infra | DEBUG | pool-5-thread-23 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:56:32,421 | test | DEBUG | pool-5-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 11:56:32,421 | test | INFO | pool-5-thread-23 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1010, in main() File "/opt/couchbase/lib/python/cbstats", line 1007, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 49, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-06-08 11:56:33,740 | infra | DEBUG | pool-5-thread-21 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:56:33,845 | infra | DEBUG | pool-5-thread-25 | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.155 2021-06-08 11:56:33,846 | infra | INFO | pool-5-thread-25 | [task:complete_task:102] Thread 'StatsWaitTask_default_ep_data_write_failed_1623178531.44' completed 2021-06-08 11:56:33,858 | test | DEBUG | pool-5-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 11:56:33,858 | test | INFO | pool-5-thread-21 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 11:56:37,421 | infra | DEBUG | pool-5-thread-23 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:56:37,540 | test | DEBUG | pool-5-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 11:56:37,540 | test | INFO | pool-5-thread-23 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 11:56:38,858 | infra | DEBUG | pool-5-thread-21 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:56:38,974 | test | DEBUG | pool-5-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 11:56:38,974 | test | INFO | pool-5-thread-21 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 11:56:39,739 | test | DEBUG | pool-5-thread-16 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 11:56:39,739 | infra | DEBUG | pool-5-thread-16 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.156': '0'} for bucket 'default' 2021-06-08 11:56:39,740 | infra | DEBUG | pool-5-thread-16 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 11:56:39,740 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 11:56:39,740 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 11:56:42,542 | infra | DEBUG | pool-5-thread-23 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:56:42,660 | test | DEBUG | pool-5-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 11:56:42,660 | test | INFO | pool-5-thread-23 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1010, in main() File "/opt/couchbase/lib/python/cbstats", line 1007, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 49, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-06-08 11:56:43,976 | infra | DEBUG | pool-5-thread-21 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:56:44,094 | test | DEBUG | pool-5-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 11:56:44,094 | test | INFO | pool-5-thread-21 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 11:56:44,742 | infra | DEBUG | pool-5-thread-16 | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.156 2021-06-08 11:56:44,743 | infra | INFO | pool-5-thread-16 | [task:complete_task:102] Thread 'StatsWaitTask_default_ep_data_write_failed_1623178531.88' completed 2021-06-08 11:56:47,661 | infra | DEBUG | pool-5-thread-23 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:56:47,776 | test | DEBUG | pool-5-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 11:56:47,776 | test | INFO | pool-5-thread-23 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 11:56:49,095 | infra | DEBUG | pool-5-thread-21 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:56:49,211 | test | DEBUG | pool-5-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 11:56:49,211 | test | INFO | pool-5-thread-21 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 11:56:52,778 | infra | DEBUG | pool-5-thread-23 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:56:52,901 | test | DEBUG | pool-5-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 11:56:52,901 | test | INFO | pool-5-thread-23 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1010, in main() File "/opt/couchbase/lib/python/cbstats", line 1007, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 49, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-06-08 11:56:54,213 | infra | DEBUG | pool-5-thread-21 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:56:54,329 | test | DEBUG | pool-5-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 11:56:54,329 | test | INFO | pool-5-thread-21 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 11:56:57,901 | infra | DEBUG | pool-5-thread-23 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:56:58,020 | test | DEBUG | pool-5-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 11:56:58,022 | test | INFO | pool-5-thread-23 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 11:56:59,329 | infra | DEBUG | pool-5-thread-21 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:56:59,447 | test | DEBUG | pool-5-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 11:56:59,447 | test | INFO | pool-5-thread-21 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 11:57:03,023 | infra | DEBUG | pool-5-thread-23 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:57:03,144 | test | DEBUG | pool-5-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 11:57:03,144 | test | INFO | pool-5-thread-23 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1010, in main() File "/opt/couchbase/lib/python/cbstats", line 1007, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 49, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-06-08 11:57:04,447 | infra | DEBUG | pool-5-thread-21 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:57:04,565 | test | DEBUG | pool-5-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 11:57:04,565 | test | INFO | pool-5-thread-21 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 11:57:08,145 | infra | DEBUG | pool-5-thread-23 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:57:08,262 | test | DEBUG | pool-5-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 11:57:08,262 | test | INFO | pool-5-thread-23 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 11:57:09,565 | infra | DEBUG | pool-5-thread-21 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:57:09,680 | test | DEBUG | pool-5-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 11:57:09,680 | test | INFO | pool-5-thread-21 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 11:57:13,262 | infra | DEBUG | pool-5-thread-23 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:57:13,562 | test | DEBUG | pool-5-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 11:57:13,562 | test | INFO | pool-5-thread-23 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1010, in main() File "/opt/couchbase/lib/python/cbstats", line 1007, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 49, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-06-08 11:57:14,681 | infra | DEBUG | pool-5-thread-21 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:57:14,801 | test | DEBUG | pool-5-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 11:57:14,802 | test | INFO | pool-5-thread-21 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 11:57:18,562 | test | DEBUG | pool-5-thread-23 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 11:57:18,562 | infra | DEBUG | pool-5-thread-23 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {} for bucket 'default' 2021-06-08 11:57:18,562 | infra | DEBUG | pool-5-thread-23 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 11:57:18,563 | infra | DEBUG | pool-5-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 11:57:18,565 | infra | DEBUG | pool-5-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 11:57:19,802 | test | DEBUG | pool-5-thread-21 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 11:57:19,802 | infra | DEBUG | pool-5-thread-21 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {} for bucket 'default' 2021-06-08 11:57:19,802 | infra | DEBUG | pool-5-thread-21 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 11:57:19,803 | infra | DEBUG | pool-5-thread-21 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 11:57:19,803 | infra | DEBUG | pool-5-thread-21 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 11:57:23,565 | infra | DEBUG | pool-5-thread-23 | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.125 2021-06-08 11:57:23,565 | infra | INFO | pool-5-thread-23 | [task:complete_task:102] Thread 'StatsWaitTask_default_ep_data_write_failed_1623178530.99' completed 2021-06-08 11:57:23,571 | infra | INFO | MainThread | [task_manager:add_new_task:23] Added new task: LoadDocsGen_default__default__default_create_11000000_12000000_1623178643.57 2021-06-08 11:57:23,571 | infra | INFO | pool-5-thread-19 | [task:start_task:87] Thread 'LoadDocsGen_default__default__default_create_11000000_12000000_1623178643.57' started 2021-06-08 11:57:23,571 | infra | DEBUG | MainThread | [task_manager:get_task_result:26] Getting task result for LoadDocsGen_default__default__default_create_11000000_12000000_1623178643.57 2021-06-08 11:57:23,628 | infra | INFO | pool-5-thread-19 | [task_manager:add_new_task:23] Added new task: PrintBucketStats_default_1623178643.62 2021-06-08 11:57:23,630 | infra | INFO | pool-5-thread-27 | [task:start_task:87] Thread 'PrintBucketStats_default_1623178643.62' started 2021-06-08 11:57:23,630 | infra | INFO | pool-5-thread-19 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_11000000_12000000_1623178643.57_create__11000000_11050000_1623178643.62 2021-06-08 11:57:23,630 | infra | INFO | pool-5-thread-8 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_11000000_12000000_1623178643.57_create__11000000_11050000_1623178643.62' started 2021-06-08 11:57:23,630 | infra | INFO | pool-5-thread-19 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_11000000_12000000_1623178643.57_create__11050000_11100000_1623178643.62 2021-06-08 11:57:23,630 | infra | INFO | pool-5-thread-24 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_11000000_12000000_1623178643.57_create__11050000_11100000_1623178643.62' started 2021-06-08 11:57:23,630 | infra | INFO | pool-5-thread-19 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_11000000_12000000_1623178643.57_create__11100000_11150000_1623178643.62 2021-06-08 11:57:23,630 | infra | INFO | pool-5-thread-9 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_11000000_12000000_1623178643.57_create__11100000_11150000_1623178643.62' started 2021-06-08 11:57:23,630 | infra | INFO | pool-5-thread-19 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_11000000_12000000_1623178643.57_create__11150000_11200000_1623178643.62 2021-06-08 11:57:23,630 | infra | INFO | pool-5-thread-12 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_11000000_12000000_1623178643.57_create__11150000_11200000_1623178643.62' started 2021-06-08 11:57:23,631 | infra | INFO | pool-5-thread-19 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_11000000_12000000_1623178643.57_create__11200000_11250000_1623178643.62 2021-06-08 11:57:23,631 | infra | INFO | pool-5-thread-19 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_11000000_12000000_1623178643.57_create__11250000_11300000_1623178643.62 2021-06-08 11:57:23,631 | infra | INFO | pool-5-thread-19 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_11000000_12000000_1623178643.57_create__11300000_11350000_1623178643.62 2021-06-08 11:57:23,631 | infra | INFO | pool-5-thread-5 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_11000000_12000000_1623178643.57_create__11200000_11250000_1623178643.62' started 2021-06-08 11:57:23,631 | infra | INFO | pool-5-thread-30 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_11000000_12000000_1623178643.57_create__11300000_11350000_1623178643.62' started 2021-06-08 11:57:23,631 | infra | INFO | pool-5-thread-13 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_11000000_12000000_1623178643.57_create__11350000_11400000_1623178643.62' started 2021-06-08 11:57:23,631 | infra | INFO | pool-5-thread-28 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_11000000_12000000_1623178643.57_create__11250000_11300000_1623178643.62' started 2021-06-08 11:57:23,631 | infra | INFO | pool-5-thread-19 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_11000000_12000000_1623178643.57_create__11350000_11400000_1623178643.62 2021-06-08 11:57:23,632 | infra | INFO | pool-5-thread-19 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_11000000_12000000_1623178643.57_create__11400000_11450000_1623178643.62 2021-06-08 11:57:23,632 | infra | INFO | pool-5-thread-20 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_11000000_12000000_1623178643.57_create__11400000_11450000_1623178643.62' started 2021-06-08 11:57:23,632 | infra | INFO | pool-5-thread-19 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_11000000_12000000_1623178643.57_create__11450000_11500000_1623178643.62 2021-06-08 11:57:23,632 | infra | INFO | pool-5-thread-19 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_11000000_12000000_1623178643.57_create__11500000_11550000_1623178643.62 2021-06-08 11:57:23,632 | infra | INFO | pool-5-thread-3 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_11000000_12000000_1623178643.57_create__11500000_11550000_1623178643.62' started 2021-06-08 11:57:23,632 | infra | INFO | pool-5-thread-19 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_11000000_12000000_1623178643.57_create__11550000_11600000_1623178643.62 2021-06-08 11:57:23,632 | infra | INFO | pool-5-thread-7 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_11000000_12000000_1623178643.57_create__11550000_11600000_1623178643.62' started 2021-06-08 11:57:23,632 | infra | INFO | pool-5-thread-11 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_11000000_12000000_1623178643.57_create__11450000_11500000_1623178643.62' started 2021-06-08 11:57:23,634 | infra | INFO | pool-5-thread-15 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_11000000_12000000_1623178643.57_create__11600000_11650000_1623178643.62' started 2021-06-08 11:57:23,634 | infra | INFO | pool-5-thread-19 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_11000000_12000000_1623178643.57_create__11600000_11650000_1623178643.62 2021-06-08 11:57:23,638 | infra | INFO | pool-5-thread-2 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_11000000_12000000_1623178643.57_create__11650000_11700000_1623178643.62' started 2021-06-08 11:57:23,638 | infra | INFO | pool-5-thread-19 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_11000000_12000000_1623178643.57_create__11650000_11700000_1623178643.62 2021-06-08 11:57:23,638 | infra | INFO | pool-5-thread-19 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_11000000_12000000_1623178643.57_create__11700000_11750000_1623178643.62 2021-06-08 11:57:23,638 | infra | INFO | pool-5-thread-19 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_11000000_12000000_1623178643.57_create__11750000_11800000_1623178643.62 2021-06-08 11:57:23,638 | infra | INFO | pool-5-thread-29 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_11000000_12000000_1623178643.57_create__11750000_11800000_1623178643.62' started 2021-06-08 11:57:23,638 | infra | INFO | pool-5-thread-19 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_11000000_12000000_1623178643.57_create__11800000_11850000_1623178643.62 2021-06-08 11:57:23,638 | infra | INFO | pool-5-thread-22 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_11000000_12000000_1623178643.57_create__11800000_11850000_1623178643.62' started 2021-06-08 11:57:23,638 | infra | INFO | pool-5-thread-6 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_11000000_12000000_1623178643.57_create__11700000_11750000_1623178643.62' started 2021-06-08 11:57:23,638 | infra | INFO | pool-5-thread-17 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_11000000_12000000_1623178643.57_create__11850000_11900000_1623178643.62' started 2021-06-08 11:57:23,638 | infra | INFO | pool-5-thread-19 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_11000000_12000000_1623178643.57_create__11850000_11900000_1623178643.62 2021-06-08 11:57:23,640 | infra | INFO | pool-5-thread-19 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_11000000_12000000_1623178643.57_create__11900000_11950000_1623178643.62 2021-06-08 11:57:23,640 | infra | INFO | pool-5-thread-4 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_11000000_12000000_1623178643.57_create__11900000_11950000_1623178643.62' started 2021-06-08 11:57:23,640 | infra | INFO | pool-5-thread-19 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_11000000_12000000_1623178643.57_create__11950000_12000000_1623178643.62 2021-06-08 11:57:23,640 | infra | INFO | pool-5-thread-18 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_11000000_12000000_1623178643.57_create__11950000_12000000_1623178643.62' started 2021-06-08 11:57:23,640 | infra | DEBUG | pool-5-thread-19 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_11000000_12000000_1623178643.57_create__11000000_11050000_1623178643.62 2021-06-08 11:57:23,852 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:57:23,854 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 0 2021-06-08 11:57:23,855 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:57:23,855 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:57:24,805 | infra | DEBUG | pool-5-thread-21 | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.122 2021-06-08 11:57:24,805 | infra | INFO | pool-5-thread-21 | [task:complete_task:102] Thread 'StatsWaitTask_default_ep_data_write_failed_1623178532.34' completed 2021-06-08 11:57:25,058 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:57:25,058 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 0 2021-06-08 11:57:25,059 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:57:25,059 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:57:26,256 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:57:26,256 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 0 2021-06-08 11:57:26,257 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:57:26,257 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:57:27,457 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:57:27,459 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 0 2021-06-08 11:57:27,460 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:57:27,460 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:57:28,654 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:57:28,654 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 0 2021-06-08 11:57:28,654 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:57:28,655 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:57:29,849 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:57:29,849 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 0 2021-06-08 11:57:29,851 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:57:29,851 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:57:31,055 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:57:31,056 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 0 2021-06-08 11:57:31,058 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:57:31,058 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:57:32,253 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 1000.000000 2021-06-08 11:57:32,253 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 0 2021-06-08 11:57:32,253 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:57:32,253 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:57:33,457 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 1000.000000 2021-06-08 11:57:33,457 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 0 2021-06-08 11:57:33,459 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:57:33,459 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:57:34,650 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 1000.000000 2021-06-08 11:57:34,650 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 0 2021-06-08 11:57:34,651 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:57:34,651 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:57:35,845 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 1000.000000 2021-06-08 11:57:35,845 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 0 2021-06-08 11:57:35,846 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:57:35,846 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:57:37,043 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 1000.000000 2021-06-08 11:57:37,043 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 0 2021-06-08 11:57:37,045 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:57:37,045 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:57:38,236 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 1000.000000 2021-06-08 11:57:38,236 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 0 2021-06-08 11:57:38,237 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:57:38,237 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:57:39,440 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 1000.000000 2021-06-08 11:57:39,440 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 0 2021-06-08 11:57:39,443 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:57:39,443 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:57:40,642 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 1000.000000 2021-06-08 11:57:40,644 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 0 2021-06-08 11:57:40,644 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:57:40,644 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:57:41,842 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:57:41,844 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 0 2021-06-08 11:57:41,845 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:57:41,845 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:57:43,042 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:57:43,042 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 0 2021-06-08 11:57:43,045 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:57:43,045 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:57:44,237 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:57:44,237 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 0 2021-06-08 11:57:44,240 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:57:44,240 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:57:45,431 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:57:45,431 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 0 2021-06-08 11:57:45,433 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:57:45,433 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:57:46,628 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:57:46,630 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 0 2021-06-08 11:57:46,631 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:57:46,631 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:57:47,823 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:57:47,825 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 0 2021-06-08 11:57:47,825 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:57:47,825 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:57:49,025 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:57:49,025 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 0 2021-06-08 11:57:49,026 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:57:49,026 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:57:50,220 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:57:50,220 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 0 2021-06-08 11:57:50,223 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:57:50,223 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:57:51,414 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:57:51,414 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 0 2021-06-08 11:57:51,414 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:57:51,414 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:57:52,617 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:57:52,617 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 0 2021-06-08 11:57:52,618 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:57:52,618 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:57:53,813 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:57:53,813 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 0 2021-06-08 11:57:53,815 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:57:53,815 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:57:55,012 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:57:55,012 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 0 2021-06-08 11:57:55,013 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:57:55,013 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:57:56,209 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:57:56,210 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 0 2021-06-08 11:57:56,210 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:57:56,210 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:57:57,404 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:57:57,405 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 0 2021-06-08 11:57:57,407 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:57:57,407 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:57:58,604 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:57:58,605 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 0 2021-06-08 11:57:58,605 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:57:58,607 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:58:01,829 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 928.800000 2021-06-08 11:58:01,829 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 0 2021-06-08 11:58:01,832 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:58:01,832 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:58:03,029 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 928.800000 2021-06-08 11:58:03,029 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 0 2021-06-08 11:58:03,032 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:58:03,032 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:58:04,236 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 928.800000 2021-06-08 11:58:04,236 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 0 2021-06-08 11:58:04,239 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:58:04,239 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:58:05,680 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 928.800000 2021-06-08 11:58:05,680 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 0 2021-06-08 11:58:05,684 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:58:05,684 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:58:06,914 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 928.800000 2021-06-08 11:58:06,915 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 0 2021-06-08 11:58:06,917 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:58:06,917 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:58:08,119 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 928.800000 2021-06-08 11:58:08,121 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 0 2021-06-08 11:58:08,121 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:58:08,121 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:58:09,316 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 928.800000 2021-06-08 11:58:09,316 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 0 2021-06-08 11:58:09,318 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:58:09,318 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:58:10,598 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 928.800000 2021-06-08 11:58:10,598 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 0 2021-06-08 11:58:10,601 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:58:10,601 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:58:11,811 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 3871.000000 2021-06-08 11:58:11,811 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 756492 2021-06-08 11:58:11,812 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:58:11,812 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:58:13,016 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 3871.000000 2021-06-08 11:58:13,017 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 756492 2021-06-08 11:58:13,019 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:58:13,019 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:58:14,463 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 3871.000000 2021-06-08 11:58:14,464 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 756492 2021-06-08 11:58:14,467 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:58:14,467 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:58:16,026 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 3871.000000 2021-06-08 11:58:16,026 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 756492 2021-06-08 11:58:16,028 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:58:16,029 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:58:17,233 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 3871.000000 2021-06-08 11:58:17,233 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 756492 2021-06-08 11:58:17,236 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:58:17,236 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:58:18,529 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 3871.000000 2021-06-08 11:58:18,529 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 756492 2021-06-08 11:58:18,532 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:58:18,532 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:58:19,769 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 3871.000000 2021-06-08 11:58:19,769 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 756492 2021-06-08 11:58:19,772 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:58:19,772 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:58:20,979 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 3871.000000 2021-06-08 11:58:20,980 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 756492 2021-06-08 11:58:20,982 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:58:20,982 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:58:22,178 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 6001.500000 2021-06-08 11:58:22,178 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 756492 2021-06-08 11:58:22,180 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:58:22,180 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:58:23,391 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 6001.500000 2021-06-08 11:58:23,392 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 756492 2021-06-08 11:58:23,394 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:58:23,394 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:58:24,592 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 6001.500000 2021-06-08 11:58:24,592 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 756492 2021-06-08 11:58:24,594 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:58:24,594 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:58:25,786 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 6001.500000 2021-06-08 11:58:25,786 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 756492 2021-06-08 11:58:25,788 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:58:25,788 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:58:26,984 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 6001.500000 2021-06-08 11:58:26,984 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 756492 2021-06-08 11:58:26,986 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:58:26,986 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:58:28,178 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 6001.500000 2021-06-08 11:58:28,178 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 756492 2021-06-08 11:58:28,180 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:58:28,180 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:58:29,377 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 6001.500000 2021-06-08 11:58:29,378 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 756492 2021-06-08 11:58:29,380 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:58:29,380 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:58:30,584 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 12377.800000 2021-06-08 11:58:30,585 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 806289 2021-06-08 11:58:30,586 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:58:30,586 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:58:31,815 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 6923.800000 2021-06-08 11:58:31,816 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 806289 2021-06-08 11:58:31,819 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:58:31,819 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:58:33,361 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 6923.800000 2021-06-08 11:58:33,361 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 806289 2021-06-08 11:58:33,362 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:58:33,362 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:58:34,569 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 6923.800000 2021-06-08 11:58:34,569 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 806289 2021-06-08 11:58:34,572 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:58:34,572 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:58:35,859 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 6923.800000 2021-06-08 11:58:35,861 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 806289 2021-06-08 11:58:35,864 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:58:35,865 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:58:37,075 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 6923.800000 2021-06-08 11:58:37,075 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 806289 2021-06-08 11:58:37,078 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:58:37,078 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:58:38,289 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 6923.800000 2021-06-08 11:58:38,289 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 806289 2021-06-08 11:58:38,292 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:58:38,292 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:58:39,767 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 6923.800000 2021-06-08 11:58:39,767 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 806289 2021-06-08 11:58:39,770 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:58:39,770 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:58:40,973 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 6923.800000 2021-06-08 11:58:40,973 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 0 2021-06-08 11:58:40,974 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:58:40,976 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:58:42,181 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 13176.100000 2021-06-08 11:58:42,181 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 0 2021-06-08 11:58:42,184 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:58:42,184 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:58:43,401 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 13176.100000 2021-06-08 11:58:43,401 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 0 2021-06-08 11:58:43,403 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:58:43,403 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:58:44,601 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 13176.100000 2021-06-08 11:58:44,601 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 0 2021-06-08 11:58:44,602 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:58:44,602 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:58:45,801 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 13176.100000 2021-06-08 11:58:45,802 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 0 2021-06-08 11:58:45,802 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:58:45,802 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:58:49,470 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 10232.656714 2021-06-08 11:58:49,471 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 829907 2021-06-08 11:58:49,473 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:58:49,473 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:58:50,684 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 10232.656714 2021-06-08 11:58:50,684 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 829907 2021-06-08 11:58:50,687 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:58:50,687 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:58:51,947 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 3482.856714 2021-06-08 11:58:51,947 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 829907 2021-06-08 11:58:51,951 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:58:51,951 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:58:53,201 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 3482.856714 2021-06-08 11:58:53,201 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 829907 2021-06-08 11:58:53,204 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:58:53,204 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:58:54,444 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 3482.856714 2021-06-08 11:58:54,444 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 829907 2021-06-08 11:58:54,448 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:58:54,448 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:58:55,664 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 3482.856714 2021-06-08 11:58:55,664 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 829907 2021-06-08 11:58:55,667 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:58:55,667 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:58:56,342 | infra | INFO | pool-5-thread-5 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_11000000_12000000_1623178643.57_create__11200000_11250000_1623178643.62' completed 2021-06-08 11:58:56,417 | infra | INFO | pool-5-thread-15 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_11000000_12000000_1623178643.57_create__11600000_11650000_1623178643.62' completed 2021-06-08 11:58:56,888 | infra | INFO | pool-5-thread-3 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_11000000_12000000_1623178643.57_create__11500000_11550000_1623178643.62' completed 2021-06-08 11:58:57,029 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 3482.856714 2021-06-08 11:58:57,029 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 829907 2021-06-08 11:58:57,030 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:58:57,030 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:58:58,229 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 3482.856714 2021-06-08 11:58:58,229 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 829907 2021-06-08 11:58:58,230 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:58:58,230 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:58:59,434 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 3482.856714 2021-06-08 11:58:59,434 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 829907 2021-06-08 11:58:59,436 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:58:59,437 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:59:00,635 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 3482.856714 2021-06-08 11:59:00,635 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 829907 2021-06-08 11:59:00,637 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:59:00,637 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:59:01,835 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 7832.856714 2021-06-08 11:59:01,835 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 0 2021-06-08 11:59:01,836 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:59:01,836 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:59:03,042 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 7832.856714 2021-06-08 11:59:03,042 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 0 2021-06-08 11:59:03,045 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:59:03,045 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:59:04,243 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 7832.856714 2021-06-08 11:59:04,243 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 0 2021-06-08 11:59:04,244 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:59:04,244 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:59:05,444 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 7832.856714 2021-06-08 11:59:05,446 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 0 2021-06-08 11:59:05,447 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:59:05,447 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:59:06,644 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 7832.856714 2021-06-08 11:59:06,645 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 0 2021-06-08 11:59:06,647 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:59:06,647 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:59:07,846 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 7832.856714 2021-06-08 11:59:07,846 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 0 2021-06-08 11:59:07,848 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:59:07,848 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:59:09,048 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 7832.856714 2021-06-08 11:59:09,049 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 0 2021-06-08 11:59:09,049 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:59:09,049 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:59:10,249 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 7832.856714 2021-06-08 11:59:10,250 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 0 2021-06-08 11:59:10,250 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:59:10,250 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:59:11,450 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 3432.856714 2021-06-08 11:59:11,450 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 0 2021-06-08 11:59:11,450 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:59:11,451 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:59:12,654 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 3432.856714 2021-06-08 11:59:12,654 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 0 2021-06-08 11:59:12,654 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:59:12,655 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:59:13,858 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 3432.856714 2021-06-08 11:59:13,858 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 0 2021-06-08 11:59:13,859 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:59:13,859 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:59:15,059 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 3432.856714 2021-06-08 11:59:15,059 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 0 2021-06-08 11:59:15,059 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:59:15,061 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:59:16,259 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 3432.856714 2021-06-08 11:59:16,259 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 0 2021-06-08 11:59:16,260 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:59:16,260 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:59:17,460 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 3432.856714 2021-06-08 11:59:17,460 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 0 2021-06-08 11:59:17,463 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:59:17,463 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:59:18,664 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 3432.856714 2021-06-08 11:59:18,665 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 0 2021-06-08 11:59:18,667 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:59:18,667 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:59:19,868 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 3432.856714 2021-06-08 11:59:19,868 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 0 2021-06-08 11:59:19,869 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:59:19,871 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:59:21,081 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 3432.856714 2021-06-08 11:59:21,081 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 0 2021-06-08 11:59:21,084 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:59:21,084 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:59:22,283 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:59:22,283 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 0 2021-06-08 11:59:22,286 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:59:22,286 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:59:23,510 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:59:23,512 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 0 2021-06-08 11:59:23,513 | infra | INFO | pool-5-thread-30 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_11000000_12000000_1623178643.57_create__11300000_11350000_1623178643.62' completed 2021-06-08 11:59:23,515 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:59:23,515 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:59:23,553 | infra | INFO | pool-5-thread-2 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_11000000_12000000_1623178643.57_create__11650000_11700000_1623178643.62' completed 2021-06-08 11:59:23,611 | infra | INFO | pool-5-thread-6 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_11000000_12000000_1623178643.57_create__11700000_11750000_1623178643.62' completed 2021-06-08 11:59:24,713 | infra | INFO | pool-5-thread-9 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_11000000_12000000_1623178643.57_create__11100000_11150000_1623178643.62' completed 2021-06-08 11:59:24,723 | infra | INFO | pool-5-thread-4 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_11000000_12000000_1623178643.57_create__11900000_11950000_1623178643.62' completed 2021-06-08 11:59:24,763 | test | DEBUG | pool-5-thread-27 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 11:59:24,763 | test | DEBUG | pool-5-thread-27 | [task:print_ep_queue_size:3508] ep_queue_size for default: 0 2021-06-08 11:59:24,765 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 11:59:24,765 | infra | DEBUG | pool-5-thread-27 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 11:59:24,792 | infra | INFO | pool-5-thread-22 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_11000000_12000000_1623178643.57_create__11800000_11850000_1623178643.62' completed 2021-06-08 11:59:24,819 | infra | INFO | pool-5-thread-12 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_11000000_12000000_1623178643.57_create__11150000_11200000_1623178643.62' completed 2021-06-08 11:59:24,845 | infra | INFO | pool-5-thread-20 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_11000000_12000000_1623178643.57_create__11400000_11450000_1623178643.62' completed 2021-06-08 11:59:24,854 | infra | INFO | pool-5-thread-11 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_11000000_12000000_1623178643.57_create__11450000_11500000_1623178643.62' completed 2021-06-08 11:59:24,882 | infra | INFO | pool-5-thread-17 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_11000000_12000000_1623178643.57_create__11850000_11900000_1623178643.62' completed 2021-06-08 11:59:24,914 | infra | INFO | pool-5-thread-13 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_11000000_12000000_1623178643.57_create__11350000_11400000_1623178643.62' completed 2021-06-08 11:59:25,039 | infra | INFO | pool-5-thread-18 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_11000000_12000000_1623178643.57_create__11950000_12000000_1623178643.62' completed 2021-06-08 11:59:25,236 | infra | INFO | pool-5-thread-7 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_11000000_12000000_1623178643.57_create__11550000_11600000_1623178643.62' completed 2021-06-08 11:59:25,526 | infra | INFO | pool-5-thread-28 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_11000000_12000000_1623178643.57_create__11250000_11300000_1623178643.62' completed 2021-06-08 11:59:25,608 | infra | INFO | pool-5-thread-24 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_11000000_12000000_1623178643.57_create__11050000_11100000_1623178643.62' completed 2021-06-08 11:59:25,618 | infra | INFO | pool-5-thread-8 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_11000000_12000000_1623178643.57_create__11000000_11050000_1623178643.62' completed 2021-06-08 11:59:25,618 | infra | DEBUG | pool-5-thread-19 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_11000000_12000000_1623178643.57_create__11000000_11050000_1623178643.62 are 50000 2021-06-08 11:59:25,618 | infra | DEBUG | pool-5-thread-19 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_11000000_12000000_1623178643.57_create__11050000_11100000_1623178643.62 2021-06-08 11:59:25,618 | infra | DEBUG | pool-5-thread-19 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_11000000_12000000_1623178643.57_create__11050000_11100000_1623178643.62 are 50000 2021-06-08 11:59:25,618 | infra | DEBUG | pool-5-thread-19 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_11000000_12000000_1623178643.57_create__11100000_11150000_1623178643.62 2021-06-08 11:59:25,618 | infra | DEBUG | pool-5-thread-19 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_11000000_12000000_1623178643.57_create__11100000_11150000_1623178643.62 are 50000 2021-06-08 11:59:25,618 | infra | DEBUG | pool-5-thread-19 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_11000000_12000000_1623178643.57_create__11150000_11200000_1623178643.62 2021-06-08 11:59:25,618 | infra | DEBUG | pool-5-thread-19 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_11000000_12000000_1623178643.57_create__11150000_11200000_1623178643.62 are 50000 2021-06-08 11:59:25,618 | infra | DEBUG | pool-5-thread-19 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_11000000_12000000_1623178643.57_create__11200000_11250000_1623178643.62 2021-06-08 11:59:25,618 | infra | DEBUG | pool-5-thread-19 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_11000000_12000000_1623178643.57_create__11200000_11250000_1623178643.62 are 50000 2021-06-08 11:59:25,618 | infra | DEBUG | pool-5-thread-19 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_11000000_12000000_1623178643.57_create__11250000_11300000_1623178643.62 2021-06-08 11:59:25,618 | infra | INFO | pool-5-thread-29 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_11000000_12000000_1623178643.57_create__11750000_11800000_1623178643.62' completed 2021-06-08 11:59:25,619 | infra | DEBUG | pool-5-thread-19 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_11000000_12000000_1623178643.57_create__11250000_11300000_1623178643.62 are 50000 2021-06-08 11:59:25,619 | infra | DEBUG | pool-5-thread-19 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_11000000_12000000_1623178643.57_create__11300000_11350000_1623178643.62 2021-06-08 11:59:25,619 | infra | DEBUG | pool-5-thread-19 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_11000000_12000000_1623178643.57_create__11300000_11350000_1623178643.62 are 50000 2021-06-08 11:59:25,619 | infra | DEBUG | pool-5-thread-19 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_11000000_12000000_1623178643.57_create__11350000_11400000_1623178643.62 2021-06-08 11:59:25,619 | infra | DEBUG | pool-5-thread-19 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_11000000_12000000_1623178643.57_create__11350000_11400000_1623178643.62 are 50000 2021-06-08 11:59:25,619 | infra | DEBUG | pool-5-thread-19 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_11000000_12000000_1623178643.57_create__11400000_11450000_1623178643.62 2021-06-08 11:59:25,619 | infra | DEBUG | pool-5-thread-19 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_11000000_12000000_1623178643.57_create__11400000_11450000_1623178643.62 are 50000 2021-06-08 11:59:25,619 | infra | DEBUG | pool-5-thread-19 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_11000000_12000000_1623178643.57_create__11450000_11500000_1623178643.62 2021-06-08 11:59:25,621 | infra | DEBUG | pool-5-thread-19 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_11000000_12000000_1623178643.57_create__11450000_11500000_1623178643.62 are 50000 2021-06-08 11:59:25,621 | infra | DEBUG | pool-5-thread-19 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_11000000_12000000_1623178643.57_create__11500000_11550000_1623178643.62 2021-06-08 11:59:25,621 | infra | DEBUG | pool-5-thread-19 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_11000000_12000000_1623178643.57_create__11500000_11550000_1623178643.62 are 50000 2021-06-08 11:59:25,621 | infra | DEBUG | pool-5-thread-19 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_11000000_12000000_1623178643.57_create__11550000_11600000_1623178643.62 2021-06-08 11:59:25,621 | infra | DEBUG | pool-5-thread-19 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_11000000_12000000_1623178643.57_create__11550000_11600000_1623178643.62 are 50000 2021-06-08 11:59:25,621 | infra | DEBUG | pool-5-thread-19 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_11000000_12000000_1623178643.57_create__11600000_11650000_1623178643.62 2021-06-08 11:59:25,621 | infra | DEBUG | pool-5-thread-19 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_11000000_12000000_1623178643.57_create__11600000_11650000_1623178643.62 are 50000 2021-06-08 11:59:25,621 | infra | DEBUG | pool-5-thread-19 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_11000000_12000000_1623178643.57_create__11650000_11700000_1623178643.62 2021-06-08 11:59:25,621 | infra | DEBUG | pool-5-thread-19 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_11000000_12000000_1623178643.57_create__11650000_11700000_1623178643.62 are 50000 2021-06-08 11:59:25,621 | infra | DEBUG | pool-5-thread-19 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_11000000_12000000_1623178643.57_create__11700000_11750000_1623178643.62 2021-06-08 11:59:25,621 | infra | DEBUG | pool-5-thread-19 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_11000000_12000000_1623178643.57_create__11700000_11750000_1623178643.62 are 50000 2021-06-08 11:59:25,621 | infra | DEBUG | pool-5-thread-19 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_11000000_12000000_1623178643.57_create__11750000_11800000_1623178643.62 2021-06-08 11:59:25,621 | infra | DEBUG | pool-5-thread-19 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_11000000_12000000_1623178643.57_create__11750000_11800000_1623178643.62 are 50000 2021-06-08 11:59:25,621 | infra | DEBUG | pool-5-thread-19 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_11000000_12000000_1623178643.57_create__11800000_11850000_1623178643.62 2021-06-08 11:59:25,621 | infra | DEBUG | pool-5-thread-19 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_11000000_12000000_1623178643.57_create__11800000_11850000_1623178643.62 are 50000 2021-06-08 11:59:25,621 | infra | DEBUG | pool-5-thread-19 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_11000000_12000000_1623178643.57_create__11850000_11900000_1623178643.62 2021-06-08 11:59:25,621 | infra | DEBUG | pool-5-thread-19 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_11000000_12000000_1623178643.57_create__11850000_11900000_1623178643.62 are 50000 2021-06-08 11:59:25,621 | infra | DEBUG | pool-5-thread-19 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_11000000_12000000_1623178643.57_create__11900000_11950000_1623178643.62 2021-06-08 11:59:25,621 | infra | DEBUG | pool-5-thread-19 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_11000000_12000000_1623178643.57_create__11900000_11950000_1623178643.62 are 50000 2021-06-08 11:59:25,621 | infra | DEBUG | pool-5-thread-19 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_11000000_12000000_1623178643.57_create__11950000_12000000_1623178643.62 2021-06-08 11:59:25,621 | infra | DEBUG | pool-5-thread-19 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_11000000_12000000_1623178643.57_create__11950000_12000000_1623178643.62 are 50000 2021-06-08 11:59:25,622 | infra | DEBUG | pool-5-thread-19 | [task_manager:get_task_result:26] Getting task result for PrintBucketStats_default_1623178643.62 2021-06-08 11:59:25,766 | test | INFO | pool-5-thread-27 | [table_view:display:72] Ops trend for bucket 'default' +-----------+--------------------------------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-----------+--------------------------------------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | *******.......*******X | 1000.000 | | 0.000 | ***************.......********......................********................*******................................................X | 12377.800 | | 6923.800 | ...................................................********................................................***X | 13176.100 | | 10232.657 | ............................................................................*X | 10232.657 | | 3482.857 | .........................********.................................*******X | 7832.857 | | 3432.857 | .........................********X | 3432.857 | | 0.000 | **X | 0.000 | +-----------+--------------------------------------------------------------------------------------------------------------------------------------+-----------+ 2021-06-08 11:59:25,766 | infra | INFO | pool-5-thread-27 | [task:complete_task:102] Thread 'PrintBucketStats_default_1623178643.62' completed 2021-06-08 11:59:25,766 | infra | DEBUG | pool-5-thread-19 | [task:call:1663] ========= Tasks in loadgen pool======= 2021-06-08 11:59:25,767 | infra | WARNING | pool-5-thread-19 | [task_manager:print_tasks_in_pool:83] Task 'LoadDocsGen_default__default__default_create_11000000_12000000_1623178643.57' not completed 2021-06-08 11:59:25,767 | infra | DEBUG | pool-5-thread-19 | [task:call:1665] ====================================== 2021-06-08 11:59:25,767 | infra | DEBUG | pool-5-thread-19 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_11000000_12000000_1623178643.57_create__11000000_11050000_1623178643.62' complete. Loaded 50000 items 2021-06-08 11:59:25,767 | infra | DEBUG | pool-5-thread-19 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_11000000_12000000_1623178643.57_create__11050000_11100000_1623178643.62' complete. Loaded 50000 items 2021-06-08 11:59:25,767 | infra | DEBUG | pool-5-thread-19 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_11000000_12000000_1623178643.57_create__11100000_11150000_1623178643.62' complete. Loaded 50000 items 2021-06-08 11:59:25,767 | infra | DEBUG | pool-5-thread-19 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_11000000_12000000_1623178643.57_create__11150000_11200000_1623178643.62' complete. Loaded 50000 items 2021-06-08 11:59:25,767 | infra | DEBUG | pool-5-thread-19 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_11000000_12000000_1623178643.57_create__11200000_11250000_1623178643.62' complete. Loaded 50000 items 2021-06-08 11:59:25,769 | infra | DEBUG | pool-5-thread-19 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_11000000_12000000_1623178643.57_create__11250000_11300000_1623178643.62' complete. Loaded 50000 items 2021-06-08 11:59:25,769 | infra | DEBUG | pool-5-thread-19 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_11000000_12000000_1623178643.57_create__11300000_11350000_1623178643.62' complete. Loaded 50000 items 2021-06-08 11:59:25,769 | infra | DEBUG | pool-5-thread-19 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_11000000_12000000_1623178643.57_create__11350000_11400000_1623178643.62' complete. Loaded 50000 items 2021-06-08 11:59:25,769 | infra | DEBUG | pool-5-thread-19 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_11000000_12000000_1623178643.57_create__11400000_11450000_1623178643.62' complete. Loaded 50000 items 2021-06-08 11:59:25,769 | infra | DEBUG | pool-5-thread-19 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_11000000_12000000_1623178643.57_create__11450000_11500000_1623178643.62' complete. Loaded 50000 items 2021-06-08 11:59:25,769 | infra | DEBUG | pool-5-thread-19 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_11000000_12000000_1623178643.57_create__11500000_11550000_1623178643.62' complete. Loaded 50000 items 2021-06-08 11:59:25,769 | infra | DEBUG | pool-5-thread-19 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_11000000_12000000_1623178643.57_create__11550000_11600000_1623178643.62' complete. Loaded 50000 items 2021-06-08 11:59:25,769 | infra | DEBUG | pool-5-thread-19 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_11000000_12000000_1623178643.57_create__11600000_11650000_1623178643.62' complete. Loaded 50000 items 2021-06-08 11:59:25,769 | infra | DEBUG | pool-5-thread-19 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_11000000_12000000_1623178643.57_create__11650000_11700000_1623178643.62' complete. Loaded 50000 items 2021-06-08 11:59:25,769 | infra | DEBUG | pool-5-thread-19 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_11000000_12000000_1623178643.57_create__11700000_11750000_1623178643.62' complete. Loaded 50000 items 2021-06-08 11:59:25,769 | infra | DEBUG | pool-5-thread-19 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_11000000_12000000_1623178643.57_create__11750000_11800000_1623178643.62' complete. Loaded 50000 items 2021-06-08 11:59:25,769 | infra | DEBUG | pool-5-thread-19 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_11000000_12000000_1623178643.57_create__11800000_11850000_1623178643.62' complete. Loaded 50000 items 2021-06-08 11:59:25,769 | infra | DEBUG | pool-5-thread-19 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_11000000_12000000_1623178643.57_create__11850000_11900000_1623178643.62' complete. Loaded 50000 items 2021-06-08 11:59:25,769 | infra | DEBUG | pool-5-thread-19 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_11000000_12000000_1623178643.57_create__11900000_11950000_1623178643.62' complete. Loaded 50000 items 2021-06-08 11:59:25,769 | infra | DEBUG | pool-5-thread-19 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_11000000_12000000_1623178643.57_create__11950000_12000000_1623178643.62' complete. Loaded 50000 items 2021-06-08 11:59:25,769 | infra | INFO | pool-5-thread-19 | [task:complete_task:102] Thread 'LoadDocsGen_default__default__default_create_11000000_12000000_1623178643.57' completed 2021-06-08 11:59:25,773 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.125 with username: root, password: couchbase 2021-06-08 11:59:25,854 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 11:59:26,062 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: ls /etc/ 2021-06-08 11:59:26,115 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 11:59:26,117 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: ls /cygdrive/c/Windows 2021-06-08 11:59:26,128 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: uname -m 2021-06-08 11:59:26,141 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: cat /proc/cpuinfo 2021-06-08 11:59:26,151 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: df -Thl 2021-06-08 11:59:26,167 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: cat /proc/meminfo 2021-06-08 11:59:26,178 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: hostname 2021-06-08 11:59:26,190 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: hostname -d 2021-06-08 11:59:26,203 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:559] Starting StatsWaitTask for ep_data_write_failed on bucket default 2021-06-08 11:59:26,203 | infra | INFO | MainThread | [task_manager:add_new_task:23] Added new task: StatsWaitTask_default_ep_data_write_failed_1623178766.2 2021-06-08 11:59:26,203 | infra | INFO | pool-5-thread-1 | [task:start_task:87] Thread 'StatsWaitTask_default_ep_data_write_failed_1623178766.2' started 2021-06-08 11:59:26,204 | infra | DEBUG | pool-5-thread-1 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:59:26,207 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.155 with username: root, password: couchbase 2021-06-08 11:59:26,292 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 11:59:26,322 | test | DEBUG | pool-5-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 11:59:26,322 | test | INFO | pool-5-thread-1 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 11:59:26,500 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: ls /etc/ 2021-06-08 11:59:26,552 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 11:59:26,553 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: ls /cygdrive/c/Windows 2021-06-08 11:59:26,566 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: uname -m 2021-06-08 11:59:26,578 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: cat /proc/cpuinfo 2021-06-08 11:59:26,589 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: df -Thl 2021-06-08 11:59:26,604 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: cat /proc/meminfo 2021-06-08 11:59:26,615 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: hostname 2021-06-08 11:59:26,628 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: hostname -d 2021-06-08 11:59:26,640 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:559] Starting StatsWaitTask for ep_data_write_failed on bucket default 2021-06-08 11:59:26,640 | infra | INFO | MainThread | [task_manager:add_new_task:23] Added new task: StatsWaitTask_default_ep_data_write_failed_1623178766.64 2021-06-08 11:59:26,641 | infra | INFO | pool-5-thread-10 | [task:start_task:87] Thread 'StatsWaitTask_default_ep_data_write_failed_1623178766.64' started 2021-06-08 11:59:26,641 | infra | DEBUG | pool-5-thread-10 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:59:26,644 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.156 with username: root, password: couchbase 2021-06-08 11:59:26,723 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 11:59:26,842 | test | DEBUG | pool-5-thread-10 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 11:59:26,842 | infra | DEBUG | pool-5-thread-10 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.155': '0'} for bucket 'default' 2021-06-08 11:59:26,842 | infra | DEBUG | pool-5-thread-10 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 11:59:26,845 | infra | DEBUG | pool-5-thread-10 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 11:59:26,845 | infra | DEBUG | pool-5-thread-10 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 11:59:26,928 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: ls /etc/ 2021-06-08 11:59:26,986 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 11:59:26,986 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: ls /cygdrive/c/Windows 2021-06-08 11:59:27,000 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: uname -m 2021-06-08 11:59:27,012 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: cat /proc/cpuinfo 2021-06-08 11:59:27,023 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: df -Thl 2021-06-08 11:59:27,039 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: cat /proc/meminfo 2021-06-08 11:59:27,051 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: hostname 2021-06-08 11:59:27,062 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: hostname -d 2021-06-08 11:59:27,075 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:559] Starting StatsWaitTask for ep_data_write_failed on bucket default 2021-06-08 11:59:27,075 | infra | INFO | MainThread | [task_manager:add_new_task:23] Added new task: StatsWaitTask_default_ep_data_write_failed_1623178767.08 2021-06-08 11:59:27,075 | infra | INFO | pool-5-thread-26 | [task:start_task:87] Thread 'StatsWaitTask_default_ep_data_write_failed_1623178767.08' started 2021-06-08 11:59:27,075 | infra | DEBUG | pool-5-thread-26 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:59:27,078 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.122 with username: root, password: couchbase 2021-06-08 11:59:27,187 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 11:59:27,395 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: ls /etc/ 2021-06-08 11:59:27,450 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 11:59:27,450 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: ls /cygdrive/c/Windows 2021-06-08 11:59:27,463 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: uname -m 2021-06-08 11:59:27,474 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: cat /proc/cpuinfo 2021-06-08 11:59:27,486 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: df -Thl 2021-06-08 11:59:27,500 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: cat /proc/meminfo 2021-06-08 11:59:27,512 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: hostname 2021-06-08 11:59:27,523 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: hostname -d 2021-06-08 11:59:27,536 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:559] Starting StatsWaitTask for ep_data_write_failed on bucket default 2021-06-08 11:59:27,536 | infra | INFO | MainThread | [task_manager:add_new_task:23] Added new task: StatsWaitTask_default_ep_data_write_failed_1623178767.54 2021-06-08 11:59:27,536 | infra | DEBUG | MainThread | [task_manager:get_task_result:26] Getting task result for StatsWaitTask_default_ep_data_write_failed_1623178766.2 2021-06-08 11:59:27,536 | infra | INFO | pool-5-thread-14 | [task:start_task:87] Thread 'StatsWaitTask_default_ep_data_write_failed_1623178767.54' started 2021-06-08 11:59:27,538 | infra | DEBUG | pool-5-thread-14 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:59:27,653 | test | DEBUG | pool-5-thread-14 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 11:59:27,653 | test | INFO | pool-5-thread-14 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 11:59:31,322 | infra | DEBUG | pool-5-thread-1 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:59:31,444 | test | DEBUG | pool-5-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 11:59:31,446 | test | INFO | pool-5-thread-1 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1010, in main() File "/opt/couchbase/lib/python/cbstats", line 1007, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 49, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-06-08 11:59:31,845 | infra | DEBUG | pool-5-thread-10 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:59:32,046 | test | DEBUG | pool-5-thread-10 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 11:59:32,046 | infra | DEBUG | pool-5-thread-10 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.155': '0'} for bucket 'default' 2021-06-08 11:59:32,046 | infra | DEBUG | pool-5-thread-10 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 11:59:32,049 | infra | DEBUG | pool-5-thread-10 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 11:59:32,049 | infra | DEBUG | pool-5-thread-10 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 11:59:32,654 | infra | DEBUG | pool-5-thread-14 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:59:32,772 | test | DEBUG | pool-5-thread-14 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 11:59:32,772 | test | INFO | pool-5-thread-14 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 11:59:36,446 | infra | DEBUG | pool-5-thread-1 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:59:36,562 | test | DEBUG | pool-5-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 11:59:36,562 | test | INFO | pool-5-thread-1 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 11:59:37,049 | infra | DEBUG | pool-5-thread-10 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:59:37,244 | test | DEBUG | pool-5-thread-10 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 11:59:37,246 | infra | DEBUG | pool-5-thread-10 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.155': '0'} for bucket 'default' 2021-06-08 11:59:37,246 | infra | DEBUG | pool-5-thread-10 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 11:59:37,247 | infra | DEBUG | pool-5-thread-10 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 11:59:37,247 | infra | DEBUG | pool-5-thread-10 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 11:59:37,772 | infra | DEBUG | pool-5-thread-14 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:59:37,888 | test | DEBUG | pool-5-thread-14 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 11:59:37,890 | test | INFO | pool-5-thread-14 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 11:59:41,563 | infra | DEBUG | pool-5-thread-1 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:59:41,686 | test | DEBUG | pool-5-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 11:59:41,687 | test | INFO | pool-5-thread-1 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1010, in main() File "/opt/couchbase/lib/python/cbstats", line 1007, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 49, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-06-08 11:59:42,247 | infra | DEBUG | pool-5-thread-10 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:59:42,446 | test | DEBUG | pool-5-thread-10 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 11:59:42,446 | infra | DEBUG | pool-5-thread-10 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.155': '0'} for bucket 'default' 2021-06-08 11:59:42,446 | infra | DEBUG | pool-5-thread-10 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 11:59:42,448 | infra | DEBUG | pool-5-thread-10 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 11:59:42,448 | infra | DEBUG | pool-5-thread-10 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 11:59:42,890 | infra | DEBUG | pool-5-thread-14 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:59:43,006 | test | DEBUG | pool-5-thread-14 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 11:59:43,006 | test | INFO | pool-5-thread-14 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 11:59:44,292 | test | DEBUG | pool-5-thread-26 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 11:59:44,292 | infra | DEBUG | pool-5-thread-26 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.156': '0'} for bucket 'default' 2021-06-08 11:59:44,292 | infra | DEBUG | pool-5-thread-26 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 11:59:44,293 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 11:59:44,293 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 11:59:46,687 | infra | DEBUG | pool-5-thread-1 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:59:46,803 | test | DEBUG | pool-5-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 11:59:46,803 | test | INFO | pool-5-thread-1 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 11:59:47,450 | infra | DEBUG | pool-5-thread-10 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:59:47,645 | test | DEBUG | pool-5-thread-10 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 11:59:47,647 | infra | DEBUG | pool-5-thread-10 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.155': '0'} for bucket 'default' 2021-06-08 11:59:47,647 | infra | DEBUG | pool-5-thread-10 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 11:59:47,648 | infra | DEBUG | pool-5-thread-10 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 11:59:47,648 | infra | DEBUG | pool-5-thread-10 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 11:59:48,006 | infra | DEBUG | pool-5-thread-14 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:59:48,130 | test | DEBUG | pool-5-thread-14 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 11:59:48,130 | test | INFO | pool-5-thread-14 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1010, in main() File "/opt/couchbase/lib/python/cbstats", line 1007, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 49, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-06-08 11:59:49,295 | infra | DEBUG | pool-5-thread-26 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:59:51,803 | infra | DEBUG | pool-5-thread-1 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:59:51,927 | test | DEBUG | pool-5-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 11:59:51,927 | test | INFO | pool-5-thread-1 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 11:59:52,648 | infra | DEBUG | pool-5-thread-10 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:59:52,845 | test | DEBUG | pool-5-thread-10 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 11:59:52,846 | infra | DEBUG | pool-5-thread-10 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.155': '0'} for bucket 'default' 2021-06-08 11:59:52,846 | infra | DEBUG | pool-5-thread-10 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 11:59:52,848 | infra | DEBUG | pool-5-thread-10 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 11:59:52,848 | infra | DEBUG | pool-5-thread-10 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 11:59:53,131 | infra | DEBUG | pool-5-thread-14 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:59:53,250 | test | DEBUG | pool-5-thread-14 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 11:59:53,250 | test | INFO | pool-5-thread-14 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 11:59:56,927 | infra | DEBUG | pool-5-thread-1 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:59:57,043 | test | DEBUG | pool-5-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 11:59:57,043 | test | INFO | pool-5-thread-1 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 11:59:57,848 | infra | DEBUG | pool-5-thread-10 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:59:58,046 | test | DEBUG | pool-5-thread-10 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 11:59:58,046 | infra | DEBUG | pool-5-thread-10 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.155': '0'} for bucket 'default' 2021-06-08 11:59:58,046 | infra | DEBUG | pool-5-thread-10 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 11:59:58,048 | infra | DEBUG | pool-5-thread-10 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 11:59:58,049 | infra | DEBUG | pool-5-thread-10 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 11:59:58,250 | infra | DEBUG | pool-5-thread-14 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 11:59:58,369 | test | DEBUG | pool-5-thread-14 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 11:59:58,371 | test | INFO | pool-5-thread-14 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1010, in main() File "/opt/couchbase/lib/python/cbstats", line 1007, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 49, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-06-08 12:00:02,043 | infra | DEBUG | pool-5-thread-1 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:00:02,161 | test | DEBUG | pool-5-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:00:02,161 | test | INFO | pool-5-thread-1 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 12:00:03,049 | infra | DEBUG | pool-5-thread-10 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:00:03,247 | test | DEBUG | pool-5-thread-10 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 12:00:03,247 | infra | DEBUG | pool-5-thread-10 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.155': '0'} for bucket 'default' 2021-06-08 12:00:03,247 | infra | DEBUG | pool-5-thread-10 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 12:00:03,250 | infra | DEBUG | pool-5-thread-10 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 12:00:03,250 | infra | DEBUG | pool-5-thread-10 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 12:00:03,371 | infra | DEBUG | pool-5-thread-14 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:00:03,494 | test | DEBUG | pool-5-thread-14 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:00:03,494 | test | INFO | pool-5-thread-14 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 12:00:05,802 | test | DEBUG | pool-5-thread-26 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 12:00:05,802 | infra | DEBUG | pool-5-thread-26 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.156': '0'} for bucket 'default' 2021-06-08 12:00:05,802 | infra | DEBUG | pool-5-thread-26 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 12:00:05,803 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 12:00:05,805 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 12:00:07,161 | infra | DEBUG | pool-5-thread-1 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:00:07,282 | test | DEBUG | pool-5-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:00:07,282 | test | INFO | pool-5-thread-1 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 12:00:08,252 | infra | DEBUG | pool-5-thread-10 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:00:08,447 | test | DEBUG | pool-5-thread-10 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 12:00:08,447 | infra | DEBUG | pool-5-thread-10 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.155': '0'} for bucket 'default' 2021-06-08 12:00:08,447 | infra | DEBUG | pool-5-thread-10 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 12:00:08,450 | infra | DEBUG | pool-5-thread-10 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 12:00:08,450 | infra | DEBUG | pool-5-thread-10 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 12:00:08,494 | infra | DEBUG | pool-5-thread-14 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:00:08,624 | test | DEBUG | pool-5-thread-14 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:00:08,625 | test | INFO | pool-5-thread-14 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1010, in main() File "/opt/couchbase/lib/python/cbstats", line 1007, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 49, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-06-08 12:00:10,805 | infra | DEBUG | pool-5-thread-26 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:00:12,282 | infra | DEBUG | pool-5-thread-1 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:00:12,400 | test | DEBUG | pool-5-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:00:12,400 | test | INFO | pool-5-thread-1 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 12:00:13,450 | infra | DEBUG | pool-5-thread-10 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:00:13,625 | infra | DEBUG | pool-5-thread-14 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:00:13,648 | test | DEBUG | pool-5-thread-10 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 12:00:13,648 | infra | DEBUG | pool-5-thread-10 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.155': '0'} for bucket 'default' 2021-06-08 12:00:13,648 | infra | DEBUG | pool-5-thread-10 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 12:00:13,650 | infra | DEBUG | pool-5-thread-10 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 12:00:13,650 | infra | DEBUG | pool-5-thread-10 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 12:00:13,743 | test | DEBUG | pool-5-thread-14 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:00:13,743 | test | INFO | pool-5-thread-14 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 12:00:17,401 | test | DEBUG | pool-5-thread-1 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 12:00:17,401 | infra | DEBUG | pool-5-thread-1 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {} for bucket 'default' 2021-06-08 12:00:17,401 | infra | DEBUG | pool-5-thread-1 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 12:00:17,404 | infra | DEBUG | pool-5-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 12:00:17,404 | infra | DEBUG | pool-5-thread-1 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 12:00:18,651 | infra | DEBUG | pool-5-thread-10 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:00:18,743 | test | DEBUG | pool-5-thread-14 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 12:00:18,743 | infra | DEBUG | pool-5-thread-14 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {} for bucket 'default' 2021-06-08 12:00:18,743 | infra | DEBUG | pool-5-thread-14 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 12:00:18,747 | infra | DEBUG | pool-5-thread-14 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 12:00:18,747 | infra | DEBUG | pool-5-thread-14 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 12:00:18,849 | test | DEBUG | pool-5-thread-10 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 12:00:18,849 | infra | DEBUG | pool-5-thread-10 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.155': '0'} for bucket 'default' 2021-06-08 12:00:18,849 | infra | DEBUG | pool-5-thread-10 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 12:00:18,852 | infra | DEBUG | pool-5-thread-10 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 12:00:18,852 | infra | DEBUG | pool-5-thread-10 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 12:00:22,404 | infra | DEBUG | pool-5-thread-1 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:00:22,523 | test | DEBUG | pool-5-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:00:22,523 | test | INFO | pool-5-thread-1 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 12:00:23,747 | infra | DEBUG | pool-5-thread-14 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:00:23,852 | infra | DEBUG | pool-5-thread-10 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:00:23,864 | test | DEBUG | pool-5-thread-14 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:00:23,865 | test | INFO | pool-5-thread-14 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 12:00:24,052 | test | DEBUG | pool-5-thread-10 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 12:00:24,052 | infra | DEBUG | pool-5-thread-10 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.155': '0'} for bucket 'default' 2021-06-08 12:00:24,052 | infra | DEBUG | pool-5-thread-10 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 12:00:24,053 | infra | DEBUG | pool-5-thread-10 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 12:00:24,053 | infra | DEBUG | pool-5-thread-10 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 12:00:27,525 | infra | DEBUG | pool-5-thread-1 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:00:27,641 | test | DEBUG | pool-5-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:00:27,641 | test | INFO | pool-5-thread-1 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 12:00:28,865 | infra | DEBUG | pool-5-thread-14 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:00:28,993 | test | DEBUG | pool-5-thread-14 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:00:28,993 | test | INFO | pool-5-thread-14 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 12:00:29,055 | infra | DEBUG | pool-5-thread-10 | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.155 2021-06-08 12:00:29,055 | infra | INFO | pool-5-thread-10 | [task:complete_task:102] Thread 'StatsWaitTask_default_ep_data_write_failed_1623178766.64' completed 2021-06-08 12:00:32,642 | infra | DEBUG | pool-5-thread-1 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:00:32,759 | test | DEBUG | pool-5-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:00:32,759 | test | INFO | pool-5-thread-1 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 12:00:33,993 | infra | DEBUG | pool-5-thread-14 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:00:34,109 | test | DEBUG | pool-5-thread-14 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:00:34,111 | test | INFO | pool-5-thread-14 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 12:00:37,759 | infra | DEBUG | pool-5-thread-1 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:00:37,881 | test | DEBUG | pool-5-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:00:37,881 | test | INFO | pool-5-thread-1 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 12:00:39,111 | infra | DEBUG | pool-5-thread-14 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:00:39,232 | test | DEBUG | pool-5-thread-14 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:00:39,232 | test | INFO | pool-5-thread-14 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 12:00:42,881 | infra | DEBUG | pool-5-thread-1 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:00:42,999 | test | DEBUG | pool-5-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:00:43,000 | test | INFO | pool-5-thread-1 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 12:00:43,398 | test | DEBUG | pool-5-thread-26 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 12:00:43,400 | infra | DEBUG | pool-5-thread-26 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.156': '0'} for bucket 'default' 2021-06-08 12:00:43,400 | infra | DEBUG | pool-5-thread-26 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 12:00:43,401 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 12:00:43,401 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 12:00:44,233 | infra | DEBUG | pool-5-thread-14 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:00:44,348 | test | DEBUG | pool-5-thread-14 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:00:44,348 | test | INFO | pool-5-thread-14 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 12:00:48,000 | infra | DEBUG | pool-5-thread-1 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:00:48,117 | test | DEBUG | pool-5-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:00:48,118 | test | INFO | pool-5-thread-1 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 12:00:48,401 | infra | DEBUG | pool-5-thread-26 | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.156 2021-06-08 12:00:48,404 | infra | INFO | pool-5-thread-26 | [task:complete_task:102] Thread 'StatsWaitTask_default_ep_data_write_failed_1623178767.08' completed 2021-06-08 12:00:49,349 | infra | DEBUG | pool-5-thread-14 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:00:49,466 | test | DEBUG | pool-5-thread-14 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:00:49,466 | test | INFO | pool-5-thread-14 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 12:00:53,118 | infra | DEBUG | pool-5-thread-1 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:00:53,233 | test | DEBUG | pool-5-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:00:53,234 | test | INFO | pool-5-thread-1 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 12:00:54,467 | infra | DEBUG | pool-5-thread-14 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:00:54,584 | test | DEBUG | pool-5-thread-14 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:00:54,585 | test | INFO | pool-5-thread-14 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 12:00:58,234 | infra | DEBUG | pool-5-thread-1 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:00:58,351 | test | DEBUG | pool-5-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:00:58,351 | test | INFO | pool-5-thread-1 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 12:00:59,585 | infra | DEBUG | pool-5-thread-14 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:00:59,707 | test | DEBUG | pool-5-thread-14 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:00:59,707 | test | INFO | pool-5-thread-14 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 12:01:03,351 | infra | DEBUG | pool-5-thread-1 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:01:03,476 | test | DEBUG | pool-5-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:01:03,476 | test | INFO | pool-5-thread-1 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 12:01:04,707 | infra | DEBUG | pool-5-thread-14 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:01:04,828 | test | DEBUG | pool-5-thread-14 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:01:04,828 | test | INFO | pool-5-thread-14 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 12:01:08,476 | infra | DEBUG | pool-5-thread-1 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:01:08,595 | test | DEBUG | pool-5-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:01:08,595 | test | INFO | pool-5-thread-1 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 12:01:09,828 | infra | DEBUG | pool-5-thread-14 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:01:09,944 | test | DEBUG | pool-5-thread-14 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:01:09,944 | test | INFO | pool-5-thread-14 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 12:01:13,595 | test | DEBUG | pool-5-thread-1 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 12:01:13,595 | infra | DEBUG | pool-5-thread-1 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {} for bucket 'default' 2021-06-08 12:01:13,595 | infra | DEBUG | pool-5-thread-1 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 12:01:13,598 | infra | DEBUG | pool-5-thread-1 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 12:01:13,598 | infra | DEBUG | pool-5-thread-1 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 12:01:14,946 | test | DEBUG | pool-5-thread-14 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 12:01:14,946 | infra | DEBUG | pool-5-thread-14 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {} for bucket 'default' 2021-06-08 12:01:14,946 | infra | DEBUG | pool-5-thread-14 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 12:01:14,947 | infra | DEBUG | pool-5-thread-14 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 12:01:14,947 | infra | DEBUG | pool-5-thread-14 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 12:01:18,598 | infra | DEBUG | pool-5-thread-1 | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.125 2021-06-08 12:01:18,598 | infra | INFO | pool-5-thread-1 | [task:complete_task:102] Thread 'StatsWaitTask_default_ep_data_write_failed_1623178766.2' completed 2021-06-08 12:01:18,604 | infra | INFO | MainThread | [task_manager:add_new_task:23] Added new task: LoadDocsGen_default__default__default_create_12000000_13000000_1623178878.6 2021-06-08 12:01:18,604 | infra | INFO | pool-5-thread-25 | [task:start_task:87] Thread 'LoadDocsGen_default__default__default_create_12000000_13000000_1623178878.6' started 2021-06-08 12:01:18,605 | infra | DEBUG | MainThread | [task_manager:get_task_result:26] Getting task result for LoadDocsGen_default__default__default_create_12000000_13000000_1623178878.6 2021-06-08 12:01:18,654 | infra | INFO | pool-5-thread-25 | [task_manager:add_new_task:23] Added new task: PrintBucketStats_default_1623178878.64 2021-06-08 12:01:18,654 | infra | INFO | pool-5-thread-16 | [task:start_task:87] Thread 'PrintBucketStats_default_1623178878.64' started 2021-06-08 12:01:18,654 | infra | INFO | pool-5-thread-25 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_12000000_13000000_1623178878.6_create__12000000_12050000_1623178878.64 2021-06-08 12:01:18,654 | infra | INFO | pool-5-thread-23 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_12000000_13000000_1623178878.6_create__12000000_12050000_1623178878.64' started 2021-06-08 12:01:18,654 | infra | INFO | pool-5-thread-21 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_12000000_13000000_1623178878.6_create__12050000_12100000_1623178878.64' started 2021-06-08 12:01:18,654 | infra | INFO | pool-5-thread-25 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_12000000_13000000_1623178878.6_create__12050000_12100000_1623178878.64 2021-06-08 12:01:18,654 | infra | INFO | pool-5-thread-25 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_12000000_13000000_1623178878.6_create__12100000_12150000_1623178878.64 2021-06-08 12:01:18,654 | infra | INFO | pool-5-thread-5 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_12000000_13000000_1623178878.6_create__12100000_12150000_1623178878.64' started 2021-06-08 12:01:18,654 | infra | INFO | pool-5-thread-25 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_12000000_13000000_1623178878.6_create__12150000_12200000_1623178878.64 2021-06-08 12:01:18,654 | infra | INFO | pool-5-thread-15 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_12000000_13000000_1623178878.6_create__12150000_12200000_1623178878.64' started 2021-06-08 12:01:18,654 | infra | INFO | pool-5-thread-25 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_12000000_13000000_1623178878.6_create__12200000_12250000_1623178878.64 2021-06-08 12:01:18,654 | infra | INFO | pool-5-thread-3 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_12000000_13000000_1623178878.6_create__12200000_12250000_1623178878.64' started 2021-06-08 12:01:18,654 | infra | INFO | pool-5-thread-25 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_12000000_13000000_1623178878.6_create__12250000_12300000_1623178878.64 2021-06-08 12:01:18,654 | infra | INFO | pool-5-thread-30 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_12000000_13000000_1623178878.6_create__12250000_12300000_1623178878.64' started 2021-06-08 12:01:18,654 | infra | INFO | pool-5-thread-25 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_12000000_13000000_1623178878.6_create__12300000_12350000_1623178878.64 2021-06-08 12:01:18,654 | infra | INFO | pool-5-thread-2 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_12000000_13000000_1623178878.6_create__12300000_12350000_1623178878.64' started 2021-06-08 12:01:18,654 | infra | INFO | pool-5-thread-25 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_12000000_13000000_1623178878.6_create__12350000_12400000_1623178878.64 2021-06-08 12:01:18,654 | infra | INFO | pool-5-thread-6 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_12000000_13000000_1623178878.6_create__12350000_12400000_1623178878.64' started 2021-06-08 12:01:18,654 | infra | INFO | pool-5-thread-25 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_12000000_13000000_1623178878.6_create__12400000_12450000_1623178878.64 2021-06-08 12:01:18,655 | infra | INFO | pool-5-thread-9 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_12000000_13000000_1623178878.6_create__12400000_12450000_1623178878.64' started 2021-06-08 12:01:18,655 | infra | INFO | pool-5-thread-4 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_12000000_13000000_1623178878.6_create__12450000_12500000_1623178878.64' started 2021-06-08 12:01:18,655 | infra | INFO | pool-5-thread-25 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_12000000_13000000_1623178878.6_create__12450000_12500000_1623178878.64 2021-06-08 12:01:18,657 | infra | INFO | pool-5-thread-25 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_12000000_13000000_1623178878.6_create__12500000_12550000_1623178878.64 2021-06-08 12:01:18,657 | infra | INFO | pool-5-thread-22 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_12000000_13000000_1623178878.6_create__12500000_12550000_1623178878.64' started 2021-06-08 12:01:18,657 | infra | INFO | pool-5-thread-25 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_12000000_13000000_1623178878.6_create__12550000_12600000_1623178878.64 2021-06-08 12:01:18,657 | infra | INFO | pool-5-thread-12 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_12000000_13000000_1623178878.6_create__12550000_12600000_1623178878.64' started 2021-06-08 12:01:18,657 | infra | INFO | pool-5-thread-20 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_12000000_13000000_1623178878.6_create__12600000_12650000_1623178878.64' started 2021-06-08 12:01:18,657 | infra | INFO | pool-5-thread-25 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_12000000_13000000_1623178878.6_create__12600000_12650000_1623178878.64 2021-06-08 12:01:18,657 | infra | INFO | pool-5-thread-25 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_12000000_13000000_1623178878.6_create__12650000_12700000_1623178878.64 2021-06-08 12:01:18,657 | infra | INFO | pool-5-thread-11 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_12000000_13000000_1623178878.6_create__12650000_12700000_1623178878.64' started 2021-06-08 12:01:18,657 | infra | INFO | pool-5-thread-17 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_12000000_13000000_1623178878.6_create__12700000_12750000_1623178878.64' started 2021-06-08 12:01:18,657 | infra | INFO | pool-5-thread-25 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_12000000_13000000_1623178878.6_create__12700000_12750000_1623178878.64 2021-06-08 12:01:18,660 | infra | INFO | pool-5-thread-13 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_12000000_13000000_1623178878.6_create__12750000_12800000_1623178878.64' started 2021-06-08 12:01:18,660 | infra | INFO | pool-5-thread-25 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_12000000_13000000_1623178878.6_create__12750000_12800000_1623178878.64 2021-06-08 12:01:18,661 | infra | INFO | pool-5-thread-25 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_12000000_13000000_1623178878.6_create__12800000_12850000_1623178878.64 2021-06-08 12:01:18,661 | infra | INFO | pool-5-thread-18 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_12000000_13000000_1623178878.6_create__12800000_12850000_1623178878.64' started 2021-06-08 12:01:18,661 | infra | INFO | pool-5-thread-25 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_12000000_13000000_1623178878.6_create__12850000_12900000_1623178878.64 2021-06-08 12:01:18,661 | infra | INFO | pool-5-thread-25 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_12000000_13000000_1623178878.6_create__12900000_12950000_1623178878.64 2021-06-08 12:01:18,661 | infra | INFO | pool-5-thread-7 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_12000000_13000000_1623178878.6_create__12850000_12900000_1623178878.64' started 2021-06-08 12:01:18,661 | infra | INFO | pool-5-thread-28 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_12000000_13000000_1623178878.6_create__12900000_12950000_1623178878.64' started 2021-06-08 12:01:18,663 | infra | INFO | pool-5-thread-25 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_12000000_13000000_1623178878.6_create__12950000_13000000_1623178878.64 2021-06-08 12:01:18,665 | infra | INFO | pool-5-thread-24 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_12000000_13000000_1623178878.6_create__12950000_13000000_1623178878.64' started 2021-06-08 12:01:18,665 | infra | DEBUG | pool-5-thread-25 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_12000000_13000000_1623178878.6_create__12000000_12050000_1623178878.64 2021-06-08 12:01:18,878 | test | DEBUG | pool-5-thread-16 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 12:01:18,878 | test | DEBUG | pool-5-thread-16 | [task:print_ep_queue_size:3508] ep_queue_size for default: 415761 2021-06-08 12:01:18,880 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 12:01:18,881 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 12:01:19,947 | infra | DEBUG | pool-5-thread-14 | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.122 2021-06-08 12:01:19,948 | infra | INFO | pool-5-thread-14 | [task:complete_task:102] Thread 'StatsWaitTask_default_ep_data_write_failed_1623178767.54' completed 2021-06-08 12:01:20,102 | test | DEBUG | pool-5-thread-16 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 12:01:20,102 | test | DEBUG | pool-5-thread-16 | [task:print_ep_queue_size:3508] ep_queue_size for default: 415761 2021-06-08 12:01:20,105 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 12:01:20,105 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 12:01:21,368 | test | DEBUG | pool-5-thread-16 | [task:record_bucket_ops:3495] Ops rate for 'default': 3693.400000 2021-06-08 12:01:21,368 | test | DEBUG | pool-5-thread-16 | [task:print_ep_queue_size:3508] ep_queue_size for default: 430569 2021-06-08 12:01:21,371 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 12:01:21,371 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 12:01:22,808 | test | DEBUG | pool-5-thread-16 | [task:record_bucket_ops:3495] Ops rate for 'default': 3693.400000 2021-06-08 12:01:22,809 | test | DEBUG | pool-5-thread-16 | [task:print_ep_queue_size:3508] ep_queue_size for default: 430569 2021-06-08 12:01:22,818 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 12:01:22,818 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 12:01:24,312 | test | DEBUG | pool-5-thread-16 | [task:record_bucket_ops:3495] Ops rate for 'default': 3693.400000 2021-06-08 12:01:24,312 | test | DEBUG | pool-5-thread-16 | [task:print_ep_queue_size:3508] ep_queue_size for default: 430569 2021-06-08 12:01:24,315 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 12:01:24,315 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 12:01:25,575 | test | DEBUG | pool-5-thread-16 | [task:record_bucket_ops:3495] Ops rate for 'default': 3693.400000 2021-06-08 12:01:25,575 | test | DEBUG | pool-5-thread-16 | [task:print_ep_queue_size:3508] ep_queue_size for default: 430569 2021-06-08 12:01:25,578 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 12:01:25,578 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 12:01:26,825 | test | DEBUG | pool-5-thread-16 | [task:record_bucket_ops:3495] Ops rate for 'default': 3693.400000 2021-06-08 12:01:26,825 | test | DEBUG | pool-5-thread-16 | [task:print_ep_queue_size:3508] ep_queue_size for default: 430569 2021-06-08 12:01:26,828 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 12:01:26,828 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 12:01:28,052 | test | DEBUG | pool-5-thread-16 | [task:record_bucket_ops:3495] Ops rate for 'default': 3693.400000 2021-06-08 12:01:28,052 | test | DEBUG | pool-5-thread-16 | [task:print_ep_queue_size:3508] ep_queue_size for default: 430569 2021-06-08 12:01:28,055 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 12:01:28,055 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 12:01:29,334 | test | DEBUG | pool-5-thread-16 | [task:record_bucket_ops:3495] Ops rate for 'default': 3693.400000 2021-06-08 12:01:29,334 | test | DEBUG | pool-5-thread-16 | [task:print_ep_queue_size:3508] ep_queue_size for default: 430569 2021-06-08 12:01:29,336 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 12:01:29,336 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 12:01:30,753 | test | DEBUG | pool-5-thread-16 | [task:record_bucket_ops:3495] Ops rate for 'default': 3693.400000 2021-06-08 12:01:30,753 | test | DEBUG | pool-5-thread-16 | [task:print_ep_queue_size:3508] ep_queue_size for default: 430569 2021-06-08 12:01:30,756 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 12:01:30,756 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 12:01:31,967 | test | DEBUG | pool-5-thread-16 | [task:record_bucket_ops:3495] Ops rate for 'default': 12691.301540 2021-06-08 12:01:31,967 | test | DEBUG | pool-5-thread-16 | [task:print_ep_queue_size:3508] ep_queue_size for default: 495397 2021-06-08 12:01:31,969 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 12:01:31,969 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 12:01:33,244 | test | DEBUG | pool-5-thread-16 | [task:record_bucket_ops:3495] Ops rate for 'default': 12691.301540 2021-06-08 12:01:33,244 | test | DEBUG | pool-5-thread-16 | [task:print_ep_queue_size:3508] ep_queue_size for default: 495397 2021-06-08 12:01:33,249 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 12:01:33,249 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 12:01:34,536 | test | DEBUG | pool-5-thread-16 | [task:record_bucket_ops:3495] Ops rate for 'default': 12691.301540 2021-06-08 12:01:34,536 | test | DEBUG | pool-5-thread-16 | [task:print_ep_queue_size:3508] ep_queue_size for default: 495397 2021-06-08 12:01:34,540 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 12:01:34,540 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 12:01:36,309 | test | DEBUG | pool-5-thread-16 | [task:record_bucket_ops:3495] Ops rate for 'default': 12691.301540 2021-06-08 12:01:36,309 | test | DEBUG | pool-5-thread-16 | [task:print_ep_queue_size:3508] ep_queue_size for default: 495397 2021-06-08 12:01:36,315 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 12:01:36,315 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 12:01:39,819 | test | DEBUG | pool-5-thread-16 | [task:record_bucket_ops:3495] Ops rate for 'default': 12691.301540 2021-06-08 12:01:39,821 | test | DEBUG | pool-5-thread-16 | [task:print_ep_queue_size:3508] ep_queue_size for default: 495397 2021-06-08 12:01:39,823 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 12:01:39,823 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 12:01:41,052 | test | DEBUG | pool-5-thread-16 | [task:record_bucket_ops:3495] Ops rate for 'default': 12691.301540 2021-06-08 12:01:41,053 | test | DEBUG | pool-5-thread-16 | [task:print_ep_queue_size:3508] ep_queue_size for default: 495397 2021-06-08 12:01:41,055 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 12:01:41,055 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 12:01:42,556 | test | DEBUG | pool-5-thread-16 | [task:record_bucket_ops:3495] Ops rate for 'default': 9281.792958 2021-06-08 12:01:42,559 | test | DEBUG | pool-5-thread-16 | [task:print_ep_queue_size:3508] ep_queue_size for default: 542206 2021-06-08 12:01:42,562 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 12:01:42,562 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 12:01:44,000 | test | DEBUG | pool-5-thread-16 | [task:record_bucket_ops:3495] Ops rate for 'default': 9281.792958 2021-06-08 12:01:44,000 | test | DEBUG | pool-5-thread-16 | [task:print_ep_queue_size:3508] ep_queue_size for default: 542206 2021-06-08 12:01:44,003 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 12:01:44,005 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 12:01:45,315 | test | DEBUG | pool-5-thread-16 | [task:record_bucket_ops:3495] Ops rate for 'default': 9281.792958 2021-06-08 12:01:45,316 | test | DEBUG | pool-5-thread-16 | [task:print_ep_queue_size:3508] ep_queue_size for default: 542206 2021-06-08 12:01:45,319 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 12:01:45,319 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 12:01:46,884 | test | DEBUG | pool-5-thread-16 | [task:record_bucket_ops:3495] Ops rate for 'default': 9281.792958 2021-06-08 12:01:46,885 | test | DEBUG | pool-5-thread-16 | [task:print_ep_queue_size:3508] ep_queue_size for default: 542206 2021-06-08 12:01:46,888 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 12:01:46,888 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 12:01:48,105 | test | DEBUG | pool-5-thread-16 | [task:record_bucket_ops:3495] Ops rate for 'default': 9281.792958 2021-06-08 12:01:48,107 | test | DEBUG | pool-5-thread-16 | [task:print_ep_queue_size:3508] ep_queue_size for default: 542206 2021-06-08 12:01:48,108 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 12:01:48,108 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 12:01:49,321 | test | DEBUG | pool-5-thread-16 | [task:record_bucket_ops:3495] Ops rate for 'default': 9281.792958 2021-06-08 12:01:49,321 | test | DEBUG | pool-5-thread-16 | [task:print_ep_queue_size:3508] ep_queue_size for default: 542206 2021-06-08 12:01:49,323 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 12:01:49,323 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 12:01:50,589 | test | DEBUG | pool-5-thread-16 | [task:record_bucket_ops:3495] Ops rate for 'default': 9281.792958 2021-06-08 12:01:50,591 | test | DEBUG | pool-5-thread-16 | [task:print_ep_queue_size:3508] ep_queue_size for default: 542206 2021-06-08 12:01:50,592 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 12:01:50,592 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 12:01:52,082 | test | DEBUG | pool-5-thread-16 | [task:record_bucket_ops:3495] Ops rate for 'default': 12530.000000 2021-06-08 12:01:52,082 | test | DEBUG | pool-5-thread-16 | [task:print_ep_queue_size:3508] ep_queue_size for default: 604929 2021-06-08 12:01:52,088 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 12:01:52,088 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 12:01:53,545 | test | DEBUG | pool-5-thread-16 | [task:record_bucket_ops:3495] Ops rate for 'default': 12530.000000 2021-06-08 12:01:53,545 | test | DEBUG | pool-5-thread-16 | [task:print_ep_queue_size:3508] ep_queue_size for default: 604929 2021-06-08 12:01:53,548 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 12:01:53,549 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 12:01:54,836 | test | DEBUG | pool-5-thread-16 | [task:record_bucket_ops:3495] Ops rate for 'default': 12530.000000 2021-06-08 12:01:54,838 | test | DEBUG | pool-5-thread-16 | [task:print_ep_queue_size:3508] ep_queue_size for default: 604929 2021-06-08 12:01:54,839 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 12:01:54,839 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 12:01:56,063 | test | DEBUG | pool-5-thread-16 | [task:record_bucket_ops:3495] Ops rate for 'default': 12530.000000 2021-06-08 12:01:56,063 | test | DEBUG | pool-5-thread-16 | [task:print_ep_queue_size:3508] ep_queue_size for default: 604929 2021-06-08 12:01:56,066 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 12:01:56,068 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 12:01:57,289 | test | DEBUG | pool-5-thread-16 | [task:record_bucket_ops:3495] Ops rate for 'default': 12530.000000 2021-06-08 12:01:57,289 | test | DEBUG | pool-5-thread-16 | [task:print_ep_queue_size:3508] ep_queue_size for default: 604929 2021-06-08 12:01:57,292 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 12:01:57,292 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 12:01:58,513 | test | DEBUG | pool-5-thread-16 | [task:record_bucket_ops:3495] Ops rate for 'default': 12530.000000 2021-06-08 12:01:58,513 | test | DEBUG | pool-5-thread-16 | [task:print_ep_queue_size:3508] ep_queue_size for default: 604929 2021-06-08 12:01:58,515 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 12:01:58,515 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 12:01:59,079 | infra | INFO | pool-5-thread-15 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_12000000_13000000_1623178878.6_create__12150000_12200000_1623178878.64' completed 2021-06-08 12:01:59,148 | infra | INFO | pool-5-thread-28 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_12000000_13000000_1623178878.6_create__12900000_12950000_1623178878.64' completed 2021-06-08 12:01:59,176 | infra | INFO | pool-5-thread-4 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_12000000_13000000_1623178878.6_create__12450000_12500000_1623178878.64' completed 2021-06-08 12:01:59,864 | test | DEBUG | pool-5-thread-16 | [task:record_bucket_ops:3495] Ops rate for 'default': 12530.000000 2021-06-08 12:01:59,865 | test | DEBUG | pool-5-thread-16 | [task:print_ep_queue_size:3508] ep_queue_size for default: 604929 2021-06-08 12:01:59,867 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 12:01:59,867 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 12:02:00,369 | infra | INFO | pool-5-thread-18 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_12000000_13000000_1623178878.6_create__12800000_12850000_1623178878.64' completed 2021-06-08 12:02:00,371 | infra | INFO | pool-5-thread-21 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_12000000_13000000_1623178878.6_create__12050000_12100000_1623178878.64' completed 2021-06-08 12:02:00,415 | infra | INFO | pool-5-thread-20 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_12000000_13000000_1623178878.6_create__12600000_12650000_1623178878.64' completed 2021-06-08 12:02:00,424 | infra | INFO | pool-5-thread-30 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_12000000_13000000_1623178878.6_create__12250000_12300000_1623178878.64' completed 2021-06-08 12:02:00,457 | infra | INFO | pool-5-thread-7 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_12000000_13000000_1623178878.6_create__12850000_12900000_1623178878.64' completed 2021-06-08 12:02:00,473 | infra | INFO | pool-5-thread-12 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_12000000_13000000_1623178878.6_create__12550000_12600000_1623178878.64' completed 2021-06-08 12:02:00,482 | infra | INFO | pool-5-thread-22 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_12000000_13000000_1623178878.6_create__12500000_12550000_1623178878.64' completed 2021-06-08 12:02:00,634 | infra | INFO | pool-5-thread-5 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_12000000_13000000_1623178878.6_create__12100000_12150000_1623178878.64' completed 2021-06-08 12:02:01,072 | infra | INFO | pool-5-thread-2 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_12000000_13000000_1623178878.6_create__12300000_12350000_1623178878.64' completed 2021-06-08 12:02:01,417 | test | DEBUG | pool-5-thread-16 | [task:record_bucket_ops:3495] Ops rate for 'default': 12530.000000 2021-06-08 12:02:01,417 | test | DEBUG | pool-5-thread-16 | [task:print_ep_queue_size:3508] ep_queue_size for default: 604929 2021-06-08 12:02:01,420 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 12:02:01,420 | infra | DEBUG | pool-5-thread-16 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 12:02:01,579 | infra | INFO | pool-5-thread-17 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_12000000_13000000_1623178878.6_create__12700000_12750000_1623178878.64' completed 2021-06-08 12:02:01,598 | infra | INFO | pool-5-thread-6 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_12000000_13000000_1623178878.6_create__12350000_12400000_1623178878.64' completed 2021-06-08 12:02:01,605 | infra | INFO | pool-5-thread-24 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_12000000_13000000_1623178878.6_create__12950000_13000000_1623178878.64' completed 2021-06-08 12:02:01,608 | infra | INFO | pool-5-thread-3 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_12000000_13000000_1623178878.6_create__12200000_12250000_1623178878.64' completed 2021-06-08 12:02:01,615 | infra | INFO | pool-5-thread-11 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_12000000_13000000_1623178878.6_create__12650000_12700000_1623178878.64' completed 2021-06-08 12:02:01,624 | infra | INFO | pool-5-thread-9 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_12000000_13000000_1623178878.6_create__12400000_12450000_1623178878.64' completed 2021-06-08 12:02:01,805 | infra | INFO | pool-5-thread-23 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_12000000_13000000_1623178878.6_create__12000000_12050000_1623178878.64' completed 2021-06-08 12:02:01,805 | infra | DEBUG | pool-5-thread-25 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_12000000_13000000_1623178878.6_create__12000000_12050000_1623178878.64 are 50000 2021-06-08 12:02:01,805 | infra | DEBUG | pool-5-thread-25 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_12000000_13000000_1623178878.6_create__12050000_12100000_1623178878.64 2021-06-08 12:02:01,805 | infra | DEBUG | pool-5-thread-25 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_12000000_13000000_1623178878.6_create__12050000_12100000_1623178878.64 are 50000 2021-06-08 12:02:01,805 | infra | DEBUG | pool-5-thread-25 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_12000000_13000000_1623178878.6_create__12100000_12150000_1623178878.64 2021-06-08 12:02:01,805 | infra | DEBUG | pool-5-thread-25 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_12000000_13000000_1623178878.6_create__12100000_12150000_1623178878.64 are 50000 2021-06-08 12:02:01,805 | infra | DEBUG | pool-5-thread-25 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_12000000_13000000_1623178878.6_create__12150000_12200000_1623178878.64 2021-06-08 12:02:01,806 | infra | DEBUG | pool-5-thread-25 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_12000000_13000000_1623178878.6_create__12150000_12200000_1623178878.64 are 50000 2021-06-08 12:02:01,806 | infra | DEBUG | pool-5-thread-25 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_12000000_13000000_1623178878.6_create__12200000_12250000_1623178878.64 2021-06-08 12:02:01,806 | infra | DEBUG | pool-5-thread-25 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_12000000_13000000_1623178878.6_create__12200000_12250000_1623178878.64 are 50000 2021-06-08 12:02:01,806 | infra | DEBUG | pool-5-thread-25 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_12000000_13000000_1623178878.6_create__12250000_12300000_1623178878.64 2021-06-08 12:02:01,806 | infra | DEBUG | pool-5-thread-25 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_12000000_13000000_1623178878.6_create__12250000_12300000_1623178878.64 are 50000 2021-06-08 12:02:01,806 | infra | DEBUG | pool-5-thread-25 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_12000000_13000000_1623178878.6_create__12300000_12350000_1623178878.64 2021-06-08 12:02:01,806 | infra | DEBUG | pool-5-thread-25 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_12000000_13000000_1623178878.6_create__12300000_12350000_1623178878.64 are 50000 2021-06-08 12:02:01,806 | infra | DEBUG | pool-5-thread-25 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_12000000_13000000_1623178878.6_create__12350000_12400000_1623178878.64 2021-06-08 12:02:01,806 | infra | DEBUG | pool-5-thread-25 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_12000000_13000000_1623178878.6_create__12350000_12400000_1623178878.64 are 50000 2021-06-08 12:02:01,806 | infra | DEBUG | pool-5-thread-25 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_12000000_13000000_1623178878.6_create__12400000_12450000_1623178878.64 2021-06-08 12:02:01,808 | infra | DEBUG | pool-5-thread-25 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_12000000_13000000_1623178878.6_create__12400000_12450000_1623178878.64 are 50000 2021-06-08 12:02:01,808 | infra | DEBUG | pool-5-thread-25 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_12000000_13000000_1623178878.6_create__12450000_12500000_1623178878.64 2021-06-08 12:02:01,808 | infra | DEBUG | pool-5-thread-25 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_12000000_13000000_1623178878.6_create__12450000_12500000_1623178878.64 are 50000 2021-06-08 12:02:01,808 | infra | DEBUG | pool-5-thread-25 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_12000000_13000000_1623178878.6_create__12500000_12550000_1623178878.64 2021-06-08 12:02:01,808 | infra | DEBUG | pool-5-thread-25 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_12000000_13000000_1623178878.6_create__12500000_12550000_1623178878.64 are 50000 2021-06-08 12:02:01,808 | infra | DEBUG | pool-5-thread-25 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_12000000_13000000_1623178878.6_create__12550000_12600000_1623178878.64 2021-06-08 12:02:01,808 | infra | DEBUG | pool-5-thread-25 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_12000000_13000000_1623178878.6_create__12550000_12600000_1623178878.64 are 50000 2021-06-08 12:02:01,808 | infra | DEBUG | pool-5-thread-25 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_12000000_13000000_1623178878.6_create__12600000_12650000_1623178878.64 2021-06-08 12:02:01,808 | infra | DEBUG | pool-5-thread-25 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_12000000_13000000_1623178878.6_create__12600000_12650000_1623178878.64 are 50000 2021-06-08 12:02:01,808 | infra | DEBUG | pool-5-thread-25 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_12000000_13000000_1623178878.6_create__12650000_12700000_1623178878.64 2021-06-08 12:02:01,809 | infra | DEBUG | pool-5-thread-25 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_12000000_13000000_1623178878.6_create__12650000_12700000_1623178878.64 are 50000 2021-06-08 12:02:01,809 | infra | DEBUG | pool-5-thread-25 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_12000000_13000000_1623178878.6_create__12700000_12750000_1623178878.64 2021-06-08 12:02:01,809 | infra | DEBUG | pool-5-thread-25 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_12000000_13000000_1623178878.6_create__12700000_12750000_1623178878.64 are 50000 2021-06-08 12:02:01,809 | infra | DEBUG | pool-5-thread-25 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_12000000_13000000_1623178878.6_create__12750000_12800000_1623178878.64 2021-06-08 12:02:01,813 | infra | INFO | pool-5-thread-13 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_12000000_13000000_1623178878.6_create__12750000_12800000_1623178878.64' completed 2021-06-08 12:02:01,813 | infra | DEBUG | pool-5-thread-25 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_12000000_13000000_1623178878.6_create__12750000_12800000_1623178878.64 are 50000 2021-06-08 12:02:01,813 | infra | DEBUG | pool-5-thread-25 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_12000000_13000000_1623178878.6_create__12800000_12850000_1623178878.64 2021-06-08 12:02:01,813 | infra | DEBUG | pool-5-thread-25 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_12000000_13000000_1623178878.6_create__12800000_12850000_1623178878.64 are 50000 2021-06-08 12:02:01,813 | infra | DEBUG | pool-5-thread-25 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_12000000_13000000_1623178878.6_create__12850000_12900000_1623178878.64 2021-06-08 12:02:01,813 | infra | DEBUG | pool-5-thread-25 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_12000000_13000000_1623178878.6_create__12850000_12900000_1623178878.64 are 50000 2021-06-08 12:02:01,813 | infra | DEBUG | pool-5-thread-25 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_12000000_13000000_1623178878.6_create__12900000_12950000_1623178878.64 2021-06-08 12:02:01,815 | infra | DEBUG | pool-5-thread-25 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_12000000_13000000_1623178878.6_create__12900000_12950000_1623178878.64 are 50000 2021-06-08 12:02:01,815 | infra | DEBUG | pool-5-thread-25 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_12000000_13000000_1623178878.6_create__12950000_13000000_1623178878.64 2021-06-08 12:02:01,815 | infra | DEBUG | pool-5-thread-25 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_12000000_13000000_1623178878.6_create__12950000_13000000_1623178878.64 are 50000 2021-06-08 12:02:01,815 | infra | DEBUG | pool-5-thread-25 | [task_manager:get_task_result:26] Getting task result for PrintBucketStats_default_1623178878.64 2021-06-08 12:02:02,420 | test | INFO | pool-5-thread-16 | [table_view:display:72] Ops trend for bucket 'default' +----------+----------------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +----------+----------------------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | **.............................********.......................................................................*****X | 12691.302 | | 9281.793 | ........................................................................*******.........................*******X | 12530.000 | +----------+----------------------------------------------------------------------------------------------------------------------+-----------+ 2021-06-08 12:02:02,421 | infra | INFO | pool-5-thread-16 | [task:complete_task:102] Thread 'PrintBucketStats_default_1623178878.64' completed 2021-06-08 12:02:02,421 | infra | DEBUG | pool-5-thread-25 | [task:call:1663] ========= Tasks in loadgen pool======= 2021-06-08 12:02:02,421 | infra | WARNING | pool-5-thread-25 | [task_manager:print_tasks_in_pool:83] Task 'LoadDocsGen_default__default__default_create_12000000_13000000_1623178878.6' not completed 2021-06-08 12:02:02,421 | infra | DEBUG | pool-5-thread-25 | [task:call:1665] ====================================== 2021-06-08 12:02:02,421 | infra | DEBUG | pool-5-thread-25 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_12000000_13000000_1623178878.6_create__12000000_12050000_1623178878.64' complete. Loaded 50000 items 2021-06-08 12:02:02,421 | infra | DEBUG | pool-5-thread-25 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_12000000_13000000_1623178878.6_create__12050000_12100000_1623178878.64' complete. Loaded 50000 items 2021-06-08 12:02:02,421 | infra | DEBUG | pool-5-thread-25 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_12000000_13000000_1623178878.6_create__12100000_12150000_1623178878.64' complete. Loaded 50000 items 2021-06-08 12:02:02,421 | infra | DEBUG | pool-5-thread-25 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_12000000_13000000_1623178878.6_create__12150000_12200000_1623178878.64' complete. Loaded 50000 items 2021-06-08 12:02:02,421 | infra | DEBUG | pool-5-thread-25 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_12000000_13000000_1623178878.6_create__12200000_12250000_1623178878.64' complete. Loaded 50000 items 2021-06-08 12:02:02,421 | infra | DEBUG | pool-5-thread-25 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_12000000_13000000_1623178878.6_create__12250000_12300000_1623178878.64' complete. Loaded 50000 items 2021-06-08 12:02:02,421 | infra | DEBUG | pool-5-thread-25 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_12000000_13000000_1623178878.6_create__12300000_12350000_1623178878.64' complete. Loaded 50000 items 2021-06-08 12:02:02,421 | infra | DEBUG | pool-5-thread-25 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_12000000_13000000_1623178878.6_create__12350000_12400000_1623178878.64' complete. Loaded 50000 items 2021-06-08 12:02:02,421 | infra | DEBUG | pool-5-thread-25 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_12000000_13000000_1623178878.6_create__12400000_12450000_1623178878.64' complete. Loaded 50000 items 2021-06-08 12:02:02,421 | infra | DEBUG | pool-5-thread-25 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_12000000_13000000_1623178878.6_create__12450000_12500000_1623178878.64' complete. Loaded 50000 items 2021-06-08 12:02:02,421 | infra | DEBUG | pool-5-thread-25 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_12000000_13000000_1623178878.6_create__12500000_12550000_1623178878.64' complete. Loaded 50000 items 2021-06-08 12:02:02,423 | infra | DEBUG | pool-5-thread-25 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_12000000_13000000_1623178878.6_create__12550000_12600000_1623178878.64' complete. Loaded 50000 items 2021-06-08 12:02:02,423 | infra | DEBUG | pool-5-thread-25 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_12000000_13000000_1623178878.6_create__12600000_12650000_1623178878.64' complete. Loaded 50000 items 2021-06-08 12:02:02,423 | infra | DEBUG | pool-5-thread-25 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_12000000_13000000_1623178878.6_create__12650000_12700000_1623178878.64' complete. Loaded 50000 items 2021-06-08 12:02:02,423 | infra | DEBUG | pool-5-thread-25 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_12000000_13000000_1623178878.6_create__12700000_12750000_1623178878.64' complete. Loaded 50000 items 2021-06-08 12:02:02,423 | infra | DEBUG | pool-5-thread-25 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_12000000_13000000_1623178878.6_create__12750000_12800000_1623178878.64' complete. Loaded 50000 items 2021-06-08 12:02:02,423 | infra | DEBUG | pool-5-thread-25 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_12000000_13000000_1623178878.6_create__12800000_12850000_1623178878.64' complete. Loaded 50000 items 2021-06-08 12:02:02,423 | infra | DEBUG | pool-5-thread-25 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_12000000_13000000_1623178878.6_create__12850000_12900000_1623178878.64' complete. Loaded 50000 items 2021-06-08 12:02:02,423 | infra | DEBUG | pool-5-thread-25 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_12000000_13000000_1623178878.6_create__12900000_12950000_1623178878.64' complete. Loaded 50000 items 2021-06-08 12:02:02,423 | infra | DEBUG | pool-5-thread-25 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_12000000_13000000_1623178878.6_create__12950000_13000000_1623178878.64' complete. Loaded 50000 items 2021-06-08 12:02:02,423 | infra | INFO | pool-5-thread-25 | [task:complete_task:102] Thread 'LoadDocsGen_default__default__default_create_12000000_13000000_1623178878.6' completed 2021-06-08 12:02:02,426 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.125 with username: root, password: couchbase 2021-06-08 12:02:02,496 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 12:02:02,706 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: ls /etc/ 2021-06-08 12:02:02,766 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 12:02:02,767 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: ls /cygdrive/c/Windows 2021-06-08 12:02:02,779 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: uname -m 2021-06-08 12:02:02,790 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: cat /proc/cpuinfo 2021-06-08 12:02:02,803 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: df -Thl 2021-06-08 12:02:02,818 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: cat /proc/meminfo 2021-06-08 12:02:02,831 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: hostname 2021-06-08 12:02:02,842 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: hostname -d 2021-06-08 12:02:02,854 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:559] Starting StatsWaitTask for ep_data_write_failed on bucket default 2021-06-08 12:02:02,854 | infra | INFO | MainThread | [task_manager:add_new_task:23] Added new task: StatsWaitTask_default_ep_data_write_failed_1623178922.85 2021-06-08 12:02:02,854 | infra | INFO | pool-5-thread-8 | [task:start_task:87] Thread 'StatsWaitTask_default_ep_data_write_failed_1623178922.85' started 2021-06-08 12:02:02,855 | infra | DEBUG | pool-5-thread-8 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:02:02,857 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.155 with username: root, password: couchbase 2021-06-08 12:02:02,946 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 12:02:02,971 | test | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:02:02,971 | test | INFO | pool-5-thread-8 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 12:02:03,154 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: ls /etc/ 2021-06-08 12:02:03,207 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 12:02:03,207 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: ls /cygdrive/c/Windows 2021-06-08 12:02:03,220 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: uname -m 2021-06-08 12:02:03,232 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: cat /proc/cpuinfo 2021-06-08 12:02:03,243 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: df -Thl 2021-06-08 12:02:03,257 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: cat /proc/meminfo 2021-06-08 12:02:03,269 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: hostname 2021-06-08 12:02:03,280 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: hostname -d 2021-06-08 12:02:03,292 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:559] Starting StatsWaitTask for ep_data_write_failed on bucket default 2021-06-08 12:02:03,293 | infra | INFO | MainThread | [task_manager:add_new_task:23] Added new task: StatsWaitTask_default_ep_data_write_failed_1623178923.29 2021-06-08 12:02:03,293 | infra | INFO | pool-5-thread-29 | [task:start_task:87] Thread 'StatsWaitTask_default_ep_data_write_failed_1623178923.29' started 2021-06-08 12:02:03,293 | infra | DEBUG | pool-5-thread-29 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:02:03,296 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.156 with username: root, password: couchbase 2021-06-08 12:02:03,368 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 12:02:03,497 | test | DEBUG | pool-5-thread-29 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 12:02:03,497 | infra | DEBUG | pool-5-thread-29 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.155': '0'} for bucket 'default' 2021-06-08 12:02:03,497 | infra | DEBUG | pool-5-thread-29 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 12:02:03,500 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 12:02:03,500 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 12:02:03,592 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: ls /etc/ 2021-06-08 12:02:03,654 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 12:02:03,654 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: ls /cygdrive/c/Windows 2021-06-08 12:02:03,667 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: uname -m 2021-06-08 12:02:03,677 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: cat /proc/cpuinfo 2021-06-08 12:02:03,690 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: df -Thl 2021-06-08 12:02:03,704 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: cat /proc/meminfo 2021-06-08 12:02:03,717 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: hostname 2021-06-08 12:02:03,730 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: hostname -d 2021-06-08 12:02:03,744 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:559] Starting StatsWaitTask for ep_data_write_failed on bucket default 2021-06-08 12:02:03,744 | infra | INFO | MainThread | [task_manager:add_new_task:23] Added new task: StatsWaitTask_default_ep_data_write_failed_1623178923.74 2021-06-08 12:02:03,744 | infra | INFO | pool-5-thread-27 | [task:start_task:87] Thread 'StatsWaitTask_default_ep_data_write_failed_1623178923.74' started 2021-06-08 12:02:03,746 | infra | DEBUG | pool-5-thread-27 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:02:03,747 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.122 with username: root, password: couchbase 2021-06-08 12:02:03,819 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 12:02:03,960 | test | DEBUG | pool-5-thread-27 | [task:_compare:2649] Comparing 3225 > 0 2021-06-08 12:02:03,960 | test | DEBUG | pool-5-thread-27 | [task:_get_all_stats_and_compare:2601] Ready: ep_data_write_failed 3225 > 0. Received: {'172.23.106.156': '3225'} for bucket 'default' 2021-06-08 12:02:03,960 | infra | INFO | pool-5-thread-27 | [task:complete_task:102] Thread 'StatsWaitTask_default_ep_data_write_failed_1623178923.74' completed 2021-06-08 12:02:04,028 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: ls /etc/ 2021-06-08 12:02:04,079 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 12:02:04,079 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: ls /cygdrive/c/Windows 2021-06-08 12:02:04,092 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: uname -m 2021-06-08 12:02:04,104 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: cat /proc/cpuinfo 2021-06-08 12:02:04,115 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: df -Thl 2021-06-08 12:02:04,130 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: cat /proc/meminfo 2021-06-08 12:02:04,141 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: hostname 2021-06-08 12:02:04,154 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: hostname -d 2021-06-08 12:02:04,165 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:559] Starting StatsWaitTask for ep_data_write_failed on bucket default 2021-06-08 12:02:04,167 | infra | INFO | MainThread | [task_manager:add_new_task:23] Added new task: StatsWaitTask_default_ep_data_write_failed_1623178924.17 2021-06-08 12:02:04,167 | infra | INFO | pool-5-thread-19 | [task:start_task:87] Thread 'StatsWaitTask_default_ep_data_write_failed_1623178924.17' started 2021-06-08 12:02:04,167 | infra | DEBUG | MainThread | [task_manager:get_task_result:26] Getting task result for StatsWaitTask_default_ep_data_write_failed_1623178922.85 2021-06-08 12:02:04,167 | infra | DEBUG | pool-5-thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:02:04,286 | test | DEBUG | pool-5-thread-19 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:02:04,286 | test | INFO | pool-5-thread-19 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 12:02:07,973 | infra | DEBUG | pool-5-thread-8 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:02:08,088 | test | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:02:08,088 | test | INFO | pool-5-thread-8 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 12:02:08,500 | infra | DEBUG | pool-5-thread-29 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:02:08,698 | test | DEBUG | pool-5-thread-29 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 12:02:08,698 | infra | DEBUG | pool-5-thread-29 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.155': '0'} for bucket 'default' 2021-06-08 12:02:08,698 | infra | DEBUG | pool-5-thread-29 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 12:02:08,700 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 12:02:08,700 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 12:02:09,286 | infra | DEBUG | pool-5-thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:02:09,404 | test | DEBUG | pool-5-thread-19 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:02:09,404 | test | INFO | pool-5-thread-19 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 12:02:13,088 | infra | DEBUG | pool-5-thread-8 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:02:13,207 | test | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:02:13,207 | test | INFO | pool-5-thread-8 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 12:02:13,701 | infra | DEBUG | pool-5-thread-29 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:02:13,901 | test | DEBUG | pool-5-thread-29 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 12:02:13,901 | infra | DEBUG | pool-5-thread-29 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.155': '0'} for bucket 'default' 2021-06-08 12:02:13,901 | infra | DEBUG | pool-5-thread-29 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 12:02:13,903 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 12:02:13,903 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 12:02:14,405 | infra | DEBUG | pool-5-thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:02:14,523 | test | DEBUG | pool-5-thread-19 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:02:14,523 | test | INFO | pool-5-thread-19 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 12:02:18,207 | infra | DEBUG | pool-5-thread-8 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:02:18,325 | test | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:02:18,325 | test | INFO | pool-5-thread-8 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 12:02:18,904 | infra | DEBUG | pool-5-thread-29 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:02:19,101 | test | DEBUG | pool-5-thread-29 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 12:02:19,101 | infra | DEBUG | pool-5-thread-29 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.155': '0'} for bucket 'default' 2021-06-08 12:02:19,101 | infra | DEBUG | pool-5-thread-29 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 12:02:19,105 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 12:02:19,105 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 12:02:19,525 | infra | DEBUG | pool-5-thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:02:19,644 | test | DEBUG | pool-5-thread-19 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:02:19,645 | test | INFO | pool-5-thread-19 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 12:02:23,325 | infra | DEBUG | pool-5-thread-8 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:02:23,450 | test | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:02:23,450 | test | INFO | pool-5-thread-8 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 12:02:24,105 | infra | DEBUG | pool-5-thread-29 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:02:24,302 | test | DEBUG | pool-5-thread-29 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 12:02:24,302 | infra | DEBUG | pool-5-thread-29 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.155': '0'} for bucket 'default' 2021-06-08 12:02:24,302 | infra | DEBUG | pool-5-thread-29 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 12:02:24,303 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 12:02:24,303 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 12:02:24,645 | infra | DEBUG | pool-5-thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:02:24,763 | test | DEBUG | pool-5-thread-19 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:02:24,763 | test | INFO | pool-5-thread-19 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 12:02:28,450 | infra | DEBUG | pool-5-thread-8 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:02:28,572 | test | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:02:28,572 | test | INFO | pool-5-thread-8 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1010, in main() File "/opt/couchbase/lib/python/cbstats", line 1007, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 49, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-06-08 12:02:29,303 | infra | DEBUG | pool-5-thread-29 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:02:29,506 | test | DEBUG | pool-5-thread-29 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 12:02:29,506 | infra | DEBUG | pool-5-thread-29 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.155': '0'} for bucket 'default' 2021-06-08 12:02:29,506 | infra | DEBUG | pool-5-thread-29 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 12:02:29,509 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 12:02:29,509 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 12:02:29,763 | infra | DEBUG | pool-5-thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:02:29,881 | test | DEBUG | pool-5-thread-19 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:02:29,881 | test | INFO | pool-5-thread-19 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 12:02:33,572 | infra | DEBUG | pool-5-thread-8 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:02:33,690 | test | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:02:33,690 | test | INFO | pool-5-thread-8 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 12:02:34,509 | infra | DEBUG | pool-5-thread-29 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:02:34,701 | test | DEBUG | pool-5-thread-29 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 12:02:34,701 | infra | DEBUG | pool-5-thread-29 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.155': '0'} for bucket 'default' 2021-06-08 12:02:34,703 | infra | DEBUG | pool-5-thread-29 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 12:02:34,703 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 12:02:34,703 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 12:02:34,881 | infra | DEBUG | pool-5-thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:02:34,996 | test | DEBUG | pool-5-thread-19 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:02:34,996 | test | INFO | pool-5-thread-19 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 12:02:38,691 | infra | DEBUG | pool-5-thread-8 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:02:38,812 | test | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:02:38,812 | test | INFO | pool-5-thread-8 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1010, in main() File "/opt/couchbase/lib/python/cbstats", line 1007, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 49, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-06-08 12:02:39,704 | infra | DEBUG | pool-5-thread-29 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:02:39,901 | test | DEBUG | pool-5-thread-29 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 12:02:39,901 | infra | DEBUG | pool-5-thread-29 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.155': '0'} for bucket 'default' 2021-06-08 12:02:39,901 | infra | DEBUG | pool-5-thread-29 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 12:02:39,904 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 12:02:39,904 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 12:02:39,997 | infra | DEBUG | pool-5-thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:02:40,115 | test | DEBUG | pool-5-thread-19 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:02:40,117 | test | INFO | pool-5-thread-19 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 12:02:43,813 | infra | DEBUG | pool-5-thread-8 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:02:43,930 | test | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:02:43,930 | test | INFO | pool-5-thread-8 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 12:02:44,904 | infra | DEBUG | pool-5-thread-29 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:02:45,098 | test | DEBUG | pool-5-thread-29 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 12:02:45,098 | infra | DEBUG | pool-5-thread-29 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.155': '0'} for bucket 'default' 2021-06-08 12:02:45,098 | infra | DEBUG | pool-5-thread-29 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 12:02:45,101 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 12:02:45,101 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 12:02:45,117 | infra | DEBUG | pool-5-thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:02:45,237 | test | DEBUG | pool-5-thread-19 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:02:45,237 | test | INFO | pool-5-thread-19 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 12:02:48,930 | infra | DEBUG | pool-5-thread-8 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:02:49,049 | test | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:02:49,049 | test | INFO | pool-5-thread-8 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1010, in main() File "/opt/couchbase/lib/python/cbstats", line 1007, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 49, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-06-08 12:02:50,101 | infra | DEBUG | pool-5-thread-29 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:02:50,237 | infra | DEBUG | pool-5-thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:02:50,302 | test | DEBUG | pool-5-thread-29 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 12:02:50,302 | infra | DEBUG | pool-5-thread-29 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.155': '0'} for bucket 'default' 2021-06-08 12:02:50,302 | infra | DEBUG | pool-5-thread-29 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 12:02:50,303 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 12:02:50,305 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 12:02:50,355 | test | DEBUG | pool-5-thread-19 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:02:50,355 | test | INFO | pool-5-thread-19 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 12:02:54,049 | test | DEBUG | pool-5-thread-8 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 12:02:54,049 | infra | DEBUG | pool-5-thread-8 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {} for bucket 'default' 2021-06-08 12:02:54,049 | infra | DEBUG | pool-5-thread-8 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 12:02:54,052 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 12:02:54,052 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 12:02:55,305 | infra | DEBUG | pool-5-thread-29 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:02:55,355 | test | DEBUG | pool-5-thread-19 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 12:02:55,355 | infra | DEBUG | pool-5-thread-19 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {} for bucket 'default' 2021-06-08 12:02:55,355 | infra | DEBUG | pool-5-thread-19 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 12:02:55,357 | infra | DEBUG | pool-5-thread-19 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 12:02:55,358 | infra | DEBUG | pool-5-thread-19 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 12:02:55,500 | test | DEBUG | pool-5-thread-29 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 12:02:55,500 | infra | DEBUG | pool-5-thread-29 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.155': '0'} for bucket 'default' 2021-06-08 12:02:55,500 | infra | DEBUG | pool-5-thread-29 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 12:02:55,502 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 12:02:55,502 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 12:02:59,052 | infra | DEBUG | pool-5-thread-8 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:02:59,181 | test | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:02:59,181 | test | INFO | pool-5-thread-8 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1010, in main() File "/opt/couchbase/lib/python/cbstats", line 1007, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 49, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-06-08 12:03:00,358 | infra | DEBUG | pool-5-thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:03:00,476 | test | DEBUG | pool-5-thread-19 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:03:00,476 | test | INFO | pool-5-thread-19 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 12:03:00,503 | infra | DEBUG | pool-5-thread-29 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:03:00,697 | test | DEBUG | pool-5-thread-29 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 12:03:00,697 | infra | DEBUG | pool-5-thread-29 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.155': '0'} for bucket 'default' 2021-06-08 12:03:00,697 | infra | DEBUG | pool-5-thread-29 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 12:03:00,700 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 12:03:00,700 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 12:03:04,183 | infra | DEBUG | pool-5-thread-8 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:03:04,301 | test | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:03:04,301 | test | INFO | pool-5-thread-8 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 12:03:05,477 | infra | DEBUG | pool-5-thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:03:05,596 | test | DEBUG | pool-5-thread-19 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:03:05,596 | test | INFO | pool-5-thread-19 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1010, in main() File "/opt/couchbase/lib/python/cbstats", line 1007, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 49, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-06-08 12:03:05,700 | infra | DEBUG | pool-5-thread-29 | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.155 2021-06-08 12:03:05,700 | infra | INFO | pool-5-thread-29 | [task:complete_task:102] Thread 'StatsWaitTask_default_ep_data_write_failed_1623178923.29' completed 2021-06-08 12:03:09,302 | infra | DEBUG | pool-5-thread-8 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:03:09,423 | test | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:03:09,423 | test | INFO | pool-5-thread-8 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1010, in main() File "/opt/couchbase/lib/python/cbstats", line 1007, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 49, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-06-08 12:03:10,598 | infra | DEBUG | pool-5-thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:03:10,716 | test | DEBUG | pool-5-thread-19 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:03:10,716 | test | INFO | pool-5-thread-19 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 12:03:14,424 | infra | DEBUG | pool-5-thread-8 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:03:14,539 | test | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:03:14,539 | test | INFO | pool-5-thread-8 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 12:03:15,716 | infra | DEBUG | pool-5-thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:03:15,842 | test | DEBUG | pool-5-thread-19 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:03:15,842 | test | INFO | pool-5-thread-19 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1010, in main() File "/opt/couchbase/lib/python/cbstats", line 1007, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 49, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-06-08 12:03:19,539 | infra | DEBUG | pool-5-thread-8 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:03:19,694 | test | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:03:19,694 | test | INFO | pool-5-thread-8 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1010, in main() File "/opt/couchbase/lib/python/cbstats", line 1007, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 49, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 79, in g mc.sasl_auth_plain(username, password) File "/opt/couchbase/lib/python/mc_bin_client.py", line 488, in sasl_auth_plain return self.sasl_auth_start('PLAIN', '\0'.join([foruser, user, password])) File "/opt/couchbase/lib/python/mc_bin_client.py", line 484, in sasl_auth_start return self._doCmd(memcacheConstants.CMD_SASL_AUTH, mech, data) File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 281, in _handleKeyedResponse cmd, errcode, opaque, cas, keylen, extralen, rv = self._recvMsg() File "/opt/couchbase/lib/python/mc_bin_client.py", line 252, in _recvMsg raise EOFError("Got empty data (remote died?).") EOFError: Got empty data (remote died?). 2021-06-08 12:03:20,844 | infra | DEBUG | pool-5-thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:03:20,963 | test | DEBUG | pool-5-thread-19 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:03:20,963 | test | INFO | pool-5-thread-19 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 12:03:24,696 | infra | DEBUG | pool-5-thread-8 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:03:24,816 | test | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:03:24,818 | test | INFO | pool-5-thread-8 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 12:03:25,963 | infra | DEBUG | pool-5-thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:03:26,088 | test | DEBUG | pool-5-thread-19 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:03:26,088 | test | INFO | pool-5-thread-19 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1010, in main() File "/opt/couchbase/lib/python/cbstats", line 1007, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 49, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-06-08 12:03:29,818 | infra | DEBUG | pool-5-thread-8 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:03:29,934 | test | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:03:29,934 | test | INFO | pool-5-thread-8 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 12:03:31,088 | infra | DEBUG | pool-5-thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:03:31,207 | test | DEBUG | pool-5-thread-19 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:03:31,207 | test | INFO | pool-5-thread-19 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 12:03:34,934 | infra | DEBUG | pool-5-thread-8 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:03:35,051 | test | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:03:35,051 | test | INFO | pool-5-thread-8 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 12:03:36,207 | infra | DEBUG | pool-5-thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:03:36,358 | test | DEBUG | pool-5-thread-19 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:03:36,358 | test | INFO | pool-5-thread-19 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1010, in main() File "/opt/couchbase/lib/python/cbstats", line 1007, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 49, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-06-08 12:03:40,051 | infra | DEBUG | pool-5-thread-8 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:03:40,168 | test | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:03:40,168 | test | INFO | pool-5-thread-8 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 12:03:41,358 | infra | DEBUG | pool-5-thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:03:41,476 | test | DEBUG | pool-5-thread-19 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:03:41,476 | test | INFO | pool-5-thread-19 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 12:03:45,168 | infra | DEBUG | pool-5-thread-8 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:03:45,286 | test | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:03:45,286 | test | INFO | pool-5-thread-8 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 12:03:46,476 | infra | DEBUG | pool-5-thread-19 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:03:46,594 | test | DEBUG | pool-5-thread-19 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:03:46,594 | test | INFO | pool-5-thread-19 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 12:03:50,286 | test | DEBUG | pool-5-thread-8 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 12:03:50,286 | infra | DEBUG | pool-5-thread-8 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {} for bucket 'default' 2021-06-08 12:03:50,286 | infra | DEBUG | pool-5-thread-8 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 12:03:50,289 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 12:03:50,289 | infra | DEBUG | pool-5-thread-8 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 12:03:51,594 | test | DEBUG | pool-5-thread-19 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 12:03:51,595 | infra | DEBUG | pool-5-thread-19 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {} for bucket 'default' 2021-06-08 12:03:51,595 | infra | DEBUG | pool-5-thread-19 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 12:03:51,596 | infra | DEBUG | pool-5-thread-19 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 12:03:51,596 | infra | DEBUG | pool-5-thread-19 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 12:03:55,289 | infra | DEBUG | pool-5-thread-8 | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.125 2021-06-08 12:03:55,293 | infra | INFO | pool-5-thread-8 | [task:complete_task:102] Thread 'StatsWaitTask_default_ep_data_write_failed_1623178922.85' completed 2021-06-08 12:03:55,299 | infra | INFO | MainThread | [task_manager:add_new_task:23] Added new task: LoadDocsGen_default__default__default_create_13000000_14000000_1623179035.3 2021-06-08 12:03:55,299 | infra | DEBUG | MainThread | [task_manager:get_task_result:26] Getting task result for LoadDocsGen_default__default__default_create_13000000_14000000_1623179035.3 2021-06-08 12:03:55,299 | infra | INFO | pool-5-thread-10 | [task:start_task:87] Thread 'LoadDocsGen_default__default__default_create_13000000_14000000_1623179035.3' started 2021-06-08 12:03:55,329 | infra | INFO | pool-5-thread-10 | [task_manager:add_new_task:23] Added new task: PrintBucketStats_default_1623179035.32 2021-06-08 12:03:55,329 | infra | INFO | pool-5-thread-26 | [task:start_task:87] Thread 'PrintBucketStats_default_1623179035.32' started 2021-06-08 12:03:55,329 | infra | INFO | pool-5-thread-10 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_13000000_14000000_1623179035.3_create__13000000_13050000_1623179035.32 2021-06-08 12:03:55,329 | infra | INFO | pool-5-thread-1 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_13000000_14000000_1623179035.3_create__13000000_13050000_1623179035.32' started 2021-06-08 12:03:55,329 | infra | INFO | pool-5-thread-10 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_13000000_14000000_1623179035.3_create__13050000_13100000_1623179035.32 2021-06-08 12:03:55,329 | infra | INFO | pool-5-thread-14 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_13000000_14000000_1623179035.3_create__13050000_13100000_1623179035.32' started 2021-06-08 12:03:55,329 | infra | INFO | pool-5-thread-10 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_13000000_14000000_1623179035.3_create__13100000_13150000_1623179035.32 2021-06-08 12:03:55,329 | infra | INFO | pool-5-thread-15 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_13000000_14000000_1623179035.3_create__13100000_13150000_1623179035.32' started 2021-06-08 12:03:55,329 | infra | INFO | pool-5-thread-10 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_13000000_14000000_1623179035.3_create__13150000_13200000_1623179035.32 2021-06-08 12:03:55,329 | infra | INFO | pool-5-thread-10 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_13000000_14000000_1623179035.3_create__13200000_13250000_1623179035.32 2021-06-08 12:03:55,331 | infra | INFO | pool-5-thread-10 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_13000000_14000000_1623179035.3_create__13250000_13300000_1623179035.32 2021-06-08 12:03:55,331 | infra | INFO | pool-5-thread-10 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_13000000_14000000_1623179035.3_create__13300000_13350000_1623179035.32 2021-06-08 12:03:55,331 | infra | INFO | pool-5-thread-10 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_13000000_14000000_1623179035.3_create__13350000_13400000_1623179035.32 2021-06-08 12:03:55,331 | infra | INFO | pool-5-thread-10 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_13000000_14000000_1623179035.3_create__13400000_13450000_1623179035.32 2021-06-08 12:03:55,331 | infra | INFO | pool-5-thread-10 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_13000000_14000000_1623179035.3_create__13450000_13500000_1623179035.32 2021-06-08 12:03:55,331 | infra | INFO | pool-5-thread-10 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_13000000_14000000_1623179035.3_create__13500000_13550000_1623179035.32 2021-06-08 12:03:55,331 | infra | INFO | pool-5-thread-4 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_13000000_14000000_1623179035.3_create__13200000_13250000_1623179035.32' started 2021-06-08 12:03:55,331 | infra | INFO | pool-5-thread-18 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_13000000_14000000_1623179035.3_create__13250000_13300000_1623179035.32' started 2021-06-08 12:03:55,332 | infra | INFO | pool-5-thread-20 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_13000000_14000000_1623179035.3_create__13350000_13400000_1623179035.32' started 2021-06-08 12:03:55,332 | infra | INFO | pool-5-thread-30 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_13000000_14000000_1623179035.3_create__13400000_13450000_1623179035.32' started 2021-06-08 12:03:55,332 | infra | INFO | pool-5-thread-12 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_13000000_14000000_1623179035.3_create__13500000_13550000_1623179035.32' started 2021-06-08 12:03:55,332 | infra | INFO | pool-5-thread-7 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_13000000_14000000_1623179035.3_create__13450000_13500000_1623179035.32' started 2021-06-08 12:03:55,332 | infra | INFO | pool-5-thread-21 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_13000000_14000000_1623179035.3_create__13300000_13350000_1623179035.32' started 2021-06-08 12:03:55,331 | infra | INFO | pool-5-thread-28 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_13000000_14000000_1623179035.3_create__13150000_13200000_1623179035.32' started 2021-06-08 12:03:55,331 | infra | INFO | pool-5-thread-10 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_13000000_14000000_1623179035.3_create__13550000_13600000_1623179035.32 2021-06-08 12:03:55,332 | infra | INFO | pool-5-thread-22 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_13000000_14000000_1623179035.3_create__13550000_13600000_1623179035.32' started 2021-06-08 12:03:55,336 | infra | INFO | pool-5-thread-10 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_13000000_14000000_1623179035.3_create__13600000_13650000_1623179035.32 2021-06-08 12:03:55,336 | infra | INFO | pool-5-thread-5 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_13000000_14000000_1623179035.3_create__13600000_13650000_1623179035.32' started 2021-06-08 12:03:55,338 | infra | INFO | pool-5-thread-10 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_13000000_14000000_1623179035.3_create__13650000_13700000_1623179035.32 2021-06-08 12:03:55,338 | infra | INFO | pool-5-thread-10 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_13000000_14000000_1623179035.3_create__13700000_13750000_1623179035.32 2021-06-08 12:03:55,338 | infra | INFO | pool-5-thread-10 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_13000000_14000000_1623179035.3_create__13750000_13800000_1623179035.32 2021-06-08 12:03:55,338 | infra | INFO | pool-5-thread-10 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_13000000_14000000_1623179035.3_create__13800000_13850000_1623179035.32 2021-06-08 12:03:55,338 | infra | INFO | pool-5-thread-10 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_13000000_14000000_1623179035.3_create__13850000_13900000_1623179035.32 2021-06-08 12:03:55,338 | infra | INFO | pool-5-thread-10 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_13000000_14000000_1623179035.3_create__13900000_13950000_1623179035.32 2021-06-08 12:03:55,338 | infra | INFO | pool-5-thread-10 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_13000000_14000000_1623179035.3_create__13950000_14000000_1623179035.32 2021-06-08 12:03:55,338 | infra | DEBUG | pool-5-thread-10 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_13000000_14000000_1623179035.3_create__13000000_13050000_1623179035.32 2021-06-08 12:03:55,338 | infra | INFO | pool-5-thread-2 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_13000000_14000000_1623179035.3_create__13650000_13700000_1623179035.32' started 2021-06-08 12:03:55,338 | infra | INFO | pool-5-thread-6 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_13000000_14000000_1623179035.3_create__13750000_13800000_1623179035.32' started 2021-06-08 12:03:55,339 | infra | INFO | pool-5-thread-24 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_13000000_14000000_1623179035.3_create__13800000_13850000_1623179035.32' started 2021-06-08 12:03:55,338 | infra | INFO | pool-5-thread-17 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_13000000_14000000_1623179035.3_create__13700000_13750000_1623179035.32' started 2021-06-08 12:03:55,339 | infra | INFO | pool-5-thread-3 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_13000000_14000000_1623179035.3_create__13850000_13900000_1623179035.32' started 2021-06-08 12:03:55,339 | infra | INFO | pool-5-thread-9 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_13000000_14000000_1623179035.3_create__13950000_14000000_1623179035.32' started 2021-06-08 12:03:55,339 | infra | INFO | pool-5-thread-11 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_13000000_14000000_1623179035.3_create__13900000_13950000_1623179035.32' started 2021-06-08 12:03:55,622 | test | DEBUG | pool-5-thread-26 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 12:03:55,622 | test | DEBUG | pool-5-thread-26 | [task:print_ep_queue_size:3508] ep_queue_size for default: 648405 2021-06-08 12:03:55,625 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 12:03:55,625 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 12:03:56,596 | infra | DEBUG | pool-5-thread-19 | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.122 2021-06-08 12:03:56,598 | infra | INFO | pool-5-thread-19 | [task:complete_task:102] Thread 'StatsWaitTask_default_ep_data_write_failed_1623178924.17' completed 2021-06-08 12:03:56,882 | test | DEBUG | pool-5-thread-26 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 12:03:56,884 | test | DEBUG | pool-5-thread-26 | [task:print_ep_queue_size:3508] ep_queue_size for default: 648405 2021-06-08 12:03:56,885 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 12:03:56,885 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 12:03:58,326 | test | DEBUG | pool-5-thread-26 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 12:03:58,326 | test | DEBUG | pool-5-thread-26 | [task:print_ep_queue_size:3508] ep_queue_size for default: 648405 2021-06-08 12:03:58,329 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 12:03:58,329 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 12:03:59,650 | test | DEBUG | pool-5-thread-26 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 12:03:59,651 | test | DEBUG | pool-5-thread-26 | [task:print_ep_queue_size:3508] ep_queue_size for default: 648405 2021-06-08 12:03:59,654 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 12:03:59,654 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 12:04:00,904 | test | DEBUG | pool-5-thread-26 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 12:04:00,904 | test | DEBUG | pool-5-thread-26 | [task:print_ep_queue_size:3508] ep_queue_size for default: 648405 2021-06-08 12:04:00,907 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 12:04:00,907 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 12:04:02,176 | test | DEBUG | pool-5-thread-26 | [task:record_bucket_ops:3495] Ops rate for 'default': 6587.100000 2021-06-08 12:04:02,177 | test | DEBUG | pool-5-thread-26 | [task:print_ep_queue_size:3508] ep_queue_size for default: 665070 2021-06-08 12:04:02,180 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 12:04:02,180 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 12:04:03,532 | test | DEBUG | pool-5-thread-26 | [task:record_bucket_ops:3495] Ops rate for 'default': 6587.100000 2021-06-08 12:04:03,532 | test | DEBUG | pool-5-thread-26 | [task:print_ep_queue_size:3508] ep_queue_size for default: 665070 2021-06-08 12:04:03,533 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 12:04:03,535 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 12:04:05,022 | test | DEBUG | pool-5-thread-26 | [task:record_bucket_ops:3495] Ops rate for 'default': 6587.100000 2021-06-08 12:04:05,022 | test | DEBUG | pool-5-thread-26 | [task:print_ep_queue_size:3508] ep_queue_size for default: 665070 2021-06-08 12:04:05,025 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 12:04:05,025 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 12:04:06,407 | test | DEBUG | pool-5-thread-26 | [task:record_bucket_ops:3495] Ops rate for 'default': 6587.100000 2021-06-08 12:04:06,407 | test | DEBUG | pool-5-thread-26 | [task:print_ep_queue_size:3508] ep_queue_size for default: 665070 2021-06-08 12:04:06,410 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 12:04:06,410 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 12:04:07,960 | test | DEBUG | pool-5-thread-26 | [task:record_bucket_ops:3495] Ops rate for 'default': 6587.100000 2021-06-08 12:04:07,961 | test | DEBUG | pool-5-thread-26 | [task:print_ep_queue_size:3508] ep_queue_size for default: 665070 2021-06-08 12:04:07,963 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 12:04:07,964 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 12:04:09,237 | test | DEBUG | pool-5-thread-26 | [task:record_bucket_ops:3495] Ops rate for 'default': 6587.100000 2021-06-08 12:04:09,239 | test | DEBUG | pool-5-thread-26 | [task:print_ep_queue_size:3508] ep_queue_size for default: 665070 2021-06-08 12:04:09,243 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 12:04:09,243 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 12:04:10,493 | test | DEBUG | pool-5-thread-26 | [task:record_bucket_ops:3495] Ops rate for 'default': 6587.100000 2021-06-08 12:04:10,494 | test | DEBUG | pool-5-thread-26 | [task:print_ep_queue_size:3508] ep_queue_size for default: 665070 2021-06-08 12:04:10,496 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 12:04:10,497 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 12:04:11,716 | test | DEBUG | pool-5-thread-26 | [task:record_bucket_ops:3495] Ops rate for 'default': 13506.900000 2021-06-08 12:04:11,717 | test | DEBUG | pool-5-thread-26 | [task:print_ep_queue_size:3508] ep_queue_size for default: 733019 2021-06-08 12:04:11,719 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 12:04:11,719 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 12:04:15,477 | test | DEBUG | pool-5-thread-26 | [task:record_bucket_ops:3495] Ops rate for 'default': 13506.900000 2021-06-08 12:04:15,477 | test | DEBUG | pool-5-thread-26 | [task:print_ep_queue_size:3508] ep_queue_size for default: 733019 2021-06-08 12:04:15,480 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 12:04:15,480 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 12:04:16,690 | test | DEBUG | pool-5-thread-26 | [task:record_bucket_ops:3495] Ops rate for 'default': 13506.900000 2021-06-08 12:04:16,690 | test | DEBUG | pool-5-thread-26 | [task:print_ep_queue_size:3508] ep_queue_size for default: 733019 2021-06-08 12:04:16,693 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 12:04:16,693 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 12:04:17,967 | test | DEBUG | pool-5-thread-26 | [task:record_bucket_ops:3495] Ops rate for 'default': 13506.900000 2021-06-08 12:04:17,967 | test | DEBUG | pool-5-thread-26 | [task:print_ep_queue_size:3508] ep_queue_size for default: 733019 2021-06-08 12:04:17,969 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 12:04:17,969 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 12:04:19,678 | test | DEBUG | pool-5-thread-26 | [task:record_bucket_ops:3495] Ops rate for 'default': 13506.900000 2021-06-08 12:04:19,678 | test | DEBUG | pool-5-thread-26 | [task:print_ep_queue_size:3508] ep_queue_size for default: 733019 2021-06-08 12:04:19,680 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 12:04:19,680 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 12:04:21,362 | test | DEBUG | pool-5-thread-26 | [task:record_bucket_ops:3495] Ops rate for 'default': 8432.300000 2021-06-08 12:04:21,365 | test | DEBUG | pool-5-thread-26 | [task:print_ep_queue_size:3508] ep_queue_size for default: 775087 2021-06-08 12:04:21,368 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 12:04:21,368 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 12:04:23,063 | test | DEBUG | pool-5-thread-26 | [task:record_bucket_ops:3495] Ops rate for 'default': 8432.300000 2021-06-08 12:04:23,063 | test | DEBUG | pool-5-thread-26 | [task:print_ep_queue_size:3508] ep_queue_size for default: 775087 2021-06-08 12:04:23,065 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 12:04:23,065 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 12:04:24,598 | test | DEBUG | pool-5-thread-26 | [task:record_bucket_ops:3495] Ops rate for 'default': 8432.300000 2021-06-08 12:04:24,598 | test | DEBUG | pool-5-thread-26 | [task:print_ep_queue_size:3508] ep_queue_size for default: 775087 2021-06-08 12:04:24,601 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 12:04:24,601 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 12:04:25,904 | test | DEBUG | pool-5-thread-26 | [task:record_bucket_ops:3495] Ops rate for 'default': 8432.300000 2021-06-08 12:04:25,904 | test | DEBUG | pool-5-thread-26 | [task:print_ep_queue_size:3508] ep_queue_size for default: 775087 2021-06-08 12:04:25,907 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 12:04:25,907 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 12:04:27,134 | test | DEBUG | pool-5-thread-26 | [task:record_bucket_ops:3495] Ops rate for 'default': 8432.300000 2021-06-08 12:04:27,134 | test | DEBUG | pool-5-thread-26 | [task:print_ep_queue_size:3508] ep_queue_size for default: 775087 2021-06-08 12:04:27,137 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 12:04:27,137 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 12:04:28,362 | test | DEBUG | pool-5-thread-26 | [task:record_bucket_ops:3495] Ops rate for 'default': 8432.300000 2021-06-08 12:04:28,364 | test | DEBUG | pool-5-thread-26 | [task:print_ep_queue_size:3508] ep_queue_size for default: 775087 2021-06-08 12:04:28,365 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 12:04:28,365 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 12:04:29,657 | test | DEBUG | pool-5-thread-26 | [task:record_bucket_ops:3495] Ops rate for 'default': 8432.300000 2021-06-08 12:04:29,657 | test | DEBUG | pool-5-thread-26 | [task:print_ep_queue_size:3508] ep_queue_size for default: 775087 2021-06-08 12:04:29,661 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 12:04:29,661 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 12:04:30,961 | test | DEBUG | pool-5-thread-26 | [task:record_bucket_ops:3495] Ops rate for 'default': 8432.300000 2021-06-08 12:04:30,963 | test | DEBUG | pool-5-thread-26 | [task:print_ep_queue_size:3508] ep_queue_size for default: 775087 2021-06-08 12:04:30,964 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 12:04:30,964 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 12:04:32,762 | test | DEBUG | pool-5-thread-26 | [task:record_bucket_ops:3495] Ops rate for 'default': 11054.600000 2021-06-08 12:04:32,763 | test | DEBUG | pool-5-thread-26 | [task:print_ep_queue_size:3508] ep_queue_size for default: 829693 2021-06-08 12:04:32,765 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 12:04:32,766 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 12:04:34,321 | test | DEBUG | pool-5-thread-26 | [task:record_bucket_ops:3495] Ops rate for 'default': 11054.600000 2021-06-08 12:04:34,321 | test | DEBUG | pool-5-thread-26 | [task:print_ep_queue_size:3508] ep_queue_size for default: 829693 2021-06-08 12:04:34,323 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 12:04:34,323 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 12:04:35,917 | test | DEBUG | pool-5-thread-26 | [task:record_bucket_ops:3495] Ops rate for 'default': 11054.600000 2021-06-08 12:04:35,917 | test | DEBUG | pool-5-thread-26 | [task:print_ep_queue_size:3508] ep_queue_size for default: 829693 2021-06-08 12:04:35,921 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 12:04:35,921 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 12:04:35,933 | infra | INFO | pool-5-thread-3 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_13000000_14000000_1623179035.3_create__13850000_13900000_1623179035.32' completed 2021-06-08 12:04:36,878 | infra | INFO | pool-5-thread-15 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_13000000_14000000_1623179035.3_create__13100000_13150000_1623179035.32' completed 2021-06-08 12:04:37,092 | infra | INFO | pool-5-thread-17 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_13000000_14000000_1623179035.3_create__13700000_13750000_1623179035.32' completed 2021-06-08 12:04:37,437 | test | DEBUG | pool-5-thread-26 | [task:record_bucket_ops:3495] Ops rate for 'default': 11054.600000 2021-06-08 12:04:37,437 | test | DEBUG | pool-5-thread-26 | [task:print_ep_queue_size:3508] ep_queue_size for default: 829693 2021-06-08 12:04:37,440 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 12:04:37,440 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 12:04:37,868 | infra | INFO | pool-5-thread-6 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_13000000_14000000_1623179035.3_create__13750000_13800000_1623179035.32' completed 2021-06-08 12:04:38,033 | infra | INFO | pool-5-thread-30 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_13000000_14000000_1623179035.3_create__13400000_13450000_1623179035.32' completed 2021-06-08 12:04:38,065 | infra | INFO | pool-5-thread-5 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_13000000_14000000_1623179035.3_create__13600000_13650000_1623179035.32' completed 2021-06-08 12:04:38,171 | infra | INFO | pool-5-thread-9 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_13000000_14000000_1623179035.3_create__13950000_14000000_1623179035.32' completed 2021-06-08 12:04:38,244 | infra | INFO | pool-5-thread-24 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_13000000_14000000_1623179035.3_create__13800000_13850000_1623179035.32' completed 2021-06-08 12:04:38,361 | infra | INFO | pool-5-thread-12 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_13000000_14000000_1623179035.3_create__13500000_13550000_1623179035.32' completed 2021-06-08 12:04:38,394 | infra | INFO | pool-5-thread-18 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_13000000_14000000_1623179035.3_create__13250000_13300000_1623179035.32' completed 2021-06-08 12:04:38,463 | infra | INFO | pool-5-thread-2 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_13000000_14000000_1623179035.3_create__13650000_13700000_1623179035.32' completed 2021-06-08 12:04:38,470 | infra | INFO | pool-5-thread-14 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_13000000_14000000_1623179035.3_create__13050000_13100000_1623179035.32' completed 2021-06-08 12:04:38,480 | infra | INFO | pool-5-thread-7 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_13000000_14000000_1623179035.3_create__13450000_13500000_1623179035.32' completed 2021-06-08 12:04:38,490 | infra | INFO | pool-5-thread-4 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_13000000_14000000_1623179035.3_create__13200000_13250000_1623179035.32' completed 2021-06-08 12:04:38,657 | test | DEBUG | pool-5-thread-26 | [task:record_bucket_ops:3495] Ops rate for 'default': 11054.600000 2021-06-08 12:04:38,657 | test | DEBUG | pool-5-thread-26 | [task:print_ep_queue_size:3508] ep_queue_size for default: 829693 2021-06-08 12:04:38,660 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 12:04:38,660 | infra | DEBUG | pool-5-thread-26 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 12:04:39,127 | infra | INFO | pool-5-thread-1 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_13000000_14000000_1623179035.3_create__13000000_13050000_1623179035.32' completed 2021-06-08 12:04:39,128 | infra | DEBUG | pool-5-thread-10 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_13000000_14000000_1623179035.3_create__13000000_13050000_1623179035.32 are 50000 2021-06-08 12:04:39,128 | infra | DEBUG | pool-5-thread-10 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_13000000_14000000_1623179035.3_create__13050000_13100000_1623179035.32 2021-06-08 12:04:39,128 | infra | DEBUG | pool-5-thread-10 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_13000000_14000000_1623179035.3_create__13050000_13100000_1623179035.32 are 50000 2021-06-08 12:04:39,128 | infra | DEBUG | pool-5-thread-10 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_13000000_14000000_1623179035.3_create__13100000_13150000_1623179035.32 2021-06-08 12:04:39,128 | infra | DEBUG | pool-5-thread-10 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_13000000_14000000_1623179035.3_create__13100000_13150000_1623179035.32 are 50000 2021-06-08 12:04:39,128 | infra | DEBUG | pool-5-thread-10 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_13000000_14000000_1623179035.3_create__13150000_13200000_1623179035.32 2021-06-08 12:04:39,134 | infra | INFO | pool-5-thread-20 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_13000000_14000000_1623179035.3_create__13350000_13400000_1623179035.32' completed 2021-06-08 12:04:39,144 | infra | INFO | pool-5-thread-21 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_13000000_14000000_1623179035.3_create__13300000_13350000_1623179035.32' completed 2021-06-08 12:04:39,155 | infra | INFO | pool-5-thread-28 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_13000000_14000000_1623179035.3_create__13150000_13200000_1623179035.32' completed 2021-06-08 12:04:39,155 | infra | DEBUG | pool-5-thread-10 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_13000000_14000000_1623179035.3_create__13150000_13200000_1623179035.32 are 50000 2021-06-08 12:04:39,155 | infra | DEBUG | pool-5-thread-10 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_13000000_14000000_1623179035.3_create__13200000_13250000_1623179035.32 2021-06-08 12:04:39,157 | infra | DEBUG | pool-5-thread-10 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_13000000_14000000_1623179035.3_create__13200000_13250000_1623179035.32 are 50000 2021-06-08 12:04:39,157 | infra | DEBUG | pool-5-thread-10 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_13000000_14000000_1623179035.3_create__13250000_13300000_1623179035.32 2021-06-08 12:04:39,157 | infra | DEBUG | pool-5-thread-10 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_13000000_14000000_1623179035.3_create__13250000_13300000_1623179035.32 are 50000 2021-06-08 12:04:39,157 | infra | DEBUG | pool-5-thread-10 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_13000000_14000000_1623179035.3_create__13300000_13350000_1623179035.32 2021-06-08 12:04:39,157 | infra | DEBUG | pool-5-thread-10 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_13000000_14000000_1623179035.3_create__13300000_13350000_1623179035.32 are 50000 2021-06-08 12:04:39,157 | infra | DEBUG | pool-5-thread-10 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_13000000_14000000_1623179035.3_create__13350000_13400000_1623179035.32 2021-06-08 12:04:39,157 | infra | DEBUG | pool-5-thread-10 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_13000000_14000000_1623179035.3_create__13350000_13400000_1623179035.32 are 50000 2021-06-08 12:04:39,157 | infra | DEBUG | pool-5-thread-10 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_13000000_14000000_1623179035.3_create__13400000_13450000_1623179035.32 2021-06-08 12:04:39,157 | infra | INFO | pool-5-thread-22 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_13000000_14000000_1623179035.3_create__13550000_13600000_1623179035.32' completed 2021-06-08 12:04:39,157 | infra | DEBUG | pool-5-thread-10 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_13000000_14000000_1623179035.3_create__13400000_13450000_1623179035.32 are 50000 2021-06-08 12:04:39,157 | infra | DEBUG | pool-5-thread-10 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_13000000_14000000_1623179035.3_create__13450000_13500000_1623179035.32 2021-06-08 12:04:39,157 | infra | INFO | pool-5-thread-11 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_13000000_14000000_1623179035.3_create__13900000_13950000_1623179035.32' completed 2021-06-08 12:04:39,157 | infra | DEBUG | pool-5-thread-10 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_13000000_14000000_1623179035.3_create__13450000_13500000_1623179035.32 are 50000 2021-06-08 12:04:39,157 | infra | DEBUG | pool-5-thread-10 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_13000000_14000000_1623179035.3_create__13500000_13550000_1623179035.32 2021-06-08 12:04:39,157 | infra | DEBUG | pool-5-thread-10 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_13000000_14000000_1623179035.3_create__13500000_13550000_1623179035.32 are 50000 2021-06-08 12:04:39,157 | infra | DEBUG | pool-5-thread-10 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_13000000_14000000_1623179035.3_create__13550000_13600000_1623179035.32 2021-06-08 12:04:39,157 | infra | DEBUG | pool-5-thread-10 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_13000000_14000000_1623179035.3_create__13550000_13600000_1623179035.32 are 50000 2021-06-08 12:04:39,157 | infra | DEBUG | pool-5-thread-10 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_13000000_14000000_1623179035.3_create__13600000_13650000_1623179035.32 2021-06-08 12:04:39,157 | infra | DEBUG | pool-5-thread-10 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_13000000_14000000_1623179035.3_create__13600000_13650000_1623179035.32 are 50000 2021-06-08 12:04:39,157 | infra | DEBUG | pool-5-thread-10 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_13000000_14000000_1623179035.3_create__13650000_13700000_1623179035.32 2021-06-08 12:04:39,158 | infra | DEBUG | pool-5-thread-10 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_13000000_14000000_1623179035.3_create__13650000_13700000_1623179035.32 are 50000 2021-06-08 12:04:39,158 | infra | DEBUG | pool-5-thread-10 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_13000000_14000000_1623179035.3_create__13700000_13750000_1623179035.32 2021-06-08 12:04:39,158 | infra | DEBUG | pool-5-thread-10 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_13000000_14000000_1623179035.3_create__13700000_13750000_1623179035.32 are 50000 2021-06-08 12:04:39,158 | infra | DEBUG | pool-5-thread-10 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_13000000_14000000_1623179035.3_create__13750000_13800000_1623179035.32 2021-06-08 12:04:39,158 | infra | DEBUG | pool-5-thread-10 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_13000000_14000000_1623179035.3_create__13750000_13800000_1623179035.32 are 50000 2021-06-08 12:04:39,158 | infra | DEBUG | pool-5-thread-10 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_13000000_14000000_1623179035.3_create__13800000_13850000_1623179035.32 2021-06-08 12:04:39,158 | infra | DEBUG | pool-5-thread-10 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_13000000_14000000_1623179035.3_create__13800000_13850000_1623179035.32 are 50000 2021-06-08 12:04:39,158 | infra | DEBUG | pool-5-thread-10 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_13000000_14000000_1623179035.3_create__13850000_13900000_1623179035.32 2021-06-08 12:04:39,158 | infra | DEBUG | pool-5-thread-10 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_13000000_14000000_1623179035.3_create__13850000_13900000_1623179035.32 are 50000 2021-06-08 12:04:39,158 | infra | DEBUG | pool-5-thread-10 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_13000000_14000000_1623179035.3_create__13900000_13950000_1623179035.32 2021-06-08 12:04:39,160 | infra | DEBUG | pool-5-thread-10 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_13000000_14000000_1623179035.3_create__13900000_13950000_1623179035.32 are 50000 2021-06-08 12:04:39,160 | infra | DEBUG | pool-5-thread-10 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_13000000_14000000_1623179035.3_create__13950000_14000000_1623179035.32 2021-06-08 12:04:39,160 | infra | DEBUG | pool-5-thread-10 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_13000000_14000000_1623179035.3_create__13950000_14000000_1623179035.32 are 50000 2021-06-08 12:04:39,160 | infra | DEBUG | pool-5-thread-10 | [task_manager:get_task_result:26] Getting task result for PrintBucketStats_default_1623179035.32 2021-06-08 12:04:39,661 | test | INFO | pool-5-thread-26 | [table_view:display:72] Ops trend for bucket 'default' +----------+-----------------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +----------+-----------------------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | *****................................................*******....................................................****X | 13506.900 | | 8432.300 | .............................................................********...................****X | 11054.600 | +----------+-----------------------------------------------------------------------------------------------------------------------+-----------+ 2021-06-08 12:04:39,661 | infra | INFO | pool-5-thread-26 | [task:complete_task:102] Thread 'PrintBucketStats_default_1623179035.32' completed 2021-06-08 12:04:39,661 | infra | DEBUG | pool-5-thread-10 | [task:call:1663] ========= Tasks in loadgen pool======= 2021-06-08 12:04:39,661 | infra | WARNING | pool-5-thread-10 | [task_manager:print_tasks_in_pool:83] Task 'LoadDocsGen_default__default__default_create_13000000_14000000_1623179035.3' not completed 2021-06-08 12:04:39,661 | infra | DEBUG | pool-5-thread-10 | [task:call:1665] ====================================== 2021-06-08 12:04:39,661 | infra | DEBUG | pool-5-thread-10 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_13000000_14000000_1623179035.3_create__13000000_13050000_1623179035.32' complete. Loaded 50000 items 2021-06-08 12:04:39,661 | infra | DEBUG | pool-5-thread-10 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_13000000_14000000_1623179035.3_create__13050000_13100000_1623179035.32' complete. Loaded 50000 items 2021-06-08 12:04:39,661 | infra | DEBUG | pool-5-thread-10 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_13000000_14000000_1623179035.3_create__13100000_13150000_1623179035.32' complete. Loaded 50000 items 2021-06-08 12:04:39,663 | infra | DEBUG | pool-5-thread-10 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_13000000_14000000_1623179035.3_create__13150000_13200000_1623179035.32' complete. Loaded 50000 items 2021-06-08 12:04:39,663 | infra | DEBUG | pool-5-thread-10 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_13000000_14000000_1623179035.3_create__13200000_13250000_1623179035.32' complete. Loaded 50000 items 2021-06-08 12:04:39,663 | infra | DEBUG | pool-5-thread-10 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_13000000_14000000_1623179035.3_create__13250000_13300000_1623179035.32' complete. Loaded 50000 items 2021-06-08 12:04:39,663 | infra | DEBUG | pool-5-thread-10 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_13000000_14000000_1623179035.3_create__13300000_13350000_1623179035.32' complete. Loaded 50000 items 2021-06-08 12:04:39,663 | infra | DEBUG | pool-5-thread-10 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_13000000_14000000_1623179035.3_create__13350000_13400000_1623179035.32' complete. Loaded 50000 items 2021-06-08 12:04:39,663 | infra | DEBUG | pool-5-thread-10 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_13000000_14000000_1623179035.3_create__13400000_13450000_1623179035.32' complete. Loaded 50000 items 2021-06-08 12:04:39,663 | infra | DEBUG | pool-5-thread-10 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_13000000_14000000_1623179035.3_create__13450000_13500000_1623179035.32' complete. Loaded 50000 items 2021-06-08 12:04:39,663 | infra | DEBUG | pool-5-thread-10 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_13000000_14000000_1623179035.3_create__13500000_13550000_1623179035.32' complete. Loaded 50000 items 2021-06-08 12:04:39,663 | infra | DEBUG | pool-5-thread-10 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_13000000_14000000_1623179035.3_create__13550000_13600000_1623179035.32' complete. Loaded 50000 items 2021-06-08 12:04:39,663 | infra | DEBUG | pool-5-thread-10 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_13000000_14000000_1623179035.3_create__13600000_13650000_1623179035.32' complete. Loaded 50000 items 2021-06-08 12:04:39,664 | infra | DEBUG | pool-5-thread-10 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_13000000_14000000_1623179035.3_create__13650000_13700000_1623179035.32' complete. Loaded 50000 items 2021-06-08 12:04:39,664 | infra | DEBUG | pool-5-thread-10 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_13000000_14000000_1623179035.3_create__13700000_13750000_1623179035.32' complete. Loaded 50000 items 2021-06-08 12:04:39,664 | infra | DEBUG | pool-5-thread-10 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_13000000_14000000_1623179035.3_create__13750000_13800000_1623179035.32' complete. Loaded 50000 items 2021-06-08 12:04:39,664 | infra | DEBUG | pool-5-thread-10 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_13000000_14000000_1623179035.3_create__13800000_13850000_1623179035.32' complete. Loaded 50000 items 2021-06-08 12:04:39,664 | infra | DEBUG | pool-5-thread-10 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_13000000_14000000_1623179035.3_create__13850000_13900000_1623179035.32' complete. Loaded 50000 items 2021-06-08 12:04:39,664 | infra | DEBUG | pool-5-thread-10 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_13000000_14000000_1623179035.3_create__13900000_13950000_1623179035.32' complete. Loaded 50000 items 2021-06-08 12:04:39,664 | infra | DEBUG | pool-5-thread-10 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_13000000_14000000_1623179035.3_create__13950000_14000000_1623179035.32' complete. Loaded 50000 items 2021-06-08 12:04:39,664 | infra | INFO | pool-5-thread-10 | [task:complete_task:102] Thread 'LoadDocsGen_default__default__default_create_13000000_14000000_1623179035.3' completed 2021-06-08 12:04:39,667 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.125 with username: root, password: couchbase 2021-06-08 12:04:39,753 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 12:04:39,961 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: ls /etc/ 2021-06-08 12:04:40,013 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 12:04:40,013 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: ls /cygdrive/c/Windows 2021-06-08 12:04:40,026 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: uname -m 2021-06-08 12:04:40,036 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: cat /proc/cpuinfo 2021-06-08 12:04:40,048 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: df -Thl 2021-06-08 12:04:40,062 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: cat /proc/meminfo 2021-06-08 12:04:40,072 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: hostname 2021-06-08 12:04:40,084 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: hostname -d 2021-06-08 12:04:40,095 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:559] Starting StatsWaitTask for ep_data_write_failed on bucket default 2021-06-08 12:04:40,095 | infra | INFO | MainThread | [task_manager:add_new_task:23] Added new task: StatsWaitTask_default_ep_data_write_failed_1623179080.1 2021-06-08 12:04:40,095 | infra | INFO | pool-5-thread-23 | [task:start_task:87] Thread 'StatsWaitTask_default_ep_data_write_failed_1623179080.1' started 2021-06-08 12:04:40,096 | infra | DEBUG | pool-5-thread-23 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:04:40,098 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.155 with username: root, password: couchbase 2021-06-08 12:04:40,171 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 12:04:40,213 | test | DEBUG | pool-5-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:04:40,213 | test | INFO | pool-5-thread-23 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 12:04:40,377 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: ls /etc/ 2021-06-08 12:04:40,427 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 12:04:40,427 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: ls /cygdrive/c/Windows 2021-06-08 12:04:40,440 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: uname -m 2021-06-08 12:04:40,450 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: cat /proc/cpuinfo 2021-06-08 12:04:40,461 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: df -Thl 2021-06-08 12:04:40,476 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: cat /proc/meminfo 2021-06-08 12:04:40,486 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: hostname 2021-06-08 12:04:40,497 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: hostname -d 2021-06-08 12:04:40,509 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:559] Starting StatsWaitTask for ep_data_write_failed on bucket default 2021-06-08 12:04:40,509 | infra | INFO | MainThread | [task_manager:add_new_task:23] Added new task: StatsWaitTask_default_ep_data_write_failed_1623179080.51 2021-06-08 12:04:40,509 | infra | INFO | pool-5-thread-13 | [task:start_task:87] Thread 'StatsWaitTask_default_ep_data_write_failed_1623179080.51' started 2021-06-08 12:04:40,510 | infra | DEBUG | pool-5-thread-13 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:04:40,513 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.156 with username: root, password: couchbase 2021-06-08 12:04:40,581 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 12:04:40,703 | test | DEBUG | pool-5-thread-13 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 12:04:40,704 | infra | DEBUG | pool-5-thread-13 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.155': '0'} for bucket 'default' 2021-06-08 12:04:40,704 | infra | DEBUG | pool-5-thread-13 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 12:04:40,707 | infra | DEBUG | pool-5-thread-13 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 12:04:40,707 | infra | DEBUG | pool-5-thread-13 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 12:04:40,799 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: ls /etc/ 2021-06-08 12:04:40,855 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 12:04:40,857 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: ls /cygdrive/c/Windows 2021-06-08 12:04:40,871 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: uname -m 2021-06-08 12:04:40,882 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: cat /proc/cpuinfo 2021-06-08 12:04:40,894 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: df -Thl 2021-06-08 12:04:40,911 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: cat /proc/meminfo 2021-06-08 12:04:40,923 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: hostname 2021-06-08 12:04:40,936 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: hostname -d 2021-06-08 12:04:40,948 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:559] Starting StatsWaitTask for ep_data_write_failed on bucket default 2021-06-08 12:04:40,948 | infra | INFO | MainThread | [task_manager:add_new_task:23] Added new task: StatsWaitTask_default_ep_data_write_failed_1623179080.95 2021-06-08 12:04:40,948 | infra | INFO | pool-5-thread-16 | [task:start_task:87] Thread 'StatsWaitTask_default_ep_data_write_failed_1623179080.95' started 2021-06-08 12:04:40,950 | infra | DEBUG | pool-5-thread-16 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:04:40,951 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.122 with username: root, password: couchbase 2021-06-08 12:04:41,019 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 12:04:41,161 | test | DEBUG | pool-5-thread-16 | [task:_compare:2649] Comparing 12053 > 0 2021-06-08 12:04:41,163 | test | DEBUG | pool-5-thread-16 | [task:_get_all_stats_and_compare:2601] Ready: ep_data_write_failed 12053 > 0. Received: {'172.23.106.156': '12053'} for bucket 'default' 2021-06-08 12:04:41,163 | infra | INFO | pool-5-thread-16 | [task:complete_task:102] Thread 'StatsWaitTask_default_ep_data_write_failed_1623179080.95' completed 2021-06-08 12:04:41,227 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: ls /etc/ 2021-06-08 12:04:41,279 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 12:04:41,279 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: ls /cygdrive/c/Windows 2021-06-08 12:04:41,290 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: uname -m 2021-06-08 12:04:41,302 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: cat /proc/cpuinfo 2021-06-08 12:04:41,312 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: df -Thl 2021-06-08 12:04:41,326 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: cat /proc/meminfo 2021-06-08 12:04:41,338 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: hostname 2021-06-08 12:04:41,348 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: hostname -d 2021-06-08 12:04:41,361 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:559] Starting StatsWaitTask for ep_data_write_failed on bucket default 2021-06-08 12:04:41,361 | infra | INFO | MainThread | [task_manager:add_new_task:23] Added new task: StatsWaitTask_default_ep_data_write_failed_1623179081.36 2021-06-08 12:04:41,361 | infra | INFO | pool-5-thread-25 | [task:start_task:87] Thread 'StatsWaitTask_default_ep_data_write_failed_1623179081.36' started 2021-06-08 12:04:41,361 | infra | DEBUG | MainThread | [task_manager:get_task_result:26] Getting task result for StatsWaitTask_default_ep_data_write_failed_1623179080.1 2021-06-08 12:04:41,362 | infra | DEBUG | pool-5-thread-25 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:04:41,479 | test | DEBUG | pool-5-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:04:41,479 | test | INFO | pool-5-thread-25 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 12:04:45,308 | infra | DEBUG | pool-5-thread-23 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:04:45,430 | test | DEBUG | pool-5-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:04:45,430 | test | INFO | pool-5-thread-23 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 12:04:45,707 | infra | DEBUG | pool-5-thread-13 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:04:45,904 | test | DEBUG | pool-5-thread-13 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 12:04:45,905 | infra | DEBUG | pool-5-thread-13 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.155': '0'} for bucket 'default' 2021-06-08 12:04:45,905 | infra | DEBUG | pool-5-thread-13 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 12:04:45,907 | infra | DEBUG | pool-5-thread-13 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 12:04:45,907 | infra | DEBUG | pool-5-thread-13 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 12:04:46,479 | infra | DEBUG | pool-5-thread-25 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:04:46,598 | test | DEBUG | pool-5-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:04:46,598 | test | INFO | pool-5-thread-25 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 12:04:50,430 | infra | DEBUG | pool-5-thread-23 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:04:50,546 | test | DEBUG | pool-5-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:04:50,546 | test | INFO | pool-5-thread-23 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 12:04:50,907 | infra | DEBUG | pool-5-thread-13 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:04:51,108 | test | DEBUG | pool-5-thread-13 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 12:04:51,108 | infra | DEBUG | pool-5-thread-13 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.155': '0'} for bucket 'default' 2021-06-08 12:04:51,108 | infra | DEBUG | pool-5-thread-13 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 12:04:51,111 | infra | DEBUG | pool-5-thread-13 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 12:04:51,111 | infra | DEBUG | pool-5-thread-13 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 12:04:51,598 | infra | DEBUG | pool-5-thread-25 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:04:51,714 | test | DEBUG | pool-5-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:04:51,714 | test | INFO | pool-5-thread-25 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 12:04:55,548 | infra | DEBUG | pool-5-thread-23 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:04:55,665 | test | DEBUG | pool-5-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:04:55,665 | test | INFO | pool-5-thread-23 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 12:04:56,111 | infra | DEBUG | pool-5-thread-13 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:04:56,306 | test | DEBUG | pool-5-thread-13 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 12:04:56,308 | infra | DEBUG | pool-5-thread-13 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.155': '0'} for bucket 'default' 2021-06-08 12:04:56,308 | infra | DEBUG | pool-5-thread-13 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 12:04:56,309 | infra | DEBUG | pool-5-thread-13 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 12:04:56,309 | infra | DEBUG | pool-5-thread-13 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 12:04:56,716 | infra | DEBUG | pool-5-thread-25 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:04:56,832 | test | DEBUG | pool-5-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:04:56,832 | test | INFO | pool-5-thread-25 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 12:05:00,665 | infra | DEBUG | pool-5-thread-23 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:05:00,786 | test | DEBUG | pool-5-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:05:00,786 | test | INFO | pool-5-thread-23 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 12:05:01,309 | infra | DEBUG | pool-5-thread-13 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:05:01,506 | test | DEBUG | pool-5-thread-13 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 12:05:01,506 | infra | DEBUG | pool-5-thread-13 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.155': '0'} for bucket 'default' 2021-06-08 12:05:01,506 | infra | DEBUG | pool-5-thread-13 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 12:05:01,507 | infra | DEBUG | pool-5-thread-13 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 12:05:01,507 | infra | DEBUG | pool-5-thread-13 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 12:05:01,832 | infra | DEBUG | pool-5-thread-25 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:05:01,953 | test | DEBUG | pool-5-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:05:01,953 | test | INFO | pool-5-thread-25 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 12:05:05,786 | infra | DEBUG | pool-5-thread-23 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:05:05,901 | test | DEBUG | pool-5-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:05:05,901 | test | INFO | pool-5-thread-23 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 12:05:06,509 | infra | DEBUG | pool-5-thread-13 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:05:06,706 | test | DEBUG | pool-5-thread-13 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 12:05:06,706 | infra | DEBUG | pool-5-thread-13 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.155': '0'} for bucket 'default' 2021-06-08 12:05:06,706 | infra | DEBUG | pool-5-thread-13 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 12:05:06,707 | infra | DEBUG | pool-5-thread-13 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 12:05:06,707 | infra | DEBUG | pool-5-thread-13 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 12:05:06,953 | infra | DEBUG | pool-5-thread-25 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:05:07,069 | test | DEBUG | pool-5-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:05:07,071 | test | INFO | pool-5-thread-25 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 12:05:10,901 | infra | DEBUG | pool-5-thread-23 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:05:11,020 | test | DEBUG | pool-5-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:05:11,020 | test | INFO | pool-5-thread-23 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 12:05:11,707 | infra | DEBUG | pool-5-thread-13 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:05:11,903 | test | DEBUG | pool-5-thread-13 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 12:05:11,903 | infra | DEBUG | pool-5-thread-13 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.155': '0'} for bucket 'default' 2021-06-08 12:05:11,904 | infra | DEBUG | pool-5-thread-13 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 12:05:11,904 | infra | DEBUG | pool-5-thread-13 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 12:05:11,904 | infra | DEBUG | pool-5-thread-13 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 12:05:12,071 | infra | DEBUG | pool-5-thread-25 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:05:12,187 | test | DEBUG | pool-5-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:05:12,187 | test | INFO | pool-5-thread-25 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 12:05:16,020 | infra | DEBUG | pool-5-thread-23 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:05:16,138 | test | DEBUG | pool-5-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:05:16,138 | test | INFO | pool-5-thread-23 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 12:05:16,905 | infra | DEBUG | pool-5-thread-13 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:05:17,099 | test | DEBUG | pool-5-thread-13 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 12:05:17,099 | infra | DEBUG | pool-5-thread-13 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.155': '0'} for bucket 'default' 2021-06-08 12:05:17,099 | infra | DEBUG | pool-5-thread-13 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 12:05:17,102 | infra | DEBUG | pool-5-thread-13 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 12:05:17,102 | infra | DEBUG | pool-5-thread-13 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 12:05:17,187 | infra | DEBUG | pool-5-thread-25 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:05:17,303 | test | DEBUG | pool-5-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:05:17,305 | test | INFO | pool-5-thread-25 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 12:05:21,138 | infra | DEBUG | pool-5-thread-23 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:05:21,256 | test | DEBUG | pool-5-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:05:21,256 | test | INFO | pool-5-thread-23 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 12:05:22,102 | infra | DEBUG | pool-5-thread-13 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:05:22,299 | test | DEBUG | pool-5-thread-13 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 12:05:22,299 | infra | DEBUG | pool-5-thread-13 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.155': '0'} for bucket 'default' 2021-06-08 12:05:22,299 | infra | DEBUG | pool-5-thread-13 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 12:05:22,302 | infra | DEBUG | pool-5-thread-13 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 12:05:22,302 | infra | DEBUG | pool-5-thread-13 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 12:05:22,305 | infra | DEBUG | pool-5-thread-25 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:05:22,426 | test | DEBUG | pool-5-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:05:22,426 | test | INFO | pool-5-thread-25 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 12:05:26,256 | infra | DEBUG | pool-5-thread-23 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:05:26,375 | test | DEBUG | pool-5-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:05:26,375 | test | INFO | pool-5-thread-23 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 12:05:27,302 | infra | DEBUG | pool-5-thread-13 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:05:27,426 | infra | DEBUG | pool-5-thread-25 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:05:27,506 | test | DEBUG | pool-5-thread-13 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 12:05:27,506 | infra | DEBUG | pool-5-thread-13 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.155': '0'} for bucket 'default' 2021-06-08 12:05:27,506 | infra | DEBUG | pool-5-thread-13 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 12:05:27,507 | infra | DEBUG | pool-5-thread-13 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 12:05:27,507 | infra | DEBUG | pool-5-thread-13 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 12:05:27,542 | test | DEBUG | pool-5-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:05:27,542 | test | INFO | pool-5-thread-25 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 12:05:31,375 | test | DEBUG | pool-5-thread-23 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 12:05:31,375 | infra | DEBUG | pool-5-thread-23 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {} for bucket 'default' 2021-06-08 12:05:31,375 | infra | DEBUG | pool-5-thread-23 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 12:05:31,377 | infra | DEBUG | pool-5-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 12:05:31,377 | infra | DEBUG | pool-5-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 12:05:32,509 | infra | DEBUG | pool-5-thread-13 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:05:32,542 | test | DEBUG | pool-5-thread-25 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 12:05:32,542 | infra | DEBUG | pool-5-thread-25 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {} for bucket 'default' 2021-06-08 12:05:32,542 | infra | DEBUG | pool-5-thread-25 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 12:05:32,545 | infra | DEBUG | pool-5-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 12:05:32,545 | infra | DEBUG | pool-5-thread-25 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 12:05:32,706 | test | DEBUG | pool-5-thread-13 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 12:05:32,707 | infra | DEBUG | pool-5-thread-13 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.155': '0'} for bucket 'default' 2021-06-08 12:05:32,707 | infra | DEBUG | pool-5-thread-13 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 12:05:32,707 | infra | DEBUG | pool-5-thread-13 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 12:05:32,709 | infra | DEBUG | pool-5-thread-13 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 12:05:36,378 | infra | DEBUG | pool-5-thread-23 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:05:36,496 | test | DEBUG | pool-5-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:05:36,496 | test | INFO | pool-5-thread-23 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 12:05:37,546 | infra | DEBUG | pool-5-thread-25 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:05:37,663 | test | DEBUG | pool-5-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:05:37,663 | test | INFO | pool-5-thread-25 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 12:05:37,709 | infra | DEBUG | pool-5-thread-13 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:05:37,907 | test | DEBUG | pool-5-thread-13 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 12:05:37,907 | infra | DEBUG | pool-5-thread-13 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.155': '0'} for bucket 'default' 2021-06-08 12:05:37,908 | infra | DEBUG | pool-5-thread-13 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 12:05:37,910 | infra | DEBUG | pool-5-thread-13 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 12:05:37,911 | infra | DEBUG | pool-5-thread-13 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 12:05:41,496 | infra | DEBUG | pool-5-thread-23 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:05:41,612 | test | DEBUG | pool-5-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:05:41,612 | test | INFO | pool-5-thread-23 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 12:05:42,664 | infra | DEBUG | pool-5-thread-25 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:05:42,780 | test | DEBUG | pool-5-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:05:42,780 | test | INFO | pool-5-thread-25 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 12:05:42,911 | infra | DEBUG | pool-5-thread-13 | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.155 2021-06-08 12:05:42,911 | infra | INFO | pool-5-thread-13 | [task:complete_task:102] Thread 'StatsWaitTask_default_ep_data_write_failed_1623179080.51' completed 2021-06-08 12:05:46,612 | infra | DEBUG | pool-5-thread-23 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:05:46,732 | test | DEBUG | pool-5-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:05:46,732 | test | INFO | pool-5-thread-23 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1010, in main() File "/opt/couchbase/lib/python/cbstats", line 1007, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 49, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-06-08 12:05:47,783 | infra | DEBUG | pool-5-thread-25 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:05:47,901 | test | DEBUG | pool-5-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:05:47,901 | test | INFO | pool-5-thread-25 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 12:05:51,733 | infra | DEBUG | pool-5-thread-23 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:05:51,849 | test | DEBUG | pool-5-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:05:51,849 | test | INFO | pool-5-thread-23 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 12:05:52,901 | infra | DEBUG | pool-5-thread-25 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:05:53,020 | test | DEBUG | pool-5-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:05:53,020 | test | INFO | pool-5-thread-25 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1010, in main() File "/opt/couchbase/lib/python/cbstats", line 1007, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 49, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-06-08 12:05:56,849 | infra | DEBUG | pool-5-thread-23 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:05:56,970 | test | DEBUG | pool-5-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:05:56,971 | test | INFO | pool-5-thread-23 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1010, in main() File "/opt/couchbase/lib/python/cbstats", line 1007, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 49, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-06-08 12:05:58,020 | infra | DEBUG | pool-5-thread-25 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:05:58,138 | test | DEBUG | pool-5-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:05:58,138 | test | INFO | pool-5-thread-25 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 12:06:01,971 | infra | DEBUG | pool-5-thread-23 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:06:02,089 | test | DEBUG | pool-5-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:06:02,089 | test | INFO | pool-5-thread-23 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 12:06:03,138 | infra | DEBUG | pool-5-thread-25 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:06:03,265 | test | DEBUG | pool-5-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:06:03,266 | test | INFO | pool-5-thread-25 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1010, in main() File "/opt/couchbase/lib/python/cbstats", line 1007, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 49, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-06-08 12:06:07,091 | infra | DEBUG | pool-5-thread-23 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:06:07,213 | test | DEBUG | pool-5-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:06:07,213 | test | INFO | pool-5-thread-23 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1010, in main() File "/opt/couchbase/lib/python/cbstats", line 1007, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 49, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-06-08 12:06:08,266 | infra | DEBUG | pool-5-thread-25 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:06:08,384 | test | DEBUG | pool-5-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:06:08,384 | test | INFO | pool-5-thread-25 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 12:06:12,213 | infra | DEBUG | pool-5-thread-23 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:06:12,328 | test | DEBUG | pool-5-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:06:12,329 | test | INFO | pool-5-thread-23 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 12:06:13,384 | infra | DEBUG | pool-5-thread-25 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:06:13,506 | test | DEBUG | pool-5-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:06:13,506 | test | INFO | pool-5-thread-25 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1010, in main() File "/opt/couchbase/lib/python/cbstats", line 1007, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 49, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-06-08 12:06:17,329 | infra | DEBUG | pool-5-thread-23 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:06:17,448 | test | DEBUG | pool-5-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:06:17,448 | test | INFO | pool-5-thread-23 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1010, in main() File "/opt/couchbase/lib/python/cbstats", line 1007, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 49, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-06-08 12:06:18,507 | infra | DEBUG | pool-5-thread-25 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:06:18,627 | test | DEBUG | pool-5-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:06:18,627 | test | INFO | pool-5-thread-25 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 12:06:22,450 | infra | DEBUG | pool-5-thread-23 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:06:22,566 | test | DEBUG | pool-5-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:06:22,566 | test | INFO | pool-5-thread-23 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 12:06:23,627 | infra | DEBUG | pool-5-thread-25 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:06:23,749 | test | DEBUG | pool-5-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:06:23,749 | test | INFO | pool-5-thread-25 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1010, in main() File "/opt/couchbase/lib/python/cbstats", line 1007, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 49, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-06-08 12:06:27,568 | test | DEBUG | pool-5-thread-23 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 12:06:27,568 | infra | DEBUG | pool-5-thread-23 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {} for bucket 'default' 2021-06-08 12:06:27,569 | infra | DEBUG | pool-5-thread-23 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 12:06:27,571 | infra | DEBUG | pool-5-thread-23 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 12:06:27,571 | infra | DEBUG | pool-5-thread-23 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 12:06:28,750 | test | DEBUG | pool-5-thread-25 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 12:06:28,750 | infra | DEBUG | pool-5-thread-25 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {} for bucket 'default' 2021-06-08 12:06:28,750 | infra | DEBUG | pool-5-thread-25 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 12:06:28,752 | infra | DEBUG | pool-5-thread-25 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 12:06:28,752 | infra | DEBUG | pool-5-thread-25 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 12:06:32,571 | infra | DEBUG | pool-5-thread-23 | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.125 2021-06-08 12:06:32,572 | infra | INFO | pool-5-thread-23 | [task:complete_task:102] Thread 'StatsWaitTask_default_ep_data_write_failed_1623179080.1' completed 2021-06-08 12:06:32,578 | infra | INFO | MainThread | [task_manager:add_new_task:23] Added new task: LoadDocsGen_default__default__default_create_14000000_15000000_1623179192.58 2021-06-08 12:06:32,578 | infra | INFO | pool-5-thread-27 | [task:start_task:87] Thread 'LoadDocsGen_default__default__default_create_14000000_15000000_1623179192.58' started 2021-06-08 12:06:32,578 | infra | DEBUG | MainThread | [task_manager:get_task_result:26] Getting task result for LoadDocsGen_default__default__default_create_14000000_15000000_1623179192.58 2021-06-08 12:06:32,655 | infra | INFO | pool-5-thread-27 | [task_manager:add_new_task:23] Added new task: PrintBucketStats_default_1623179192.64 2021-06-08 12:06:32,655 | infra | INFO | pool-5-thread-29 | [task:start_task:87] Thread 'PrintBucketStats_default_1623179192.64' started 2021-06-08 12:06:32,655 | infra | INFO | pool-5-thread-27 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_14000000_15000000_1623179192.58_create__14000000_14050000_1623179192.64 2021-06-08 12:06:32,655 | infra | INFO | pool-5-thread-8 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_14000000_15000000_1623179192.58_create__14000000_14050000_1623179192.64' started 2021-06-08 12:06:32,655 | infra | INFO | pool-5-thread-27 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_14000000_15000000_1623179192.58_create__14050000_14100000_1623179192.64 2021-06-08 12:06:32,655 | infra | INFO | pool-5-thread-19 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_14000000_15000000_1623179192.58_create__14050000_14100000_1623179192.64' started 2021-06-08 12:06:32,657 | infra | INFO | pool-5-thread-27 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_14000000_15000000_1623179192.58_create__14100000_14150000_1623179192.64 2021-06-08 12:06:32,657 | infra | INFO | pool-5-thread-3 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_14000000_15000000_1623179192.58_create__14100000_14150000_1623179192.64' started 2021-06-08 12:06:32,657 | infra | INFO | pool-5-thread-27 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_14000000_15000000_1623179192.58_create__14150000_14200000_1623179192.64 2021-06-08 12:06:32,657 | infra | INFO | pool-5-thread-15 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_14000000_15000000_1623179192.58_create__14150000_14200000_1623179192.64' started 2021-06-08 12:06:32,657 | infra | INFO | pool-5-thread-27 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_14000000_15000000_1623179192.58_create__14200000_14250000_1623179192.64 2021-06-08 12:06:32,657 | infra | INFO | pool-5-thread-27 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_14000000_15000000_1623179192.58_create__14250000_14300000_1623179192.64 2021-06-08 12:06:32,657 | infra | INFO | pool-5-thread-17 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_14000000_15000000_1623179192.58_create__14200000_14250000_1623179192.64' started 2021-06-08 12:06:32,657 | infra | INFO | pool-5-thread-6 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_14000000_15000000_1623179192.58_create__14250000_14300000_1623179192.64' started 2021-06-08 12:06:32,657 | infra | INFO | pool-5-thread-27 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_14000000_15000000_1623179192.58_create__14300000_14350000_1623179192.64 2021-06-08 12:06:32,657 | infra | INFO | pool-5-thread-30 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_14000000_15000000_1623179192.58_create__14300000_14350000_1623179192.64' started 2021-06-08 12:06:32,657 | infra | INFO | pool-5-thread-27 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_14000000_15000000_1623179192.58_create__14350000_14400000_1623179192.64 2021-06-08 12:06:32,657 | infra | INFO | pool-5-thread-5 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_14000000_15000000_1623179192.58_create__14350000_14400000_1623179192.64' started 2021-06-08 12:06:32,657 | infra | INFO | pool-5-thread-27 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_14000000_15000000_1623179192.58_create__14400000_14450000_1623179192.64 2021-06-08 12:06:32,657 | infra | INFO | pool-5-thread-9 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_14000000_15000000_1623179192.58_create__14400000_14450000_1623179192.64' started 2021-06-08 12:06:32,657 | infra | INFO | pool-5-thread-27 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_14000000_15000000_1623179192.58_create__14450000_14500000_1623179192.64 2021-06-08 12:06:32,657 | infra | INFO | pool-5-thread-24 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_14000000_15000000_1623179192.58_create__14450000_14500000_1623179192.64' started 2021-06-08 12:06:32,657 | infra | INFO | pool-5-thread-12 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_14000000_15000000_1623179192.58_create__14500000_14550000_1623179192.64' started 2021-06-08 12:06:32,657 | infra | INFO | pool-5-thread-27 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_14000000_15000000_1623179192.58_create__14500000_14550000_1623179192.64 2021-06-08 12:06:32,657 | infra | INFO | pool-5-thread-27 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_14000000_15000000_1623179192.58_create__14550000_14600000_1623179192.64 2021-06-08 12:06:32,657 | infra | INFO | pool-5-thread-18 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_14000000_15000000_1623179192.58_create__14550000_14600000_1623179192.64' started 2021-06-08 12:06:32,658 | infra | INFO | pool-5-thread-2 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_14000000_15000000_1623179192.58_create__14600000_14650000_1623179192.64' started 2021-06-08 12:06:32,658 | infra | INFO | pool-5-thread-27 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_14000000_15000000_1623179192.58_create__14600000_14650000_1623179192.64 2021-06-08 12:06:32,660 | infra | INFO | pool-5-thread-27 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_14000000_15000000_1623179192.58_create__14650000_14700000_1623179192.64 2021-06-08 12:06:32,660 | infra | INFO | pool-5-thread-14 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_14000000_15000000_1623179192.58_create__14650000_14700000_1623179192.64' started 2021-06-08 12:06:32,660 | infra | INFO | pool-5-thread-27 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_14000000_15000000_1623179192.58_create__14700000_14750000_1623179192.64 2021-06-08 12:06:32,660 | infra | INFO | pool-5-thread-7 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_14000000_15000000_1623179192.58_create__14700000_14750000_1623179192.64' started 2021-06-08 12:06:32,660 | infra | INFO | pool-5-thread-27 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_14000000_15000000_1623179192.58_create__14750000_14800000_1623179192.64 2021-06-08 12:06:32,660 | infra | INFO | pool-5-thread-4 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_14000000_15000000_1623179192.58_create__14750000_14800000_1623179192.64' started 2021-06-08 12:06:32,660 | infra | INFO | pool-5-thread-27 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_14000000_15000000_1623179192.58_create__14800000_14850000_1623179192.64 2021-06-08 12:06:32,661 | infra | INFO | pool-5-thread-1 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_14000000_15000000_1623179192.58_create__14800000_14850000_1623179192.64' started 2021-06-08 12:06:32,661 | infra | INFO | pool-5-thread-20 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_14000000_15000000_1623179192.58_create__14850000_14900000_1623179192.64' started 2021-06-08 12:06:32,663 | infra | INFO | pool-5-thread-27 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_14000000_15000000_1623179192.58_create__14850000_14900000_1623179192.64 2021-06-08 12:06:32,663 | infra | INFO | pool-5-thread-27 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_14000000_15000000_1623179192.58_create__14900000_14950000_1623179192.64 2021-06-08 12:06:32,663 | infra | INFO | pool-5-thread-21 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_14000000_15000000_1623179192.58_create__14900000_14950000_1623179192.64' started 2021-06-08 12:06:32,664 | infra | INFO | pool-5-thread-27 | [task_manager:add_new_task:23] Added new task: LoadDocs_LoadDocsGen_default__default__default_create_14000000_15000000_1623179192.58_create__14950000_15000000_1623179192.64 2021-06-08 12:06:32,664 | infra | INFO | pool-5-thread-28 | [task:start_task:87] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_14000000_15000000_1623179192.58_create__14950000_15000000_1623179192.64' started 2021-06-08 12:06:32,664 | infra | DEBUG | pool-5-thread-27 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_14000000_15000000_1623179192.58_create__14000000_14050000_1623179192.64 2021-06-08 12:06:32,924 | test | DEBUG | pool-5-thread-29 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 12:06:32,924 | test | DEBUG | pool-5-thread-29 | [task:print_ep_queue_size:3508] ep_queue_size for default: 882702 2021-06-08 12:06:32,927 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 12:06:32,927 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 12:06:33,755 | infra | DEBUG | pool-5-thread-25 | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.122 2021-06-08 12:06:33,756 | infra | INFO | pool-5-thread-25 | [task:complete_task:102] Thread 'StatsWaitTask_default_ep_data_write_failed_1623179081.36' completed 2021-06-08 12:06:34,206 | test | DEBUG | pool-5-thread-29 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 12:06:34,206 | test | DEBUG | pool-5-thread-29 | [task:print_ep_queue_size:3508] ep_queue_size for default: 882702 2021-06-08 12:06:34,210 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 12:06:34,210 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 12:06:35,470 | test | DEBUG | pool-5-thread-29 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 12:06:35,470 | test | DEBUG | pool-5-thread-29 | [task:print_ep_queue_size:3508] ep_queue_size for default: 882702 2021-06-08 12:06:35,471 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 12:06:35,471 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 12:06:36,819 | test | DEBUG | pool-5-thread-29 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 12:06:36,819 | test | DEBUG | pool-5-thread-29 | [task:print_ep_queue_size:3508] ep_queue_size for default: 882702 2021-06-08 12:06:36,822 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 12:06:36,823 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 12:06:38,145 | test | DEBUG | pool-5-thread-29 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 12:06:38,147 | test | DEBUG | pool-5-thread-29 | [task:print_ep_queue_size:3508] ep_queue_size for default: 882702 2021-06-08 12:06:38,150 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 12:06:38,150 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 12:06:42,219 | test | DEBUG | pool-5-thread-29 | [task:record_bucket_ops:3495] Ops rate for 'default': 0.000000 2021-06-08 12:06:42,219 | test | DEBUG | pool-5-thread-29 | [task:print_ep_queue_size:3508] ep_queue_size for default: 882702 2021-06-08 12:06:42,220 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 12:06:42,220 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 12:06:43,431 | test | DEBUG | pool-5-thread-29 | [task:record_bucket_ops:3495] Ops rate for 'default': 9259.300000 2021-06-08 12:06:43,431 | test | DEBUG | pool-5-thread-29 | [task:print_ep_queue_size:3508] ep_queue_size for default: 929000 2021-06-08 12:06:43,434 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 12:06:43,434 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 12:06:44,709 | test | DEBUG | pool-5-thread-29 | [task:record_bucket_ops:3495] Ops rate for 'default': 9259.300000 2021-06-08 12:06:44,710 | test | DEBUG | pool-5-thread-29 | [task:print_ep_queue_size:3508] ep_queue_size for default: 929000 2021-06-08 12:06:44,713 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 12:06:44,713 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 12:06:45,931 | test | DEBUG | pool-5-thread-29 | [task:record_bucket_ops:3495] Ops rate for 'default': 9259.300000 2021-06-08 12:06:45,933 | test | DEBUG | pool-5-thread-29 | [task:print_ep_queue_size:3508] ep_queue_size for default: 929000 2021-06-08 12:06:45,934 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 12:06:45,934 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 12:06:47,168 | test | DEBUG | pool-5-thread-29 | [task:record_bucket_ops:3495] Ops rate for 'default': 9259.300000 2021-06-08 12:06:47,168 | test | DEBUG | pool-5-thread-29 | [task:print_ep_queue_size:3508] ep_queue_size for default: 929000 2021-06-08 12:06:47,171 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 12:06:47,173 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 12:06:48,476 | test | DEBUG | pool-5-thread-29 | [task:record_bucket_ops:3495] Ops rate for 'default': 9259.300000 2021-06-08 12:06:48,476 | test | DEBUG | pool-5-thread-29 | [task:print_ep_queue_size:3508] ep_queue_size for default: 929000 2021-06-08 12:06:48,480 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 12:06:48,480 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 12:06:49,743 | test | DEBUG | pool-5-thread-29 | [task:record_bucket_ops:3495] Ops rate for 'default': 9259.300000 2021-06-08 12:06:49,743 | test | DEBUG | pool-5-thread-29 | [task:print_ep_queue_size:3508] ep_queue_size for default: 929000 2021-06-08 12:06:49,746 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 12:06:49,747 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 12:06:51,625 | test | DEBUG | pool-5-thread-29 | [task:record_bucket_ops:3495] Ops rate for 'default': 8618.100000 2021-06-08 12:06:51,628 | test | DEBUG | pool-5-thread-29 | [task:print_ep_queue_size:3508] ep_queue_size for default: 971690 2021-06-08 12:06:51,631 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 12:06:51,631 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 12:06:53,578 | test | DEBUG | pool-5-thread-29 | [task:record_bucket_ops:3495] Ops rate for 'default': 8618.100000 2021-06-08 12:06:53,579 | test | DEBUG | pool-5-thread-29 | [task:print_ep_queue_size:3508] ep_queue_size for default: 971690 2021-06-08 12:06:53,582 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 12:06:53,582 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 12:06:55,183 | test | DEBUG | pool-5-thread-29 | [task:record_bucket_ops:3495] Ops rate for 'default': 8618.100000 2021-06-08 12:06:55,183 | test | DEBUG | pool-5-thread-29 | [task:print_ep_queue_size:3508] ep_queue_size for default: 971690 2021-06-08 12:06:55,184 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 12:06:55,184 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 12:06:56,671 | test | DEBUG | pool-5-thread-29 | [task:record_bucket_ops:3495] Ops rate for 'default': 8618.100000 2021-06-08 12:06:56,671 | test | DEBUG | pool-5-thread-29 | [task:print_ep_queue_size:3508] ep_queue_size for default: 971690 2021-06-08 12:06:56,673 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 12:06:56,673 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 12:06:57,908 | test | DEBUG | pool-5-thread-29 | [task:record_bucket_ops:3495] Ops rate for 'default': 8618.100000 2021-06-08 12:06:57,908 | test | DEBUG | pool-5-thread-29 | [task:print_ep_queue_size:3508] ep_queue_size for default: 971690 2021-06-08 12:06:57,913 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 12:06:57,914 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 12:06:59,164 | test | DEBUG | pool-5-thread-29 | [task:record_bucket_ops:3495] Ops rate for 'default': 8618.100000 2021-06-08 12:06:59,164 | test | DEBUG | pool-5-thread-29 | [task:print_ep_queue_size:3508] ep_queue_size for default: 971690 2021-06-08 12:06:59,167 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 12:06:59,167 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 12:07:00,398 | test | DEBUG | pool-5-thread-29 | [task:record_bucket_ops:3495] Ops rate for 'default': 8618.100000 2021-06-08 12:07:00,400 | test | DEBUG | pool-5-thread-29 | [task:print_ep_queue_size:3508] ep_queue_size for default: 971690 2021-06-08 12:07:00,401 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 12:07:00,401 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 12:07:01,726 | test | DEBUG | pool-5-thread-29 | [task:record_bucket_ops:3495] Ops rate for 'default': 12419.300000 2021-06-08 12:07:01,726 | test | DEBUG | pool-5-thread-29 | [task:print_ep_queue_size:3508] ep_queue_size for default: 1033863 2021-06-08 12:07:01,730 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 12:07:01,730 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 12:07:03,239 | test | DEBUG | pool-5-thread-29 | [task:record_bucket_ops:3495] Ops rate for 'default': 12419.300000 2021-06-08 12:07:03,239 | test | DEBUG | pool-5-thread-29 | [task:print_ep_queue_size:3508] ep_queue_size for default: 1033863 2021-06-08 12:07:03,240 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 12:07:03,242 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 12:07:04,483 | test | DEBUG | pool-5-thread-29 | [task:record_bucket_ops:3495] Ops rate for 'default': 12419.300000 2021-06-08 12:07:04,483 | test | DEBUG | pool-5-thread-29 | [task:print_ep_queue_size:3508] ep_queue_size for default: 1033863 2021-06-08 12:07:04,486 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 12:07:04,486 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 12:07:05,698 | test | DEBUG | pool-5-thread-29 | [task:record_bucket_ops:3495] Ops rate for 'default': 12419.300000 2021-06-08 12:07:05,698 | test | DEBUG | pool-5-thread-29 | [task:print_ep_queue_size:3508] ep_queue_size for default: 1033863 2021-06-08 12:07:05,701 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 12:07:05,701 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 12:07:06,943 | test | DEBUG | pool-5-thread-29 | [task:record_bucket_ops:3495] Ops rate for 'default': 12419.300000 2021-06-08 12:07:06,943 | test | DEBUG | pool-5-thread-29 | [task:print_ep_queue_size:3508] ep_queue_size for default: 1033863 2021-06-08 12:07:06,946 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 12:07:06,946 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 12:07:08,474 | test | DEBUG | pool-5-thread-29 | [task:record_bucket_ops:3495] Ops rate for 'default': 12419.300000 2021-06-08 12:07:08,476 | test | DEBUG | pool-5-thread-29 | [task:print_ep_queue_size:3508] ep_queue_size for default: 1033863 2021-06-08 12:07:08,477 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 12:07:08,477 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 12:07:09,714 | test | DEBUG | pool-5-thread-29 | [task:record_bucket_ops:3495] Ops rate for 'default': 12419.300000 2021-06-08 12:07:09,716 | test | DEBUG | pool-5-thread-29 | [task:print_ep_queue_size:3508] ep_queue_size for default: 1033863 2021-06-08 12:07:09,717 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 12:07:09,717 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 12:07:10,996 | test | DEBUG | pool-5-thread-29 | [task:record_bucket_ops:3495] Ops rate for 'default': 12419.300000 2021-06-08 12:07:10,996 | test | DEBUG | pool-5-thread-29 | [task:print_ep_queue_size:3508] ep_queue_size for default: 1033863 2021-06-08 12:07:11,000 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 12:07:11,000 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 12:07:12,219 | test | DEBUG | pool-5-thread-29 | [task:record_bucket_ops:3495] Ops rate for 'default': 13204.000000 2021-06-08 12:07:12,219 | test | DEBUG | pool-5-thread-29 | [task:print_ep_queue_size:3508] ep_queue_size for default: 1098951 2021-06-08 12:07:12,221 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 12:07:12,221 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 12:07:13,509 | test | DEBUG | pool-5-thread-29 | [task:record_bucket_ops:3495] Ops rate for 'default': 13204.000000 2021-06-08 12:07:13,510 | test | DEBUG | pool-5-thread-29 | [task:print_ep_queue_size:3508] ep_queue_size for default: 1098951 2021-06-08 12:07:13,513 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 12:07:13,513 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 12:07:17,111 | test | DEBUG | pool-5-thread-29 | [task:record_bucket_ops:3495] Ops rate for 'default': 13204.000000 2021-06-08 12:07:17,111 | test | DEBUG | pool-5-thread-29 | [task:print_ep_queue_size:3508] ep_queue_size for default: 1098951 2021-06-08 12:07:17,114 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 12:07:17,114 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 12:07:18,318 | test | DEBUG | pool-5-thread-29 | [task:record_bucket_ops:3495] Ops rate for 'default': 13204.000000 2021-06-08 12:07:18,318 | test | DEBUG | pool-5-thread-29 | [task:print_ep_queue_size:3508] ep_queue_size for default: 1098951 2021-06-08 12:07:18,319 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 12:07:18,319 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 12:07:19,319 | infra | INFO | pool-5-thread-4 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_14000000_15000000_1623179192.58_create__14750000_14800000_1623179192.64' completed 2021-06-08 12:07:19,628 | test | DEBUG | pool-5-thread-29 | [task:record_bucket_ops:3495] Ops rate for 'default': 13204.000000 2021-06-08 12:07:19,628 | test | DEBUG | pool-5-thread-29 | [task:print_ep_queue_size:3508] ep_queue_size for default: 1098951 2021-06-08 12:07:19,631 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 12:07:19,631 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 12:07:20,127 | infra | INFO | pool-5-thread-18 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_14000000_15000000_1623179192.58_create__14550000_14600000_1623179192.64' completed 2021-06-08 12:07:20,187 | infra | INFO | pool-5-thread-7 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_14000000_15000000_1623179192.58_create__14700000_14750000_1623179192.64' completed 2021-06-08 12:07:20,986 | infra | INFO | pool-5-thread-8 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_14000000_15000000_1623179192.58_create__14000000_14050000_1623179192.64' completed 2021-06-08 12:07:20,986 | infra | DEBUG | pool-5-thread-27 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_14000000_15000000_1623179192.58_create__14000000_14050000_1623179192.64 are 50000 2021-06-08 12:07:20,986 | infra | DEBUG | pool-5-thread-27 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_14000000_15000000_1623179192.58_create__14050000_14100000_1623179192.64 2021-06-08 12:07:21,009 | infra | INFO | pool-5-thread-6 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_14000000_15000000_1623179192.58_create__14250000_14300000_1623179192.64' completed 2021-06-08 12:07:21,059 | test | DEBUG | pool-5-thread-29 | [task:record_bucket_ops:3495] Ops rate for 'default': 13204.000000 2021-06-08 12:07:21,059 | test | DEBUG | pool-5-thread-29 | [task:print_ep_queue_size:3508] ep_queue_size for default: 1098951 2021-06-08 12:07:21,062 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> call():L3540 2021-06-08 12:07:21,062 | infra | DEBUG | pool-5-thread-29 | [common_lib:sleep:24] Sleep 1 seconds. Reason: None 2021-06-08 12:07:21,078 | infra | INFO | pool-5-thread-19 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_14000000_15000000_1623179192.58_create__14050000_14100000_1623179192.64' completed 2021-06-08 12:07:21,078 | infra | DEBUG | pool-5-thread-27 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_14000000_15000000_1623179192.58_create__14050000_14100000_1623179192.64 are 50000 2021-06-08 12:07:21,078 | infra | DEBUG | pool-5-thread-27 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_14000000_15000000_1623179192.58_create__14100000_14150000_1623179192.64 2021-06-08 12:07:21,078 | infra | INFO | pool-5-thread-12 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_14000000_15000000_1623179192.58_create__14500000_14550000_1623179192.64' completed 2021-06-08 12:07:21,118 | infra | INFO | pool-5-thread-28 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_14000000_15000000_1623179192.58_create__14950000_15000000_1623179192.64' completed 2021-06-08 12:07:21,128 | infra | INFO | pool-5-thread-20 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_14000000_15000000_1623179192.58_create__14850000_14900000_1623179192.64' completed 2021-06-08 12:07:21,148 | infra | INFO | pool-5-thread-21 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_14000000_15000000_1623179192.58_create__14900000_14950000_1623179192.64' completed 2021-06-08 12:07:21,388 | infra | INFO | pool-5-thread-2 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_14000000_15000000_1623179192.58_create__14600000_14650000_1623179192.64' completed 2021-06-08 12:07:21,401 | infra | INFO | pool-5-thread-15 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_14000000_15000000_1623179192.58_create__14150000_14200000_1623179192.64' completed 2021-06-08 12:07:21,411 | infra | INFO | pool-5-thread-3 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_14000000_15000000_1623179192.58_create__14100000_14150000_1623179192.64' completed 2021-06-08 12:07:21,411 | infra | DEBUG | pool-5-thread-27 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_14000000_15000000_1623179192.58_create__14100000_14150000_1623179192.64 are 50000 2021-06-08 12:07:21,411 | infra | DEBUG | pool-5-thread-27 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_14000000_15000000_1623179192.58_create__14150000_14200000_1623179192.64 2021-06-08 12:07:21,413 | infra | DEBUG | pool-5-thread-27 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_14000000_15000000_1623179192.58_create__14150000_14200000_1623179192.64 are 50000 2021-06-08 12:07:21,413 | infra | DEBUG | pool-5-thread-27 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_14000000_15000000_1623179192.58_create__14200000_14250000_1623179192.64 2021-06-08 12:07:21,447 | infra | INFO | pool-5-thread-5 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_14000000_15000000_1623179192.58_create__14350000_14400000_1623179192.64' completed 2021-06-08 12:07:21,463 | infra | INFO | pool-5-thread-24 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_14000000_15000000_1623179192.58_create__14450000_14500000_1623179192.64' completed 2021-06-08 12:07:21,482 | infra | INFO | pool-5-thread-1 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_14000000_15000000_1623179192.58_create__14800000_14850000_1623179192.64' completed 2021-06-08 12:07:21,507 | infra | INFO | pool-5-thread-9 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_14000000_15000000_1623179192.58_create__14400000_14450000_1623179192.64' completed 2021-06-08 12:07:21,510 | infra | INFO | pool-5-thread-30 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_14000000_15000000_1623179192.58_create__14300000_14350000_1623179192.64' completed 2021-06-08 12:07:21,513 | infra | INFO | pool-5-thread-17 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_14000000_15000000_1623179192.58_create__14200000_14250000_1623179192.64' completed 2021-06-08 12:07:21,513 | infra | DEBUG | pool-5-thread-27 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_14000000_15000000_1623179192.58_create__14200000_14250000_1623179192.64 are 50000 2021-06-08 12:07:21,513 | infra | DEBUG | pool-5-thread-27 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_14000000_15000000_1623179192.58_create__14250000_14300000_1623179192.64 2021-06-08 12:07:21,513 | infra | DEBUG | pool-5-thread-27 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_14000000_15000000_1623179192.58_create__14250000_14300000_1623179192.64 are 50000 2021-06-08 12:07:21,513 | infra | DEBUG | pool-5-thread-27 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_14000000_15000000_1623179192.58_create__14300000_14350000_1623179192.64 2021-06-08 12:07:21,513 | infra | DEBUG | pool-5-thread-27 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_14000000_15000000_1623179192.58_create__14300000_14350000_1623179192.64 are 50000 2021-06-08 12:07:21,513 | infra | DEBUG | pool-5-thread-27 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_14000000_15000000_1623179192.58_create__14350000_14400000_1623179192.64 2021-06-08 12:07:21,513 | infra | DEBUG | pool-5-thread-27 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_14000000_15000000_1623179192.58_create__14350000_14400000_1623179192.64 are 50000 2021-06-08 12:07:21,513 | infra | DEBUG | pool-5-thread-27 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_14000000_15000000_1623179192.58_create__14400000_14450000_1623179192.64 2021-06-08 12:07:21,513 | infra | DEBUG | pool-5-thread-27 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_14000000_15000000_1623179192.58_create__14400000_14450000_1623179192.64 are 50000 2021-06-08 12:07:21,513 | infra | DEBUG | pool-5-thread-27 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_14000000_15000000_1623179192.58_create__14450000_14500000_1623179192.64 2021-06-08 12:07:21,513 | infra | DEBUG | pool-5-thread-27 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_14000000_15000000_1623179192.58_create__14450000_14500000_1623179192.64 are 50000 2021-06-08 12:07:21,513 | infra | DEBUG | pool-5-thread-27 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_14000000_15000000_1623179192.58_create__14500000_14550000_1623179192.64 2021-06-08 12:07:21,515 | infra | DEBUG | pool-5-thread-27 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_14000000_15000000_1623179192.58_create__14500000_14550000_1623179192.64 are 50000 2021-06-08 12:07:21,515 | infra | DEBUG | pool-5-thread-27 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_14000000_15000000_1623179192.58_create__14550000_14600000_1623179192.64 2021-06-08 12:07:21,515 | infra | DEBUG | pool-5-thread-27 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_14000000_15000000_1623179192.58_create__14550000_14600000_1623179192.64 are 50000 2021-06-08 12:07:21,515 | infra | DEBUG | pool-5-thread-27 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_14000000_15000000_1623179192.58_create__14600000_14650000_1623179192.64 2021-06-08 12:07:21,515 | infra | DEBUG | pool-5-thread-27 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_14000000_15000000_1623179192.58_create__14600000_14650000_1623179192.64 are 50000 2021-06-08 12:07:21,515 | infra | DEBUG | pool-5-thread-27 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_14000000_15000000_1623179192.58_create__14650000_14700000_1623179192.64 2021-06-08 12:07:21,717 | infra | INFO | pool-5-thread-14 | [task:complete_task:102] Thread 'LoadDocs_LoadDocsGen_default__default__default_create_14000000_15000000_1623179192.58_create__14650000_14700000_1623179192.64' completed 2021-06-08 12:07:21,717 | infra | DEBUG | pool-5-thread-27 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_14000000_15000000_1623179192.58_create__14650000_14700000_1623179192.64 are 50000 2021-06-08 12:07:21,717 | infra | DEBUG | pool-5-thread-27 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_14000000_15000000_1623179192.58_create__14700000_14750000_1623179192.64 2021-06-08 12:07:21,717 | infra | DEBUG | pool-5-thread-27 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_14000000_15000000_1623179192.58_create__14700000_14750000_1623179192.64 are 50000 2021-06-08 12:07:21,717 | infra | DEBUG | pool-5-thread-27 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_14000000_15000000_1623179192.58_create__14750000_14800000_1623179192.64 2021-06-08 12:07:21,717 | infra | DEBUG | pool-5-thread-27 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_14000000_15000000_1623179192.58_create__14750000_14800000_1623179192.64 are 50000 2021-06-08 12:07:21,717 | infra | DEBUG | pool-5-thread-27 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_14000000_15000000_1623179192.58_create__14800000_14850000_1623179192.64 2021-06-08 12:07:21,717 | infra | DEBUG | pool-5-thread-27 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_14000000_15000000_1623179192.58_create__14800000_14850000_1623179192.64 are 50000 2021-06-08 12:07:21,717 | infra | DEBUG | pool-5-thread-27 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_14000000_15000000_1623179192.58_create__14850000_14900000_1623179192.64 2021-06-08 12:07:21,717 | infra | DEBUG | pool-5-thread-27 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_14000000_15000000_1623179192.58_create__14850000_14900000_1623179192.64 are 50000 2021-06-08 12:07:21,717 | infra | DEBUG | pool-5-thread-27 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_14000000_15000000_1623179192.58_create__14900000_14950000_1623179192.64 2021-06-08 12:07:21,717 | infra | DEBUG | pool-5-thread-27 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_14000000_15000000_1623179192.58_create__14900000_14950000_1623179192.64 are 50000 2021-06-08 12:07:21,719 | infra | DEBUG | pool-5-thread-27 | [task_manager:get_task_result:26] Getting task result for LoadDocs_LoadDocsGen_default__default__default_create_14000000_15000000_1623179192.58_create__14950000_15000000_1623179192.64 2021-06-08 12:07:21,719 | infra | DEBUG | pool-5-thread-27 | [task:call:1631] Items loaded in task LoadDocs_LoadDocsGen_default__default__default_create_14000000_15000000_1623179192.58_create__14950000_15000000_1623179192.64 are 50000 2021-06-08 12:07:21,719 | infra | DEBUG | pool-5-thread-27 | [task_manager:get_task_result:26] Getting task result for PrintBucketStats_default_1623179192.64 2021-06-08 12:07:22,062 | test | INFO | pool-5-thread-29 | [table_view:display:72] Ops trend for bucket 'default' +----------+--------------------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +----------+--------------------------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | ******......................................................................*****X | 9259.300 | | 8618.100 | ................................................................*******.............................********......*****X | 13204.000 | +----------+--------------------------------------------------------------------------------------------------------------------------+-----------+ 2021-06-08 12:07:22,062 | infra | INFO | pool-5-thread-29 | [task:complete_task:102] Thread 'PrintBucketStats_default_1623179192.64' completed 2021-06-08 12:07:22,062 | infra | DEBUG | pool-5-thread-27 | [task:call:1663] ========= Tasks in loadgen pool======= 2021-06-08 12:07:22,063 | infra | WARNING | pool-5-thread-27 | [task_manager:print_tasks_in_pool:83] Task 'LoadDocsGen_default__default__default_create_14000000_15000000_1623179192.58' not completed 2021-06-08 12:07:22,063 | infra | DEBUG | pool-5-thread-27 | [task:call:1665] ====================================== 2021-06-08 12:07:22,063 | infra | DEBUG | pool-5-thread-27 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_14000000_15000000_1623179192.58_create__14000000_14050000_1623179192.64' complete. Loaded 50000 items 2021-06-08 12:07:22,063 | infra | DEBUG | pool-5-thread-27 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_14000000_15000000_1623179192.58_create__14050000_14100000_1623179192.64' complete. Loaded 50000 items 2021-06-08 12:07:22,063 | infra | DEBUG | pool-5-thread-27 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_14000000_15000000_1623179192.58_create__14100000_14150000_1623179192.64' complete. Loaded 50000 items 2021-06-08 12:07:22,063 | infra | DEBUG | pool-5-thread-27 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_14000000_15000000_1623179192.58_create__14150000_14200000_1623179192.64' complete. Loaded 50000 items 2021-06-08 12:07:22,063 | infra | DEBUG | pool-5-thread-27 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_14000000_15000000_1623179192.58_create__14200000_14250000_1623179192.64' complete. Loaded 50000 items 2021-06-08 12:07:22,063 | infra | DEBUG | pool-5-thread-27 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_14000000_15000000_1623179192.58_create__14250000_14300000_1623179192.64' complete. Loaded 50000 items 2021-06-08 12:07:22,065 | infra | DEBUG | pool-5-thread-27 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_14000000_15000000_1623179192.58_create__14300000_14350000_1623179192.64' complete. Loaded 50000 items 2021-06-08 12:07:22,065 | infra | DEBUG | pool-5-thread-27 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_14000000_15000000_1623179192.58_create__14350000_14400000_1623179192.64' complete. Loaded 50000 items 2021-06-08 12:07:22,065 | infra | DEBUG | pool-5-thread-27 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_14000000_15000000_1623179192.58_create__14400000_14450000_1623179192.64' complete. Loaded 50000 items 2021-06-08 12:07:22,065 | infra | DEBUG | pool-5-thread-27 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_14000000_15000000_1623179192.58_create__14450000_14500000_1623179192.64' complete. Loaded 50000 items 2021-06-08 12:07:22,065 | infra | DEBUG | pool-5-thread-27 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_14000000_15000000_1623179192.58_create__14500000_14550000_1623179192.64' complete. Loaded 50000 items 2021-06-08 12:07:22,065 | infra | DEBUG | pool-5-thread-27 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_14000000_15000000_1623179192.58_create__14550000_14600000_1623179192.64' complete. Loaded 50000 items 2021-06-08 12:07:22,065 | infra | DEBUG | pool-5-thread-27 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_14000000_15000000_1623179192.58_create__14600000_14650000_1623179192.64' complete. Loaded 50000 items 2021-06-08 12:07:22,065 | infra | DEBUG | pool-5-thread-27 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_14000000_15000000_1623179192.58_create__14650000_14700000_1623179192.64' complete. Loaded 50000 items 2021-06-08 12:07:22,065 | infra | DEBUG | pool-5-thread-27 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_14000000_15000000_1623179192.58_create__14700000_14750000_1623179192.64' complete. Loaded 50000 items 2021-06-08 12:07:22,065 | infra | DEBUG | pool-5-thread-27 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_14000000_15000000_1623179192.58_create__14750000_14800000_1623179192.64' complete. Loaded 50000 items 2021-06-08 12:07:22,065 | infra | DEBUG | pool-5-thread-27 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_14000000_15000000_1623179192.58_create__14800000_14850000_1623179192.64' complete. Loaded 50000 items 2021-06-08 12:07:22,065 | infra | DEBUG | pool-5-thread-27 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_14000000_15000000_1623179192.58_create__14850000_14900000_1623179192.64' complete. Loaded 50000 items 2021-06-08 12:07:22,065 | infra | DEBUG | pool-5-thread-27 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_14000000_15000000_1623179192.58_create__14900000_14950000_1623179192.64' complete. Loaded 50000 items 2021-06-08 12:07:22,065 | infra | DEBUG | pool-5-thread-27 | [task:call:1668] Task 'LoadDocs_LoadDocsGen_default__default__default_create_14000000_15000000_1623179192.58_create__14950000_15000000_1623179192.64' complete. Loaded 50000 items 2021-06-08 12:07:22,065 | infra | INFO | pool-5-thread-27 | [task:complete_task:102] Thread 'LoadDocsGen_default__default__default_create_14000000_15000000_1623179192.58' completed 2021-06-08 12:07:22,069 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.125 with username: root, password: couchbase 2021-06-08 12:07:22,135 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 12:07:22,345 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: ls /etc/ 2021-06-08 12:07:22,398 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 12:07:22,398 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: ls /cygdrive/c/Windows 2021-06-08 12:07:22,411 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: uname -m 2021-06-08 12:07:22,423 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: cat /proc/cpuinfo 2021-06-08 12:07:22,434 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: df -Thl 2021-06-08 12:07:22,450 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: cat /proc/meminfo 2021-06-08 12:07:22,461 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: hostname 2021-06-08 12:07:22,473 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: hostname -d 2021-06-08 12:07:22,486 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:559] Starting StatsWaitTask for ep_data_write_failed on bucket default 2021-06-08 12:07:22,486 | infra | INFO | MainThread | [task_manager:add_new_task:23] Added new task: StatsWaitTask_default_ep_data_write_failed_1623179242.49 2021-06-08 12:07:22,486 | infra | INFO | pool-5-thread-22 | [task:start_task:87] Thread 'StatsWaitTask_default_ep_data_write_failed_1623179242.49' started 2021-06-08 12:07:22,486 | infra | DEBUG | pool-5-thread-22 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:07:22,489 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.155 with username: root, password: couchbase 2021-06-08 12:07:22,558 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 12:07:22,605 | test | DEBUG | pool-5-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:07:22,605 | test | INFO | pool-5-thread-22 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 12:07:22,766 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: ls /etc/ 2021-06-08 12:07:22,839 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 12:07:22,839 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: ls /cygdrive/c/Windows 2021-06-08 12:07:22,852 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: uname -m 2021-06-08 12:07:22,865 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: cat /proc/cpuinfo 2021-06-08 12:07:22,877 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: df -Thl 2021-06-08 12:07:22,891 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: cat /proc/meminfo 2021-06-08 12:07:22,903 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: hostname 2021-06-08 12:07:22,914 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: hostname -d 2021-06-08 12:07:22,927 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:559] Starting StatsWaitTask for ep_data_write_failed on bucket default 2021-06-08 12:07:22,927 | infra | INFO | MainThread | [task_manager:add_new_task:23] Added new task: StatsWaitTask_default_ep_data_write_failed_1623179242.93 2021-06-08 12:07:22,927 | infra | INFO | pool-5-thread-11 | [task:start_task:87] Thread 'StatsWaitTask_default_ep_data_write_failed_1623179242.93' started 2021-06-08 12:07:22,930 | infra | DEBUG | pool-5-thread-11 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:07:22,930 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.156 with username: root, password: couchbase 2021-06-08 12:07:22,999 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 12:07:23,125 | test | DEBUG | pool-5-thread-11 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 12:07:23,125 | infra | DEBUG | pool-5-thread-11 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.155': '0'} for bucket 'default' 2021-06-08 12:07:23,125 | infra | DEBUG | pool-5-thread-11 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 12:07:23,127 | infra | DEBUG | pool-5-thread-11 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 12:07:23,127 | infra | DEBUG | pool-5-thread-11 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 12:07:23,210 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: ls /etc/ 2021-06-08 12:07:23,266 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 12:07:23,266 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: ls /cygdrive/c/Windows 2021-06-08 12:07:23,279 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: uname -m 2021-06-08 12:07:23,292 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: cat /proc/cpuinfo 2021-06-08 12:07:23,303 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: df -Thl 2021-06-08 12:07:23,319 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: cat /proc/meminfo 2021-06-08 12:07:23,332 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: hostname 2021-06-08 12:07:23,345 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: hostname -d 2021-06-08 12:07:23,358 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:559] Starting StatsWaitTask for ep_data_write_failed on bucket default 2021-06-08 12:07:23,358 | infra | INFO | MainThread | [task_manager:add_new_task:23] Added new task: StatsWaitTask_default_ep_data_write_failed_1623179243.36 2021-06-08 12:07:23,358 | infra | INFO | pool-5-thread-26 | [task:start_task:87] Thread 'StatsWaitTask_default_ep_data_write_failed_1623179243.36' started 2021-06-08 12:07:23,359 | infra | DEBUG | pool-5-thread-26 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:07:23,361 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.122 with username: root, password: couchbase 2021-06-08 12:07:23,444 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 12:07:23,579 | test | DEBUG | pool-5-thread-26 | [task:_compare:2649] Comparing 19825 > 0 2021-06-08 12:07:23,581 | test | DEBUG | pool-5-thread-26 | [task:_get_all_stats_and_compare:2601] Ready: ep_data_write_failed 19825 > 0. Received: {'172.23.106.156': '19825'} for bucket 'default' 2021-06-08 12:07:23,581 | infra | INFO | pool-5-thread-26 | [task:complete_task:102] Thread 'StatsWaitTask_default_ep_data_write_failed_1623179243.36' completed 2021-06-08 12:07:23,651 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: ls /etc/ 2021-06-08 12:07:23,704 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 12:07:23,704 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: ls /cygdrive/c/Windows 2021-06-08 12:07:23,717 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: uname -m 2021-06-08 12:07:23,730 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: cat /proc/cpuinfo 2021-06-08 12:07:23,740 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: df -Thl 2021-06-08 12:07:23,756 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: cat /proc/meminfo 2021-06-08 12:07:23,766 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: hostname 2021-06-08 12:07:23,779 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: hostname -d 2021-06-08 12:07:23,790 | infra | DEBUG | MainThread | [cluster:async_wait_for_stats:559] Starting StatsWaitTask for ep_data_write_failed on bucket default 2021-06-08 12:07:23,792 | infra | INFO | MainThread | [task_manager:add_new_task:23] Added new task: StatsWaitTask_default_ep_data_write_failed_1623179243.79 2021-06-08 12:07:23,792 | infra | INFO | pool-5-thread-10 | [task:start_task:87] Thread 'StatsWaitTask_default_ep_data_write_failed_1623179243.79' started 2021-06-08 12:07:23,792 | infra | DEBUG | MainThread | [task_manager:get_task_result:26] Getting task result for StatsWaitTask_default_ep_data_write_failed_1623179242.49 2021-06-08 12:07:23,792 | infra | DEBUG | pool-5-thread-10 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:07:23,908 | test | DEBUG | pool-5-thread-10 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:07:23,908 | test | INFO | pool-5-thread-10 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 12:07:27,605 | infra | DEBUG | pool-5-thread-22 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:07:27,726 | test | DEBUG | pool-5-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:07:27,726 | test | INFO | pool-5-thread-22 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 12:07:28,127 | infra | DEBUG | pool-5-thread-11 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:07:28,325 | test | DEBUG | pool-5-thread-11 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 12:07:28,325 | infra | DEBUG | pool-5-thread-11 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.155': '0'} for bucket 'default' 2021-06-08 12:07:28,326 | infra | DEBUG | pool-5-thread-11 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 12:07:28,328 | infra | DEBUG | pool-5-thread-11 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 12:07:28,328 | infra | DEBUG | pool-5-thread-11 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 12:07:28,908 | infra | DEBUG | pool-5-thread-10 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:07:29,026 | test | DEBUG | pool-5-thread-10 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:07:29,026 | test | INFO | pool-5-thread-10 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 12:07:32,726 | infra | DEBUG | pool-5-thread-22 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:07:32,844 | test | DEBUG | pool-5-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:07:32,844 | test | INFO | pool-5-thread-22 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 12:07:33,328 | infra | DEBUG | pool-5-thread-11 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:07:33,532 | test | DEBUG | pool-5-thread-11 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 12:07:33,532 | infra | DEBUG | pool-5-thread-11 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.155': '0'} for bucket 'default' 2021-06-08 12:07:33,532 | infra | DEBUG | pool-5-thread-11 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 12:07:33,533 | infra | DEBUG | pool-5-thread-11 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 12:07:33,533 | infra | DEBUG | pool-5-thread-11 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 12:07:34,026 | infra | DEBUG | pool-5-thread-10 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:07:34,142 | test | DEBUG | pool-5-thread-10 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:07:34,144 | test | INFO | pool-5-thread-10 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 12:07:37,844 | infra | DEBUG | pool-5-thread-22 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:07:37,959 | test | DEBUG | pool-5-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:07:37,959 | test | INFO | pool-5-thread-22 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 12:07:38,535 | infra | DEBUG | pool-5-thread-11 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:07:38,730 | test | DEBUG | pool-5-thread-11 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 12:07:38,730 | infra | DEBUG | pool-5-thread-11 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.155': '0'} for bucket 'default' 2021-06-08 12:07:38,730 | infra | DEBUG | pool-5-thread-11 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 12:07:38,732 | infra | DEBUG | pool-5-thread-11 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 12:07:38,732 | infra | DEBUG | pool-5-thread-11 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 12:07:39,144 | infra | DEBUG | pool-5-thread-10 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:07:39,260 | test | DEBUG | pool-5-thread-10 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:07:39,260 | test | INFO | pool-5-thread-10 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 12:07:42,960 | infra | DEBUG | pool-5-thread-22 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:07:43,079 | test | DEBUG | pool-5-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:07:43,079 | test | INFO | pool-5-thread-22 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 12:07:43,733 | infra | DEBUG | pool-5-thread-11 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:07:43,927 | test | DEBUG | pool-5-thread-11 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 12:07:43,927 | infra | DEBUG | pool-5-thread-11 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.155': '0'} for bucket 'default' 2021-06-08 12:07:43,927 | infra | DEBUG | pool-5-thread-11 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 12:07:43,930 | infra | DEBUG | pool-5-thread-11 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 12:07:43,930 | infra | DEBUG | pool-5-thread-11 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 12:07:44,260 | infra | DEBUG | pool-5-thread-10 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:07:44,378 | test | DEBUG | pool-5-thread-10 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:07:44,380 | test | INFO | pool-5-thread-10 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 12:07:48,081 | infra | DEBUG | pool-5-thread-22 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:07:48,197 | test | DEBUG | pool-5-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:07:48,197 | test | INFO | pool-5-thread-22 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 12:07:48,930 | infra | DEBUG | pool-5-thread-11 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:07:49,128 | test | DEBUG | pool-5-thread-11 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 12:07:49,128 | infra | DEBUG | pool-5-thread-11 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.155': '0'} for bucket 'default' 2021-06-08 12:07:49,128 | infra | DEBUG | pool-5-thread-11 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 12:07:49,130 | infra | DEBUG | pool-5-thread-11 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 12:07:49,131 | infra | DEBUG | pool-5-thread-11 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 12:07:49,380 | infra | DEBUG | pool-5-thread-10 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:07:49,500 | test | DEBUG | pool-5-thread-10 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:07:49,500 | test | INFO | pool-5-thread-10 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 12:07:53,197 | infra | DEBUG | pool-5-thread-22 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:07:53,313 | test | DEBUG | pool-5-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:07:53,313 | test | INFO | pool-5-thread-22 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 12:07:54,131 | infra | DEBUG | pool-5-thread-11 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:07:54,329 | test | DEBUG | pool-5-thread-11 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 12:07:54,329 | infra | DEBUG | pool-5-thread-11 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.155': '0'} for bucket 'default' 2021-06-08 12:07:54,329 | infra | DEBUG | pool-5-thread-11 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 12:07:54,332 | infra | DEBUG | pool-5-thread-11 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 12:07:54,332 | infra | DEBUG | pool-5-thread-11 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 12:07:54,500 | infra | DEBUG | pool-5-thread-10 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:07:54,621 | test | DEBUG | pool-5-thread-10 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:07:54,621 | test | INFO | pool-5-thread-10 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 12:07:58,313 | infra | DEBUG | pool-5-thread-22 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:07:58,431 | test | DEBUG | pool-5-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:07:58,431 | test | INFO | pool-5-thread-22 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 12:07:59,332 | infra | DEBUG | pool-5-thread-11 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:07:59,528 | test | DEBUG | pool-5-thread-11 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 12:07:59,528 | infra | DEBUG | pool-5-thread-11 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.155': '0'} for bucket 'default' 2021-06-08 12:07:59,528 | infra | DEBUG | pool-5-thread-11 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 12:07:59,529 | infra | DEBUG | pool-5-thread-11 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 12:07:59,529 | infra | DEBUG | pool-5-thread-11 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 12:07:59,621 | infra | DEBUG | pool-5-thread-10 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:07:59,737 | test | DEBUG | pool-5-thread-10 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:07:59,737 | test | INFO | pool-5-thread-10 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 12:08:03,433 | infra | DEBUG | pool-5-thread-22 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:08:03,549 | test | DEBUG | pool-5-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:08:03,549 | test | INFO | pool-5-thread-22 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 12:08:04,529 | infra | DEBUG | pool-5-thread-11 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:08:04,729 | test | DEBUG | pool-5-thread-11 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 12:08:04,730 | infra | DEBUG | pool-5-thread-11 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.155': '0'} for bucket 'default' 2021-06-08 12:08:04,730 | infra | DEBUG | pool-5-thread-11 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 12:08:04,730 | infra | DEBUG | pool-5-thread-11 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 12:08:04,732 | infra | DEBUG | pool-5-thread-11 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 12:08:04,737 | infra | DEBUG | pool-5-thread-10 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:08:04,859 | test | DEBUG | pool-5-thread-10 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:08:04,859 | test | INFO | pool-5-thread-10 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 12:08:08,549 | infra | DEBUG | pool-5-thread-22 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:08:08,667 | test | DEBUG | pool-5-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:08:08,667 | test | INFO | pool-5-thread-22 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 12:08:09,732 | infra | DEBUG | pool-5-thread-11 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:08:09,861 | infra | DEBUG | pool-5-thread-10 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:08:09,930 | test | DEBUG | pool-5-thread-11 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 12:08:09,930 | infra | DEBUG | pool-5-thread-11 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.155': '0'} for bucket 'default' 2021-06-08 12:08:09,930 | infra | DEBUG | pool-5-thread-11 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 12:08:09,933 | infra | DEBUG | pool-5-thread-11 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 12:08:09,933 | infra | DEBUG | pool-5-thread-11 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 12:08:09,977 | test | DEBUG | pool-5-thread-10 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:08:09,977 | test | INFO | pool-5-thread-10 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 12:08:13,667 | test | DEBUG | pool-5-thread-22 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 12:08:13,668 | infra | DEBUG | pool-5-thread-22 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {} for bucket 'default' 2021-06-08 12:08:13,668 | infra | DEBUG | pool-5-thread-22 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 12:08:13,671 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 12:08:13,671 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 12:08:14,933 | infra | DEBUG | pool-5-thread-11 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:08:14,977 | test | DEBUG | pool-5-thread-10 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 12:08:14,977 | infra | DEBUG | pool-5-thread-10 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {} for bucket 'default' 2021-06-08 12:08:14,977 | infra | DEBUG | pool-5-thread-10 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 12:08:14,980 | infra | DEBUG | pool-5-thread-10 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 12:08:14,980 | infra | DEBUG | pool-5-thread-10 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 12:08:15,128 | test | DEBUG | pool-5-thread-11 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 12:08:15,130 | infra | DEBUG | pool-5-thread-11 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.155': '0'} for bucket 'default' 2021-06-08 12:08:15,130 | infra | DEBUG | pool-5-thread-11 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 12:08:15,131 | infra | DEBUG | pool-5-thread-11 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 12:08:15,131 | infra | DEBUG | pool-5-thread-11 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 12:08:18,671 | infra | DEBUG | pool-5-thread-22 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:08:18,789 | test | DEBUG | pool-5-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:08:18,789 | test | INFO | pool-5-thread-22 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 12:08:19,982 | infra | DEBUG | pool-5-thread-10 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:08:20,098 | test | DEBUG | pool-5-thread-10 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:08:20,098 | test | INFO | pool-5-thread-10 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 12:08:20,131 | infra | DEBUG | pool-5-thread-11 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:08:20,325 | test | DEBUG | pool-5-thread-11 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 12:08:20,325 | infra | DEBUG | pool-5-thread-11 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {'172.23.106.155': '0'} for bucket 'default' 2021-06-08 12:08:20,325 | infra | DEBUG | pool-5-thread-11 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 12:08:20,328 | infra | DEBUG | pool-5-thread-11 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 12:08:20,328 | infra | DEBUG | pool-5-thread-11 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 12:08:23,789 | infra | DEBUG | pool-5-thread-22 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:08:23,907 | test | DEBUG | pool-5-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:08:23,907 | test | INFO | pool-5-thread-22 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 12:08:25,099 | infra | DEBUG | pool-5-thread-10 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:08:25,220 | test | DEBUG | pool-5-thread-10 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:08:25,220 | test | INFO | pool-5-thread-10 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 12:08:25,328 | infra | DEBUG | pool-5-thread-11 | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.155 2021-06-08 12:08:25,328 | infra | INFO | pool-5-thread-11 | [task:complete_task:102] Thread 'StatsWaitTask_default_ep_data_write_failed_1623179242.93' completed 2021-06-08 12:08:28,907 | infra | DEBUG | pool-5-thread-22 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:08:29,023 | test | DEBUG | pool-5-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:08:29,023 | test | INFO | pool-5-thread-22 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 12:08:30,221 | infra | DEBUG | pool-5-thread-10 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:08:30,338 | test | DEBUG | pool-5-thread-10 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:08:30,338 | test | INFO | pool-5-thread-10 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 12:08:34,023 | infra | DEBUG | pool-5-thread-22 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:08:34,141 | test | DEBUG | pool-5-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:08:34,141 | test | INFO | pool-5-thread-22 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 12:08:35,339 | infra | DEBUG | pool-5-thread-10 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:08:35,457 | test | DEBUG | pool-5-thread-10 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:08:35,457 | test | INFO | pool-5-thread-10 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 12:08:39,142 | infra | DEBUG | pool-5-thread-22 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:08:39,259 | test | DEBUG | pool-5-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:08:39,259 | test | INFO | pool-5-thread-22 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 12:08:40,459 | infra | DEBUG | pool-5-thread-10 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:08:40,575 | test | DEBUG | pool-5-thread-10 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:08:40,575 | test | INFO | pool-5-thread-10 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 12:08:44,259 | infra | DEBUG | pool-5-thread-22 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:08:44,381 | test | DEBUG | pool-5-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:08:44,381 | test | INFO | pool-5-thread-22 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1010, in main() File "/opt/couchbase/lib/python/cbstats", line 1007, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 49, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-06-08 12:08:45,575 | infra | DEBUG | pool-5-thread-10 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:08:45,691 | test | DEBUG | pool-5-thread-10 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:08:45,691 | test | INFO | pool-5-thread-10 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 12:08:49,381 | infra | DEBUG | pool-5-thread-22 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:08:49,500 | test | DEBUG | pool-5-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:08:49,500 | test | INFO | pool-5-thread-22 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 12:08:50,693 | infra | DEBUG | pool-5-thread-10 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:08:50,812 | test | DEBUG | pool-5-thread-10 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:08:50,812 | test | INFO | pool-5-thread-10 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1010, in main() File "/opt/couchbase/lib/python/cbstats", line 1007, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 49, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-06-08 12:08:54,502 | infra | DEBUG | pool-5-thread-22 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:08:54,625 | test | DEBUG | pool-5-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:08:54,625 | test | INFO | pool-5-thread-22 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1010, in main() File "/opt/couchbase/lib/python/cbstats", line 1007, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 49, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-06-08 12:08:55,813 | infra | DEBUG | pool-5-thread-10 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:08:55,930 | test | DEBUG | pool-5-thread-10 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:08:55,930 | test | INFO | pool-5-thread-10 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 12:08:59,627 | infra | DEBUG | pool-5-thread-22 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:08:59,743 | test | DEBUG | pool-5-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:08:59,743 | test | INFO | pool-5-thread-22 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 12:09:00,931 | infra | DEBUG | pool-5-thread-10 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:09:01,051 | test | DEBUG | pool-5-thread-10 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:09:01,051 | test | INFO | pool-5-thread-10 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1010, in main() File "/opt/couchbase/lib/python/cbstats", line 1007, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 49, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-06-08 12:09:04,743 | infra | DEBUG | pool-5-thread-22 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:09:04,868 | test | DEBUG | pool-5-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:09:04,868 | test | INFO | pool-5-thread-22 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. Traceback (most recent call last): File "/opt/couchbase/lib/python/cbstats", line 1010, in main() File "/opt/couchbase/lib/python/cbstats", line 1007, in main c.execute() File "/opt/couchbase/lib/python/clitool.py", line 83, in execute f[0](mc, *args[2:], **opts.__dict__) File "/opt/couchbase/lib/python/cbstats", line 49, in g f(*args, **kwargs) File "/opt/couchbase/lib/python/cli_auth_utils.py", line 105, in g mc.bucket_select(bucket) File "/opt/couchbase/lib/python/mc_bin_client.py", line 679, in bucket_select return self._doCmd(memcacheConstants.CMD_SELECT_BUCKET, name, '') File "/opt/couchbase/lib/python/mc_bin_client.py", line 303, in _doCmd return self._handleSingleResponse(opaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 296, in _handleSingleResponse cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) File "/opt/couchbase/lib/python/mc_bin_client.py", line 292, in _handleKeyedResponse raise MemcachedError(errcode, msg) mc_bin_client.ErrorKeyEnoent: Memcached error #1: KEY_ENOENT : Not Found : 2021-06-08 12:09:06,052 | infra | DEBUG | pool-5-thread-10 | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: /opt/couchbase/bin/cbstats localhost:11210 -u Administrator -p password -b default all 2021-06-08 12:09:06,170 | test | DEBUG | pool-5-thread-10 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2586 2021-06-08 12:09:06,170 | test | INFO | pool-5-thread-10 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'ep_data_write_failed' 2021-06-08 12:09:09,868 | test | DEBUG | pool-5-thread-22 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 12:09:09,868 | infra | DEBUG | pool-5-thread-22 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {} for bucket 'default' 2021-06-08 12:09:09,868 | infra | DEBUG | pool-5-thread-22 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 12:09:09,871 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 12:09:09,871 | infra | DEBUG | pool-5-thread-22 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 12:09:11,171 | test | DEBUG | pool-5-thread-10 | [task:_compare:2649] Comparing 0 > 0 2021-06-08 12:09:11,171 | infra | DEBUG | pool-5-thread-10 | [task:_get_all_stats_and_compare:2594] Not Ready: ep_data_write_failed 0 > 0. Received: {} for bucket 'default' 2021-06-08 12:09:11,171 | infra | DEBUG | pool-5-thread-10 | [task:_get_all_stats_and_compare:2598] Wait before next StatWaitTask check 2021-06-08 12:09:11,173 | infra | DEBUG | pool-5-thread-10 | [common_lib:sleep:17] Sleep is called from lib/Jython_tasks/task.py -> _get_all_stats_and_compare():L2599 2021-06-08 12:09:11,173 | infra | DEBUG | pool-5-thread-10 | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 12:09:14,871 | infra | DEBUG | pool-5-thread-22 | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.125 2021-06-08 12:09:14,872 | infra | INFO | pool-5-thread-22 | [task:complete_task:102] Thread 'StatsWaitTask_default_ep_data_write_failed_1623179242.49' completed 2021-06-08 12:09:14,880 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.156 with username: root, password: couchbase ERROR 2021-06-08 12:09:14,969 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 12:09:15,232 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: ls /etc/ 2021-06-08 12:09:15,308 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 12:09:15,309 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: ls /cygdrive/c/Windows 2021-06-08 12:09:15,328 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: uname -m 2021-06-08 12:09:15,345 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: cat /proc/cpuinfo 2021-06-08 12:09:15,361 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: df -Thl 2021-06-08 12:09:15,381 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: cat /proc/meminfo 2021-06-08 12:09:15,395 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: hostname 2021-06-08 12:09:15,414 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: hostname -d 2021-06-08 12:09:15,433 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: rm -rf /data/full_disk_* 2021-06-08 12:09:15,749 | infra | DEBUG | MainThread | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.156 2021-06-08 12:09:15,755 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/magma/magma_disk_full.py -> free_disk():L109 2021-06-08 12:09:15,755 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: Wait for files to clean up from the disk 2021-06-08 12:09:16,174 | infra | DEBUG | pool-5-thread-10 | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.122 2021-06-08 12:09:16,174 | infra | INFO | pool-5-thread-10 | [task:complete_task:102] Thread 'StatsWaitTask_default_ep_data_write_failed_1623179243.79' completed 2021-06-08 12:09:25,757 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.125 with username: root, password: couchbase 2021-06-08 12:09:25,829 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 12:09:26,042 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: ls /etc/ 2021-06-08 12:09:26,095 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 12:09:26,095 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: ls /cygdrive/c/Windows 2021-06-08 12:09:26,108 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: uname -m 2021-06-08 12:09:26,118 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: cat /proc/cpuinfo 2021-06-08 12:09:26,131 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: df -Thl 2021-06-08 12:09:26,144 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: cat /proc/meminfo 2021-06-08 12:09:26,157 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: hostname 2021-06-08 12:09:26,168 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: hostname -d 2021-06-08 12:09:26,180 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: rm -rf /data/full_disk_* 2021-06-08 12:09:26,359 | infra | DEBUG | MainThread | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.125 2021-06-08 12:09:26,365 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/magma/magma_disk_full.py -> free_disk():L109 2021-06-08 12:09:26,365 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: Wait for files to clean up from the disk 2021-06-08 12:09:36,368 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.122 with username: root, password: couchbase 2021-06-08 12:09:36,464 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 12:09:36,677 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: ls /etc/ 2021-06-08 12:09:36,732 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 12:09:36,732 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: ls /cygdrive/c/Windows 2021-06-08 12:09:36,744 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: uname -m 2021-06-08 12:09:36,757 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: cat /proc/cpuinfo 2021-06-08 12:09:36,769 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: df -Thl 2021-06-08 12:09:36,783 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: cat /proc/meminfo 2021-06-08 12:09:36,796 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: hostname 2021-06-08 12:09:36,808 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: hostname -d 2021-06-08 12:09:36,821 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: rm -rf /data/full_disk_* 2021-06-08 12:09:36,989 | infra | DEBUG | MainThread | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.122 2021-06-08 12:09:36,994 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/magma/magma_disk_full.py -> free_disk():L109 2021-06-08 12:09:36,994 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: Wait for files to clean up from the disk 2021-06-08 12:09:46,996 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.155 with username: root, password: couchbase 2021-06-08 12:09:47,086 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 12:09:47,299 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: ls /etc/ 2021-06-08 12:09:47,357 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 12:09:47,358 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: ls /cygdrive/c/Windows 2021-06-08 12:09:47,371 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: uname -m 2021-06-08 12:09:47,382 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: cat /proc/cpuinfo 2021-06-08 12:09:47,394 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: df -Thl 2021-06-08 12:09:47,411 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: cat /proc/meminfo 2021-06-08 12:09:47,423 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: hostname 2021-06-08 12:09:47,434 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: hostname -d 2021-06-08 12:09:47,447 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: rm -rf /data/full_disk_* 2021-06-08 12:09:47,510 | infra | DEBUG | MainThread | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.155 2021-06-08 12:09:47,516 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from pytests/magma/magma_disk_full.py -> free_disk():L109 2021-06-08 12:09:47,516 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: Wait for files to clean up from the disk 2021-06-08 12:09:57,548 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+-------------+-------------+---------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+-------------+-------------+---------------+-------------------+-----------------------+ | 172.23.106.155 | kv | 2.34335839599 | 12294176768 | 5030047744 | 0 / 0 | 1259120 / 3773423 | 7.0.0-5270-enterprise | | 172.23.106.156 | kv | 13.1346299571 | 12294176768 | 6273277952 | 0 / 0 | 3093056 / 5299553 | 7.0.0-5270-enterprise | | 172.23.106.122 | kv | 6.49218650743 | 12294176768 | 11398524928 | 0 / 0 | 0 / 0 | 7.0.0-5270-enterprise | | 172.23.106.125 | kv | 9.57923008057 | 12294176768 | 11308118016 | 0 / 0 | 0 / 0 | 7.0.0-5270-enterprise | +----------------+----------+-----------------+-------------+-------------+---------------+-------------------+-----------------------+ 2021-06-08 12:09:57,773 | test | INFO | MainThread | [magma_base:tearDown:430] ## Active Resident Threshold of default is 59.6416137583 ## 2021-06-08 12:09:57,773 | infra | INFO | MainThread | [task_manager:shutdown:65] Running TaskManager shutdown 2021-06-08 12:09:57,776 | infra | INFO | MainThread | [task_manager:shutdown:65] Running TaskManager shutdown 2021-06-08 12:09:57,776 | infra | WARNING | MainThread | [cluster:shutdown:702] Cluster instance shutdown with force 2021-06-08 12:09:57,778 | infra | INFO | MainThread | [task_manager:abort_all_tasks:87] Stopping task StatsWaitTask_default_ep_data_write_failed_1623178923.74 2021-06-08 12:09:57,778 | infra | DEBUG | MainThread | [task_manager:abort_all_tasks:92] Task StatsWaitTask_default_ep_data_write_failed_1623178923.74 could not be cancelled or already finished. 2021-06-08 12:09:57,778 | infra | INFO | MainThread | [task_manager:abort_all_tasks:87] Stopping task StatsWaitTask_default_ep_data_write_failed_1623178182.39 2021-06-08 12:09:57,778 | infra | DEBUG | MainThread | [task_manager:abort_all_tasks:92] Task StatsWaitTask_default_ep_data_write_failed_1623178182.39 could not be cancelled or already finished. 2021-06-08 12:09:57,778 | infra | INFO | MainThread | [task_manager:abort_all_tasks:87] Stopping task StatsWaitTask_default_ep_data_write_failed_1623179081.36 2021-06-08 12:09:57,778 | infra | DEBUG | MainThread | [task_manager:abort_all_tasks:92] Task StatsWaitTask_default_ep_data_write_failed_1623179081.36 could not be cancelled or already finished. 2021-06-08 12:09:57,778 | infra | INFO | MainThread | [task_manager:abort_all_tasks:87] Stopping task StatsWaitTask_default_ep_data_write_failed_1623177799.11 2021-06-08 12:09:57,779 | infra | DEBUG | MainThread | [task_manager:abort_all_tasks:92] Task StatsWaitTask_default_ep_data_write_failed_1623177799.11 could not be cancelled or already finished. 2021-06-08 12:09:57,779 | infra | INFO | MainThread | [task_manager:abort_all_tasks:87] Stopping task StatsWaitTask_default_ep_data_write_failed_1623177799.55 2021-06-08 12:09:57,779 | infra | DEBUG | MainThread | [task_manager:abort_all_tasks:92] Task StatsWaitTask_default_ep_data_write_failed_1623177799.55 could not be cancelled or already finished. 2021-06-08 12:09:57,779 | infra | INFO | MainThread | [task_manager:abort_all_tasks:87] Stopping task StatsWaitTask_default_ep_data_write_failed_1623178766.64 2021-06-08 12:09:57,779 | infra | DEBUG | MainThread | [task_manager:abort_all_tasks:92] Task StatsWaitTask_default_ep_data_write_failed_1623178766.64 could not be cancelled or already finished. 2021-06-08 12:09:57,779 | infra | INFO | MainThread | [task_manager:abort_all_tasks:87] Stopping task StatsWaitTask_default_ep_data_write_failed_1623178924.17 2021-06-08 12:09:57,779 | infra | DEBUG | MainThread | [task_manager:abort_all_tasks:92] Task StatsWaitTask_default_ep_data_write_failed_1623178924.17 could not be cancelled or already finished. 2021-06-08 12:09:57,779 | infra | INFO | MainThread | [task_manager:abort_all_tasks:87] Stopping task StatsWaitTask_default_ep_data_write_failed_1623177711.52 2021-06-08 12:09:57,779 | infra | DEBUG | MainThread | [task_manager:abort_all_tasks:92] Task StatsWaitTask_default_ep_data_write_failed_1623177711.52 could not be cancelled or already finished. 2021-06-08 12:09:57,779 | infra | INFO | MainThread | [task_manager:abort_all_tasks:87] Stopping task StatsWaitTask_default_ep_data_write_failed_1623177999.41 2021-06-08 12:09:57,779 | infra | DEBUG | MainThread | [task_manager:abort_all_tasks:92] Task StatsWaitTask_default_ep_data_write_failed_1623177999.41 could not be cancelled or already finished. 2021-06-08 12:09:57,779 | infra | INFO | MainThread | [task_manager:abort_all_tasks:87] Stopping task StatsWaitTask_default_ep_data_write_failed_1623178531.44 2021-06-08 12:09:57,779 | infra | DEBUG | MainThread | [task_manager:abort_all_tasks:92] Task StatsWaitTask_default_ep_data_write_failed_1623178531.44 could not be cancelled or already finished. 2021-06-08 12:09:57,779 | infra | INFO | MainThread | [task_manager:abort_all_tasks:87] Stopping task StatsWaitTask_default_ep_data_write_failed_1623178531.88 2021-06-08 12:09:57,779 | infra | DEBUG | MainThread | [task_manager:abort_all_tasks:92] Task StatsWaitTask_default_ep_data_write_failed_1623178531.88 could not be cancelled or already finished. 2021-06-08 12:09:57,780 | infra | INFO | MainThread | [task_manager:abort_all_tasks:87] Stopping task StatsWaitTask_default_ep_data_write_failed_1623179242.49 2021-06-08 12:09:57,780 | infra | DEBUG | MainThread | [task_manager:abort_all_tasks:92] Task StatsWaitTask_default_ep_data_write_failed_1623179242.49 could not be cancelled or already finished. 2021-06-08 12:09:57,780 | infra | INFO | MainThread | [task_manager:abort_all_tasks:87] Stopping task StatsWaitTask_default_ep_data_write_failed_1623178346.63 2021-06-08 12:09:57,780 | infra | DEBUG | MainThread | [task_manager:abort_all_tasks:92] Task StatsWaitTask_default_ep_data_write_failed_1623178346.63 could not be cancelled or already finished. 2021-06-08 12:09:57,780 | infra | INFO | MainThread | [task_manager:abort_all_tasks:87] Stopping task StatsWaitTask_default_ep_data_write_failed_1623179080.51 2021-06-08 12:09:57,780 | infra | DEBUG | MainThread | [task_manager:abort_all_tasks:92] Task StatsWaitTask_default_ep_data_write_failed_1623179080.51 could not be cancelled or already finished. 2021-06-08 12:09:57,780 | infra | INFO | MainThread | [task_manager:abort_all_tasks:87] Stopping task StatsWaitTask_default_ep_data_write_failed_1623179080.95 2021-06-08 12:09:57,780 | infra | DEBUG | MainThread | [task_manager:abort_all_tasks:92] Task StatsWaitTask_default_ep_data_write_failed_1623179080.95 could not be cancelled or already finished. 2021-06-08 12:09:57,780 | infra | INFO | MainThread | [task_manager:abort_all_tasks:87] Stopping task StatsWaitTask_default_ep_data_write_failed_1623178767.08 2021-06-08 12:09:57,780 | infra | DEBUG | MainThread | [task_manager:abort_all_tasks:92] Task StatsWaitTask_default_ep_data_write_failed_1623178767.08 could not be cancelled or already finished. 2021-06-08 12:09:57,782 | infra | INFO | MainThread | [task_manager:abort_all_tasks:87] Stopping task StatsWaitTask_default_ep_data_write_failed_1623179080.1 2021-06-08 12:09:57,782 | infra | DEBUG | MainThread | [task_manager:abort_all_tasks:92] Task StatsWaitTask_default_ep_data_write_failed_1623179080.1 could not be cancelled or already finished. 2021-06-08 12:09:57,782 | infra | INFO | MainThread | [task_manager:abort_all_tasks:87] Stopping task StatsWaitTask_default_ep_data_write_failed_1623177710.79 2021-06-08 12:09:57,782 | infra | DEBUG | MainThread | [task_manager:abort_all_tasks:92] Task StatsWaitTask_default_ep_data_write_failed_1623177710.79 could not be cancelled or already finished. 2021-06-08 12:09:57,782 | infra | INFO | MainThread | [task_manager:abort_all_tasks:87] Stopping task StatsWaitTask_default_ep_data_write_failed_1623178922.85 2021-06-08 12:09:57,782 | infra | DEBUG | MainThread | [task_manager:abort_all_tasks:92] Task StatsWaitTask_default_ep_data_write_failed_1623178922.85 could not be cancelled or already finished. 2021-06-08 12:09:57,782 | infra | INFO | MainThread | [task_manager:abort_all_tasks:87] Stopping task StatsWaitTask_default_ep_data_write_failed_1623179242.93 2021-06-08 12:09:57,782 | infra | DEBUG | MainThread | [task_manager:abort_all_tasks:92] Task StatsWaitTask_default_ep_data_write_failed_1623179242.93 could not be cancelled or already finished. 2021-06-08 12:09:57,782 | infra | INFO | MainThread | [task_manager:abort_all_tasks:87] Stopping task StatsWaitTask_default_ep_data_write_failed_1623177998.93 2021-06-08 12:09:57,782 | infra | DEBUG | MainThread | [task_manager:abort_all_tasks:92] Task StatsWaitTask_default_ep_data_write_failed_1623177998.93 could not be cancelled or already finished. 2021-06-08 12:09:57,782 | infra | INFO | MainThread | [task_manager:abort_all_tasks:87] Stopping task StatsWaitTask_default_ep_data_write_failed_1623178766.2 2021-06-08 12:09:57,782 | infra | DEBUG | MainThread | [task_manager:abort_all_tasks:92] Task StatsWaitTask_default_ep_data_write_failed_1623178766.2 could not be cancelled or already finished. 2021-06-08 12:09:57,782 | infra | INFO | MainThread | [task_manager:abort_all_tasks:87] Stopping task StatsWaitTask_default_ep_data_write_failed_1623178346.2 2021-06-08 12:09:57,782 | infra | DEBUG | MainThread | [task_manager:abort_all_tasks:92] Task StatsWaitTask_default_ep_data_write_failed_1623178346.2 could not be cancelled or already finished. 2021-06-08 12:09:57,782 | infra | INFO | MainThread | [task_manager:abort_all_tasks:87] Stopping task StatsWaitTask_default_ep_data_write_failed_1623178767.54 2021-06-08 12:09:57,782 | infra | DEBUG | MainThread | [task_manager:abort_all_tasks:92] Task StatsWaitTask_default_ep_data_write_failed_1623178767.54 could not be cancelled or already finished. 2021-06-08 12:09:57,782 | infra | INFO | MainThread | [task_manager:abort_all_tasks:87] Stopping task StatsWaitTask_default_ep_data_write_failed_1623178923.29 2021-06-08 12:09:57,782 | infra | DEBUG | MainThread | [task_manager:abort_all_tasks:92] Task StatsWaitTask_default_ep_data_write_failed_1623178923.29 could not be cancelled or already finished. 2021-06-08 12:09:57,782 | infra | INFO | MainThread | [task_manager:abort_all_tasks:87] Stopping task StatsWaitTask_default_ep_data_write_failed_1623178345.79 2021-06-08 12:09:57,782 | infra | DEBUG | MainThread | [task_manager:abort_all_tasks:92] Task StatsWaitTask_default_ep_data_write_failed_1623178345.79 could not be cancelled or already finished. 2021-06-08 12:09:57,783 | infra | INFO | MainThread | [task_manager:abort_all_tasks:87] Stopping task StatsWaitTask_default_ep_data_write_failed_1623178345.36 2021-06-08 12:09:57,783 | infra | DEBUG | MainThread | [task_manager:abort_all_tasks:92] Task StatsWaitTask_default_ep_data_write_failed_1623178345.36 could not be cancelled or already finished. 2021-06-08 12:09:57,783 | infra | INFO | MainThread | [task_manager:abort_all_tasks:87] Stopping task StatsWaitTask_default_ep_data_write_failed_1623178182.87 2021-06-08 12:09:57,783 | infra | DEBUG | MainThread | [task_manager:abort_all_tasks:92] Task StatsWaitTask_default_ep_data_write_failed_1623178182.87 could not be cancelled or already finished. 2021-06-08 12:09:57,783 | infra | INFO | MainThread | [task_manager:abort_all_tasks:87] Stopping task StatsWaitTask_default_ep_data_write_failed_1623178530.99 2021-06-08 12:09:57,783 | infra | DEBUG | MainThread | [task_manager:abort_all_tasks:92] Task StatsWaitTask_default_ep_data_write_failed_1623178530.99 could not be cancelled or already finished. 2021-06-08 12:09:57,783 | infra | INFO | MainThread | [task_manager:abort_all_tasks:87] Stopping task StatsWaitTask_default_ep_data_write_failed_1623179243.36 2021-06-08 12:09:57,783 | infra | DEBUG | MainThread | [task_manager:abort_all_tasks:92] Task StatsWaitTask_default_ep_data_write_failed_1623179243.36 could not be cancelled or already finished. 2021-06-08 12:09:57,783 | infra | INFO | MainThread | [task_manager:abort_all_tasks:87] Stopping task StatsWaitTask_default_ep_data_write_failed_1623178532.34 2021-06-08 12:09:57,783 | infra | DEBUG | MainThread | [task_manager:abort_all_tasks:92] Task StatsWaitTask_default_ep_data_write_failed_1623178532.34 could not be cancelled or already finished. 2021-06-08 12:09:57,785 | infra | INFO | MainThread | [task_manager:abort_all_tasks:87] Stopping task StatsWaitTask_default_ep_data_write_failed_1623179243.79 2021-06-08 12:09:57,785 | infra | DEBUG | MainThread | [task_manager:abort_all_tasks:92] Task StatsWaitTask_default_ep_data_write_failed_1623179243.79 could not be cancelled or already finished. 2021-06-08 12:09:57,785 | infra | DEBUG | MainThread | [sdk_client3:shutdown:76] Closing clients from SDKClientPool 2021-06-08 12:09:57,785 | test | DEBUG | MainThread | [sdk_client3:close:307] Closing SDK for bucket 'default' 2021-06-08 12:09:57,798 | test | DEBUG | MainThread | [sdk_client3:close:311] Cluster disconnected and env shutdown 2021-06-08 12:09:57,798 | test | DEBUG | MainThread | [sdk_client3:close:307] Closing SDK for bucket 'default' 2021-06-08 12:09:57,811 | test | DEBUG | MainThread | [sdk_client3:close:311] Cluster disconnected and env shutdown 2021-06-08 12:09:57,811 | test | DEBUG | MainThread | [sdk_client3:close:307] Closing SDK for bucket 'default' 2021-06-08 12:09:57,822 | test | DEBUG | MainThread | [sdk_client3:close:311] Cluster disconnected and env shutdown 2021-06-08 12:09:57,822 | test | DEBUG | MainThread | [sdk_client3:close:307] Closing SDK for bucket 'default' 2021-06-08 12:09:57,835 | test | DEBUG | MainThread | [sdk_client3:close:311] Cluster disconnected and env shutdown 2021-06-08 12:09:57,835 | test | DEBUG | MainThread | [sdk_client3:close:307] Closing SDK for bucket 'default' 2021-06-08 12:09:57,848 | test | DEBUG | MainThread | [sdk_client3:close:311] Cluster disconnected and env shutdown 2021-06-08 12:09:57,848 | test | DEBUG | MainThread | [sdk_client3:close:307] Closing SDK for bucket 'default' 2021-06-08 12:09:57,859 | test | DEBUG | MainThread | [sdk_client3:close:311] Cluster disconnected and env shutdown 2021-06-08 12:09:57,859 | test | DEBUG | MainThread | [sdk_client3:close:307] Closing SDK for bucket 'default' 2021-06-08 12:09:57,871 | test | DEBUG | MainThread | [sdk_client3:close:311] Cluster disconnected and env shutdown 2021-06-08 12:09:57,872 | test | DEBUG | MainThread | [sdk_client3:close:307] Closing SDK for bucket 'default' 2021-06-08 12:09:57,884 | test | DEBUG | MainThread | [sdk_client3:close:311] Cluster disconnected and env shutdown 2021-06-08 12:09:57,884 | test | DEBUG | MainThread | [sdk_client3:close:307] Closing SDK for bucket 'default' 2021-06-08 12:09:57,897 | test | DEBUG | MainThread | [sdk_client3:close:311] Cluster disconnected and env shutdown 2021-06-08 12:09:57,897 | test | DEBUG | MainThread | [sdk_client3:close:307] Closing SDK for bucket 'default' 2021-06-08 12:09:57,910 | test | DEBUG | MainThread | [sdk_client3:close:311] Cluster disconnected and env shutdown 2021-06-08 12:09:57,910 | test | DEBUG | MainThread | [sdk_client3:close:307] Closing SDK for bucket 'default' 2021-06-08 12:09:57,921 | test | DEBUG | MainThread | [sdk_client3:close:311] Cluster disconnected and env shutdown 2021-06-08 12:09:57,921 | test | DEBUG | MainThread | [sdk_client3:close:307] Closing SDK for bucket 'default' 2021-06-08 12:09:57,934 | test | DEBUG | MainThread | [sdk_client3:close:311] Cluster disconnected and env shutdown 2021-06-08 12:09:57,934 | test | DEBUG | MainThread | [sdk_client3:close:307] Closing SDK for bucket 'default' 2021-06-08 12:09:57,947 | test | DEBUG | MainThread | [sdk_client3:close:311] Cluster disconnected and env shutdown 2021-06-08 12:09:57,947 | test | DEBUG | MainThread | [sdk_client3:close:307] Closing SDK for bucket 'default' 2021-06-08 12:09:57,959 | test | DEBUG | MainThread | [sdk_client3:close:311] Cluster disconnected and env shutdown 2021-06-08 12:09:57,959 | test | DEBUG | MainThread | [sdk_client3:close:307] Closing SDK for bucket 'default' 2021-06-08 12:09:57,970 | test | DEBUG | MainThread | [sdk_client3:close:311] Cluster disconnected and env shutdown 2021-06-08 12:09:57,970 | test | DEBUG | MainThread | [sdk_client3:close:307] Closing SDK for bucket 'default' 2021-06-08 12:09:57,983 | test | DEBUG | MainThread | [sdk_client3:close:311] Cluster disconnected and env shutdown 2021-06-08 12:09:57,983 | test | DEBUG | MainThread | [sdk_client3:close:307] Closing SDK for bucket 'default' 2021-06-08 12:09:57,996 | test | DEBUG | MainThread | [sdk_client3:close:311] Cluster disconnected and env shutdown 2021-06-08 12:09:58,069 | test | DEBUG | MainThread | [sdk_client3:close:307] Closing SDK for bucket 'default' 2021-06-08 12:09:58,082 | test | DEBUG | MainThread | [sdk_client3:close:311] Cluster disconnected and env shutdown 2021-06-08 12:09:58,082 | test | DEBUG | MainThread | [sdk_client3:close:307] Closing SDK for bucket 'default' 2021-06-08 12:09:58,095 | test | DEBUG | MainThread | [sdk_client3:close:311] Cluster disconnected and env shutdown 2021-06-08 12:09:58,095 | test | DEBUG | MainThread | [sdk_client3:close:307] Closing SDK for bucket 'default' 2021-06-08 12:09:58,108 | test | DEBUG | MainThread | [sdk_client3:close:311] Cluster disconnected and env shutdown 2021-06-08 12:09:58,111 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.156 with username: root, password: couchbase 2021-06-08 12:09:58,201 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 12:09:58,424 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: ls /etc/ 2021-06-08 12:09:58,483 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 12:09:58,483 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: ls /cygdrive/c/Windows 2021-06-08 12:09:58,497 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: uname -m 2021-06-08 12:09:58,509 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: cat /proc/cpuinfo 2021-06-08 12:09:58,522 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: df -Thl 2021-06-08 12:09:58,539 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: cat /proc/meminfo 2021-06-08 12:09:58,552 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: hostname 2021-06-08 12:09:58,566 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: hostname -d 2021-06-08 12:09:58,579 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-06-08 12:09:58,595 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:744] 172.23.106.156: No crash files found 2021-06-08 12:09:58,618 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-06-08 12:09:58,632 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.001038.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' 2021-06-08 12:09:58,658 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.001038.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-06-08 12:09:58,688 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.001038.txt 2021-06-08 12:09:58,709 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.001038.txt 2021-06-08 12:09:58,730 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.001039.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' 2021-06-08 12:09:58,757 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.001039.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-06-08 12:09:58,789 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.001039.txt 2021-06-08 12:09:58,811 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.001039.txt 2021-06-08 12:09:58,829 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.001040.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' 2021-06-08 12:09:58,855 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.001040.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-06-08 12:09:58,881 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.001040.txt 2021-06-08 12:09:58,901 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.001040.txt 2021-06-08 12:09:58,921 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.001041.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' 2021-06-08 12:09:58,944 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.001041.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-06-08 12:09:58,973 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.001041.txt 2021-06-08 12:09:58,992 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.001041.txt 2021-06-08 12:09:59,010 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.001042.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' 2021-06-08 12:09:59,035 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.001042.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-06-08 12:09:59,062 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.001042.txt 2021-06-08 12:09:59,084 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.001042.txt 2021-06-08 12:09:59,102 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.001043.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' 2021-06-08 12:09:59,128 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.001043.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-06-08 12:09:59,157 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.001043.txt 2021-06-08 12:09:59,177 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.001043.txt 2021-06-08 12:09:59,196 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.001044.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' 2021-06-08 12:09:59,220 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.001044.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-06-08 12:09:59,249 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.001044.txt 2021-06-08 12:09:59,267 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.001044.txt 2021-06-08 12:09:59,288 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.001045.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' 2021-06-08 12:09:59,313 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.001045.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-06-08 12:09:59,341 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.001045.txt 2021-06-08 12:09:59,359 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.001045.txt 2021-06-08 12:09:59,378 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.001046.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' 2021-06-08 12:09:59,403 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.001046.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-06-08 12:09:59,430 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.001046.txt 2021-06-08 12:09:59,450 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.001046.txt 2021-06-08 12:09:59,469 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.001047.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' 2021-06-08 12:09:59,493 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.001047.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-06-08 12:09:59,523 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.001047.txt 2021-06-08 12:09:59,542 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.001047.txt 2021-06-08 12:09:59,562 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.001048.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' 2021-06-08 12:09:59,586 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.001048.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-06-08 12:09:59,615 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.001048.txt 2021-06-08 12:09:59,635 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.001048.txt 2021-06-08 12:09:59,654 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.001049.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' 2021-06-08 12:09:59,680 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.001049.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-06-08 12:09:59,709 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.001049.txt 2021-06-08 12:09:59,729 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.001049.txt 2021-06-08 12:09:59,746 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.001050.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' 2021-06-08 12:09:59,770 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.001050.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-06-08 12:09:59,798 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.001050.txt 2021-06-08 12:09:59,816 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.001050.txt 2021-06-08 12:09:59,835 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.001051.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' 2021-06-08 12:09:59,859 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.001051.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-06-08 12:09:59,888 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.001051.txt 2021-06-08 12:09:59,908 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.001051.txt 2021-06-08 12:09:59,926 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.001052.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' 2021-06-08 12:09:59,951 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.001052.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-06-08 12:09:59,979 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.001052.txt 2021-06-08 12:09:59,997 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.001052.txt 2021-06-08 12:10:00,016 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.001053.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' 2021-06-08 12:10:00,039 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.001053.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-06-08 12:10:00,065 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.001053.txt 2021-06-08 12:10:00,085 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.001053.txt 2021-06-08 12:10:00,104 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.001054.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' 2021-06-08 12:10:00,128 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.001054.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-06-08 12:10:00,154 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.001054.txt 2021-06-08 12:10:00,174 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.001054.txt 2021-06-08 12:10:00,191 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.001055.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' 2021-06-08 12:10:00,217 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.001055.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-06-08 12:10:00,244 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.001055.txt 2021-06-08 12:10:00,263 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.001055.txt 2021-06-08 12:10:00,282 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.001056.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' 2021-06-08 12:10:00,306 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.001056.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-06-08 12:10:00,335 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.001056.txt 2021-06-08 12:10:00,354 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.001056.txt 2021-06-08 12:10:00,371 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.001057.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' 2021-06-08 12:10:00,394 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.001057.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-06-08 12:10:00,417 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.001057.txt 2021-06-08 12:10:00,433 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.001057.txt 2021-06-08 12:10:00,448 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-06-08 12:10:00,460 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:763] 172.23.106.156: No 'sanitizers.log.*' files found 2021-06-08 12:10:00,460 | infra | DEBUG | MainThread | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.156 2021-06-08 12:10:00,463 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.125 with username: root, password: couchbase 2021-06-08 12:10:00,555 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 12:10:00,767 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: ls /etc/ 2021-06-08 12:10:00,822 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 12:10:00,822 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: ls /cygdrive/c/Windows 2021-06-08 12:10:00,835 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: uname -m 2021-06-08 12:10:00,846 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: cat /proc/cpuinfo 2021-06-08 12:10:00,858 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: df -Thl 2021-06-08 12:10:00,874 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: cat /proc/meminfo 2021-06-08 12:10:00,885 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: hostname 2021-06-08 12:10:00,898 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: hostname -d 2021-06-08 12:10:00,911 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-06-08 12:10:00,926 | test | CRITICAL | MainThread | [basetestcase:check_coredump_exist:735] 172.23.106.125: 118 core dump seen 2021-06-08 12:10:00,927 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: rm -rf /opt/couchbase/var/lib/couchbase/crash/6d198271-692e-4dc2-241c0392-6d90dc71.core 2021-06-08 12:10:00,938 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: //opt/couchbase/bin/minidump-2-core /opt/couchbase/var/lib/couchbase/crash/6d198271-692e-4dc2-241c0392-6d90dc71.dmp > /opt/couchbase/var/lib/couchbase/crash/6d198271-692e-4dc2-241c0392-6d90dc71.core 2021-06-08 12:10:00,960 | test | INFO | MainThread | [basetestcase:get_gdb:677] running gdb --batch /opt/couchbase/bin/memcached -c /opt/couchbase/var/lib/couchbase/crash/6d198271-692e-4dc2-241c0392-6d90dc71.core -ex "bt full" -ex quit 2021-06-08 12:10:00,960 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: gdb --batch /opt/couchbase/bin/memcached -c /opt/couchbase/var/lib/couchbase/crash/6d198271-692e-4dc2-241c0392-6d90dc71.core -ex "bt full" -ex quit 2021-06-08 12:10:20,674 | test | CRITICAL | MainThread | [basetestcase:check_coredump_exist:736] 172.23.106.125: Stack Trace of first crash - 6d198271-692e-4dc2-241c0392-6d90dc71.dmp Core was generated by `/opt/couchbase/bin/memcached -C /opt/couchbase/var/lib/couchbase/config/memcach'. #0 0x00007f0cded87337 in raise () from /lib64/libc.so.6 #0 0x00007f0cded87337 in raise () from /lib64/libc.so.6 No symbol table info available. #1 0x00007f0cded88a28 in abort () from /lib64/libc.so.6 No symbol table info available. #2 0x00007f0cdf6df7f2 in __gnu_cxx::__verbose_terminate_handler() () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/libsupc++/vterminate.cc:50 terminating = false t = #3 0x0000000000a7c65f in backtrace_terminate_handler() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/utilities/terminate_handler.cc:87 No locals. #4 0x00007f0cdf6dd8f6 in __cxxabiv1::__terminate(void (*)()) () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/libsupc++/eh_terminate.cc:48 No locals. #5 0x00007f0cdf6dd961 in std::terminate () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/libsupc++/eh_terminate.cc:58 No locals. #6 0x00007f0cdf6ddbf4 in __cxxabiv1::__cxa_throw (obj=obj@entry=0x7f094c000940, tinfo=0xf676e0 , dest=0x4435c0 <_ZNSt13runtime_errorD1Ev@plt>) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/libsupc++/eh_throw.cc:95 globals = header = 0x7f094c0008c0 #7 0x00000000004c7874 in MagmaKVStore::makeCompactionContext (this=0x7f0cd8190300, vbid=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/magma-kvstore/magma-kvstore.cc:2623 ss = { >> = { >> = { >> = { = {_vptr.ios_base = 0xf67358 , std::allocator >+24>, static boolalpha = std::_S_boolalpha, static dec = std::_S_dec, static fixed = std::_S_fixed, static hex = std::_S_hex, static internal = std::_S_internal, static left = std::_S_left, static oct = std::_S_oct, static right = std::_S_right, static scientific = std::_S_scientific, static showbase = std::_S_showbase, static showpoint = std::_S_showpoint, static showpos = std::_S_showpos, static skipws = std::_S_skipws, static unitbuf = std::_S_unitbuf, static uppercase = std::_S_uppercase, static adjustfield = std::_S_adjustfield, static basefield = std::_S_basefield, static floatfield = std::_S_floatfield, static badbit = std::_S_badbit, static eofbit = std::_S_eofbit, static failbit = std::_S_failbit, static goodbit = std::_S_goodbit, static app = std::_S_app, static ate = std::_S_ate, static binary = std::_S_bin, static in = std::_S_in, static out = std::_S_out, static trunc = std::_S_trunc, static beg = std::_S_beg, static cur = std::_S_cur, static end = std::_S_end, _M_precision = 0, _M_width = 16151424, _M_flags = 16150424, _M_exception = std::_S_goodbit, _M_streambuf_state = -669624448, _M_callbacks = 0x7f0cd8165780, _M_word_zero = {_M_pword = 0x7f0cd8165790, _M_iword = 139693141677952}, _M_local_word = {{_M_pword = 0x7f0cd8165817, _M_iword = 139693141678464}, {_M_pword = 0x7f0cdfa04dc0 <(anonymous namespace)::c_locale_impl>, _M_iword = 24}, {_M_pword = 0x7f0cd8165780, _M_iword = 16}, {_M_pword = 0x200, _M_iword = 30744794945974132}, {_M_pword = 0xf673a8 , std::allocator >+104>, _M_iword = 6}, {_M_pword = 0x0, _M_iword = 4098}, {_M_pword = 0x0, _M_iword = 0}, {_M_pword = 0x0, _M_iword = 0}}, _M_word_size = 0, _M_word = 0x0, _M_ios_locale = {static none = 0, static ctype = 1, static numeric = 2, static collate = 4, static time = 8, static monetary = 16, static messages = 32, static all = 63, _M_impl = 0x0, static _S_classic = 0x0, static _S_global = 0x0, static _S_categories = 0x3bd680, static _S_once = 0, static _S_twinned_facets = 0x7f0cdf9f7f00 }}, _M_tie = 0x0, _M_fill = 0 '\000', _M_fill_init = false, _M_streambuf = 0x0, _M_ctype = 0x0, _M_num_put = 0x0, _M_num_get = 0x0}, _vptr.basic_istream = 0xf67358 , std::allocator >+24>, _M_gcount = 0}, >> = {_vptr.basic_ostream = 0xf67380 , std::allocator >+64>}, }, _M_stringbuf = { >> = {_vptr.basic_streambuf = 0xf66f98 , std::allocator >+16>, _M_in_beg = 0x7f0cd8165780
, _M_in_cur = 0x7f0cd8165780
, _M_in_end = 0x7f0cd8165790
, _M_out_beg = 0x7f0cd8165780
, _M_out_cur = 0x7f0cd8165817
, _M_out_end = 0x7f0cd8165980
, _M_buf_locale = {static none = 0, static ctype = 1, static numeric = 2, static collate = 4, static time = 8, static monetary = 16, static messages = 32, static all = 63, _M_impl = 0x7f0cdfa04dc0 <(anonymous namespace)::c_locale_impl>, static _S_classic = 0x0, static _S_global = 0x0, static _S_categories = 0x3bd680, static _S_once = 0, static _S_twinned_facets = 0x7f0cdf9f7f00 }}, _M_mode = 24, _M_string = {static npos = 18446744073709551615, _M_dataplus = {> = {<__gnu_cxx::new_allocator> = {}, }, _M_p = 0x7f0cd8165780
}, _M_string_length = 16, {_M_local_buf = "\000\002\000\000\000\000\000\000tore::m", _M_allocated_capacity = 512}}}} config = {purge_before_ts = 1622919032, purge_before_seq = 0, drop_deletes = false, retain_erroneous_tombstones = false} ctx = {> = {> = {}, _M_ptr = 0x7f0cc82d1e10, _M_refcount = {_M_pi = 0x7f0cc82d1e00}}, } status = dropped = readState = {status = {s = {_M_t = { >> = {_M_t = { >> = { >> = {, true>> = {> = {}, }, }, > = {_M_head_impl = 0x7f0cd01a8190}, }, }}, }}}, vbstate = {static CurrentVersion = 4, maxDeletedSeqno = {counter = {_M_elems = "\000\000\000\000\000"}}, highSeqno = 0, purgeSeqno = 0, lastSnapStart = 0, lastSnapEnd = 0, maxCas = 0, hlcCasEpochSeqno = -1, mightContainXattrs = false, supportsNamespaces = true, version = 4, persistedCompletedSeqno = 0, persistedPreparedSeqno = 0, highPreparedSeqno = 0, maxVisibleSeqno = 0, onDiskPrepares = 0, onDiskPrepareBytes = 0, checkpointType = Memory, transition = {failovers = {static npos = 18446744073709551615, _M_dataplus = {> = {<__gnu_cxx::new_allocator> = {}, }, _M_p = 0x7f0c36fe2960 ""}, _M_string_length = 0, {_M_local_buf = '\000' , _M_allocated_capacity = 0}}, replicationTopology = {m_type = nlohmann::detail::null, m_value = {object = 0x0, array = 0x0, string = 0x0, boolean = false, number_integer = 0, number_unsigned = 0, number_float = 0}}, state = vbucket_state_dead}}, kvstoreRev = 1} #8 0x0000000000859585 in MagmaKVStore::compactionCallBack(MagmaKVStore::MagmaCompactionCB&, magma::Slice const&, magma::Slice const&, magma::Slice const&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/magma-kvstore/magma-kvstore.cc:278 vbid = {vbid = 934} itemString = @0x7f0c7c7a4418: seqno = exptime = #9 0x000000000094a6bd in magma::DocSeqGCContext::Transform (this=0x7f0c30cd8500, key=..., value=..., buffer=..., outputValue=0x7f0c36fe2d50) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/kvstore/format.h:44 drop = false #10 0x00000000009cf755 in magma::ItemGCFilter::Process(std::vector, std::allocator > >&, std::vector, std::allocator > >*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/lsm/mvcc.cc:106 newValue = {data = 0x0, len = 0} deleted = oldValue = 0x0 kv = {> = {}, first = {data = 0x7f0c2744b400
, len = 27}, second = {data = 0x7f0c2744b41b
, len = 51}} item = {key = {data = 0x7f0c2744b400
, len = 18}, sn = 576460752303423488, op = magma::Update} gcOffset = 1 lastSn = #11 0x0000000000962bb2 in magma::LSMTree::mergeSSTables(std::vector, std::allocator > >&, magma::ItemGCFilter&, unsigned long, double, std::vector, std::allocator > >*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/lsm/lsm_tree.cc:649 magma::rdBuf = std::__ioinit = {static _S_refcount = 0, static _S_synced_with_stdio = true} fmt::v5::internal::basic_data::POWERS_OF_10_32 = {0, 10, 100, 1000, 10000, 100000, 1000000, 10000000, 100000000, 1000000000} fmt::v5::internal::basic_data::POWERS_OF_10_64 = {0, 10, 100, 1000, 10000, 100000, 1000000, 10000000, 100000000, 1000000000, 10000000000, 100000000000, 1000000000000, 10000000000000, 100000000000000, 1000000000000000, 10000000000000000, 100000000000000000, 1000000000000000000, 10000000000000000000} fmt::v5::internal::basic_data::DIGITS = "00010203040506070809101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899" boost::exception_detail::exception_ptr_static_exception_object::e = {ptr_ = {px = 0x0, pn = {pi_ = 0x0}}} boost::exception_detail::exception_ptr_static_exception_object::e = {ptr_ = {px = 0x0, pn = {pi_ = 0x0}}} #12 0x00000000009c919d in magma::LSMTree::compactLevelForInputs(int, int, std::unique_lock&, std::vector, std::allocator > >&, std::vector, std::allocator > >&, bool, std::function > (bool)>) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/lsm/compaction.cc:168 status = {s = {_M_t = { >> = {_M_t = { >> = { >> = {, true>> = {> = {}, }, }, > = {_M_head_impl = 0x0}, }, }}, }}} compactID = 2 _c = {fn = {> = {}, = {static _M_max_size = 16, static _M_max_align = 8, _M_functor = {_M_unused = {_M_object = 0x7f0c7c570960, _M_const_object = 0x7f0c7c570960, _M_function_pointer = 0x7f0c7c570960, _M_member_pointer = (void (std::_Undefined_class::*)(std::_Undefined_class * const)) 0x7f0c7c570960}, _M_pod_data = "`\tW|\f\177\000\000\000\000\000\000\000\000\000"}, _M_manager = 0x9c5290 &, magma::TableList&, magma::TableList&, bool, magma::GCContextBuilder):: >::_M_manager(std::_Any_data &, const std::_Any_data &, std::_Manager_operation)>}, _M_invoker = 0x9c5a60 &, magma::TableList&, magma::TableList&, bool, magma::GCContextBuilder):: >::_M_invoke(const std::_Any_data &)>}} srcLevelSize = 8192 targetSize = 4194304 levelSizeRatio = 0 srcDataSize = dstDataSize = srcLogicalDataSz = 8192 dstLogicalDataSz = 147456 height = isHighestLevel = writeAmp = 18 inputList = {, std::allocator > >> = {_M_impl = { >> = {<__gnu_cxx::new_allocator >> = {}, }, , std::allocator > >::_Vector_impl_data> = {_M_start = 0x7f0c29e27000, _M_finish = 0x7f0c29e27020, _M_end_of_storage = 0x7f0c29e27020}, }}, } outputList = {, std::allocator > >> = {_M_impl = { >> = {<__gnu_cxx::new_allocator >> = {}, }, , std::allocator > >::_Vector_impl_data> = {_M_start = 0x0, _M_finish = 0x0, _M_end_of_storage = 0x0}, }}, } needSSTablesMerging = true bloomFilterAccuracy = 0.94999998807907104 levelSize = 2097152 fileCompaction = true gcFilter = {_M_t = { >> = {_M_t = { >> = { >> = {, true>> = {> = {}, }, }, > = {_M_head_impl = 0x7f0c2ac2acc0}, }, }}, }} _ = {fn = {> = {}, = {static _M_max_size = 16, static _M_max_align = 8, _M_functor = {_M_unused = {_M_object = 0x7f0c36fe3398, _M_const_object = 0x7f0c36fe3398, _M_function_pointer = 0x7f0c36fe3398, _M_member_pointer = (void (std::_Undefined_class::*)(std::_Undefined_class * const)) 0x7f0c36fe3398, this adjustment 139690438964176}, _M_pod_data = "\230\063\376\066\f\177\000\000\320\063\376\066\f\177\000"}, _M_manager = 0x9c4f80 &, magma::TableList&, magma::TableList&, bool, magma::GCContextBuilder):: >::_M_manager(std::_Any_data &, const std::_Any_data &, std::_Manager_operation)>}, _M_invoker = 0x9c5320 &, std::vector, std::allocator > >&, std::vector, std::allocator > >&, bool, std::function > (bool)>)::{lambda()#2}>::_M_invoke(std::_Any_data const&)>}} resultState = outDataSize = newSnapshot = {> = {> = {}, _M_ptr = 0x7f0c29e77210, _M_refcount = {_M_pi = 0x0}}, } dur = 382580694 secs = MBps = rMBps = 6.9016246944896283e-310 wMBps = 5.0652662371470238e-317 gcRatio = 819200 newLevelRatio = 0 #13 0x00000000009cb5b4 in magma::LSMTree::compactLevel(int, std::unique_lock&, bool, std::function > (bool)>) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/lsm/compaction.cc:73 srcLevelList = {, std::allocator > >> = {_M_impl = { >> = {<__gnu_cxx::new_allocator >> = {}, }, , std::allocator > >::_Vector_impl_data> = {_M_start = 0x7f0c314241b0, _M_finish = 0x7f0c314241c0, _M_end_of_storage = 0x7f0c314241c0}, }}, } dstLevelList = {, std::allocator > >> = {_M_impl = { >> = {<__gnu_cxx::new_allocator >> = {}, }, , std::allocator > >::_Vector_impl_data> = {_M_start = 0x7f0c308da120, _M_finish = 0x7f0c308da130, _M_end_of_storage = 0x7f0c308da130}, }}, } inputList = {, std::allocator > >> = {_M_impl = { >> = {<__gnu_cxx::new_allocator >> = {}, }, , std::allocator > >::_Vector_impl_data> = {_M_start = 0x0, _M_finish = 0x0, _M_end_of_storage = 0x0}, }}, } outputList = {, std::allocator > >> = {_M_impl = { >> = {<__gnu_cxx::new_allocator >> = {}, }, , std::allocator > >::_Vector_impl_data> = {_M_start = 0x0, _M_finish = 0x0, _M_end_of_storage = 0x0}, }}, } status = {s = {_M_t = { >> = {_M_t = { >> = { >> = {, true>> = {> = {}, }, }, > = {_M_head_impl = 0x0}, }, }}, }}} dstLevel = 4 #14 0x00000000009cb855 in magma::LSMTree::compact(std::unique_lock&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/lsm/compaction.cc:339 level = status = {s = {_M_t = { >> = {_M_t = { >> = { >> = {, true>> = {> = {}, }, }, > = {_M_head_impl = 0x0}, }, }}, }}} ratio = @0x7f0c29ad38e0: __for_range = @0x7f0c36fe36f0: {, std::allocator > >> = {_M_impl = { >> = {<__gnu_cxx::new_allocator >> = {}, }, , std::allocator > >::_Vector_impl_data> = {_M_start = 0x7f0c29ad38e0, _M_finish = 0x7f0c29ad38f0, _M_end_of_storage = 0x7f0c29ad38f0}, }}, } compacted = false sizeRatios = {, std::allocator > >> = {_M_impl = { >> = {<__gnu_cxx::new_allocator >> = {}, }, , std::allocator > >::_Vector_impl_data> = {_M_start = 0x7f0c29ad38e0, _M_finish = 0x7f0c29ad38f0, _M_end_of_storage = 0x7f0c29ad38f0}, }}, } #15 0x0000000000957fc8 in std::_Function_handler::_M_invoke(std::_Any_data const&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/lsm/lsm_tree.cc:297 magma::rdBuf = std::__ioinit = {static _S_refcount = 0, static _S_synced_with_stdio = true} fmt::v5::internal::basic_data::POWERS_OF_10_32 = {0, 10, 100, 1000, 10000, 100000, 1000000, 10000000, 100000000, 1000000000} fmt::v5::internal::basic_data::POWERS_OF_10_64 = {0, 10, 100, 1000, 10000, 100000, 1000000, 10000000, 100000000, 1000000000, 10000000000, 100000000000, 1000000000000, 10000000000000, 100000000000000, 1000000000000000, 10000000000000000, 100000000000000000, 1000000000000000000, 10000000000000000000} fmt::v5::internal::basic_data::DIGITS = "00010203040506070809101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899" boost::exception_detail::exception_ptr_static_exception_object::e = {ptr_ = {px = 0x0, pn = {pi_ = 0x0}}} boost::exception_detail::exception_ptr_static_exception_object::e = {ptr_ = {px = 0x0, pn = {pi_ = 0x0}}} #16 0x000000000099653d in operator() (this=0x7f0c30a3dcd8) at /opt/gcc-10.2.0/include/c++/10.2.0/bits/std_function.h:248 No locals. #17 magma::TimedTask::Complete(bool) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:143 wg = 0x7f0c5c54a740 envGuard = {previous = 0x0} currTask = {> = {> = {}, _M_ptr = 0x7f0c30a3dcd0, _M_refcount = {_M_pi = 0x7f0c30a3dcc0}}, } #18 0x0000000000997d31 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:160 task = {> = {> = {}, _M_ptr = 0x7f0c29233b10, _M_refcount = {_M_pi = 0x7f0c29233b00}}, } worker = 0x7f0cd002d340 #19 0x0000000000aab9e9 in run (this=0x7f0cd9b3a740) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 No locals. #20 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 No locals. #21 0x00007f0ce12eae65 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #22 0x00007f0cdee4f88d in clone () from /lib64/libc.so.6 No symbol table info available. 2021-06-08 12:10:20,676 | test | INFO | MainThread | [basetestcase:get_full_thread_dump:686] running gdb -p `(pidof memcached)` -ex "thread apply all bt" -ex detach -ex quit 2021-06-08 12:10:20,676 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: gdb -p `(pidof memcached)` -ex "thread apply all bt" -ex detach -ex quit [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Loaded symbols for /lib64/libpthread.so.0 Reading symbols from /lib64/librt.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/librt.so.1 Reading symbols from /opt/couchbase/bin/../lib/libsnappy.so.1...Reading symbols from /usr/lib/debug/opt/couchbase/lib/libsnappy.so.1.2.0.debug...done. done. Loaded symbols for /opt/couchbase/bin/../lib/libsnappy.so.1 Reading symbols from /opt/couchbase/bin/../lib/libjemalloc.so.2...Reading symbols from /usr/lib/debug/opt/couchbase/lib/libjemalloc.so.2.debug...done. done. Loaded symbols for /opt/couchbase/bin/../lib/libjemalloc.so.2 Reading symbols from /opt/couchbase/bin/../lib/libevent_core-2.1.so.7...Reading symbols from /usr/lib/debug/opt/couchbase/lib/libevent_core-2.1.so.7.0.0.debug...done. done. Loaded symbols for /opt/couchbase/bin/../lib/libevent_core-2.1.so.7 Reading symbols from /opt/couchbase/bin/../lib/libevent_extra-2.1.so.7...Reading symbols from /usr/lib/debug/opt/couchbase/lib/libevent_extra-2.1.so.7.0.0.debug...done. done. Loaded symbols for /opt/couchbase/bin/../lib/libevent_extra-2.1.so.7 Reading symbols from /opt/couchbase/bin/../lib/libevent_pthreads-2.1.so.7...Reading symbols from /usr/lib/debug/opt/couchbase/lib/libevent_pthreads-2.1.so.7.0.0.debug...done. done. Loaded symbols for /opt/couchbase/bin/../lib/libevent_pthreads-2.1.so.7 Reading symbols from /opt/couchbase/bin/../lib/libevent_openssl-2.1.so.7...Reading symbols from /usr/lib/debug/opt/couchbase/lib/libevent_openssl-2.1.so.7.0.0.debug...done. done. Loaded symbols for /opt/couchbase/bin/../lib/libevent_openssl-2.1.so.7 Reading symbols from /opt/couchbase/bin/../lib/libssl.so.1.1...Reading symbols from /usr/lib/debug/opt/couchbase/lib/libssl.so.1.1.debug...(no debugging symbols found)...done. (no debugging symbols found)...done. Loaded symbols for /opt/couchbase/bin/../lib/libssl.so.1.1 Reading symbols from /opt/couchbase/bin/../lib/libcrypto.so.1.1...Reading symbols from /usr/lib/debug/opt/couchbase/lib/libcrypto.so.1.1.debug...(no debugging symbols found)...done. (no debugging symbols found)...done. Loaded symbols for /opt/couchbase/bin/../lib/libcrypto.so.1.1 Reading symbols from /opt/couchbase/bin/../lib/libstdc++.so.6...done. Loaded symbols for /opt/couchbase/bin/../lib/libstdc++.so.6 Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done. Loaded symbols for /lib64/libm.so.6 Reading symbols from /opt/couchbase/bin/../lib/libgcc_s.so.1...done. Loaded symbols for /opt/couchbase/bin/../lib/libgcc_s.so.1 Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done. Loaded symbols for /lib64/libc.so.6 Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/ld-linux-x86-64.so.2 0x00007ff8371aae63 in epoll_wait () from /lib64/libc.so.6 To enable execution of this file add add-auto-load-safe-path /opt/couchbase/lib/libstdc++.so.6.0.28-gdb.py line to your configuration file "/root/.gdbinit". To completely disable this security protection add set auto-load safe-path / line to your configuration file "/root/.gdbinit". For more information about this security protection see the "Auto-loading safe path" section in the GDB manual. E.g., run from the shell: info "(gdb)Auto-loading safe path" Thread 61 (Thread 0x7ff8367ff700 (LWP 4548)): #0 0x00007ff839649da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007ff838d865a7 in background_thread_sleep (tsdn=, interval=10000000000, info=) at src/background_thread.c:255 #2 background_work_sleep_once (ind=0, info=, tsdn=) at src/background_thread.c:307 #3 background_thread0_work (tsd=0x7ff8367ee228) at src/background_thread.c:452 #4 background_work (ind=, tsd=0x7ff8367ee228) at src/background_thread.c:490 #5 background_thread_entry (ind_arg=) at src/background_thread.c:522 #6 0x00007ff839645e65 in start_thread () from /lib64/libpthread.so.0 #7 0x00007ff8371aa88d in clone () from /lib64/libc.so.6 Thread 60 (Thread 0x7ff835a7e700 (LWP 4549)): #0 0x00007ff839649da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000000000a855a5 in __gthread_cond_timedwait ( __abs_timeout=0x7ff835a6c760, __mutex=, __cond=0x7ff835fca138) at /opt/gcc-10.2.0/include/c++/10.2.0/x86_64-pc-linux-gnu/bits/gthr-default.h:872 #2 __wait_until_impl > > (__lock=..., __lock=..., __atime=, this=0x7ff835fca138) at /opt/gcc-10.2.0/include/c++/10.2.0/condition_variable:232 #3 wait_until > > (__atime=..., __lock=..., this=0x7ff835fca138) at /opt/gcc-10.2.0/include/c++/10.2.0/condition_variable:141 #4 wait_until >, spdlog::details::mpmc_blocking_queue::dequeue_for:: > (__p=..., __atime=..., __lock=..., this=0x7ff835fca138) at /opt/gcc-10.2.0/include/c++/10.2.0/condition_variable:158 #5 wait_for, spdlog::details::mpmc_blocking_queue::dequeue_for:: > ( __rtime=, __rtime=, __p=..., __lock=..., this=0x7ff835fca138) at /opt/gcc-10.2.0/include/c++/10.2.0/condition_variable:185 #6 dequeue_for (wait_duration=..., popped_item=..., this=0x7ff835fca110) at /home/couchbase/jenkins/workspace/couchbase-server-unix/third_party/spdlog/include/spdlog/details/mpmc_blocking_q.h:61 #7 spdlog::details::thread_pool::process_next_msg_ ( this=this@entry=0x7ff835fca110) at /home/couchbase/jenkins/workspace/couchbase-server-unix/third_party/spdlog/include/spdlog/details/thread_pool.h:193 #8 0x0000000000a85bc0 in spdlog::details::thread_pool::worker_loop_ ( this=0x7ff835fca110) at /home/couchbase/jenkins/workspace/couchbase-server-unix/third_party/spdlog/include/spdlog/details/thread_pool.h:184 #9 0x00007ff837a61d40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #10 0x00007ff839645e65 in start_thread () from /lib64/libpthread.so.0 #11 0x00007ff8371aa88d in clone () from /lib64/libc.so.6 Thread 59 (Thread 0x7ff83527d700 (LWP 4550)): #0 0x00007ff839649da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000000000a84d5b in __gthread_cond_timedwait ( __abs_timeout=0x7ff83526b9b0, __mutex=, __cond=) at /opt/gcc-10.2.0/include/c++/10.2.0/x86_64-pc-linux-gnu/bits/gthr-default.h:872 #2 __wait_until_impl > > (__lock=..., __lock=..., __atime=, this=) at /opt/gcc-10.2.0/include/c++/10.2.0/condition_variable:232 #3 wait_until > > (__atime=..., __lock=..., this=) at /opt/gcc-10.2.0/include/c++/10.2.0/condition_variable:141 #4 wait_until >, spdlog::details::periodic_worker::periodic_worker(const std::function&, std::chrono::seconds):::: > (__p=..., __atime=..., __lock=..., this=) at /opt/gcc-10.2.0/include/c++/10.2.0/condition_variable:158 #5 wait_for, spdlog::details::periodic_worker::periodic_worker(const std::function&, std::chrono::seconds):::: > (__rtime=..., __rtime=..., __p=..., __lock=..., this=) at /opt/gcc-10.2.0/include/c++/10.2.0/condition_variable:185 #6 operator() (__closure=0x7ff835fcb648) at /home/couchbase/jenkins/workspace/couchbase-server-unix/third_party/spdlog/include/spdlog/details/periodic_worker.h:38 #7 __invoke_impl&, std::chrono::seconds):: > ( __f=) at /opt/gcc-10.2.0/include/c++/10.2.0/bits/invoke.h:60 #8 __invoke&, std::chrono::seconds):: > ( __fn=) at /opt/gcc-10.2.0/include/c++/10.2.0/bits/invoke.h:95 #9 _M_invoke<0> (this=0x7ff835fcb648) at /opt/gcc-10.2.0/include/c++/10.2.0/thread:264 #10 operator() (this=0x7ff835fcb648) at /opt/gcc-10.2.0/include/c++/10.2.0/thread:271 #11 _ZNSt6thread11_State_implINS_8_InvokerISt5tupleIJZN6spdlog7details15periodic_workerC4ERKSt8functionIFvvEENSt6chrono8durationIlSt5ratioILl1ELl1EEEEEUlvE_EEEEE6_M_runEv (this=0x7ff835fcb640) at /opt/gcc-10.2.0/include/c++/10.2.0/thread:215 #12 0x00007ff837a61d40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #13 0x00007ff839645e65 in start_thread () from /lib64/libpthread.so.0 #14 0x00007ff8371aa88d in clone () from /lib64/libc.so.6 Thread 58 (Thread 0x7ff834a7c700 (LWP 4551)): #0 0x00007ff83719fbed in poll () from /lib64/libc.so.6 #1 0x000000000058a841 in get_command (buffsize=80, buffer=0x7ff834a6a990 "@\006\310\071\370\177") at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/stdin_check.cc:57 #2 check_stdin_thread (arg=) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/stdin_check.cc:98 #3 0x0000000000aab9e9 in run (this=0x7ff835fc61b0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #4 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #5 0x00007ff839645e65 in start_thread () from /lib64/libpthread.so.0 #6 0x00007ff8371aa88d in clone () from /lib64/libc.so.6 Thread 57 (Thread 0x7ff83427b700 (LWP 4552)): #0 0x00007ff8396499f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00000000008c9e4b in consume_socket (thread_index=, sp=0x20390e0, ctx=0x2037bc0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/prometheus-cpp/prometheus-cpp-prefix/src/prometheus-cpp/3rdparty/civetweb/src/civetweb.c:18846 #2 worker_thread_run (conn=0x20387f0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/prometheus-cpp/prometheus-cpp-prefix/src/prometheus-cpp/3rdparty/civetweb/src/civetweb.c:18987 #3 worker_thread (thread_func_param=0x20387f0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/prometheus-cpp/prometheus-cpp-prefix/src/prometheus-cpp/3rdparty/civetweb/src/civetweb.c:19145 #4 0x00007ff839645e65 in start_thread () from /lib64/libpthread.so.0 #5 0x00007ff8371aa88d in clone () from /lib64/libc.so.6 Thread 56 (Thread 0x7ff833a7a700 (LWP 4553)): #0 0x00007ff8396499f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00000000008c9e4b in consume_socket (thread_index=, sp=0x2039ac8, ctx=0x2037bc0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/prometheus-cpp/prometheus-cpp-prefix/src/prometheus-cpp/3rdparty/civetweb/src/civetweb.c:18846 #2 worker_thread_run (conn=0x20391d8) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/prometheus-cpp/prometheus-cpp-prefix/src/prometheus-cpp/3rdparty/civetweb/src/civetweb.c:18987 #3 worker_thread (thread_func_param=0x20391d8) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/prometheus-cpp/prometheus-cpp-prefix/src/prometheus-cpp/3rdparty/civetweb/src/civetweb.c:19145 #4 0x00007ff839645e65 in start_thread () from /lib64/libpthread.so.0 #5 0x00007ff8371aa88d in clone () from /lib64/libc.so.6 Thread 55 (Thread 0x7ff833279700 (LWP 4554)): #0 0x00007ff83719fbed in poll () from /lib64/libc.so.6 #1 0x00000000008c0db8 in master_thread_run (ctx=0x2037bc0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/prometheus-cpp/prometheus-cpp-prefix/src/prometheus-cpp/3rdparty/civetweb/src/civetweb.c:19291 #2 master_thread (thread_func_param=0x2037bc0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/prometheus-cpp/prometheus-cpp-prefix/src/prometheus-cpp/3rdparty/civetweb/src/civetweb.c:19386 #3 0x00007ff839645e65 in start_thread () from /lib64/libpthread.so.0 #4 0x00007ff8371aa88d in clone () from /lib64/libc.so.6 Thread 54 (Thread 0x7ff832a78700 (LWP 4555)): #0 0x00007ff839649da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00000000005bb7d6 in ExternalAuthManagerThread::run() () at /opt/gcc-10.2.0/include/c++/10.2.0/x86_64-pc-linux-gnu/bits/gthr-default.h:872 #2 0x0000000000ab4cc3 in Couchbase::Thread::thread_entry() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/thread.cc:38 #3 0x0000000000aab9e9 in run (this=0x7ff835fc6900) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #4 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #5 0x00007ff839645e65 in start_thread () from /lib64/libpthread.so.0 #6 0x00007ff8371aa88d in clone () from /lib64/libc.so.6 Thread 53 (Thread 0x7ff831ef7700 (LWP 4556)): #0 0x00007ff839649da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000000000629ee7 in AuditImpl::consume_events() () at /opt/gcc-10.2.0/include/c++/10.2.0/x86_64-pc-linux-gnu/bits/gthr-default.h:872 #2 0x0000000000aab9e9 in run (this=0x7ff835fc6660) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #3 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #4 0x00007ff839645e65 in start_thread () from /lib64/libpthread.so.0 #5 0x00007ff8371aa88d in clone () from /lib64/libc.so.6 Thread 52 (Thread 0x7ff8316f6700 (LWP 4557)): #0 0x00007ff8371aae63 in epoll_wait () from /lib64/libc.so.6 #1 0x00007ff838b4d3c5 in epoll_dispatch (base=0x7ff835fdd300, tv=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/epoll.c:465 #2 0x00007ff838b44376 in event_base_loop (base=0x7ff835fdd300, flags=flags@entry=0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:1998 #3 0x000000000058adcb in worker_libevent (arg=0x7ff835d2f000) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/thread.cc:202 #4 0x0000000000aab9e9 in run (this=0x7ff835fe3b60) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #5 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #6 0x00007ff839645e65 in start_thread () from /lib64/libpthread.so.0 #7 0x00007ff8371aa88d in clone () from /lib64/libc.so.6 Thread 51 (Thread 0x7ff830ef5700 (LWP 4558)): #0 0x00007ff8371aae63 in epoll_wait () from /lib64/libc.so.6 #1 0x00007ff838b4d3c5 in epoll_dispatch (base=0x7ff835fdd600, tv=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/epoll.c:465 #2 0x00007ff838b44376 in event_base_loop (base=0x7ff835fdd600, flags=flags@entry=0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:1998 #3 0x000000000058adcb in worker_libevent (arg=0x7ff835d2faa8) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/thread.cc:202 #4 0x0000000000aab9e9 in run (this=0x7ff835fe3920) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #5 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #6 0x00007ff839645e65 in start_thread () from /lib64/libpthread.so.0 #7 0x00007ff8371aa88d in clone () from /lib64/libc.so.6 Thread 50 (Thread 0x7ff82bfff700 (LWP 4559)): #0 0x00007ff8371aae63 in epoll_wait () from /lib64/libc.so.6 #1 0x00007ff838b4d3c5 in epoll_dispatch (base=0x7ff835fdd900, tv=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/epoll.c:465 #2 0x00007ff838b44376 in event_base_loop (base=0x7ff835fdd900, flags=flags@entry=0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:1998 #3 0x000000000058adcb in worker_libevent (arg=0x7ff835d30550) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/thread.cc:202 #4 0x0000000000aab9e9 in run (this=0x7ff835fe3b00) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #5 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #6 0x00007ff839645e65 in start_thread () from /lib64/libpthread.so.0 #7 0x00007ff8371aa88d in clone () from /lib64/libc.so.6 Thread 49 (Thread 0x7ff82b7fe700 (LWP 4560)): #0 0x00007ff8371aae63 in epoll_wait () from /lib64/libc.so.6 #1 0x00007ff838b4d3c5 in epoll_dispatch (base=0x7ff835fddc00, tv=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/epoll.c:465 #2 0x00007ff838b44376 in event_base_loop (base=0x7ff835fddc00, flags=flags@entry=0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:1998 #3 0x000000000058adcb in worker_libevent (arg=0x7ff835d30ff8) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/thread.cc:202 #4 0x0000000000aab9e9 in run (this=0x7ff835fe3c20) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #5 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #6 0x00007ff839645e65 in start_thread () from /lib64/libpthread.so.0 #7 0x00007ff8371aa88d in clone () from /lib64/libc.so.6 Thread 48 (Thread 0x7ff82affd700 (LWP 4561)): #0 0x00007ff8371aae63 in epoll_wait () from /lib64/libc.so.6 #1 0x00007ff838b4d3c5 in epoll_dispatch (base=0x7ff835fddf00, tv=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/epoll.c:465 #2 0x00007ff838b44376 in event_base_loop (base=0x7ff835fddf00, flags=flags@entry=0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:1998 #3 0x000000000058adcb in worker_libevent (arg=0x7ff835d31aa0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/thread.cc:202 #4 0x0000000000aab9e9 in run (this=0x7ff835fe3aa0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #5 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #6 0x00007ff839645e65 in start_thread () from /lib64/libpthread.so.0 #7 0x00007ff8371aa88d in clone () from /lib64/libc.so.6 Thread 47 (Thread 0x7ff82a7fc700 (LWP 4562)): #0 0x00007ff8371aae63 in epoll_wait () from /lib64/libc.so.6 #1 0x00007ff838b4d3c5 in epoll_dispatch (base=0x7ff835fde200, tv=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/epoll.c:465 #2 0x00007ff838b44376 in event_base_loop (base=0x7ff835fde200, flags=flags@entry=0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:1998 #3 0x000000000058adcb in worker_libevent (arg=0x7ff835d32548) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/thread.cc:202 #4 0x0000000000aab9e9 in run (this=0x7ff835fe3c80) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #5 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #6 0x00007ff839645e65 in start_thread () from /lib64/libpthread.so.0 #7 0x00007ff8371aa88d in clone () from /lib64/libc.so.6 Thread 46 (Thread 0x7ff829ffb700 (LWP 4563)): #0 0x00007ff8371aae63 in epoll_wait () from /lib64/libc.so.6 #1 0x00007ff838b4d3c5 in epoll_dispatch (base=0x7ff835fde500, tv=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/epoll.c:465 #2 0x00007ff838b44376 in event_base_loop (base=0x7ff835fde500, flags=flags@entry=0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:1998 #3 0x000000000058adcb in worker_libevent (arg=0x7ff835d32ff0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/thread.cc:202 #4 0x0000000000aab9e9 in run (this=0x7ff835fe3cb0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #5 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #6 0x00007ff839645e65 in start_thread () from /lib64/libpthread.so.0 #7 0x00007ff8371aa88d in clone () from /lib64/libc.so.6 Thread 45 (Thread 0x7ff8297fa700 (LWP 4564)): #0 0x00007ff8396499f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007ff837a5c8bc in __gthread_cond_wait (__mutex=, __cond=__cond@entry=0x7ff832069158) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=this@entry=0x7ff832069158, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000005ce701 in Executor::run() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/executor.cc:38 #4 0x0000000000ab4cc3 in Couchbase::Thread::thread_entry() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/thread.cc:38 #5 0x0000000000aab9e9 in run (this=0x7ff835fe3a70) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007ff839645e65 in start_thread () from /lib64/libpthread.so.0 #8 0x00007ff8371aa88d in clone () from /lib64/libc.so.6 Thread 44 (Thread 0x7ff828ff9700 (LWP 4565)): #0 0x00007ff8396499f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007ff837a5c8bc in __gthread_cond_wait (__mutex=, __cond=__cond@entry=0x7ff832069318) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=this@entry=0x7ff832069318, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000005ce701 in Executor::run() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/executor.cc:38 #4 0x0000000000ab4cc3 in Couchbase::Thread::thread_entry() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/thread.cc:38 #5 0x0000000000aab9e9 in run (this=0x7ff835fe3c50) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007ff839645e65 in start_thread () from /lib64/libpthread.so.0 #8 0x00007ff8371aa88d in clone () from /lib64/libc.so.6 Thread 43 (Thread 0x7ff823fff700 (LWP 4566)): #0 0x00007ff8396499f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007ff837a5c8bc in __gthread_cond_wait (__mutex=, __cond=__cond@entry=0x7ff8320694d8) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=this@entry=0x7ff8320694d8, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000005ce701 in Executor::run() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/executor.cc:38 #4 0x0000000000ab4cc3 in Couchbase::Thread::thread_entry() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/thread.cc:38 #5 0x0000000000aab9e9 in run (this=0x7ff835fe3d70) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007ff839645e65 in start_thread () from /lib64/libpthread.so.0 #8 0x00007ff8371aa88d in clone () from /lib64/libc.so.6 Thread 42 (Thread 0x7ff8237fe700 (LWP 4567)): #0 0x00007ff8396499f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007ff837a5c8bc in __gthread_cond_wait (__mutex=, __cond=__cond@entry=0x7ff832069698) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=this@entry=0x7ff832069698, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000005ce701 in Executor::run() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/executor.cc:38 #4 0x0000000000ab4cc3 in Couchbase::Thread::thread_entry() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/thread.cc:38 #5 0x0000000000aab9e9 in run (this=0x7ff835fe3bf0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007ff839645e65 in start_thread () from /lib64/libpthread.so.0 #8 0x00007ff8371aa88d in clone () from /lib64/libc.so.6 Thread 41 (Thread 0x7ff822ffd700 (LWP 4568)): #0 0x00007ff8396499f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007ff837a5c8bc in __gthread_cond_wait (__mutex=, __cond=__cond@entry=0x7ff832069858) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=this@entry=0x7ff832069858, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000005ce701 in Executor::run() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/executor.cc:38 #4 0x0000000000ab4cc3 in Couchbase::Thread::thread_entry() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/thread.cc:38 #5 0x0000000000aab9e9 in run (this=0x7ff835fe3dd0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007ff839645e65 in start_thread () from /lib64/libpthread.so.0 #8 0x00007ff8371aa88d in clone () from /lib64/libc.so.6 Thread 40 (Thread 0x7ff8227fc700 (LWP 4569)): #0 0x00007ff8396499f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007ff837a5c8bc in __gthread_cond_wait (__mutex=, __cond=__cond@entry=0x7ff832069a18) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=this@entry=0x7ff832069a18, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000005ce701 in Executor::run() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/executor.cc:38 #4 0x0000000000ab4cc3 in Couchbase::Thread::thread_entry() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/thread.cc:38 #5 0x0000000000aab9e9 in run (this=0x7ff835fe3b90) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007ff839645e65 in start_thread () from /lib64/libpthread.so.0 #8 0x00007ff8371aa88d in clone () from /lib64/libc.so.6 Thread 39 (Thread 0x7ff821ffb700 (LWP 4570)): #0 0x00007ff8396499f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007ff837a5c8bc in __gthread_cond_wait (__mutex=, __cond=__cond@entry=0x7ff832069bd8) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=this@entry=0x7ff832069bd8, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000005ce701 in Executor::run() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/executor.cc:38 #4 0x0000000000ab4cc3 in Couchbase::Thread::thread_entry() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/thread.cc:38 #5 0x0000000000aab9e9 in run (this=0x7ff835fe3bc0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007ff839645e65 in start_thread () from /lib64/libpthread.so.0 #8 0x00007ff8371aa88d in clone () from /lib64/libc.so.6 Thread 38 (Thread 0x7ff81bfff700 (LWP 4572)): #0 0x00007ff8396499f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007ff837a5c8bc in __gthread_cond_wait (__mutex=, __cond=__cond@entry=0x7ff81bfeb9f8) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=this@entry=0x7ff81bfeb9f8, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x000000000097633c in magma::WaitGroup::Wait ( this=this@entry=0x7ff81bfeb9c0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/common.cc:76 #4 0x00000000009dd0fd in magma::SSTableLoader::Load() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/lsm/table_loader.cc:58 #5 0x00000000009c2e19 in magma::TreeSnapshot::decode(std::__cxx11::basic_string, std::allocator > const&, magma::TableLoader&, std::function > ()>) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/lsm/encoding.cc:191 #6 0x00000000009c391e in magma::TreeSnapshot::Read(unsigned long, magma::TableLoader&, std::function > ()>) () at /opt/gcc-10.2.0/include/c++/10.2.0/bits/std_function.h:248 #7 0x000000000096be69 in magma::LSMTree::doRecovery(std::vector > const&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/lsm/checkpoint.cc:523 #8 0x000000000095fcf4 in magma::LSMTree::Open(std::vector > const&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/lsm/lsm_tree.cc:204 #9 0x00000000009426c5 in magma::KVStore::Open() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/kvstore/kvstore.cc:119 #10 0x0000000000907d6b in magma::Magma::Impl::createKVStore(unsigned short, unsigned int, std::shared_ptr&)::{lambda()#1}::operator()() const () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/db.cc:212 #11 0x0000000000907f3d in std::_Function_handler >(), magma::Magma::Impl::createKVStore(magma::Magma::KVStoreID, magma::Magma::KVStoreRevision, magma::KVStoreHandle&):: >::_M_invoke(const std::_Any_data &) (__functor=...) at /opt/gcc-10.2.0/include/c++/10.2.0/bits/std_function.h:289 #12 0x0000000000935b43 in operator() (this=0x7ff81bfecaf0) at /opt/gcc-10.2.0/include/c++/10.2.0/bits/std_function.h:248 #13 magma::KVStoreSet::CreateKVStore(std::function > ()>) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/kvstore_set.cc:27 #14 0x000000000090b0fb in magma::Magma::Impl::createKVStore(unsigned short, unsigned int, std::shared_ptr&) () at /opt/gcc-10.2.0/include/c++/10.2.0/bits/std_function.h:248 #15 0x0000000000970618 in magma::Magma::Impl::recovery() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/db_recovery.cc:76 #16 0x00000000008ffd3a in magma::Magma::Impl::Open() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/db.cc:129 #17 0x00000000008ffdfd in magma::Magma::Open (this=) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/db.cc:142 #18 0x00000000008545ae in MagmaKVStore::MagmaKVStore(MagmaKVStoreConfig&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/magma-kvstore/magma-kvstore.cc:540 #19 0x0000000000804993 in KVStoreFactory::create(KVStoreConfig&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/kvstore.cc:192 #20 0x00000000006c8ea4 in KVShard::KVShard(EventuallyPersistentEngine&, unsigned short) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/kvshard.cc:52 #21 0x0000000000702d10 in VBucketMap::VBucketMap(Configuration&, KVBucket&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/vbucketmap.cc:24 #22 0x00000000006b356e in KVBucket::KVBucket(EventuallyPersistentEngine&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/ep_engine.h:674 #23 0x00000000007edc86 in EPBucket::EPBucket(EventuallyPersistentEngine&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/ep_bucket.cc:240 #24 0x0000000000655ff8 in EventuallyPersistentEngine::makeBucket(Configuration&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/ep_engine.cc:6560 #25 0x000000000066ee6d in EventuallyPersistentEngine::initialize(char const*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/ep_engine.cc:2038 #26 0x00000000005380cc in CreateBucketThread::create() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/memcached.cc:894 #27 0x000000000053865f in CreateBucketThread::run() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/memcached.cc:949 #28 0x0000000000ab4cc3 in Couchbase::Thread::thread_entry() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/thread.cc:38 #29 0x0000000000aab9e9 in run (this=0x7ff835fe4b20) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #30 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #31 0x00007ff839645e65 in start_thread () from /lib64/libpthread.so.0 #32 0x00007ff8371aa88d in clone () from /lib64/libc.so.6 Thread 37 (Thread 0x7ff81b7fe700 (LWP 4573)): #0 0x00007ff839649da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007ff838d85e51 in background_thread_sleep (tsdn=, interval=10000000000, info=) at src/background_thread.c:255 #2 background_work_sleep_once (ind=, info=, tsdn=) at src/background_thread.c:307 #3 background_work (ind=, tsd=0x7ff81b7ed228) at src/background_thread.c:497 #4 background_thread_entry (ind_arg=) at src/background_thread.c:522 #5 0x00007ff839645e65 in start_thread () from /lib64/libpthread.so.0 #6 0x00007ff8371aa88d in clone () from /lib64/libc.so.6 Thread 36 (Thread 0x7ff81affd700 (LWP 4574)): #0 0x00007ff8371a4ba9 in syscall () from /lib64/libc.so.6 #1 0x0000000000aea3bd in nativeFutexWaitImpl ( waitMask=waitMask@entry=452900360, absSteadyTime=absSteadyTime@entry=0x7ff81afeb5e8, absSystemTime=absSystemTime@entry=0x0, expected=expected@entry=2, addr=addr@entry=0x7ff8027db018) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:123 #2 folly::detail::futexWaitImpl (futex=futex@entry=0x7ff8027db018, expected=expected@entry=2, absSystemTime=absSystemTime@entry=0x0, absSteadyTime=absSteadyTime@entry=0x7ff81afeb608, waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:254 #3 0x0000000000af3f5a in futexWaitImpl const, std::chrono::time_point > > > (waitMask=4294967295, deadline=..., expected=2, futex=0x7ff8027db018) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:85 #4 folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > > ( futex=futex@entry=0x7ff8027db018, expected=expected@entry=2, deadline=..., waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:123 #5 0x0000000000af9ef7 in futexWaitPreIdle, std::chrono::time_point > >, std::chrono::duration > > (timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, deadline=..., expected=2, fut=..., _ret=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:194 #6 futexWaitUntil, std::chrono::time_point > > > (timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, deadline=..., expected=2, fut=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:151 #7 folly::SaturatingSemaphore::tryWaitSlow > > ( this=this@entry=0x7ff8027db018, deadline=..., opt=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:318 #8 0x0000000000afa426 in try_wait_until > > (opt=..., deadline=..., this=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:179 #9 folly::detail::LifoSemBase, std::atomic>::try_wait_until > > (this=this@entry=0x7ff810000a40, deadline=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:521 #10 0x0000000000afdb80 in try_wait_for > ( timeout=, this=0x7ff810000a40) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:498 #11 folly::UnboundedBlockingQueue::try_take_for (this=0x7ff810000a00, time=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/task_queue/UnboundedBlockingQueue.h:41 #12 0x0000000000af20f7 in folly::CPUThreadPoolExecutor::threadRun ( this=0x7ff831f1fc00, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:243 #13 0x0000000000b0d369 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> ( __t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #14 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #15 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #16 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #17 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #18 0x0000000000699d4d in folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/folly_executorpool.cc:48 #19 0x00007ff837a61d40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #20 0x00007ff839645e65 in start_thread () from /lib64/libpthread.so.0 #21 0x00007ff8371aa88d in clone () from /lib64/libc.so.6 Thread 35 (Thread 0x7ff81a7fc700 (LWP 4575)): #0 0x00007ff8371a4ba9 in syscall () from /lib64/libc.so.6 #1 0x0000000000aea3bd in nativeFutexWaitImpl ( waitMask=waitMask@entry=444507656, absSteadyTime=absSteadyTime@entry=0x7ff81a7ea5e8, absSystemTime=absSystemTime@entry=0x0, expected=expected@entry=2, addr=addr@entry=0x7ff8027db030) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:123 #2 folly::detail::futexWaitImpl (futex=futex@entry=0x7ff8027db030, expected=expected@entry=2, absSystemTime=absSystemTime@entry=0x0, absSteadyTime=absSteadyTime@entry=0x7ff81a7ea608, waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:254 #3 0x0000000000af3f5a in futexWaitImpl const, std::chrono::time_point > > > (waitMask=4294967295, deadline=..., expected=2, futex=0x7ff8027db030) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:85 #4 folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > > ( futex=futex@entry=0x7ff8027db030, expected=expected@entry=2, deadline=..., waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:123 #5 0x0000000000af9ef7 in futexWaitPreIdle, std::chrono::time_point > >, std::chrono::duration > > (timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, deadline=..., expected=2, fut=..., _ret=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:194 #6 futexWaitUntil, std::chrono::time_point > > > (timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, deadline=..., expected=2, fut=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:151 #7 folly::SaturatingSemaphore::tryWaitSlow > > ( this=this@entry=0x7ff8027db030, deadline=..., opt=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:318 #8 0x0000000000afa426 in try_wait_until > > (opt=..., deadline=..., this=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:179 #9 folly::detail::LifoSemBase, std::atomic>::try_wait_until > > (this=this@entry=0x7ff810000a40, deadline=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:521 #10 0x0000000000afdb80 in try_wait_for > ( timeout=, this=0x7ff810000a40) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:498 #11 folly::UnboundedBlockingQueue::try_take_for (this=0x7ff810000a00, time=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/task_queue/UnboundedBlockingQueue.h:41 #12 0x0000000000af20f7 in folly::CPUThreadPoolExecutor::threadRun ( this=0x7ff831f1fc00, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:243 #13 0x0000000000b0d369 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> ( __t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #14 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #15 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #16 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #17 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #18 0x0000000000699d4d in folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/folly_executorpool.cc:48 #19 0x00007ff837a61d40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #20 0x00007ff839645e65 in start_thread () from /lib64/libpthread.so.0 #21 0x00007ff8371aa88d in clone () from /lib64/libc.so.6 Thread 34 (Thread 0x7ff819ffb700 (LWP 4576)): #0 0x00007ff8371a4ba9 in syscall () from /lib64/libc.so.6 #1 0x0000000000aea3bd in nativeFutexWaitImpl ( waitMask=waitMask@entry=436114952, absSteadyTime=absSteadyTime@entry=0x7ff819fe95e8, absSystemTime=absSystemTime@entry=0x0, expected=expected@entry=2, addr=addr@entry=0x7ff8027db048) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:123 #2 folly::detail::futexWaitImpl (futex=futex@entry=0x7ff8027db048, expected=expected@entry=2, absSystemTime=absSystemTime@entry=0x0, absSteadyTime=absSteadyTime@entry=0x7ff819fe9608, waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:254 #3 0x0000000000af3f5a in futexWaitImpl const, std::chrono::time_point > > > (waitMask=4294967295, deadline=..., expected=2, futex=0x7ff8027db048) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:85 #4 folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > > ( futex=futex@entry=0x7ff8027db048, expected=expected@entry=2, deadline=..., waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:123 #5 0x0000000000af9ef7 in futexWaitPreIdle, std::chrono::time_point > >, std::chrono::duration > > (timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, deadline=..., expected=2, fut=..., _ret=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:194 #6 futexWaitUntil, std::chrono::time_point > > > (timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, deadline=..., expected=2, fut=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:151 #7 folly::SaturatingSemaphore::tryWaitSlow > > ( this=this@entry=0x7ff8027db048, deadline=..., opt=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:318 #8 0x0000000000afa426 in try_wait_until > > (opt=..., deadline=..., this=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:179 #9 folly::detail::LifoSemBase, std::atomic>::try_wait_until > > (this=this@entry=0x7ff810000a40, deadline=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:521 #10 0x0000000000afdb80 in try_wait_for > ( timeout=, this=0x7ff810000a40) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:498 #11 folly::UnboundedBlockingQueue::try_take_for (this=0x7ff810000a00, time=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/task_queue/UnboundedBlockingQueue.h:41 #12 0x0000000000af20f7 in folly::CPUThreadPoolExecutor::threadRun ( this=0x7ff831f1fc00, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:243 #13 0x0000000000b0d369 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> ( __t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #14 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #15 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #16 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #17 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #18 0x0000000000699d4d in folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/folly_executorpool.cc:48 #19 0x00007ff837a61d40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #20 0x00007ff839645e65 in start_thread () from /lib64/libpthread.so.0 #21 0x00007ff8371aa88d in clone () from /lib64/libc.so.6 Thread 33 (Thread 0x7ff8197fa700 (LWP 4577)): #0 0x00007ff8371a4ba9 in syscall () from /lib64/libc.so.6 #1 0x0000000000aea3bd in nativeFutexWaitImpl ( waitMask=waitMask@entry=427722248, absSteadyTime=absSteadyTime@entry=0x7ff8197e85e8, absSystemTime=absSystemTime@entry=0x0, expected=expected@entry=2, addr=addr@entry=0x7ff8027db060) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:123 #2 folly::detail::futexWaitImpl (futex=futex@entry=0x7ff8027db060, expected=expected@entry=2, absSystemTime=absSystemTime@entry=0x0, absSteadyTime=absSteadyTime@entry=0x7ff8197e8608, waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:254 #3 0x0000000000af3f5a in futexWaitImpl const, std::chrono::time_point > > > (waitMask=4294967295, deadline=..., expected=2, futex=0x7ff8027db060) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:85 #4 folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > > ( futex=futex@entry=0x7ff8027db060, expected=expected@entry=2, deadline=..., waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:123 #5 0x0000000000af9ef7 in futexWaitPreIdle, std::chrono::time_point > >, std::chrono::duration > > (timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, deadline=..., expected=2, fut=..., _ret=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:194 #6 futexWaitUntil, std::chrono::time_point > > > (timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, deadline=..., expected=2, fut=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:151 #7 folly::SaturatingSemaphore::tryWaitSlow > > ( this=this@entry=0x7ff8027db060, deadline=..., opt=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:318 #8 0x0000000000afa426 in try_wait_until > > (opt=..., deadline=..., this=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:179 #9 folly::detail::LifoSemBase, std::atomic>::try_wait_until > > (this=this@entry=0x7ff810000a40, deadline=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:521 #10 0x0000000000afdb80 in try_wait_for > ( timeout=, this=0x7ff810000a40) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:498 #11 folly::UnboundedBlockingQueue::try_take_for (this=0x7ff810000a00, time=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/task_queue/UnboundedBlockingQueue.h:41 #12 0x0000000000af20f7 in folly::CPUThreadPoolExecutor::threadRun ( this=0x7ff831f1fc00, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:243 #13 0x0000000000b0d369 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> ( __t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #14 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #15 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #16 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #17 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #18 0x0000000000699d4d in folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/folly_executorpool.cc:48 #19 0x00007ff837a61d40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #20 0x00007ff839645e65 in start_thread () from /lib64/libpthread.so.0 #21 0x00007ff8371aa88d in clone () from /lib64/libc.so.6 Thread 32 (Thread 0x7ff818ff9700 (LWP 4578)): #0 0x00007ff8371a4ba9 in syscall () from /lib64/libc.so.6 #1 0x0000000000aea3bd in nativeFutexWaitImpl ( waitMask=waitMask@entry=419329544, absSteadyTime=absSteadyTime@entry=0x7ff818fe75e8, absSystemTime=absSystemTime@entry=0x0, expected=expected@entry=2, addr=addr@entry=0x7ff8027db078) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:123 #2 folly::detail::futexWaitImpl (futex=futex@entry=0x7ff8027db078, expected=expected@entry=2, absSystemTime=absSystemTime@entry=0x0, absSteadyTime=absSteadyTime@entry=0x7ff818fe7608, waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:254 #3 0x0000000000af3f5a in futexWaitImpl const, std::chrono::time_point > > > (waitMask=4294967295, deadline=..., expected=2, futex=0x7ff8027db078) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:85 #4 folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > > ( futex=futex@entry=0x7ff8027db078, expected=expected@entry=2, deadline=..., waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:123 #5 0x0000000000af9ef7 in futexWaitPreIdle, std::chrono::time_point > >, std::chrono::duration > > (timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, deadline=..., expected=2, fut=..., _ret=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:194 #6 futexWaitUntil, std::chrono::time_point > > > (timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, deadline=..., expected=2, fut=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:151 #7 folly::SaturatingSemaphore::tryWaitSlow > > ( this=this@entry=0x7ff8027db078, deadline=..., opt=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:318 #8 0x0000000000afa426 in try_wait_until > > (opt=..., deadline=..., this=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:179 #9 folly::detail::LifoSemBase, std::atomic>::try_wait_until > > (this=this@entry=0x7ff810000a40, deadline=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:521 #10 0x0000000000afdb80 in try_wait_for > ( timeout=, this=0x7ff810000a40) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:498 #11 folly::UnboundedBlockingQueue::try_take_for (this=0x7ff810000a00, time=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/task_queue/UnboundedBlockingQueue.h:41 #12 0x0000000000af20f7 in folly::CPUThreadPoolExecutor::threadRun ( this=0x7ff831f1fc00, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:243 #13 0x0000000000b0d369 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> ( __t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #14 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #15 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #16 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #17 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #18 0x0000000000699d4d in folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/folly_executorpool.cc:48 #19 0x00007ff837a61d40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #20 0x00007ff839645e65 in start_thread () from /lib64/libpthread.so.0 #21 0x00007ff8371aa88d in clone () from /lib64/libc.so.6 Thread 31 (Thread 0x7ff8027da700 (LWP 4579)): #0 0x00007ff8371a4ba9 in syscall () from /lib64/libc.so.6 #1 0x0000000000aea3bd in nativeFutexWaitImpl ( waitMask=waitMask@entry=41715208, absSteadyTime=absSteadyTime@entry=0x7ff8027c85e8, absSystemTime=absSystemTime@entry=0x0, expected=expected@entry=2, addr=addr@entry=0x7ff8027db090) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:123 #2 folly::detail::futexWaitImpl (futex=futex@entry=0x7ff8027db090, expected=expected@entry=2, absSystemTime=absSystemTime@entry=0x0, absSteadyTime=absSteadyTime@entry=0x7ff8027c8608, waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:254 #3 0x0000000000af3f5a in futexWaitImpl const, std::chrono::time_point > > > (waitMask=4294967295, deadline=..., expected=2, futex=0x7ff8027db090) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:85 #4 folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > > ( futex=futex@entry=0x7ff8027db090, expected=expected@entry=2, deadline=..., waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:123 #5 0x0000000000af9ef7 in futexWaitPreIdle, std::chrono::time_point > >, std::chrono::duration > > (timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, deadline=..., expected=2, fut=..., _ret=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:194 #6 futexWaitUntil, std::chrono::time_point > > > (timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, deadline=..., expected=2, fut=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:151 #7 folly::SaturatingSemaphore::tryWaitSlow > > ( this=this@entry=0x7ff8027db090, deadline=..., opt=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:318 #8 0x0000000000afa426 in try_wait_until > > (opt=..., deadline=..., this=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:179 #9 folly::detail::LifoSemBase, std::atomic>::try_wait_until > > (this=this@entry=0x7ff810000a40, deadline=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:521 #10 0x0000000000afdb80 in try_wait_for > ( timeout=, this=0x7ff810000a40) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:498 #11 folly::UnboundedBlockingQueue::try_take_for (this=0x7ff810000a00, time=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/task_queue/UnboundedBlockingQueue.h:41 #12 0x0000000000af20f7 in folly::CPUThreadPoolExecutor::threadRun ( this=0x7ff831f1fc00, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:243 #13 0x0000000000b0d369 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> ( __t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #14 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #15 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #16 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #17 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #18 0x0000000000699d4d in folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/folly_executorpool.cc:48 #19 0x00007ff837a61d40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #20 0x00007ff839645e65 in start_thread () from /lib64/libpthread.so.0 #21 0x00007ff8371aa88d in clone () from /lib64/libc.so.6 Thread 30 (Thread 0x7ff801fd9700 (LWP 4580)): #0 0x00007ff8371a4ba9 in syscall () from /lib64/libc.so.6 #1 0x0000000000aea3bd in nativeFutexWaitImpl ( waitMask=waitMask@entry=33322504, absSteadyTime=absSteadyTime@entry=0x7ff801fc75e8, absSystemTime=absSystemTime@entry=0x0, expected=expected@entry=2, addr=addr@entry=0x7ff8027db0a8) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:123 #2 folly::detail::futexWaitImpl (futex=futex@entry=0x7ff8027db0a8, expected=expected@entry=2, absSystemTime=absSystemTime@entry=0x0, absSteadyTime=absSteadyTime@entry=0x7ff801fc7608, waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:254 #3 0x0000000000af3f5a in futexWaitImpl const, std::chrono::time_point > > > (waitMask=4294967295, deadline=..., expected=2, futex=0x7ff8027db0a8) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:85 #4 folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > > ( futex=futex@entry=0x7ff8027db0a8, expected=expected@entry=2, deadline=..., waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:123 #5 0x0000000000af9ef7 in futexWaitPreIdle, std::chrono::time_point > >, std::chrono::duration > > (timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, deadline=..., expected=2, fut=..., _ret=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:194 #6 futexWaitUntil, std::chrono::time_point > > > (timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, deadline=..., expected=2, fut=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:151 #7 folly::SaturatingSemaphore::tryWaitSlow > > ( this=this@entry=0x7ff8027db0a8, deadline=..., opt=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:318 #8 0x0000000000afa426 in try_wait_until > > (opt=..., deadline=..., this=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:179 #9 folly::detail::LifoSemBase, std::atomic>::try_wait_until > > (this=this@entry=0x7ff810000a40, deadline=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:521 #10 0x0000000000afdb80 in try_wait_for > ( timeout=, this=0x7ff810000a40) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:498 #11 folly::UnboundedBlockingQueue::try_take_for (this=0x7ff810000a00, time=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/task_queue/UnboundedBlockingQueue.h:41 #12 0x0000000000af20f7 in folly::CPUThreadPoolExecutor::threadRun ( this=0x7ff831f1fc00, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:243 #13 0x0000000000b0d369 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> ( __t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #14 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #15 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #16 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #17 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #18 0x0000000000699d4d in folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/folly_executorpool.cc:48 #19 0x00007ff837a61d40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #20 0x00007ff839645e65 in start_thread () from /lib64/libpthread.so.0 #21 0x00007ff8371aa88d in clone () from /lib64/libc.so.6 Thread 29 (Thread 0x7ff8017d8700 (LWP 4581)): #0 0x00007ff8371a4ba9 in syscall () from /lib64/libc.so.6 #1 0x0000000000aea3bd in nativeFutexWaitImpl ( waitMask=waitMask@entry=24929800, absSteadyTime=absSteadyTime@entry=0x7ff8017c65e8, absSystemTime=absSystemTime@entry=0x0, expected=expected@entry=2, addr=addr@entry=0x7ff8027db0c0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:123 #2 folly::detail::futexWaitImpl (futex=futex@entry=0x7ff8027db0c0, expected=expected@entry=2, absSystemTime=absSystemTime@entry=0x0, absSteadyTime=absSteadyTime@entry=0x7ff8017c6608, waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:254 #3 0x0000000000af3f5a in futexWaitImpl const, std::chrono::time_point > > > (waitMask=4294967295, deadline=..., expected=2, futex=0x7ff8027db0c0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:85 #4 folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > > ( futex=futex@entry=0x7ff8027db0c0, expected=expected@entry=2, deadline=..., waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:123 #5 0x0000000000af9ef7 in futexWaitPreIdle, std::chrono::time_point > >, std::chrono::duration > > (timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, deadline=..., expected=2, fut=..., _ret=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:194 #6 futexWaitUntil, std::chrono::time_point > > > (timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, deadline=..., expected=2, fut=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:151 #7 folly::SaturatingSemaphore::tryWaitSlow > > ( this=this@entry=0x7ff8027db0c0, deadline=..., opt=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:318 #8 0x0000000000afa426 in try_wait_until > > (opt=..., deadline=..., this=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:179 #9 folly::detail::LifoSemBase, std::atomic>::try_wait_until > > (this=this@entry=0x7ff810000a40, deadline=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:521 #10 0x0000000000afdb80 in try_wait_for > ( timeout=, this=0x7ff810000a40) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:498 #11 folly::UnboundedBlockingQueue::try_take_for (this=0x7ff810000a00, time=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/task_queue/UnboundedBlockingQueue.h:41 #12 0x0000000000af20f7 in folly::CPUThreadPoolExecutor::threadRun ( this=0x7ff831f1fc00, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:243 #13 0x0000000000b0d369 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> ( __t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #14 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #15 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #16 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #17 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #18 0x0000000000699d4d in folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/folly_executorpool.cc:48 #19 0x00007ff837a61d40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #20 0x00007ff839645e65 in start_thread () from /lib64/libpthread.so.0 #21 0x00007ff8371aa88d in clone () from /lib64/libc.so.6 Thread 28 (Thread 0x7ff800fd7700 (LWP 4582)): #0 0x00007ff8371a4ba9 in syscall () from /lib64/libc.so.6 #1 0x0000000000aea3bd in nativeFutexWaitImpl ( waitMask=waitMask@entry=16537096, absSteadyTime=absSteadyTime@entry=0x7ff800fc55e8, absSystemTime=absSystemTime@entry=0x0, expected=expected@entry=2, addr=addr@entry=0x7ff8027db0d8) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:123 #2 folly::detail::futexWaitImpl (futex=futex@entry=0x7ff8027db0d8, expected=expected@entry=2, absSystemTime=absSystemTime@entry=0x0, absSteadyTime=absSteadyTime@entry=0x7ff800fc5608, waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:254 #3 0x0000000000af3f5a in futexWaitImpl const, std::chrono::time_point > > > (waitMask=4294967295, deadline=..., expected=2, futex=0x7ff8027db0d8) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:85 #4 folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > > ( futex=futex@entry=0x7ff8027db0d8, expected=expected@entry=2, deadline=..., waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:123 #5 0x0000000000af9ef7 in futexWaitPreIdle, std::chrono::time_point > >, std::chrono::duration > > (timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, deadline=..., expected=2, fut=..., _ret=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:194 #6 futexWaitUntil, std::chrono::time_point > > > (timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, deadline=..., expected=2, fut=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:151 #7 folly::SaturatingSemaphore::tryWaitSlow > > ( this=this@entry=0x7ff8027db0d8, deadline=..., opt=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:318 #8 0x0000000000afa426 in try_wait_until > > (opt=..., deadline=..., this=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:179 #9 folly::detail::LifoSemBase, std::atomic>::try_wait_until > > (this=this@entry=0x7ff810002140, deadline=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:521 #10 0x0000000000afdb80 in try_wait_for > ( timeout=, this=0x7ff810002140) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:498 #11 folly::UnboundedBlockingQueue::try_take_for (this=0x7ff810002100, time=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/task_queue/UnboundedBlockingQueue.h:41 #12 0x0000000000af20f7 in folly::CPUThreadPoolExecutor::threadRun ( this=0x7ff831f20800, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:243 #13 0x0000000000b0d369 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> ( __t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #14 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #15 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #16 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #17 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #18 0x0000000000699d4d in folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/folly_executorpool.cc:48 #19 0x00007ff837a61d40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #20 0x00007ff839645e65 in start_thread () from /lib64/libpthread.so.0 #21 0x00007ff8371aa88d in clone () from /lib64/libc.so.6 Thread 27 (Thread 0x7ff7ebfff700 (LWP 4583)): #0 0x00007ff8371a4ba9 in syscall () from /lib64/libc.so.6 #1 0x0000000000aea3bd in nativeFutexWaitImpl ( waitMask=waitMask@entry=3959346696, absSteadyTime=absSteadyTime@entry=0x7ff7ebfed5e8, absSystemTime=absSystemTime@entry=0x0, expected=expected@entry=2, addr=addr@entry=0x7ff8027db0f0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:123 #2 folly::detail::futexWaitImpl (futex=futex@entry=0x7ff8027db0f0, expected=expected@entry=2, absSystemTime=absSystemTime@entry=0x0, absSteadyTime=absSteadyTime@entry=0x7ff7ebfed608, waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:254 #3 0x0000000000af3f5a in futexWaitImpl const, std::chrono::time_point > > > (waitMask=4294967295, deadline=..., expected=2, futex=0x7ff8027db0f0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:85 #4 folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > > ( futex=futex@entry=0x7ff8027db0f0, expected=expected@entry=2, deadline=..., waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:123 #5 0x0000000000af9ef7 in futexWaitPreIdle, std::chrono::time_point > >, std::chrono::duration > > (timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, deadline=..., expected=2, fut=..., _ret=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:194 #6 futexWaitUntil, std::chrono::time_point > > > (timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, deadline=..., expected=2, fut=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:151 #7 folly::SaturatingSemaphore::tryWaitSlow > > ( this=this@entry=0x7ff8027db0f0, deadline=..., opt=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:318 #8 0x0000000000afa426 in try_wait_until > > (opt=..., deadline=..., this=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:179 #9 folly::detail::LifoSemBase, std::atomic>::try_wait_until > > (this=this@entry=0x7ff810002140, deadline=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:521 #10 0x0000000000afdb80 in try_wait_for > ( timeout=, this=0x7ff810002140) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:498 #11 folly::UnboundedBlockingQueue::try_take_for (this=0x7ff810002100, time=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/task_queue/UnboundedBlockingQueue.h:41 #12 0x0000000000af20f7 in folly::CPUThreadPoolExecutor::threadRun ( this=0x7ff831f20800, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:243 #13 0x0000000000b0d369 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> ( __t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #14 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #15 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #16 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #17 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #18 0x0000000000699d4d in folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/folly_executorpool.cc:48 #19 0x00007ff837a61d40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #20 0x00007ff839645e65 in start_thread () from /lib64/libpthread.so.0 #21 0x00007ff8371aa88d in clone () from /lib64/libc.so.6 Thread 26 (Thread 0x7ff7eb7fe700 (LWP 4584)): #0 0x00007ff8371a4ba9 in syscall () from /lib64/libc.so.6 #1 0x0000000000aea3bd in nativeFutexWaitImpl ( waitMask=waitMask@entry=3950953992, absSteadyTime=absSteadyTime@entry=0x7ff7eb7ec5e8, absSystemTime=absSystemTime@entry=0x0, expected=expected@entry=2, addr=addr@entry=0x7ff8027db108) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:123 #2 folly::detail::futexWaitImpl (futex=futex@entry=0x7ff8027db108, expected=expected@entry=2, absSystemTime=absSystemTime@entry=0x0, absSteadyTime=absSteadyTime@entry=0x7ff7eb7ec608, waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:254 #3 0x0000000000af3f5a in futexWaitImpl const, std::chrono::time_point > > > (waitMask=4294967295, deadline=..., expected=2, futex=0x7ff8027db108) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:85 #4 folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > > ( futex=futex@entry=0x7ff8027db108, expected=expected@entry=2, deadline=..., waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:123 #5 0x0000000000af9ef7 in futexWaitPreIdle, std::chrono::time_point > >, std::chrono::duration > > (timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, deadline=..., expected=2, fut=..., _ret=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:194 #6 futexWaitUntil, std::chrono::time_point > > > (timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, deadline=..., expected=2, fut=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:151 #7 folly::SaturatingSemaphore::tryWaitSlow > > ( this=this@entry=0x7ff8027db108, deadline=..., opt=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:318 #8 0x0000000000afa426 in try_wait_until > > (opt=..., deadline=..., this=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:179 #9 folly::detail::LifoSemBase, std::atomic>::try_wait_until > > (this=this@entry=0x7ff810002140, deadline=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:521 #10 0x0000000000afdb80 in try_wait_for > ( timeout=, this=0x7ff810002140) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:498 #11 folly::UnboundedBlockingQueue::try_take_for (this=0x7ff810002100, time=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/task_queue/UnboundedBlockingQueue.h:41 #12 0x0000000000af20f7 in folly::CPUThreadPoolExecutor::threadRun ( this=0x7ff831f20800, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:243 #13 0x0000000000b0d369 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> ( __t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #14 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #15 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #16 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #17 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #18 0x0000000000699d4d in folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/folly_executorpool.cc:48 #19 0x00007ff837a61d40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #20 0x00007ff839645e65 in start_thread () from /lib64/libpthread.so.0 #21 0x00007ff8371aa88d in clone () from /lib64/libc.so.6 Thread 25 (Thread 0x7ff7eaffd700 (LWP 4585)): #0 0x00007ff8371a4ba9 in syscall () from /lib64/libc.so.6 #1 0x0000000000aea3bd in nativeFutexWaitImpl ( waitMask=waitMask@entry=3942561288, absSteadyTime=absSteadyTime@entry=0x7ff7eafeb5e8, absSystemTime=absSystemTime@entry=0x0, expected=expected@entry=2, addr=addr@entry=0x7ff8027db120) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:123 #2 folly::detail::futexWaitImpl (futex=futex@entry=0x7ff8027db120, expected=expected@entry=2, absSystemTime=absSystemTime@entry=0x0, absSteadyTime=absSteadyTime@entry=0x7ff7eafeb608, waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:254 #3 0x0000000000af3f5a in futexWaitImpl const, std::chrono::time_point > > > (waitMask=4294967295, deadline=..., expected=2, futex=0x7ff8027db120) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:85 #4 folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > > ( futex=futex@entry=0x7ff8027db120, expected=expected@entry=2, deadline=..., waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:123 #5 0x0000000000af9ef7 in futexWaitPreIdle, std::chrono::time_point > >, std::chrono::duration > > (timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, deadline=..., expected=2, fut=..., _ret=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:194 #6 futexWaitUntil, std::chrono::time_point > > > (timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, deadline=..., expected=2, fut=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:151 #7 folly::SaturatingSemaphore::tryWaitSlow > > ( this=this@entry=0x7ff8027db120, deadline=..., opt=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:318 #8 0x0000000000afa426 in try_wait_until > > (opt=..., deadline=..., this=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:179 #9 folly::detail::LifoSemBase, std::atomic>::try_wait_until > > (this=this@entry=0x7ff810002140, deadline=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:521 #10 0x0000000000afdb80 in try_wait_for > ( timeout=, this=0x7ff810002140) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:498 #11 folly::UnboundedBlockingQueue::try_take_for (this=0x7ff810002100, time=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/task_queue/UnboundedBlockingQueue.h:41 #12 0x0000000000af20f7 in folly::CPUThreadPoolExecutor::threadRun ( this=0x7ff831f20800, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:243 #13 0x0000000000b0d369 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> ( __t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #14 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #15 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #16 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #17 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #18 0x0000000000699d4d in folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/folly_executorpool.cc:48 #19 0x00007ff837a61d40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #20 0x00007ff839645e65 in start_thread () from /lib64/libpthread.so.0 #21 0x00007ff8371aa88d in clone () from /lib64/libc.so.6 Thread 24 (Thread 0x7ff7ea7fc700 (LWP 4586)): #0 0x00007ff8371a4ba9 in syscall () from /lib64/libc.so.6 #1 0x0000000000aea3bd in nativeFutexWaitImpl ( waitMask=waitMask@entry=3934168584, absSteadyTime=absSteadyTime@entry=0x7ff7ea7ea5e8, absSystemTime=absSystemTime@entry=0x0, expected=expected@entry=2, addr=addr@entry=0x7ff8027db138) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:123 #2 folly::detail::futexWaitImpl (futex=futex@entry=0x7ff8027db138, expected=expected@entry=2, absSystemTime=absSystemTime@entry=0x0, absSteadyTime=absSteadyTime@entry=0x7ff7ea7ea608, waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:254 #3 0x0000000000af3f5a in futexWaitImpl const, std::chrono::time_point > > > (waitMask=4294967295, deadline=..., expected=2, futex=0x7ff8027db138) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:85 #4 folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > > ( futex=futex@entry=0x7ff8027db138, expected=expected@entry=2, deadline=..., waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:123 #5 0x0000000000af9ef7 in futexWaitPreIdle, std::chrono::time_point > >, std::chrono::duration > > (timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, deadline=..., expected=2, fut=..., _ret=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:194 #6 futexWaitUntil, std::chrono::time_point > > > (timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, deadline=..., expected=2, fut=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:151 #7 folly::SaturatingSemaphore::tryWaitSlow > > ( this=this@entry=0x7ff8027db138, deadline=..., opt=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:318 #8 0x0000000000afa426 in try_wait_until > > (opt=..., deadline=..., this=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:179 #9 folly::detail::LifoSemBase, std::atomic>::try_wait_until > > (this=this@entry=0x7ff810002e40, deadline=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:521 #10 0x0000000000afdb80 in try_wait_for > ( timeout=, this=0x7ff810002e40) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:498 #11 folly::UnboundedBlockingQueue::try_take_for (this=0x7ff810002e00, time=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/task_queue/UnboundedBlockingQueue.h:41 #12 0x0000000000af20f7 in folly::CPUThreadPoolExecutor::threadRun ( this=0x7ff831f21400, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:243 #13 0x0000000000b0d369 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> ( __t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #14 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #15 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #16 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #17 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #18 0x0000000000699d4d in folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/folly_executorpool.cc:48 #19 0x00007ff837a61d40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #20 0x00007ff839645e65 in start_thread () from /lib64/libpthread.so.0 #21 0x00007ff8371aa88d in clone () from /lib64/libc.so.6 Thread 23 (Thread 0x7ff7e9ffb700 (LWP 4587)): #0 0x00007ff8371a4ba9 in syscall () from /lib64/libc.so.6 #1 0x0000000000aea3bd in nativeFutexWaitImpl ( waitMask=waitMask@entry=3925775880, absSteadyTime=absSteadyTime@entry=0x7ff7e9fe95e8, absSystemTime=absSystemTime@entry=0x0, expected=expected@entry=2, addr=addr@entry=0x7ff8027db150) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:123 #2 folly::detail::futexWaitImpl (futex=futex@entry=0x7ff8027db150, expected=expected@entry=2, absSystemTime=absSystemTime@entry=0x0, absSteadyTime=absSteadyTime@entry=0x7ff7e9fe9608, waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:254 #3 0x0000000000af3f5a in futexWaitImpl const, std::chrono::time_point > > > (waitMask=4294967295, deadline=..., expected=2, futex=0x7ff8027db150) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:85 #4 folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > > ( futex=futex@entry=0x7ff8027db150, expected=expected@entry=2, deadline=..., waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:123 #5 0x0000000000af9ef7 in futexWaitPreIdle, std::chrono::time_point > >, std::chrono::duration > > (timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, deadline=..., expected=2, fut=..., _ret=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:194 #6 futexWaitUntil, std::chrono::time_point > > > (timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, deadline=..., expected=2, fut=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:151 #7 folly::SaturatingSemaphore::tryWaitSlow > > ( this=this@entry=0x7ff8027db150, deadline=..., opt=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:318 #8 0x0000000000afa426 in try_wait_until > > (opt=..., deadline=..., this=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:179 #9 folly::detail::LifoSemBase, std::atomic>::try_wait_until > > (this=this@entry=0x7ff810003340, deadline=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:521 #10 0x0000000000afdb80 in try_wait_for > ( timeout=, this=0x7ff810003340) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:498 #11 folly::UnboundedBlockingQueue::try_take_for (this=0x7ff810003300, time=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/task_queue/UnboundedBlockingQueue.h:41 #12 0x0000000000af20f7 in folly::CPUThreadPoolExecutor::threadRun ( this=0x7ff831f51000, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:243 #13 0x0000000000b0d369 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> ( __t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #14 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #15 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #16 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #17 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #18 0x0000000000699d4d in folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/folly_executorpool.cc:48 #19 0x00007ff837a61d40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #20 0x00007ff839645e65 in start_thread () from /lib64/libpthread.so.0 #21 0x00007ff8371aa88d in clone () from /lib64/libc.so.6 Thread 22 (Thread 0x7ff7e97fa700 (LWP 4588)): #0 0x00007ff8371a4ba9 in syscall () from /lib64/libc.so.6 #1 0x0000000000aea3bd in nativeFutexWaitImpl ( waitMask=waitMask@entry=3917383176, absSteadyTime=absSteadyTime@entry=0x7ff7e97e85e8, absSystemTime=absSystemTime@entry=0x0, expected=expected@entry=2, addr=addr@entry=0x7ff8027db168) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:123 #2 folly::detail::futexWaitImpl (futex=futex@entry=0x7ff8027db168, expected=expected@entry=2, absSystemTime=absSystemTime@entry=0x0, absSteadyTime=absSteadyTime@entry=0x7ff7e97e8608, waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:254 #3 0x0000000000af3f5a in futexWaitImpl const, std::chrono::time_point > > > (waitMask=4294967295, deadline=..., expected=2, futex=0x7ff8027db168) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:85 #4 folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > > ( futex=futex@entry=0x7ff8027db168, expected=expected@entry=2, deadline=..., waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:123 #5 0x0000000000af9ef7 in futexWaitPreIdle, std::chrono::time_point > >, std::chrono::duration > > (timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, deadline=..., expected=2, fut=..., _ret=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:194 #6 futexWaitUntil, std::chrono::time_point > > > (timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, deadline=..., expected=2, fut=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:151 #7 folly::SaturatingSemaphore::tryWaitSlow > > ( this=this@entry=0x7ff8027db168, deadline=..., opt=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:318 #8 0x0000000000afa426 in try_wait_until > > (opt=..., deadline=..., this=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:179 #9 folly::detail::LifoSemBase, std::atomic>::try_wait_until > > (this=this@entry=0x7ff810003340, deadline=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:521 #10 0x0000000000afdb80 in try_wait_for > ( timeout=, this=0x7ff810003340) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:498 #11 folly::UnboundedBlockingQueue::try_take_for (this=0x7ff810003300, time=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/task_queue/UnboundedBlockingQueue.h:41 #12 0x0000000000af20f7 in folly::CPUThreadPoolExecutor::threadRun ( this=0x7ff831f51000, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:243 #13 0x0000000000b0d369 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> ( __t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #14 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #15 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #16 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #17 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #18 0x0000000000699d4d in folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/folly_executorpool.cc:48 #19 0x00007ff837a61d40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #20 0x00007ff839645e65 in start_thread () from /lib64/libpthread.so.0 #21 0x00007ff8371aa88d in clone () from /lib64/libc.so.6 Thread 21 (Thread 0x7ff7e8ff9700 (LWP 4589)): #0 0x00007ff839649da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000000000997bcf in __gthread_cond_timedwait ( __abs_timeout=0x7ff7e8fe7960, __mutex=, __cond=0x7ff835d61958) at /opt/gcc-10.2.0/include/c++/10.2.0/x86_64-pc-linux-gnu/bits/gthr-default.h:872 #2 __wait_until_impl > > (__lock=..., __lock=..., __atime=, this=0x7ff835d61958) at /opt/gcc-10.2.0/include/c++/10.2.0/condition_variable:232 #3 wait_until > > (__atime=..., __lock=..., this=0x7ff835d61958) at /opt/gcc-10.2.0/include/c++/10.2.0/condition_variable:141 #4 wait_for > (__rtime=..., __rtime=..., __lock=..., this=0x7ff835d61958) at /opt/gcc-10.2.0/include/c++/10.2.0/condition_variable:172 #5 magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:98 #6 0x0000000000997d21 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:159 #7 0x0000000000aab9e9 in run (this=0x7ff835d3a6a0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #8 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #9 0x00007ff839645e65 in start_thread () from /lib64/libpthread.so.0 #10 0x00007ff8371aa88d in clone () from /lib64/libc.so.6 Thread 20 (Thread 0x7ff7c7fff700 (LWP 4590)): #0 0x00007ff839649da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000000000997bcf in __gthread_cond_timedwait ( __abs_timeout=0x7ff7c7fed960, __mutex=, __cond=0x7ff835d61958) at /opt/gcc-10.2.0/include/c++/10.2.0/x86_64-pc-linux-gnu/bits/gthr-default.h:872 #2 __wait_until_impl > > (__lock=..., __lock=..., __atime=, this=0x7ff835d61958) at /opt/gcc-10.2.0/include/c++/10.2.0/condition_variable:232 #3 wait_until > > (__atime=..., __lock=..., this=0x7ff835d61958) at /opt/gcc-10.2.0/include/c++/10.2.0/condition_variable:141 #4 wait_for > (__rtime=..., __rtime=..., __lock=..., this=0x7ff835d61958) at /opt/gcc-10.2.0/include/c++/10.2.0/condition_variable:172 #5 magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:98 #6 0x0000000000997d21 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:159 #7 0x0000000000aab9e9 in run (this=0x7ff835d3a6d0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #8 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #9 0x00007ff839645e65 in start_thread () from /lib64/libpthread.so.0 #10 0x00007ff8371aa88d in clone () from /lib64/libc.so.6 Thread 19 (Thread 0x7ff7c77fe700 (LWP 4591)): #0 0x00007ff839649da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000000000997bcf in __gthread_cond_timedwait ( __abs_timeout=0x7ff7c77ec960, __mutex=, __cond=0x7ff835d61958) at /opt/gcc-10.2.0/include/c++/10.2.0/x86_64-pc-linux-gnu/bits/gthr-default.h:872 #2 __wait_until_impl > > (__lock=..., __lock=..., __atime=, this=0x7ff835d61958) at /opt/gcc-10.2.0/include/c++/10.2.0/condition_variable:232 #3 wait_until > > (__atime=..., __lock=..., this=0x7ff835d61958) at /opt/gcc-10.2.0/include/c++/10.2.0/condition_variable:141 #4 wait_for > (__rtime=..., __rtime=..., __lock=..., this=0x7ff835d61958) at /opt/gcc-10.2.0/include/c++/10.2.0/condition_variable:172 #5 magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:98 #6 0x0000000000997d21 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:159 #7 0x0000000000aab9e9 in run (this=0x7ff835d3a700) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #8 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #9 0x00007ff839645e65 in start_thread () from /lib64/libpthread.so.0 #10 0x00007ff8371aa88d in clone () from /lib64/libc.so.6 Thread 18 (Thread 0x7ff7c6ffd700 (LWP 4592)): #0 0x00007ff839649da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000000000997bcf in __gthread_cond_timedwait ( __abs_timeout=0x7ff7c6feb960, __mutex=, __cond=0x7ff835d61958) at /opt/gcc-10.2.0/include/c++/10.2.0/x86_64-pc-linux-gnu/bits/gthr-default.h:872 #2 __wait_until_impl > > (__lock=..., __lock=..., __atime=, this=0x7ff835d61958) at /opt/gcc-10.2.0/include/c++/10.2.0/condition_variable:232 #3 wait_until > > (__atime=..., __lock=..., this=0x7ff835d61958) at /opt/gcc-10.2.0/include/c++/10.2.0/condition_variable:141 #4 wait_for > (__rtime=..., __rtime=..., __lock=..., this=0x7ff835d61958) at /opt/gcc-10.2.0/include/c++/10.2.0/condition_variable:172 #5 magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:98 #6 0x0000000000997d21 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:159 #7 0x0000000000aab9e9 in run (this=0x7ff835d3a730) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #8 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #9 0x00007ff839645e65 in start_thread () from /lib64/libpthread.so.0 #10 0x00007ff8371aa88d in clone () from /lib64/libc.so.6 Thread 17 (Thread 0x7ff7c67fc700 (LWP 4593)): #0 0x00007ff8396499f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007ff837a5c8bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x0000000000997938 in magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:100 #4 0x0000000000997d21 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:159 #5 0x0000000000aab9e9 in run (this=0x7ff835d3a760) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007ff839645e65 in start_thread () from /lib64/libpthread.so.0 #8 0x00007ff8371aa88d in clone () from /lib64/libc.so.6 Thread 16 (Thread 0x7ff7c5ffb700 (LWP 4594)): #0 0x00007ff8396499f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007ff837a5c8bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x0000000000997938 in magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:100 #4 0x0000000000997d21 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:159 #5 0x0000000000aab9e9 in run (this=0x7ff835d3a790) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007ff839645e65 in start_thread () from /lib64/libpthread.so.0 #8 0x00007ff8371aa88d in clone () from /lib64/libc.so.6 Thread 15 (Thread 0x7ff7c57fa700 (LWP 4595)): #0 0x00007ff8396499f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007ff837a5c8bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x0000000000997938 in magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:100 #4 0x0000000000997d21 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:159 #5 0x0000000000aab9e9 in run (this=0x7ff835d3a7c0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007ff839645e65 in start_thread () from /lib64/libpthread.so.0 #8 0x00007ff8371aa88d in clone () from /lib64/libc.so.6 Thread 14 (Thread 0x7ff7c4ff9700 (LWP 4596)): #0 0x00007ff8396499f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007ff837a5c8bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x0000000000997938 in magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:100 #4 0x0000000000997d21 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:159 #5 0x0000000000aab9e9 in run (this=0x7ff835d3a7f0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007ff839645e65 in start_thread () from /lib64/libpthread.so.0 #8 0x00007ff8371aa88d in clone () from /lib64/libc.so.6 Thread 13 (Thread 0x7ff7c47f8700 (LWP 4597)): #0 0x00007ff8396499f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007ff837a5c8bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x0000000000997938 in magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:100 #4 0x0000000000997d21 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:159 #5 0x0000000000aab9e9 in run (this=0x7ff835d3a820) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007ff839645e65 in start_thread () from /lib64/libpthread.so.0 #8 0x00007ff8371aa88d in clone () from /lib64/libc.so.6 Thread 12 (Thread 0x7ff7c3ff7700 (LWP 4598)): #0 0x00007ff8396499f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007ff837a5c8bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x0000000000997938 in magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:100 #4 0x0000000000997d21 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:159 #5 0x0000000000aab9e9 in run (this=0x7ff835d3a850) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007ff839645e65 in start_thread () from /lib64/libpthread.so.0 #8 0x00007ff8371aa88d in clone () from /lib64/libc.so.6 Thread 11 (Thread 0x7ff7c37f6700 (LWP 4599)): #0 0x00007ff8396499f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007ff837a5c8bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x0000000000997938 in magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:100 #4 0x0000000000997d21 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:159 #5 0x0000000000aab9e9 in run (this=0x7ff835d3a880) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007ff839645e65 in start_thread () from /lib64/libpthread.so.0 #8 0x00007ff8371aa88d in clone () from /lib64/libc.so.6 Thread 10 (Thread 0x7ff7c2ff5700 (LWP 4600)): #0 0x00007ff8396499f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007ff837a5c8bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x0000000000997938 in magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:100 #4 0x0000000000997d21 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:159 #5 0x0000000000aab9e9 in run (this=0x7ff835d3a8b0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007ff839645e65 in start_thread () from /lib64/libpthread.so.0 #8 0x00007ff8371aa88d in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x7ff7c19ff700 (LWP 4601)): #0 0x00007ff8396499f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007ff837a5c8bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x0000000000997938 in magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:100 #4 0x0000000000997d21 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:159 #5 0x0000000000aab9e9 in run (this=0x7ff835d3a8e0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007ff839645e65 in start_thread () from /lib64/libpthread.so.0 #8 0x00007ff8371aa88d in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7ff7c11fe700 (LWP 4602)): #0 0x00007ff8396499f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007ff837a5c8bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x0000000000997938 in magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:100 #4 0x0000000000997d21 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:159 #5 0x0000000000aab9e9 in run (this=0x7ff835d3a910) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007ff839645e65 in start_thread () from /lib64/libpthread.so.0 #8 0x00007ff8371aa88d in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7ff7c09fd700 (LWP 4603)): #0 0x00007ff8396499f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007ff837a5c8bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x0000000000997938 in magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:100 #4 0x0000000000997d21 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:159 #5 0x0000000000aab9e9 in run (this=0x7ff83220a900) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007ff839645e65 in start_thread () from /lib64/libpthread.so.0 #8 0x00007ff8371aa88d in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7ff7c01fc700 (LWP 4604)): #0 0x00007ff8396499f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007ff837a5c8bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x0000000000997938 in magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:100 #4 0x0000000000997d21 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:159 #5 0x0000000000aab9e9 in run (this=0x7ff83220a930) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007ff839645e65 in start_thread () from /lib64/libpthread.so.0 #8 0x00007ff8371aa88d in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7ff7bf9fb700 (LWP 4605)): #0 0x00007ff8396499f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007ff837a5c8bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x0000000000997938 in magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:100 #4 0x0000000000997d21 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:159 #5 0x0000000000aab9e9 in run (this=0x7ff83220a960) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007ff839645e65 in start_thread () from /lib64/libpthread.so.0 #8 0x00007ff8371aa88d in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7ff7bf1fa700 (LWP 4606)): #0 0x00007ff8396499f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007ff837a5c8bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x0000000000997938 in magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:100 #4 0x0000000000997d21 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:159 #5 0x0000000000aab9e9 in run (this=0x7ff83220a990) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007ff839645e65 in start_thread () from /lib64/libpthread.so.0 #8 0x00007ff8371aa88d in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7ff7be9f9700 (LWP 4607)): #0 0x00007ff8396499f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007ff837a5c8bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x0000000000997938 in magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:100 #4 0x0000000000997d21 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:159 #5 0x0000000000aab9e9 in run (this=0x7ff83220a9c0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007ff839645e65 in start_thread () from /lib64/libpthread.so.0 #8 0x00007ff8371aa88d in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7ff7be1f8700 (LWP 4608)): #0 0x00007ff8396499f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007ff837a5c8bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x0000000000997938 in magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:100 #4 0x0000000000997d21 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:159 #5 0x0000000000aab9e9 in run (this=0x7ff83220a9f0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007ff839645e65 in start_thread () from /lib64/libpthread.so.0 #8 0x00007ff8371aa88d in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7ff83a4f1c40 (LWP 4547)): #0 0x00007ff8371aae63 in epoll_wait () from /lib64/libc.so.6 #1 0x00007ff838b4d3c5 in epoll_dispatch (base=0x7ff835fdd000, tv=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/epoll.c:465 #2 0x00007ff838b44376 in event_base_loop (base=0x7ff835fdd000, flags=0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:1998 #3 0x000000000053bb26 in memcached_main(int, char**) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/memcached.cc:1525 #4 0x00007ff8370ce505 in __libc_start_main () from /lib64/libc.so.6 #5 0x0000000000531d94 in _start () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/include/platform/compression/allocator.h:68 Detaching from program: /opt/couchbase/bin/memcached, process 4547 2021-06-08 12:10:38,977 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-06-08 12:10:38,992 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.001108.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' 2021-06-08 12:10:39,017 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.001108.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-06-08 12:10:39,046 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.001108.txt 2021-06-08 12:10:39,065 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.001108.txt 2021-06-08 12:10:39,084 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-06-08 12:10:39,096 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:763] 172.23.106.125: No 'sanitizers.log.*' files found 2021-06-08 12:10:39,096 | infra | DEBUG | MainThread | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.125 2021-06-08 12:10:39,099 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.122 with username: root, password: couchbase 2021-06-08 12:10:39,178 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 12:10:39,388 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: ls /etc/ 2021-06-08 12:10:39,440 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 12:10:39,440 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: ls /cygdrive/c/Windows 2021-06-08 12:10:39,453 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: uname -m 2021-06-08 12:10:39,466 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: cat /proc/cpuinfo 2021-06-08 12:10:39,477 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: df -Thl 2021-06-08 12:10:39,492 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: cat /proc/meminfo 2021-06-08 12:10:39,505 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: hostname 2021-06-08 12:10:39,516 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: hostname -d 2021-06-08 12:10:39,530 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-06-08 12:10:39,546 | test | CRITICAL | MainThread | [basetestcase:check_coredump_exist:735] 172.23.106.122: 126 core dump seen 2021-06-08 12:10:39,546 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: rm -rf /opt/couchbase/var/lib/couchbase/crash/95ac37ba-5db0-4e55-140fb383-34a48bd6.core 2021-06-08 12:10:39,561 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: //opt/couchbase/bin/minidump-2-core /opt/couchbase/var/lib/couchbase/crash/95ac37ba-5db0-4e55-140fb383-34a48bd6.dmp > /opt/couchbase/var/lib/couchbase/crash/95ac37ba-5db0-4e55-140fb383-34a48bd6.core 2021-06-08 12:10:39,582 | test | INFO | MainThread | [basetestcase:get_gdb:677] running gdb --batch /opt/couchbase/bin/memcached -c /opt/couchbase/var/lib/couchbase/crash/95ac37ba-5db0-4e55-140fb383-34a48bd6.core -ex "bt full" -ex quit 2021-06-08 12:10:39,582 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: gdb --batch /opt/couchbase/bin/memcached -c /opt/couchbase/var/lib/couchbase/crash/95ac37ba-5db0-4e55-140fb383-34a48bd6.core -ex "bt full" -ex quit 2021-06-08 12:10:59,213 | test | CRITICAL | MainThread | [basetestcase:check_coredump_exist:736] 172.23.106.122: Stack Trace of first crash - 95ac37ba-5db0-4e55-140fb383-34a48bd6.dmp Core was generated by `/opt/couchbase/bin/memcached -C /opt/couchbase/var/lib/couchbase/config/memcach'. #0 0x00007f673cfa0337 in raise () from /lib64/libc.so.6 #0 0x00007f673cfa0337 in raise () from /lib64/libc.so.6 No symbol table info available. #1 0x00007f673cfa1a28 in abort () from /lib64/libc.so.6 No symbol table info available. #2 0x00007f673d8eb63c in __gnu_cxx::__verbose_terminate_handler () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/libsupc++/vterminate.cc:95 terminating = false t = #3 0x0000000000a7c65f in backtrace_terminate_handler() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/utilities/terminate_handler.cc:87 No locals. #4 0x00007f673d8f68f6 in __cxxabiv1::__terminate(void (*)()) () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/libsupc++/eh_terminate.cc:48 No locals. #5 0x00007f673d8f6961 in std::terminate () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/libsupc++/eh_terminate.cc:58 No locals. #6 0x00007f673d8f6bf4 in __cxxabiv1::__cxa_throw (obj=obj@entry=0x7f63b4000940, tinfo=0xf676e0 , dest=0x4435c0 <_ZNSt13runtime_errorD1Ev@plt>) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/libsupc++/eh_throw.cc:95 globals = header = 0x7f63b40008c0 #7 0x00000000004c7874 in MagmaKVStore::makeCompactionContext (this=0x7f6725e91400, vbid=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/magma-kvstore/magma-kvstore.cc:2623 ss = { >> = { >> = { >> = { = {_vptr.ios_base = 0xf67358 , std::allocator >+24>, static boolalpha = std::_S_boolalpha, static dec = std::_S_dec, static fixed = std::_S_fixed, static hex = std::_S_hex, static internal = std::_S_internal, static left = std::_S_left, static oct = std::_S_oct, static right = std::_S_right, static scientific = std::_S_scientific, static showbase = std::_S_showbase, static showpoint = std::_S_showpoint, static showpos = std::_S_showpos, static skipws = std::_S_skipws, static unitbuf = std::_S_unitbuf, static uppercase = std::_S_uppercase, static adjustfield = std::_S_adjustfield, static basefield = std::_S_basefield, static floatfield = std::_S_floatfield, static badbit = std::_S_badbit, static eofbit = std::_S_eofbit, static failbit = std::_S_failbit, static goodbit = std::_S_goodbit, static app = std::_S_app, static ate = std::_S_ate, static binary = std::_S_bin, static in = std::_S_in, static out = std::_S_out, static trunc = std::_S_trunc, static beg = std::_S_beg, static cur = std::_S_cur, static end = std::_S_end, _M_precision = 0, _M_width = 16151424, _M_flags = 16150424, _M_exception = std::_S_goodbit, _M_streambuf_state = 635876224, _M_callbacks = 0x7f6725e6b380, _M_word_zero = {_M_pword = 0x7f6725e6b390, _M_iword = 140080994235264}, _M_local_word = {{_M_pword = 0x7f6725e6b417, _M_iword = 140080994235776}, {_M_pword = 0x7f673dc1ddc0 <(anonymous namespace)::c_locale_impl>, _M_iword = 24}, {_M_pword = 0x7f6725e6b380, _M_iword = 16}, {_M_pword = 0x200, _M_iword = 30744794945974132}, {_M_pword = 0xf673a8 , std::allocator >+104>, _M_iword = 6}, {_M_pword = 0x0, _M_iword = 4098}, {_M_pword = 0x0, _M_iword = 0}, {_M_pword = 0x0, _M_iword = 0}}, _M_word_size = 0, _M_word = 0x0, _M_ios_locale = {static none = 0, static ctype = 1, static numeric = 2, static collate = 4, static time = 8, static monetary = 16, static messages = 32, static all = 63, _M_impl = 0x0, static _S_classic = 0x0, static _S_global = 0x0, static _S_categories = 0x3bd680, static _S_once = 0, static _S_twinned_facets = 0x7f673dc10f00 }}, _M_tie = 0x0, _M_fill = 0 '\000', _M_fill_init = false, _M_streambuf = 0x0, _M_ctype = 0x0, _M_num_put = 0x0, _M_num_get = 0x0}, _vptr.basic_istream = 0xf67358 , std::allocator >+24>, _M_gcount = 0}, >> = {_vptr.basic_ostream = 0xf67380 , std::allocator >+64>}, }, _M_stringbuf = { >> = {_vptr.basic_streambuf = 0xf66f98 , std::allocator >+16>, _M_in_beg = 0x7f6725e6b380
, _M_in_cur = 0x7f6725e6b380
, _M_in_end = 0x7f6725e6b390
, _M_out_beg = 0x7f6725e6b380
, _M_out_cur = 0x7f6725e6b417
, _M_out_end = 0x7f6725e6b580
, _M_buf_locale = {static none = 0, static ctype = 1, static numeric = 2, static collate = 4, static time = 8, static monetary = 16, static messages = 32, static all = 63, _M_impl = 0x7f673dc1ddc0 <(anonymous namespace)::c_locale_impl>, static _S_classic = 0x0, static _S_global = 0x0, static _S_categories = 0x3bd680, static _S_once = 0, static _S_twinned_facets = 0x7f673dc10f00 }}, _M_mode = 24, _M_string = {static npos = 18446744073709551615, _M_dataplus = {> = {<__gnu_cxx::new_allocator> = {}, }, _M_p = 0x7f6725e6b380
}, _M_string_length = 16, {_M_local_buf = "\000\002\000\000\000\000\000\000tore::m", _M_allocated_capacity = 512}}}} config = {purge_before_ts = 1622919015, purge_before_seq = 0, drop_deletes = false, retain_erroneous_tombstones = false} ctx = {> = {> = {}, _M_ptr = 0x7f66dc533f90, _M_refcount = {_M_pi = 0x7f66dc533f80}}, } status = dropped = readState = {status = {s = {_M_t = { >> = {_M_t = { >> = { >> = {, true>> = {> = {}, }, }, > = {_M_head_impl = 0x7f672617f1a0}, }, }}, }}}, vbstate = {static CurrentVersion = 4, maxDeletedSeqno = {counter = {_M_elems = "\000\000\000\000\000"}}, highSeqno = 0, purgeSeqno = 0, lastSnapStart = 0, lastSnapEnd = 0, maxCas = 0, hlcCasEpochSeqno = -1, mightContainXattrs = false, supportsNamespaces = true, version = 4, persistedCompletedSeqno = 0, persistedPreparedSeqno = 0, highPreparedSeqno = 0, maxVisibleSeqno = 0, onDiskPrepares = 0, onDiskPrepareBytes = 0, checkpointType = Memory, transition = {failovers = {static npos = 18446744073709551615, _M_dataplus = {> = {<__gnu_cxx::new_allocator> = {}, }, _M_p = 0x7f669bfec960 ""}, _M_string_length = 0, {_M_local_buf = '\000' , _M_allocated_capacity = 0}}, replicationTopology = {m_type = nlohmann::detail::null, m_value = {object = 0x0, array = 0x0, string = 0x0, boolean = false, number_integer = 0, number_unsigned = 0, number_float = 0}}, state = vbucket_state_dead}}, kvstoreRev = 1} #8 0x0000000000859585 in MagmaKVStore::compactionCallBack(MagmaKVStore::MagmaCompactionCB&, magma::Slice const&, magma::Slice const&, magma::Slice const&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/magma-kvstore/magma-kvstore.cc:278 vbid = {vbid = 723} itemString = @0x7f6634f07418: seqno = exptime = #9 0x000000000094a6bd in magma::DocSeqGCContext::Transform (this=0x7f66d47c3140, key=..., value=..., buffer=..., outputValue=0x7f669bfecd50) at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/magma/kvstore/format.h:44 drop = false #10 0x00000000009cf755 in magma::ItemGCFilter::Process(std::vector, std::allocator > >&, std::vector, std::allocator > >*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/lsm/mvcc.cc:106 newValue = {data = 0x0, len = 0} deleted = oldValue = 0x0 kv = {> = {}, first = {data = 0x7f657f7212c0
, len = 26}, second = {data = 0x7f657f7212da
, len = 51}} item = {key = {data = 0x7f657f7212c0
, len = 17}, sn = -8861676691781582848, op = magma::Update} gcOffset = 1 lastSn = #11 0x0000000000962bb2 in magma::LSMTree::mergeSSTables(std::vector, std::allocator > >&, magma::ItemGCFilter&, unsigned long, double, std::vector, std::allocator > >*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/lsm/lsm_tree.cc:649 magma::rdBuf = std::__ioinit = {static _S_refcount = 0, static _S_synced_with_stdio = true} fmt::v5::internal::basic_data::POWERS_OF_10_32 = {0, 10, 100, 1000, 10000, 100000, 1000000, 10000000, 100000000, 1000000000} fmt::v5::internal::basic_data::POWERS_OF_10_64 = {0, 10, 100, 1000, 10000, 100000, 1000000, 10000000, 100000000, 1000000000, 10000000000, 100000000000, 1000000000000, 10000000000000, 100000000000000, 1000000000000000, 10000000000000000, 100000000000000000, 1000000000000000000, 10000000000000000000} fmt::v5::internal::basic_data::DIGITS = "00010203040506070809101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899" boost::exception_detail::exception_ptr_static_exception_object::e = {ptr_ = {px = 0x0, pn = {pi_ = 0x0}}} boost::exception_detail::exception_ptr_static_exception_object::e = {ptr_ = {px = 0x0, pn = {pi_ = 0x0}}} #12 0x00000000009c919d in magma::LSMTree::compactLevelForInputs(int, int, std::unique_lock&, std::vector, std::allocator > >&, std::vector, std::allocator > >&, bool, std::function > (bool)>) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/lsm/compaction.cc:168 status = {s = {_M_t = { >> = {_M_t = { >> = { >> = {, true>> = {> = {}, }, }, > = {_M_head_impl = 0x0}, }, }}, }}} compactID = 2 _c = {fn = {> = {}, = {static _M_max_size = 16, static _M_max_align = 8, _M_functor = {_M_unused = {_M_object = 0x7f65532608e0, _M_const_object = 0x7f65532608e0, _M_function_pointer = 0x7f65532608e0, _M_member_pointer = (void (std::_Undefined_class::*)(std::_Undefined_class * const)) 0x7f65532608e0}, _M_pod_data = "\340\b&Se\177\000\000\000\000\000\000\000\000\000"}, _M_manager = 0x9c5290 &, magma::TableList&, magma::TableList&, bool, magma::GCContextBuilder):: >::_M_manager(std::_Any_data &, const std::_Any_data &, std::_Manager_operation)>}, _M_invoker = 0x9c5a60 &, magma::TableList&, magma::TableList&, bool, magma::GCContextBuilder):: >::_M_invoke(const std::_Any_data &)>}} srcLevelSize = 8192 targetSize = 4194304 levelSizeRatio = 0 srcDataSize = dstDataSize = srcLogicalDataSz = 8192 dstLogicalDataSz = 147456 height = isHighestLevel = writeAmp = 18 inputList = {, std::allocator > >> = {_M_impl = { >> = {<__gnu_cxx::new_allocator >> = {}, }, , std::allocator > >::_Vector_impl_data> = {_M_start = 0x7f6584bd2080, _M_finish = 0x7f6584bd20a0, _M_end_of_storage = 0x7f6584bd20a0}, }}, } outputList = {, std::allocator > >> = {_M_impl = { >> = {<__gnu_cxx::new_allocator >> = {}, }, , std::allocator > >::_Vector_impl_data> = {_M_start = 0x0, _M_finish = 0x0, _M_end_of_storage = 0x0}, }}, } needSSTablesMerging = true bloomFilterAccuracy = 0.94999998807907104 levelSize = 2097152 fileCompaction = true gcFilter = {_M_t = { >> = {_M_t = { >> = { >> = {, true>> = {> = {}, }, }, > = {_M_head_impl = 0x7f668de99ac0}, }, }}, }} _ = {fn = {> = {}, = {static _M_max_size = 16, static _M_max_align = 8, _M_functor = {_M_unused = {_M_object = 0x7f669bfed398, _M_const_object = 0x7f669bfed398, _M_function_pointer = 0x7f669bfed398, _M_member_pointer = (void (std::_Undefined_class::*)(std::_Undefined_class * const)) 0x7f669bfed398, this adjustment 140078680560592}, _M_pod_data = "\230\323\376\233f\177\000\000\320\323\376\233f\177\000"}, _M_manager = 0x9c4f80 &, magma::TableList&, magma::TableList&, bool, magma::GCContextBuilder):: >::_M_manager(std::_Any_data &, const std::_Any_data &, std::_Manager_operation)>}, _M_invoker = 0x9c5320 &, std::vector, std::allocator > >&, std::vector, std::allocator > >&, bool, std::function > (bool)>)::{lambda()#2}>::_M_invoke(std::_Any_data const&)>}} resultState = outDataSize = newSnapshot = {> = {> = {}, _M_ptr = 0x7f668d60e210, _M_refcount = {_M_pi = 0x0}}, } dur = 391455696 secs = MBps = rMBps = 6.920806377997199e-310 wMBps = 5.0652662371470238e-317 gcRatio = 819200 newLevelRatio = 0 #13 0x00000000009cb5b4 in magma::LSMTree::compactLevel(int, std::unique_lock&, bool, std::function > (bool)>) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/lsm/compaction.cc:73 srcLevelList = {, std::allocator > >> = {_M_impl = { >> = {<__gnu_cxx::new_allocator >> = {}, }, , std::allocator > >::_Vector_impl_data> = {_M_start = 0x7f66a42dccc0, _M_finish = 0x7f66a42dccd0, _M_end_of_storage = 0x7f66a42dccd0}, }}, } dstLevelList = {, std::allocator > >> = {_M_impl = { >> = {<__gnu_cxx::new_allocator >> = {}, }, , std::allocator > >::_Vector_impl_data> = {_M_start = 0x7f6667ed7b80, _M_finish = 0x7f6667ed7b90, _M_end_of_storage = 0x7f6667ed7b90}, }}, } inputList = {, std::allocator > >> = {_M_impl = { >> = {<__gnu_cxx::new_allocator >> = {}, }, , std::allocator > >::_Vector_impl_data> = {_M_start = 0x0, _M_finish = 0x0, _M_end_of_storage = 0x0}, }}, } outputList = {, std::allocator > >> = {_M_impl = { >> = {<__gnu_cxx::new_allocator >> = {}, }, , std::allocator > >::_Vector_impl_data> = {_M_start = 0x0, _M_finish = 0x0, _M_end_of_storage = 0x0}, }}, } status = {s = {_M_t = { >> = {_M_t = { >> = { >> = {, true>> = {> = {}, }, }, > = {_M_head_impl = 0x0}, }, }}, }}} dstLevel = 4 #14 0x00000000009cb855 in magma::LSMTree::compact(std::unique_lock&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/lsm/compaction.cc:339 level = status = {s = {_M_t = { >> = {_M_t = { >> = { >> = {, true>> = {> = {}, }, }, > = {_M_head_impl = 0x0}, }, }}, }}} ratio = @0x7f667b3778d0: __for_range = @0x7f669bfed6f0: {, std::allocator > >> = {_M_impl = { >> = {<__gnu_cxx::new_allocator >> = {}, }, , std::allocator > >::_Vector_impl_data> = {_M_start = 0x7f667b3778d0, _M_finish = 0x7f667b3778e0, _M_end_of_storage = 0x7f667b3778e0}, }}, } compacted = false sizeRatios = {, std::allocator > >> = {_M_impl = { >> = {<__gnu_cxx::new_allocator >> = {}, }, , std::allocator > >::_Vector_impl_data> = {_M_start = 0x7f667b3778d0, _M_finish = 0x7f667b3778e0, _M_end_of_storage = 0x7f667b3778e0}, }}, } #15 0x0000000000957fc8 in std::_Function_handler::_M_invoke(std::_Any_data const&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/lsm/lsm_tree.cc:297 magma::rdBuf = std::__ioinit = {static _S_refcount = 0, static _S_synced_with_stdio = true} fmt::v5::internal::basic_data::POWERS_OF_10_32 = {0, 10, 100, 1000, 10000, 100000, 1000000, 10000000, 100000000, 1000000000} fmt::v5::internal::basic_data::POWERS_OF_10_64 = {0, 10, 100, 1000, 10000, 100000, 1000000, 10000000, 100000000, 1000000000, 10000000000, 100000000000, 1000000000000, 10000000000000, 100000000000000, 1000000000000000, 10000000000000000, 100000000000000000, 1000000000000000000, 10000000000000000000} fmt::v5::internal::basic_data::DIGITS = "00010203040506070809101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899" boost::exception_detail::exception_ptr_static_exception_object::e = {ptr_ = {px = 0x0, pn = {pi_ = 0x0}}} boost::exception_detail::exception_ptr_static_exception_object::e = {ptr_ = {px = 0x0, pn = {pi_ = 0x0}}} #16 0x000000000099653d in operator() (this=0x7f66873113d8) at /opt/gcc-10.2.0/include/c++/10.2.0/bits/std_function.h:248 No locals. #17 magma::TimedTask::Complete(bool) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:143 wg = 0x7f6698682d40 envGuard = {previous = 0x0} currTask = {> = {> = {}, _M_ptr = 0x7f66873113d0, _M_refcount = {_M_pi = 0x7f66873113c0}}, } #18 0x0000000000997d31 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:160 task = {> = {> = {}, _M_ptr = 0x7f668ecca660, _M_refcount = {_M_pi = 0x7f668ecca650}}, } worker = 0x7f672603f140 #19 0x0000000000aab9e9 in run (this=0x7f6733cc16b0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 No locals. #20 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 No locals. #21 0x00007f673f503e65 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #22 0x00007f673d06888d in clone () from /lib64/libc.so.6 No symbol table info available. 2021-06-08 12:10:59,216 | test | INFO | MainThread | [basetestcase:get_full_thread_dump:686] running gdb -p `(pidof memcached)` -ex "thread apply all bt" -ex detach -ex quit 2021-06-08 12:10:59,216 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: gdb -p `(pidof memcached)` -ex "thread apply all bt" -ex detach -ex quit Attaching to process 71145 2021-06-08 12:10:59,321 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-06-08 12:10:59,334 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.001220.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' 2021-06-08 12:10:59,359 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.001220.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-06-08 12:10:59,387 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.001220.txt 2021-06-08 12:10:59,407 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.001220.txt 2021-06-08 12:10:59,426 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-06-08 12:10:59,437 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:763] 172.23.106.122: No 'sanitizers.log.*' files found 2021-06-08 12:10:59,437 | infra | DEBUG | MainThread | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.122 2021-06-08 12:10:59,440 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.155 with username: root, password: couchbase 2021-06-08 12:10:59,517 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 12:10:59,724 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: ls /etc/ 2021-06-08 12:10:59,776 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 12:10:59,776 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: ls /cygdrive/c/Windows 2021-06-08 12:10:59,789 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: uname -m 2021-06-08 12:10:59,801 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: cat /proc/cpuinfo 2021-06-08 12:10:59,812 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: df -Thl 2021-06-08 12:10:59,825 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: cat /proc/meminfo 2021-06-08 12:10:59,838 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: hostname 2021-06-08 12:10:59,848 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: hostname -d 2021-06-08 12:10:59,861 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: ls -lt /opt/couchbase/var/lib/couchbase/crash/ 2021-06-08 12:10:59,875 | test | CRITICAL | MainThread | [basetestcase:check_coredump_exist:735] 172.23.106.155: 1 core dump seen 2021-06-08 12:10:59,875 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: rm -rf /opt/couchbase/var/lib/couchbase/crash/56a055cd-e49f-4cc6-2290879e-68333067.core 2021-06-08 12:10:59,885 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: //opt/couchbase/bin/minidump-2-core /opt/couchbase/var/lib/couchbase/crash/56a055cd-e49f-4cc6-2290879e-68333067.dmp > /opt/couchbase/var/lib/couchbase/crash/56a055cd-e49f-4cc6-2290879e-68333067.core 2021-06-08 12:10:59,915 | test | INFO | MainThread | [basetestcase:get_gdb:677] running gdb --batch /opt/couchbase/bin/memcached -c /opt/couchbase/var/lib/couchbase/crash/56a055cd-e49f-4cc6-2290879e-68333067.core -ex "bt full" -ex quit 2021-06-08 12:10:59,915 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: gdb --batch /opt/couchbase/bin/memcached -c /opt/couchbase/var/lib/couchbase/crash/56a055cd-e49f-4cc6-2290879e-68333067.core -ex "bt full" -ex quit Traceback (most recent call last): File "/opt/jython/Lib/logging/__init__.py", line 853, in emit msg = self.format(record) File "/opt/jython/Lib/logging/__init__.py", line 726, in format return fmt.format(record) File "/opt/jython/Lib/logging/__init__.py", line 469, in format s = self._fmt % record.__dict__ UnicodeDecodeError: 'ascii' codec can't decode byte 0xcc in position 12904: ordinal not in range(128) Logged from file basetestcase.py, line 736 Traceback (most recent call last): File "/opt/jython/Lib/logging/__init__.py", line 853, in emit msg = self.format(record) 2021-06-08 12:11:19,634 | test | INFO | MainThread | [basetestcase:get_full_thread_dump:686] running gdb -p `(pidof memcached)` -ex "thread apply all bt" -ex detach -ex quit File "/opt/jython/Lib/logging/__init__.py", line 726, in format return fmt.format(record) File "/opt/jython/Lib/logging/__init__.py", line 469, in format s = self._fmt % record.__dict__ UnicodeDecodeError: 'ascii' codec can't decode byte 0xcc in position 12904: ordinal not in range(128) Logged from file basetestcase.py, line 736 2021-06-08 12:11:19,634 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: gdb -p `(pidof memcached)` -ex "thread apply all bt" -ex detach -ex quit [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Loaded symbols for /lib64/libpthread.so.0 Reading symbols from /lib64/librt.so.1...(no debugging symbols found)...done. Loaded symbols for /lib64/librt.so.1 Reading symbols from /opt/couchbase/bin/../lib/libsnappy.so.1...Reading symbols from /usr/lib/debug/opt/couchbase/lib/libsnappy.so.1.2.0.debug...done. done. Loaded symbols for /opt/couchbase/bin/../lib/libsnappy.so.1 Reading symbols from /opt/couchbase/bin/../lib/libjemalloc.so.2...Reading symbols from /usr/lib/debug/opt/couchbase/lib/libjemalloc.so.2.debug...done. done. Loaded symbols for /opt/couchbase/bin/../lib/libjemalloc.so.2 Reading symbols from /opt/couchbase/bin/../lib/libevent_core-2.1.so.7...Reading symbols from /usr/lib/debug/opt/couchbase/lib/libevent_core-2.1.so.7.0.0.debug...done. done. Loaded symbols for /opt/couchbase/bin/../lib/libevent_core-2.1.so.7 Reading symbols from /opt/couchbase/bin/../lib/libevent_extra-2.1.so.7...Reading symbols from /usr/lib/debug/opt/couchbase/lib/libevent_extra-2.1.so.7.0.0.debug...done. done. Loaded symbols for /opt/couchbase/bin/../lib/libevent_extra-2.1.so.7 Reading symbols from /opt/couchbase/bin/../lib/libevent_pthreads-2.1.so.7...Reading symbols from /usr/lib/debug/opt/couchbase/lib/libevent_pthreads-2.1.so.7.0.0.debug...done. done. Loaded symbols for /opt/couchbase/bin/../lib/libevent_pthreads-2.1.so.7 Reading symbols from /opt/couchbase/bin/../lib/libevent_openssl-2.1.so.7...Reading symbols from /usr/lib/debug/opt/couchbase/lib/libevent_openssl-2.1.so.7.0.0.debug...done. done. Loaded symbols for /opt/couchbase/bin/../lib/libevent_openssl-2.1.so.7 Reading symbols from /opt/couchbase/bin/../lib/libssl.so.1.1...Reading symbols from /usr/lib/debug/opt/couchbase/lib/libssl.so.1.1.debug...(no debugging symbols found)...done. (no debugging symbols found)...done. Loaded symbols for /opt/couchbase/bin/../lib/libssl.so.1.1 Reading symbols from /opt/couchbase/bin/../lib/libcrypto.so.1.1...Reading symbols from /usr/lib/debug/opt/couchbase/lib/libcrypto.so.1.1.debug...(no debugging symbols found)...done. (no debugging symbols found)...done. Loaded symbols for /opt/couchbase/bin/../lib/libcrypto.so.1.1 Reading symbols from /opt/couchbase/bin/../lib/libstdc++.so.6...done. Loaded symbols for /opt/couchbase/bin/../lib/libstdc++.so.6 Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done. Loaded symbols for /lib64/libm.so.6 Reading symbols from /opt/couchbase/bin/../lib/libgcc_s.so.1...done. Loaded symbols for /opt/couchbase/bin/../lib/libgcc_s.so.1 Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done. Loaded symbols for /lib64/libc.so.6 Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done. Loaded symbols for /lib64/ld-linux-x86-64.so.2 0x00007fe97bd96e63 in epoll_wait () from /lib64/libc.so.6 To enable execution of this file add add-auto-load-safe-path /opt/couchbase/lib/libstdc++.so.6.0.28-gdb.py line to your configuration file "/root/.gdbinit". To completely disable this security protection add set auto-load safe-path / line to your configuration file "/root/.gdbinit". For more information about this security protection see the "Auto-loading safe path" section in the GDB manual. E.g., run from the shell: info "(gdb)Auto-loading safe path" Thread 61 (Thread 0x7fe97b3ff700 (LWP 96530)): #0 0x00007fe97e235da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fe97d9725a7 in background_thread_sleep (tsdn=, interval=10000000000, info=) at src/background_thread.c:255 #2 background_work_sleep_once (ind=0, info=, tsdn=) at src/background_thread.c:307 #3 background_thread0_work (tsd=0x7fe97b3ee228) at src/background_thread.c:452 #4 background_work (ind=, tsd=0x7fe97b3ee228) at src/background_thread.c:490 #5 background_thread_entry (ind_arg=) at src/background_thread.c:522 #6 0x00007fe97e231e65 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fe97bd9688d in clone () from /lib64/libc.so.6 Thread 60 (Thread 0x7fe97a67e700 (LWP 96531)): #0 0x00007fe97e235da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000000000a855a5 in __gthread_cond_timedwait ( __abs_timeout=0x7fe97a66c760, __mutex=, __cond=0x7fe97abca138) at /opt/gcc-10.2.0/include/c++/10.2.0/x86_64-pc-linux-gnu/bits/gthr-default.h:872 #2 __wait_until_impl > > (__lock=..., __lock=..., __atime=, this=0x7fe97abca138) at /opt/gcc-10.2.0/include/c++/10.2.0/condition_variable:232 #3 wait_until > > (__atime=..., __lock=..., this=0x7fe97abca138) at /opt/gcc-10.2.0/include/c++/10.2.0/condition_variable:141 #4 wait_until >, spdlog::details::mpmc_blocking_queue::dequeue_for:: > (__p=..., __atime=..., __lock=..., this=0x7fe97abca138) at /opt/gcc-10.2.0/include/c++/10.2.0/condition_variable:158 #5 wait_for, spdlog::details::mpmc_blocking_queue::dequeue_for:: > ( __rtime=, __rtime=, __p=..., __lock=..., this=0x7fe97abca138) at /opt/gcc-10.2.0/include/c++/10.2.0/condition_variable:185 #6 dequeue_for (wait_duration=..., popped_item=..., this=0x7fe97abca110) at /home/couchbase/jenkins/workspace/couchbase-server-unix/third_party/spdlog/include/spdlog/details/mpmc_blocking_q.h:61 #7 spdlog::details::thread_pool::process_next_msg_ ( this=this@entry=0x7fe97abca110) at /home/couchbase/jenkins/workspace/couchbase-server-unix/third_party/spdlog/include/spdlog/details/thread_pool.h:193 #8 0x0000000000a85bc0 in spdlog::details::thread_pool::worker_loop_ ( this=0x7fe97abca110) at /home/couchbase/jenkins/workspace/couchbase-server-unix/third_party/spdlog/include/spdlog/details/thread_pool.h:184 #9 0x00007fe97c64dd40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #10 0x00007fe97e231e65 in start_thread () from /lib64/libpthread.so.0 #11 0x00007fe97bd9688d in clone () from /lib64/libc.so.6 Thread 59 (Thread 0x7fe979e7d700 (LWP 96532)): #0 0x00007fe97e235da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000000000a84d5b in __gthread_cond_timedwait ( __abs_timeout=0x7fe979e6b9b0, __mutex=, __cond=) at /opt/gcc-10.2.0/include/c++/10.2.0/x86_64-pc-linux-gnu/bits/gthr-default.h:872 #2 __wait_until_impl > > (__lock=..., __lock=..., __atime=, this=) at /opt/gcc-10.2.0/include/c++/10.2.0/condition_variable:232 #3 wait_until > > (__atime=..., __lock=..., this=) at /opt/gcc-10.2.0/include/c++/10.2.0/condition_variable:141 #4 wait_until >, spdlog::details::periodic_worker::periodic_worker(const std::function&, std::chrono::seconds):::: > (__p=..., __atime=..., __lock=..., this=) at /opt/gcc-10.2.0/include/c++/10.2.0/condition_variable:158 #5 wait_for, spdlog::details::periodic_worker::periodic_worker(const std::function&, std::chrono::seconds):::: > (__rtime=..., __rtime=..., __p=..., __lock=..., this=) at /opt/gcc-10.2.0/include/c++/10.2.0/condition_variable:185 #6 operator() (__closure=0x7fe97abcb648) at /home/couchbase/jenkins/workspace/couchbase-server-unix/third_party/spdlog/include/spdlog/details/periodic_worker.h:38 #7 __invoke_impl&, std::chrono::seconds):: > ( __f=) at /opt/gcc-10.2.0/include/c++/10.2.0/bits/invoke.h:60 #8 __invoke&, std::chrono::seconds):: > ( __fn=) at /opt/gcc-10.2.0/include/c++/10.2.0/bits/invoke.h:95 #9 _M_invoke<0> (this=0x7fe97abcb648) at /opt/gcc-10.2.0/include/c++/10.2.0/thread:264 #10 operator() (this=0x7fe97abcb648) at /opt/gcc-10.2.0/include/c++/10.2.0/thread:271 #11 _ZNSt6thread11_State_implINS_8_InvokerISt5tupleIJZN6spdlog7details15periodic_workerC4ERKSt8functionIFvvEENSt6chrono8durationIlSt5ratioILl1ELl1EEEEEUlvE_EEEEE6_M_runEv (this=0x7fe97abcb640) at /opt/gcc-10.2.0/include/c++/10.2.0/thread:215 #12 0x00007fe97c64dd40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #13 0x00007fe97e231e65 in start_thread () from /lib64/libpthread.so.0 #14 0x00007fe97bd9688d in clone () from /lib64/libc.so.6 Thread 58 (Thread 0x7fe97967c700 (LWP 96533)): #0 0x00007fe97bd8bbed in poll () from /lib64/libc.so.6 #1 0x000000000058a841 in get_command (buffsize=80, buffer=0x7fe97966a990 "@Ɔ~\351\177") at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/stdin_check.cc:57 #2 check_stdin_thread (arg=) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/stdin_check.cc:98 #3 0x0000000000aab9e9 in run (this=0x7fe97abc61b0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #4 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #5 0x00007fe97e231e65 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fe97bd9688d in clone () from /lib64/libc.so.6 Thread 57 (Thread 0x7fe978e7b700 (LWP 96534)): #0 0x00007fe97e2359f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00000000008c9e4b in consume_socket (thread_index=, sp=0x26b60e0, ctx=0x26b4bc0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/prometheus-cpp/prometheus-cpp-prefix/src/prometheus-cpp/3rdparty/civetweb/src/civetweb.c:18846 #2 worker_thread_run (conn=0x26b57f0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/prometheus-cpp/prometheus-cpp-prefix/src/prometheus-cpp/3rdparty/civetweb/src/civetweb.c:18987 #3 worker_thread (thread_func_param=0x26b57f0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/prometheus-cpp/prometheus-cpp-prefix/src/prometheus-cpp/3rdparty/civetweb/src/civetweb.c:19145 #4 0x00007fe97e231e65 in start_thread () from /lib64/libpthread.so.0 #5 0x00007fe97bd9688d in clone () from /lib64/libc.so.6 Thread 56 (Thread 0x7fe97867a700 (LWP 96535)): #0 0x00007fe97e2359f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00000000008c9e4b in consume_socket (thread_index=, sp=0x26b6ac8, ctx=0x26b4bc0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/prometheus-cpp/prometheus-cpp-prefix/src/prometheus-cpp/3rdparty/civetweb/src/civetweb.c:18846 #2 worker_thread_run (conn=0x26b61d8) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/prometheus-cpp/prometheus-cpp-prefix/src/prometheus-cpp/3rdparty/civetweb/src/civetweb.c:18987 #3 worker_thread (thread_func_param=0x26b61d8) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/prometheus-cpp/prometheus-cpp-prefix/src/prometheus-cpp/3rdparty/civetweb/src/civetweb.c:19145 #4 0x00007fe97e231e65 in start_thread () from /lib64/libpthread.so.0 #5 0x00007fe97bd9688d in clone () from /lib64/libc.so.6 Thread 55 (Thread 0x7fe977e79700 (LWP 96536)): #0 0x00007fe97bd8bbed in poll () from /lib64/libc.so.6 #1 0x00000000008c0db8 in master_thread_run (ctx=0x26b4bc0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/prometheus-cpp/prometheus-cpp-prefix/src/prometheus-cpp/3rdparty/civetweb/src/civetweb.c:19291 #2 master_thread (thread_func_param=0x26b4bc0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/prometheus-cpp/prometheus-cpp-prefix/src/prometheus-cpp/3rdparty/civetweb/src/civetweb.c:19386 #3 0x00007fe97e231e65 in start_thread () from /lib64/libpthread.so.0 #4 0x00007fe97bd9688d in clone () from /lib64/libc.so.6 Thread 54 (Thread 0x7fe977678700 (LWP 96537)): #0 0x00007fe97e235da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00000000005bb7d6 in ExternalAuthManagerThread::run() () at /opt/gcc-10.2.0/include/c++/10.2.0/x86_64-pc-linux-gnu/bits/gthr-default.h:872 #2 0x0000000000ab4cc3 in Couchbase::Thread::thread_entry() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/thread.cc:38 #3 0x0000000000aab9e9 in run (this=0x7fe97abc6900) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #4 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #5 0x00007fe97e231e65 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fe97bd9688d in clone () from /lib64/libc.so.6 Thread 53 (Thread 0x7fe976af7700 (LWP 96538)): #0 0x00007fe97e235da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000000000629ee7 in AuditImpl::consume_events() () at /opt/gcc-10.2.0/include/c++/10.2.0/x86_64-pc-linux-gnu/bits/gthr-default.h:872 #2 0x0000000000aab9e9 in run (this=0x7fe97abc6660) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #3 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #4 0x00007fe97e231e65 in start_thread () from /lib64/libpthread.so.0 #5 0x00007fe97bd9688d in clone () from /lib64/libc.so.6 Thread 52 (Thread 0x7fe9762f6700 (LWP 96539)): #0 0x00007fe97bd96e63 in epoll_wait () from /lib64/libc.so.6 #1 0x00007fe97d7393c5 in epoll_dispatch (base=0x7fe97abdf300, tv=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/epoll.c:465 #2 0x00007fe97d730376 in event_base_loop (base=0x7fe97abdf300, flags=flags@entry=0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:1998 #3 0x000000000058adcb in worker_libevent (arg=0x7fe97a92f000) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/thread.cc:202 #4 0x0000000000aab9e9 in run (this=0x7fe97abe5b60) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #5 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #6 0x00007fe97e231e65 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fe97bd9688d in clone () from /lib64/libc.so.6 Thread 51 (Thread 0x7fe975af5700 (LWP 96540)): #0 0x00007fe97bd96e63 in epoll_wait () from /lib64/libc.so.6 #1 0x00007fe97d7393c5 in epoll_dispatch (base=0x7fe97abdf600, tv=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/epoll.c:465 #2 0x00007fe97d730376 in event_base_loop (base=0x7fe97abdf600, flags=flags@entry=0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:1998 #3 0x000000000058adcb in worker_libevent (arg=0x7fe97a92faa8) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/thread.cc:202 #4 0x0000000000aab9e9 in run (this=0x7fe97abe5920) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #5 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #6 0x00007fe97e231e65 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fe97bd9688d in clone () from /lib64/libc.so.6 Thread 50 (Thread 0x7fe9752f4700 (LWP 96541)): #0 0x00007fe97bd96e63 in epoll_wait () from /lib64/libc.so.6 #1 0x00007fe97d7393c5 in epoll_dispatch (base=0x7fe97abdf900, tv=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/epoll.c:465 #2 0x00007fe97d730376 in event_base_loop (base=0x7fe97abdf900, flags=flags@entry=0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:1998 #3 0x000000000058adcb in worker_libevent (arg=0x7fe97a930550) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/thread.cc:202 #4 0x0000000000aab9e9 in run (this=0x7fe97abe5b00) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #5 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #6 0x00007fe97e231e65 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fe97bd9688d in clone () from /lib64/libc.so.6 Thread 49 (Thread 0x7fe974af3700 (LWP 96542)): #0 0x00007fe97bd96e63 in epoll_wait () from /lib64/libc.so.6 #1 0x00007fe97d7393c5 in epoll_dispatch (base=0x7fe97abdfc00, tv=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/epoll.c:465 #2 0x00007fe97d730376 in event_base_loop (base=0x7fe97abdfc00, flags=flags@entry=0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:1998 #3 0x000000000058adcb in worker_libevent (arg=0x7fe97a930ff8) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/thread.cc:202 #4 0x0000000000aab9e9 in run (this=0x7fe97abe5c20) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #5 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #6 0x00007fe97e231e65 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fe97bd9688d in clone () from /lib64/libc.so.6 Thread 48 (Thread 0x7fe96ffff700 (LWP 96543)): #0 0x00007fe97bd96e63 in epoll_wait () from /lib64/libc.so.6 #1 0x00007fe97d7393c5 in epoll_dispatch (base=0x7fe97abdff00, tv=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/epoll.c:465 #2 0x00007fe97d730376 in event_base_loop (base=0x7fe97abdff00, flags=flags@entry=0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:1998 #3 0x000000000058adcb in worker_libevent (arg=0x7fe97a931aa0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/thread.cc:202 #4 0x0000000000aab9e9 in run (this=0x7fe97abe5aa0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #5 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #6 0x00007fe97e231e65 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fe97bd9688d in clone () from /lib64/libc.so.6 Thread 47 (Thread 0x7fe96f7fe700 (LWP 96544)): #0 0x00007fe97bd96e63 in epoll_wait () from /lib64/libc.so.6 #1 0x00007fe97d7393c5 in epoll_dispatch (base=0x7fe97abe0200, tv=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/epoll.c:465 #2 0x00007fe97d730376 in event_base_loop (base=0x7fe97abe0200, flags=flags@entry=0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:1998 #3 0x000000000058adcb in worker_libevent (arg=0x7fe97a932548) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/thread.cc:202 #4 0x0000000000aab9e9 in run (this=0x7fe97abe5c80) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #5 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #6 0x00007fe97e231e65 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fe97bd9688d in clone () from /lib64/libc.so.6 Thread 46 (Thread 0x7fe96effd700 (LWP 96545)): #0 0x00007fe97bd96e63 in epoll_wait () from /lib64/libc.so.6 #1 0x00007fe97d7393c5 in epoll_dispatch (base=0x7fe97abe0500, tv=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/epoll.c:465 #2 0x00007fe97d730376 in event_base_loop (base=0x7fe97abe0500, flags=flags@entry=0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:1998 #3 0x000000000058adcb in worker_libevent (arg=0x7fe97a932ff0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/thread.cc:202 #4 0x0000000000aab9e9 in run (this=0x7fe97abe5cb0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #5 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #6 0x00007fe97e231e65 in start_thread () from /lib64/libpthread.so.0 #7 0x00007fe97bd9688d in clone () from /lib64/libc.so.6 Thread 45 (Thread 0x7fe96e7fc700 (LWP 96546)): #0 0x00007fe97e2359f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fe97c6488bc in __gthread_cond_wait (__mutex=, __cond=__cond@entry=0x7fe976c6b158) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=this@entry=0x7fe976c6b158, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000005ce701 in Executor::run() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/executor.cc:38 #4 0x0000000000ab4cc3 in Couchbase::Thread::thread_entry() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/thread.cc:38 #5 0x0000000000aab9e9 in run (this=0x7fe97abe5a70) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007fe97e231e65 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fe97bd9688d in clone () from /lib64/libc.so.6 Thread 44 (Thread 0x7fe96dffb700 (LWP 96547)): #0 0x00007fe97e2359f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fe97c6488bc in __gthread_cond_wait (__mutex=, __cond=__cond@entry=0x7fe976c6b318) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=this@entry=0x7fe976c6b318, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000005ce701 in Executor::run() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/executor.cc:38 #4 0x0000000000ab4cc3 in Couchbase::Thread::thread_entry() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/thread.cc:38 #5 0x0000000000aab9e9 in run (this=0x7fe97abe5c50) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007fe97e231e65 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fe97bd9688d in clone () from /lib64/libc.so.6 Thread 43 (Thread 0x7fe96d7fa700 (LWP 96548)): #0 0x00007fe97e2359f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fe97c6488bc in __gthread_cond_wait (__mutex=, __cond=__cond@entry=0x7fe976c6b4d8) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=this@entry=0x7fe976c6b4d8, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000005ce701 in Executor::run() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/executor.cc:38 #4 0x0000000000ab4cc3 in Couchbase::Thread::thread_entry() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/thread.cc:38 #5 0x0000000000aab9e9 in run (this=0x7fe97abe5d70) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007fe97e231e65 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fe97bd9688d in clone () from /lib64/libc.so.6 Thread 42 (Thread 0x7fe96cff9700 (LWP 96549)): #0 0x00007fe97e2359f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fe97c6488bc in __gthread_cond_wait (__mutex=, __cond=__cond@entry=0x7fe976c6b698) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=this@entry=0x7fe976c6b698, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000005ce701 in Executor::run() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/executor.cc:38 #4 0x0000000000ab4cc3 in Couchbase::Thread::thread_entry() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/thread.cc:38 #5 0x0000000000aab9e9 in run (this=0x7fe97abe5bf0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007fe97e231e65 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fe97bd9688d in clone () from /lib64/libc.so.6 Thread 41 (Thread 0x7fe967fff700 (LWP 96550)): #0 0x00007fe97e2359f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fe97c6488bc in __gthread_cond_wait (__mutex=, __cond=__cond@entry=0x7fe976c6b858) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=this@entry=0x7fe976c6b858, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000005ce701 in Executor::run() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/executor.cc:38 #4 0x0000000000ab4cc3 in Couchbase::Thread::thread_entry() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/thread.cc:38 #5 0x0000000000aab9e9 in run (this=0x7fe97abe5dd0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007fe97e231e65 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fe97bd9688d in clone () from /lib64/libc.so.6 Thread 40 (Thread 0x7fe9677fe700 (LWP 96551)): #0 0x00007fe97e2359f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fe97c6488bc in __gthread_cond_wait (__mutex=, __cond=__cond@entry=0x7fe976c6ba18) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=this@entry=0x7fe976c6ba18, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000005ce701 in Executor::run() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/executor.cc:38 #4 0x0000000000ab4cc3 in Couchbase::Thread::thread_entry() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/thread.cc:38 #5 0x0000000000aab9e9 in run (this=0x7fe97abe5b90) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007fe97e231e65 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fe97bd9688d in clone () from /lib64/libc.so.6 Thread 39 (Thread 0x7fe966ffd700 (LWP 96552)): #0 0x00007fe97e2359f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fe97c6488bc in __gthread_cond_wait (__mutex=, __cond=__cond@entry=0x7fe976c6bbd8) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=this@entry=0x7fe976c6bbd8, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x00000000005ce701 in Executor::run() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/executor.cc:38 #4 0x0000000000ab4cc3 in Couchbase::Thread::thread_entry() () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/thread.cc:38 #5 0x0000000000aab9e9 in run (this=0x7fe97abe5bc0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007fe97e231e65 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fe97bd9688d in clone () from /lib64/libc.so.6 Thread 38 (Thread 0x7fe927fff700 (LWP 96554)): #0 0x00007fe97e235da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fe97d971e51 in background_thread_sleep (tsdn=, interval=10000000000, info=) at src/background_thread.c:255 #2 background_work_sleep_once (ind=, info=, tsdn=) at src/background_thread.c:307 #3 background_work (ind=, tsd=0x7fe927fee228) at src/background_thread.c:497 #4 background_thread_entry (ind_arg=) at src/background_thread.c:522 #5 0x00007fe97e231e65 in start_thread () from /lib64/libpthread.so.0 #6 0x00007fe97bd9688d in clone () from /lib64/libc.so.6 Thread 37 (Thread 0x7fe9277fe700 (LWP 96555)): #0 0x00007fe97bd90ba9 in syscall () from /lib64/libc.so.6 #1 0x0000000000aea3bd in nativeFutexWaitImpl ( waitMask=waitMask@entry=662619656, absSteadyTime=absSteadyTime@entry=0x7fe9277ec5e8, absSystemTime=absSystemTime@entry=0x0, expected=expected@entry=2, addr=addr@entry=0x7fe9167db270) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:123 #2 folly::detail::futexWaitImpl (futex=futex@entry=0x7fe9167db270, expected=expected@entry=2, absSystemTime=absSystemTime@entry=0x0, absSteadyTime=absSteadyTime@entry=0x7fe9277ec608, waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:254 #3 0x0000000000af3f5a in futexWaitImpl const, std::chrono::time_point > > > (waitMask=4294967295, deadline=..., expected=2, futex=0x7fe9167db270) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:85 #4 folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > > ( futex=futex@entry=0x7fe9167db270, expected=expected@entry=2, deadline=..., waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:123 #5 0x0000000000af9dbd in futexWaitUntil, std::chrono::time_point > > > (timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, deadline=..., expected=2, fut=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:164 #6 folly::SaturatingSemaphore::tryWaitSlow > > ( this=this@entry=0x7fe9167db270, deadline=..., opt=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:318 #7 0x0000000000afa426 in try_wait_until > > (opt=..., deadline=..., this=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:179 #8 folly::detail::LifoSemBase, std::atomic>::try_wait_until > > (this=this@entry=0x7fe92c000a40, deadline=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:521 #9 0x0000000000afdb80 in try_wait_for > ( timeout=, this=0x7fe92c000a40) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:498 #10 folly::UnboundedBlockingQueue::try_take_for (this=0x7fe92c000a00, time=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/task_queue/UnboundedBlockingQueue.h:41 #11 0x0000000000af20f7 in folly::CPUThreadPoolExecutor::threadRun ( this=0x7fe976b19c00, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:243 #12 0x0000000000b0d369 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> ( __t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #13 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #14 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #15 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #16 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #17 0x0000000000699d4d in operator() (this=0x7fe96c0aa9c0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/folly_executorpool.cc:48 #18 operator() (__closure=0x7fe96c0aa9c0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/folly_executorpool.cc:48 #19 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #20 0x00007fe97c64dd40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #21 0x00007fe97e231e65 in start_thread () from /lib64/libpthread.so.0 #22 0x00007fe97bd9688d in clone () from /lib64/libc.so.6 Thread 36 (Thread 0x7fe926ffd700 (LWP 96556)): #0 0x00007fe97bd90ba9 in syscall () from /lib64/libc.so.6 #1 0x0000000000aea3bd in nativeFutexWaitImpl ( waitMask=waitMask@entry=654226952, absSteadyTime=absSteadyTime@entry=0x7fe926feb5e8, absSystemTime=absSystemTime@entry=0x0, expected=expected@entry=2, addr=addr@entry=0x7fe9167db150) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:123 #2 folly::detail::futexWaitImpl (futex=futex@entry=0x7fe9167db150, expected=expected@entry=2, absSystemTime=absSystemTime@entry=0x0, absSteadyTime=absSteadyTime@entry=0x7fe926feb608, waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:254 #3 0x0000000000af3f5a in futexWaitImpl const, std::chrono::time_point > > > (waitMask=4294967295, deadline=..., expected=2, futex=0x7fe9167db150) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:85 #4 folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > > ( futex=futex@entry=0x7fe9167db150, expected=expected@entry=2, deadline=..., waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:123 #5 0x0000000000af9dbd in futexWaitUntil, std::chrono::time_point > > > (timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, deadline=..., expected=2, fut=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:164 #6 folly::SaturatingSemaphore::tryWaitSlow > > ( this=this@entry=0x7fe9167db150, deadline=..., opt=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:318 #7 0x0000000000afa426 in try_wait_until > > (opt=..., deadline=..., this=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:179 #8 folly::detail::LifoSemBase, std::atomic>::try_wait_until > > (this=this@entry=0x7fe92c000a40, deadline=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:521 #9 0x0000000000afdb80 in try_wait_for > ( timeout=, this=0x7fe92c000a40) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:498 #10 folly::UnboundedBlockingQueue::try_take_for (this=0x7fe92c000a00, time=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/task_queue/UnboundedBlockingQueue.h:41 #11 0x0000000000af20f7 in folly::CPUThreadPoolExecutor::threadRun ( this=0x7fe976b19c00, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:243 #12 0x0000000000b0d369 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> ( __t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #13 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #14 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #15 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #16 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #17 0x0000000000699d4d in operator() (this=0x7fe96c0aaac0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/folly_executorpool.cc:48 #18 operator() (__closure=0x7fe96c0aaac0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/folly_executorpool.cc:48 #19 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #20 0x00007fe97c64dd40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #21 0x00007fe97e231e65 in start_thread () from /lib64/libpthread.so.0 #22 0x00007fe97bd9688d in clone () from /lib64/libc.so.6 Thread 35 (Thread 0x7fe9267fc700 (LWP 96557)): #0 0x00007fe97bd90ba9 in syscall () from /lib64/libc.so.6 #1 0x0000000000aea3bd in nativeFutexWaitImpl ( waitMask=waitMask@entry=645834248, absSteadyTime=absSteadyTime@entry=0x7fe9267ea5e8, absSystemTime=absSystemTime@entry=0x0, expected=expected@entry=2, addr=addr@entry=0x7fe9167db1c8) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:123 #2 folly::detail::futexWaitImpl (futex=futex@entry=0x7fe9167db1c8, expected=expected@entry=2, absSystemTime=absSystemTime@entry=0x0, absSteadyTime=absSteadyTime@entry=0x7fe9267ea608, waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:254 #3 0x0000000000af3f5a in futexWaitImpl const, std::chrono::time_point > > > (waitMask=4294967295, deadline=..., expected=2, futex=0x7fe9167db1c8) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:85 #4 folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > > ( futex=futex@entry=0x7fe9167db1c8, expected=expected@entry=2, deadline=..., waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:123 #5 0x0000000000af9dbd in futexWaitUntil, std::chrono::time_point > > > (timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, deadline=..., expected=2, fut=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:164 #6 folly::SaturatingSemaphore::tryWaitSlow > > ( this=this@entry=0x7fe9167db1c8, deadline=..., opt=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:318 #7 0x0000000000afa426 in try_wait_until > > (opt=..., deadline=..., this=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:179 #8 folly::detail::LifoSemBase, std::atomic>::try_wait_until > > (this=this@entry=0x7fe92c000a40, deadline=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:521 #9 0x0000000000afdb80 in try_wait_for > ( timeout=, this=0x7fe92c000a40) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:498 #10 folly::UnboundedBlockingQueue::try_take_for (this=0x7fe92c000a00, time=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/task_queue/UnboundedBlockingQueue.h:41 #11 0x0000000000af20f7 in folly::CPUThreadPoolExecutor::threadRun ( this=0x7fe976b19c00, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:243 #12 0x0000000000b0d369 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> ( __t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #13 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #14 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #15 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #16 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #17 0x0000000000699d4d in operator() (this=0x7fe96c0aab00) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/folly_executorpool.cc:48 #18 operator() (__closure=0x7fe96c0aab00) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/folly_executorpool.cc:48 #19 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #20 0x00007fe97c64dd40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #21 0x00007fe97e231e65 in start_thread () from /lib64/libpthread.so.0 #22 0x00007fe97bd9688d in clone () from /lib64/libc.so.6 Thread 34 (Thread 0x7fe925ffb700 (LWP 96558)): #0 0x00007fe97bd90ba9 in syscall () from /lib64/libc.so.6 #1 0x0000000000aea3bd in nativeFutexWaitImpl ( waitMask=waitMask@entry=637441544, absSteadyTime=absSteadyTime@entry=0x7fe925fe95e8, absSystemTime=absSystemTime@entry=0x0, expected=expected@entry=2, addr=addr@entry=0x7fe9167db228) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:123 #2 folly::detail::futexWaitImpl (futex=futex@entry=0x7fe9167db228, expected=expected@entry=2, absSystemTime=absSystemTime@entry=0x0, absSteadyTime=absSteadyTime@entry=0x7fe925fe9608, waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:254 #3 0x0000000000af3f5a in futexWaitImpl const, std::chrono::time_point > > > (waitMask=4294967295, deadline=..., expected=2, futex=0x7fe9167db228) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:85 #4 folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > > ( futex=futex@entry=0x7fe9167db228, expected=expected@entry=2, deadline=..., waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:123 #5 0x0000000000af9dbd in futexWaitUntil, std::chrono::time_point > > > (timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, deadline=..., expected=2, fut=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:164 #6 folly::SaturatingSemaphore::tryWaitSlow > > ( this=this@entry=0x7fe9167db228, deadline=..., opt=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:318 #7 0x0000000000afa426 in try_wait_until > > (opt=..., deadline=..., this=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:179 #8 folly::detail::LifoSemBase, std::atomic>::try_wait_until > > (this=this@entry=0x7fe92c000a40, deadline=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:521 #9 0x0000000000afdb80 in try_wait_for > ( timeout=, this=0x7fe92c000a40) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:498 #10 folly::UnboundedBlockingQueue::try_take_for (this=0x7fe92c000a00, time=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/task_queue/UnboundedBlockingQueue.h:41 #11 0x0000000000af20f7 in folly::CPUThreadPoolExecutor::threadRun ( this=0x7fe976b19c00, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:243 #12 0x0000000000b0d369 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> ( __t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #13 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #14 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #15 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #16 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #17 0x0000000000699d4d in operator() (this=0x7fe96c0aab80) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/folly_executorpool.cc:48 #18 operator() (__closure=0x7fe96c0aab80) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/folly_executorpool.cc:48 #19 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #20 0x00007fe97c64dd40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #21 0x00007fe97e231e65 in start_thread () from /lib64/libpthread.so.0 #22 0x00007fe97bd9688d in clone () from /lib64/libc.so.6 Thread 33 (Thread 0x7fe9257fa700 (LWP 96559)): #0 0x00007fe97bd90ba9 in syscall () from /lib64/libc.so.6 #1 0x0000000000aea3bd in nativeFutexWaitImpl ( waitMask=waitMask@entry=629048840, absSteadyTime=absSteadyTime@entry=0x7fe9257e85e8, absSystemTime=absSystemTime@entry=0x0, expected=expected@entry=2, addr=addr@entry=0x7fe9167db258) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:123 #2 folly::detail::futexWaitImpl (futex=futex@entry=0x7fe9167db258, expected=expected@entry=2, absSystemTime=absSystemTime@entry=0x0, absSteadyTime=absSteadyTime@entry=0x7fe9257e8608, waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:254 #3 0x0000000000af3f5a in futexWaitImpl const, std::chrono::time_point > > > (waitMask=4294967295, deadline=..., expected=2, futex=0x7fe9167db258) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:85 #4 folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > > ( futex=futex@entry=0x7fe9167db258, expected=expected@entry=2, deadline=..., waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:123 #5 0x0000000000af9dbd in futexWaitUntil, std::chrono::time_point > > > (timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, deadline=..., expected=2, fut=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:164 #6 folly::SaturatingSemaphore::tryWaitSlow > > ( this=this@entry=0x7fe9167db258, deadline=..., opt=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:318 #7 0x0000000000afa426 in try_wait_until > > (opt=..., deadline=..., this=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:179 #8 folly::detail::LifoSemBase, std::atomic>::try_wait_until > > (this=this@entry=0x7fe92c000a40, deadline=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:521 #9 0x0000000000afdb80 in try_wait_for > ( timeout=, this=0x7fe92c000a40) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:498 #10 folly::UnboundedBlockingQueue::try_take_for (this=0x7fe92c000a00, time=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/task_queue/UnboundedBlockingQueue.h:41 #11 0x0000000000af20f7 in folly::CPUThreadPoolExecutor::threadRun ( this=0x7fe976b19c00, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:243 #12 0x0000000000b0d369 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> ( __t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #13 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #14 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #15 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #16 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #17 0x0000000000699d4d in operator() (this=0x7fe96c0aabc0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/folly_executorpool.cc:48 #18 operator() (__closure=0x7fe96c0aabc0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/folly_executorpool.cc:48 #19 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #20 0x00007fe97c64dd40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #21 0x00007fe97e231e65 in start_thread () from /lib64/libpthread.so.0 #22 0x00007fe97bd9688d in clone () from /lib64/libc.so.6 Thread 32 (Thread 0x7fe924ff9700 (LWP 96560)): #0 0x00007fe97bd90ba9 in syscall () from /lib64/libc.so.6 #1 0x0000000000aea3bd in nativeFutexWaitImpl ( waitMask=waitMask@entry=620656136, absSteadyTime=absSteadyTime@entry=0x7fe924fe75e8, absSystemTime=absSystemTime@entry=0x0, expected=expected@entry=2, addr=addr@entry=0x7fe9167db240) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:123 #2 folly::detail::futexWaitImpl (futex=futex@entry=0x7fe9167db240, expected=expected@entry=2, absSystemTime=absSystemTime@entry=0x0, absSteadyTime=absSteadyTime@entry=0x7fe924fe7608, waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:254 #3 0x0000000000af3f5a in futexWaitImpl const, std::chrono::time_point > > > (waitMask=4294967295, deadline=..., expected=2, futex=0x7fe9167db240) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:85 #4 folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > > ( futex=futex@entry=0x7fe9167db240, expected=expected@entry=2, deadline=..., waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:123 #5 0x0000000000af9dbd in futexWaitUntil, std::chrono::time_point > > > (timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, deadline=..., expected=2, fut=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:164 #6 folly::SaturatingSemaphore::tryWaitSlow > > ( this=this@entry=0x7fe9167db240, deadline=..., opt=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:318 #7 0x0000000000afa426 in try_wait_until > > (opt=..., deadline=..., this=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:179 #8 folly::detail::LifoSemBase, std::atomic>::try_wait_until > > (this=this@entry=0x7fe92c000a40, deadline=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:521 #9 0x0000000000afdb80 in try_wait_for > ( timeout=, this=0x7fe92c000a40) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:498 #10 folly::UnboundedBlockingQueue::try_take_for (this=0x7fe92c000a00, time=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/task_queue/UnboundedBlockingQueue.h:41 #11 0x0000000000af20f7 in folly::CPUThreadPoolExecutor::threadRun ( this=0x7fe976b19c00, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:243 #12 0x0000000000b0d369 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> ( __t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #13 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #14 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #15 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #16 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #17 0x0000000000699d4d in operator() (this=0x7fe96c0aab40) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/folly_executorpool.cc:48 #18 operator() (__closure=0x7fe96c0aab40) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/folly_executorpool.cc:48 #19 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #20 0x00007fe97c64dd40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #21 0x00007fe97e231e65 in start_thread () from /lib64/libpthread.so.0 #22 0x00007fe97bd9688d in clone () from /lib64/libc.so.6 Thread 31 (Thread 0x7fe9167da700 (LWP 96561)): #0 0x00007fe97bd90ba9 in syscall () from /lib64/libc.so.6 #1 0x0000000000aea3bd in nativeFutexWaitImpl ( waitMask=waitMask@entry=377259528, absSteadyTime=absSteadyTime@entry=0x7fe9167c85e8, absSystemTime=absSystemTime@entry=0x0, expected=expected@entry=2, addr=addr@entry=0x7fe9167db030) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:123 #2 folly::detail::futexWaitImpl (futex=futex@entry=0x7fe9167db030, expected=expected@entry=2, absSystemTime=absSystemTime@entry=0x0, absSteadyTime=absSteadyTime@entry=0x7fe9167c8608, waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:254 #3 0x0000000000af3f5a in futexWaitImpl const, std::chrono::time_point > > > (waitMask=4294967295, deadline=..., expected=2, futex=0x7fe9167db030) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:85 #4 folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > > ( futex=futex@entry=0x7fe9167db030, expected=expected@entry=2, deadline=..., waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:123 #5 0x0000000000af9dbd in futexWaitUntil, std::chrono::time_point > > > (timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, deadline=..., expected=2, fut=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:164 #6 folly::SaturatingSemaphore::tryWaitSlow > > ( this=this@entry=0x7fe9167db030, deadline=..., opt=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:318 #7 0x0000000000afa426 in try_wait_until > > (opt=..., deadline=..., this=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:179 #8 folly::detail::LifoSemBase, std::atomic>::try_wait_until > > (this=this@entry=0x7fe92c000a40, deadline=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:521 #9 0x0000000000afdb80 in try_wait_for > ( timeout=, this=0x7fe92c000a40) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:498 #10 folly::UnboundedBlockingQueue::try_take_for (this=0x7fe92c000a00, time=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/task_queue/UnboundedBlockingQueue.h:41 #11 0x0000000000af20f7 in folly::CPUThreadPoolExecutor::threadRun ( this=0x7fe976b19c00, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:243 #12 0x0000000000b0d369 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> ( __t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #13 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #14 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #15 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #16 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #17 0x0000000000699d4d in operator() (this=0x7fe96c0aac00) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/folly_executorpool.cc:48 #18 operator() (__closure=0x7fe96c0aac00) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/folly_executorpool.cc:48 #19 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #20 0x00007fe97c64dd40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #21 0x00007fe97e231e65 in start_thread () from /lib64/libpthread.so.0 #22 0x00007fe97bd9688d in clone () from /lib64/libc.so.6 Thread 30 (Thread 0x7fe915fd9700 (LWP 96562)): #0 0x00007fe97bd90ba9 in syscall () from /lib64/libc.so.6 #1 0x0000000000aea3bd in nativeFutexWaitImpl ( waitMask=waitMask@entry=368866824, absSteadyTime=absSteadyTime@entry=0x7fe915fc75e8, absSystemTime=absSystemTime@entry=0x0, expected=expected@entry=2, addr=addr@entry=0x7fe9167db198) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:123 #2 folly::detail::futexWaitImpl (futex=futex@entry=0x7fe9167db198, expected=expected@entry=2, absSystemTime=absSystemTime@entry=0x0, absSteadyTime=absSteadyTime@entry=0x7fe915fc7608, waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:254 #3 0x0000000000af3f5a in futexWaitImpl const, std::chrono::time_point > > > (waitMask=4294967295, deadline=..., expected=2, futex=0x7fe9167db198) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:85 #4 folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > > ( futex=futex@entry=0x7fe9167db198, expected=expected@entry=2, deadline=..., waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:123 #5 0x0000000000af9dbd in futexWaitUntil, std::chrono::time_point > > > (timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, deadline=..., expected=2, fut=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:164 #6 folly::SaturatingSemaphore::tryWaitSlow > > ( this=this@entry=0x7fe9167db198, deadline=..., opt=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:318 #7 0x0000000000afa426 in try_wait_until > > (opt=..., deadline=..., this=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:179 #8 folly::detail::LifoSemBase, std::atomic>::try_wait_until > > (this=this@entry=0x7fe92c000a40, deadline=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:521 #9 0x0000000000afdb80 in try_wait_for > ( timeout=, this=0x7fe92c000a40) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:498 #10 folly::UnboundedBlockingQueue::try_take_for (this=0x7fe92c000a00, time=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/task_queue/UnboundedBlockingQueue.h:41 #11 0x0000000000af20f7 in folly::CPUThreadPoolExecutor::threadRun ( this=0x7fe976b19c00, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:243 #12 0x0000000000b0d369 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> ( __t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #13 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #14 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #15 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #16 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #17 0x0000000000699d4d in operator() (this=0x7fe96c0aac40) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/folly_executorpool.cc:48 #18 operator() (__closure=0x7fe96c0aac40) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/folly_executorpool.cc:48 #19 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #20 0x00007fe97c64dd40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #21 0x00007fe97e231e65 in start_thread () from /lib64/libpthread.so.0 #22 0x00007fe97bd9688d in clone () from /lib64/libc.so.6 Thread 29 (Thread 0x7fe9157d8700 (LWP 96563)): #0 0x00007fe97bd90ba9 in syscall () from /lib64/libc.so.6 #1 0x0000000000aea3bd in nativeFutexWaitImpl ( waitMask=waitMask@entry=360474120, absSteadyTime=absSteadyTime@entry=0x7fe9157c65e8, absSystemTime=absSystemTime@entry=0x0, expected=expected@entry=2, addr=addr@entry=0x7fe9167db0d8) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:123 #2 folly::detail::futexWaitImpl (futex=futex@entry=0x7fe9167db0d8, expected=expected@entry=2, absSystemTime=absSystemTime@entry=0x0, absSteadyTime=absSteadyTime@entry=0x7fe9157c6608, waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:254 #3 0x0000000000af3f5a in futexWaitImpl const, std::chrono::time_point > > > (waitMask=4294967295, deadline=..., expected=2, futex=0x7fe9167db0d8) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:85 #4 folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > > ( futex=futex@entry=0x7fe9167db0d8, expected=expected@entry=2, deadline=..., waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:123 #5 0x0000000000af9dbd in futexWaitUntil, std::chrono::time_point > > > (timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, deadline=..., expected=2, fut=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:164 #6 folly::SaturatingSemaphore::tryWaitSlow > > ( this=this@entry=0x7fe9167db0d8, deadline=..., opt=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:318 #7 0x0000000000afa426 in try_wait_until > > (opt=..., deadline=..., this=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:179 #8 folly::detail::LifoSemBase, std::atomic>::try_wait_until > > (this=this@entry=0x7fe92c002140, deadline=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:521 #9 0x0000000000afdb80 in try_wait_for > ( timeout=, this=0x7fe92c002140) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:498 #10 folly::UnboundedBlockingQueue::try_take_for (this=0x7fe92c002100, time=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/task_queue/UnboundedBlockingQueue.h:41 #11 0x0000000000af20f7 in folly::CPUThreadPoolExecutor::threadRun ( this=0x7fe976b1a800, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:243 #12 0x0000000000b0d369 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> ( __t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #13 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #14 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #15 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #16 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #17 0x0000000000699d4d in operator() (this=0x7fe96c0aad80) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/folly_executorpool.cc:48 #18 operator() (__closure=0x7fe96c0aad80) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/folly_executorpool.cc:48 #19 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #20 0x00007fe97c64dd40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #21 0x00007fe97e231e65 in start_thread () from /lib64/libpthread.so.0 #22 0x00007fe97bd9688d in clone () from /lib64/libc.so.6 Thread 28 (Thread 0x7fe914fd7700 (LWP 96564)): #0 0x00007fe97bd90ba9 in syscall () from /lib64/libc.so.6 #1 0x0000000000aea3bd in nativeFutexWaitImpl ( waitMask=waitMask@entry=352081416, absSteadyTime=absSteadyTime@entry=0x7fe914fc55e8, absSystemTime=absSystemTime@entry=0x0, expected=expected@entry=2, addr=addr@entry=0x7fe9167db0f0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:123 #2 folly::detail::futexWaitImpl (futex=futex@entry=0x7fe9167db0f0, expected=expected@entry=2, absSystemTime=absSystemTime@entry=0x0, absSteadyTime=absSteadyTime@entry=0x7fe914fc5608, waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:254 #3 0x0000000000af3f5a in futexWaitImpl const, std::chrono::time_point > > > (waitMask=4294967295, deadline=..., expected=2, futex=0x7fe9167db0f0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:85 #4 folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > > ( futex=futex@entry=0x7fe9167db0f0, expected=expected@entry=2, deadline=..., waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:123 #5 0x0000000000af9dbd in futexWaitUntil, std::chrono::time_point > > > (timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, deadline=..., expected=2, fut=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:164 #6 folly::SaturatingSemaphore::tryWaitSlow > > ( this=this@entry=0x7fe9167db0f0, deadline=..., opt=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:318 #7 0x0000000000afa426 in try_wait_until > > (opt=..., deadline=..., this=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:179 #8 folly::detail::LifoSemBase, std::atomic>::try_wait_until > > (this=this@entry=0x7fe92c002140, deadline=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:521 #9 0x0000000000afdb80 in try_wait_for > ( timeout=, this=0x7fe92c002140) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:498 #10 folly::UnboundedBlockingQueue::try_take_for (this=0x7fe92c002100, time=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/task_queue/UnboundedBlockingQueue.h:41 #11 0x0000000000af20f7 in folly::CPUThreadPoolExecutor::threadRun ( this=0x7fe976b1a800, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:243 #12 0x0000000000b0d369 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> ( __t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #13 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #14 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #15 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #16 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #17 0x0000000000699d4d in operator() (this=0x7fe96c0aadc0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/folly_executorpool.cc:48 #18 operator() (__closure=0x7fe96c0aadc0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/folly_executorpool.cc:48 #19 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #20 0x00007fe97c64dd40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #21 0x00007fe97e231e65 in start_thread () from /lib64/libpthread.so.0 #22 0x00007fe97bd9688d in clone () from /lib64/libc.so.6 Thread 27 (Thread 0x7fe8f7fff700 (LWP 96565)): #0 0x00007fe97bd90ba9 in syscall () from /lib64/libc.so.6 #1 0x0000000000aea3bd in nativeFutexWaitImpl ( waitMask=waitMask@entry=4160673288, absSteadyTime=absSteadyTime@entry=0x7fe8f7fed5e8, absSystemTime=absSystemTime@entry=0x0, expected=expected@entry=2, addr=addr@entry=0x7fe9167db5a0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:123 #2 folly::detail::futexWaitImpl (futex=futex@entry=0x7fe9167db5a0, expected=expected@entry=2, absSystemTime=absSystemTime@entry=0x0, absSteadyTime=absSteadyTime@entry=0x7fe8f7fed608, waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:254 #3 0x0000000000af3f5a in futexWaitImpl const, std::chrono::time_point > > > (waitMask=4294967295, deadline=..., expected=2, futex=0x7fe9167db5a0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:85 #4 folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > > ( futex=futex@entry=0x7fe9167db5a0, expected=expected@entry=2, deadline=..., waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:123 #5 0x0000000000af9dbd in futexWaitUntil, std::chrono::time_point > > > (timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, deadline=..., expected=2, fut=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:164 #6 folly::SaturatingSemaphore::tryWaitSlow > > ( this=this@entry=0x7fe9167db5a0, deadline=..., opt=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:318 #7 0x0000000000afa426 in try_wait_until > > (opt=..., deadline=..., this=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:179 #8 folly::detail::LifoSemBase, std::atomic>::try_wait_until > > (this=this@entry=0x7fe92c002140, deadline=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:521 #9 0x0000000000afdb80 in try_wait_for > ( timeout=, this=0x7fe92c002140) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:498 #10 folly::UnboundedBlockingQueue::try_take_for (this=0x7fe92c002100, time=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/task_queue/UnboundedBlockingQueue.h:41 #11 0x0000000000af20f7 in folly::CPUThreadPoolExecutor::threadRun ( this=0x7fe976b1a800, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:243 #12 0x0000000000b0d369 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> ( __t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #13 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #14 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #15 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #16 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #17 0x0000000000699d4d in operator() (this=0x7fe96c0aae00) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/folly_executorpool.cc:48 #18 operator() (__closure=0x7fe96c0aae00) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/folly_executorpool.cc:48 #19 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #20 0x00007fe97c64dd40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #21 0x00007fe97e231e65 in start_thread () from /lib64/libpthread.so.0 #22 0x00007fe97bd9688d in clone () from /lib64/libc.so.6 Thread 26 (Thread 0x7fe8f77fe700 (LWP 96566)): #0 0x00007fe97bd90ba9 in syscall () from /lib64/libc.so.6 #1 0x0000000000aea3bd in nativeFutexWaitImpl ( waitMask=waitMask@entry=4152280584, absSteadyTime=absSteadyTime@entry=0x7fe8f77ec5e8, absSystemTime=absSystemTime@entry=0x0, expected=expected@entry=2, addr=addr@entry=0x7fe9167db690) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:123 #2 folly::detail::futexWaitImpl (futex=futex@entry=0x7fe9167db690, expected=expected@entry=2, absSystemTime=absSystemTime@entry=0x0, absSteadyTime=absSteadyTime@entry=0x7fe8f77ec608, waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:254 #3 0x0000000000af3f5a in futexWaitImpl const, std::chrono::time_point > > > (waitMask=4294967295, deadline=..., expected=2, futex=0x7fe9167db690) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:85 #4 folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > > ( futex=futex@entry=0x7fe9167db690, expected=expected@entry=2, deadline=..., waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:123 #5 0x0000000000af9dbd in futexWaitUntil, std::chrono::time_point > > > (timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, deadline=..., expected=2, fut=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:164 #6 folly::SaturatingSemaphore::tryWaitSlow > > ( this=this@entry=0x7fe9167db690, deadline=..., opt=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:318 #7 0x0000000000afa426 in try_wait_until > > (opt=..., deadline=..., this=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:179 #8 folly::detail::LifoSemBase, std::atomic>::try_wait_until > > (this=this@entry=0x7fe92c002140, deadline=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:521 #9 0x0000000000afdb80 in try_wait_for > ( timeout=, this=0x7fe92c002140) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:498 #10 folly::UnboundedBlockingQueue::try_take_for (this=0x7fe92c002100, time=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/task_queue/UnboundedBlockingQueue.h:41 #11 0x0000000000af20f7 in folly::CPUThreadPoolExecutor::threadRun ( this=0x7fe976b1a800, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:243 #12 0x0000000000b0d369 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> ( __t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #13 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #14 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #15 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #16 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #17 0x0000000000699d4d in operator() (this=0x7fe96c0aae80) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/folly_executorpool.cc:48 #18 operator() (__closure=0x7fe96c0aae80) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/folly_executorpool.cc:48 #19 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #20 0x00007fe97c64dd40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #21 0x00007fe97e231e65 in start_thread () from /lib64/libpthread.so.0 #22 0x00007fe97bd9688d in clone () from /lib64/libc.so.6 Thread 25 (Thread 0x7fe8f6ffd700 (LWP 96567)): #0 0x00007fe97bd90ba9 in syscall () from /lib64/libc.so.6 #1 0x0000000000aea3bd in nativeFutexWaitImpl ( waitMask=waitMask@entry=4143887880, absSteadyTime=absSteadyTime@entry=0x7fe8f6feb5e8, absSystemTime=absSystemTime@entry=0x0, expected=expected@entry=2, addr=addr@entry=0x7fe9167db138) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:123 #2 folly::detail::futexWaitImpl (futex=futex@entry=0x7fe9167db138, expected=expected@entry=2, absSystemTime=absSystemTime@entry=0x0, absSteadyTime=absSteadyTime@entry=0x7fe8f6feb608, waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:254 #3 0x0000000000af3f5a in futexWaitImpl const, std::chrono::time_point > > > (waitMask=4294967295, deadline=..., expected=2, futex=0x7fe9167db138) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:85 #4 folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > > ( futex=futex@entry=0x7fe9167db138, expected=expected@entry=2, deadline=..., waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:123 #5 0x0000000000af9dbd in futexWaitUntil, std::chrono::time_point > > > (timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, deadline=..., expected=2, fut=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:164 #6 folly::SaturatingSemaphore::tryWaitSlow > > ( this=this@entry=0x7fe9167db138, deadline=..., opt=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:318 #7 0x0000000000afa426 in try_wait_until > > (opt=..., deadline=..., this=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:179 #8 folly::detail::LifoSemBase, std::atomic>::try_wait_until > > (this=this@entry=0x7fe92c002e40, deadline=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:521 #9 0x0000000000afdb80 in try_wait_for > ( timeout=, this=0x7fe92c002e40) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:498 #10 folly::UnboundedBlockingQueue::try_take_for (this=0x7fe92c002e00, time=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/task_queue/UnboundedBlockingQueue.h:41 #11 0x0000000000af20f7 in folly::CPUThreadPoolExecutor::threadRun ( this=0x7fe976b1b400, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:243 #12 0x0000000000b0d369 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> ( __t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #13 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #14 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #15 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #16 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #17 0x0000000000699d4d in operator() (this=0x7fe96c0aaf40) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/folly_executorpool.cc:48 #18 operator() (__closure=0x7fe96c0aaf40) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/folly_executorpool.cc:48 #19 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #20 0x00007fe97c64dd40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #21 0x00007fe97e231e65 in start_thread () from /lib64/libpthread.so.0 #22 0x00007fe97bd9688d in clone () from /lib64/libc.so.6 Thread 24 (Thread 0x7fe8f67fc700 (LWP 96568)): #0 0x00007fe97bd90ba9 in syscall () from /lib64/libc.so.6 #1 0x0000000000aea3bd in nativeFutexWaitImpl ( waitMask=waitMask@entry=4135495176, absSteadyTime=absSteadyTime@entry=0x7fe8f67ea5e8, absSystemTime=absSystemTime@entry=0x0, expected=expected@entry=2, addr=addr@entry=0x7fe9167db600) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:123 #2 folly::detail::futexWaitImpl (futex=futex@entry=0x7fe9167db600, expected=expected@entry=2, absSystemTime=absSystemTime@entry=0x0, absSteadyTime=absSteadyTime@entry=0x7fe8f67ea608, waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:254 #3 0x0000000000af3f5a in futexWaitImpl const, std::chrono::time_point > > > (waitMask=4294967295, deadline=..., expected=2, futex=0x7fe9167db600) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:85 #4 folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > > ( futex=futex@entry=0x7fe9167db600, expected=expected@entry=2, deadline=..., waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:123 #5 0x0000000000af9ef7 in futexWaitPreIdle, std::chrono::time_point > >, std::chrono::duration > > (timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, deadline=..., expected=2, fut=..., _ret=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:194 #6 futexWaitUntil, std::chrono::time_point > > > (timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, deadline=..., expected=2, fut=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:151 #7 folly::SaturatingSemaphore::tryWaitSlow > > ( this=this@entry=0x7fe9167db600, deadline=..., opt=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:318 #8 0x0000000000afa426 in try_wait_until > > (opt=..., deadline=..., this=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:179 #9 folly::detail::LifoSemBase, std::atomic>::try_wait_until > > (this=this@entry=0x7fe92c003340, deadline=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:521 #10 0x0000000000afdb80 in try_wait_for > ( timeout=, this=0x7fe92c003340) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:498 #11 folly::UnboundedBlockingQueue::try_take_for (this=0x7fe92c003300, time=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/task_queue/UnboundedBlockingQueue.h:41 #12 0x0000000000af20f7 in folly::CPUThreadPoolExecutor::threadRun ( this=0x7fe96c0ef000, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:243 #13 0x0000000000b0d369 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> ( __t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #14 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #15 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #16 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #17 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #18 0x0000000000699d4d in operator() (this=0x7fe96c0c3040) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/folly_executorpool.cc:48 #19 operator() (__closure=0x7fe96c0c3040) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/folly_executorpool.cc:48 #20 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #21 0x00007fe97c64dd40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #22 0x00007fe97e231e65 in start_thread () from /lib64/libpthread.so.0 #23 0x00007fe97bd9688d in clone () from /lib64/libc.so.6 Thread 23 (Thread 0x7fe8f5ffb700 (LWP 96569)): #0 0x00007fe97bd90ba9 in syscall () from /lib64/libc.so.6 #1 0x0000000000aea3bd in nativeFutexWaitImpl ( waitMask=waitMask@entry=4127102472, absSteadyTime=absSteadyTime@entry=0x7fe8f5fe95e8, absSystemTime=absSystemTime@entry=0x0, expected=expected@entry=2, addr=addr@entry=0x7fe9167db4e0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:123 #2 folly::detail::futexWaitImpl (futex=futex@entry=0x7fe9167db4e0, expected=expected@entry=2, absSystemTime=absSystemTime@entry=0x0, absSteadyTime=absSteadyTime@entry=0x7fe8f5fe9608, waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex.cpp:254 #3 0x0000000000af3f5a in futexWaitImpl const, std::chrono::time_point > > > (waitMask=4294967295, deadline=..., expected=2, futex=0x7fe9167db4e0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:85 #4 folly::detail::futexWaitUntil, std::chrono::_V2::steady_clock, std::chrono::duration > > ( futex=futex@entry=0x7fe9167db4e0, expected=expected@entry=2, deadline=..., waitMask=waitMask@entry=4294967295) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/Futex-inl.h:123 #5 0x0000000000af9ef7 in futexWaitPreIdle, std::chrono::time_point > >, std::chrono::duration > > (timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, deadline=..., expected=2, fut=..., _ret=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:194 #6 futexWaitUntil, std::chrono::time_point > > > (timeoutVariationFrac=0.5, stackToRetain=1024, idleTimeout=..., waitMask=4294967295, deadline=..., expected=2, fut=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/detail/MemoryIdler.h:151 #7 folly::SaturatingSemaphore::tryWaitSlow > > ( this=this@entry=0x7fe9167db4e0, deadline=..., opt=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:318 #8 0x0000000000afa426 in try_wait_until > > (opt=..., deadline=..., this=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/SaturatingSemaphore.h:179 #9 folly::detail::LifoSemBase, std::atomic>::try_wait_until > > (this=this@entry=0x7fe92c003340, deadline=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:521 #10 0x0000000000afdb80 in try_wait_for > ( timeout=, this=0x7fe92c003340) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/synchronization/LifoSem.h:498 #11 folly::UnboundedBlockingQueue::try_take_for (this=0x7fe92c003300, time=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/task_queue/UnboundedBlockingQueue.h:41 #12 0x0000000000af20f7 in folly::CPUThreadPoolExecutor::threadRun ( this=0x7fe96c0ef000, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/CPUThreadPoolExecutor.cpp:243 #13 0x0000000000b0d369 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> ( __t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #14 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #15 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #16 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #17 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #18 0x0000000000699d4d in operator() (this=0x7fe96c0c3080) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/folly_executorpool.cc:48 #19 operator() (__closure=0x7fe96c0c3080) at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/engines/ep/src/folly_executorpool.cc:48 #20 folly::detail::function::FunctionTraits::callBig&&)::{lambda()#1}>(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/couchbase-server-unix/server_build/tlm/deps/folly.exploded/include/folly/Function.h:401 #21 0x00007fe97c64dd40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #22 0x00007fe97e231e65 in start_thread () from /lib64/libpthread.so.0 #23 0x00007fe97bd9688d in clone () from /lib64/libc.so.6 Thread 22 (Thread 0x7fe8f57fa700 (LWP 96570)): #0 0x00007fe97e235da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000000000997bcf in __gthread_cond_timedwait ( __abs_timeout=0x7fe8f57e8960, __mutex=, __cond=0x7fe976d2a758) at /opt/gcc-10.2.0/include/c++/10.2.0/x86_64-pc-linux-gnu/bits/gthr-default.h:872 #2 __wait_until_impl > > (__lock=..., __lock=..., __atime=, this=0x7fe976d2a758) at /opt/gcc-10.2.0/include/c++/10.2.0/condition_variable:232 #3 wait_until > > (__atime=..., __lock=..., this=0x7fe976d2a758) at /opt/gcc-10.2.0/include/c++/10.2.0/condition_variable:141 #4 wait_for > (__rtime=..., __rtime=..., __lock=..., this=0x7fe976d2a758) at /opt/gcc-10.2.0/include/c++/10.2.0/condition_variable:172 #5 magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:98 #6 0x0000000000997d21 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:159 #7 0x0000000000aab9e9 in run (this=0x7fe96c005f20) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #8 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #9 0x00007fe97e231e65 in start_thread () from /lib64/libpthread.so.0 #10 0x00007fe97bd9688d in clone () from /lib64/libc.so.6 Thread 21 (Thread 0x7fe8f4ff9700 (LWP 96571)): #0 0x00007fe97e235da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000000000997bcf in __gthread_cond_timedwait ( __abs_timeout=0x7fe8f4fe7960, __mutex=, __cond=0x7fe976d2a758) at /opt/gcc-10.2.0/include/c++/10.2.0/x86_64-pc-linux-gnu/bits/gthr-default.h:872 #2 __wait_until_impl > > (__lock=..., __lock=..., __atime=, this=0x7fe976d2a758) at /opt/gcc-10.2.0/include/c++/10.2.0/condition_variable:232 #3 wait_until > > (__atime=..., __lock=..., this=0x7fe976d2a758) at /opt/gcc-10.2.0/include/c++/10.2.0/condition_variable:141 #4 wait_for > (__rtime=..., __rtime=..., __lock=..., this=0x7fe976d2a758) at /opt/gcc-10.2.0/include/c++/10.2.0/condition_variable:172 #5 magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:98 #6 0x0000000000997d21 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:159 #7 0x0000000000aab9e9 in run (this=0x7fe96c005f50) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #8 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #9 0x00007fe97e231e65 in start_thread () from /lib64/libpthread.so.0 #10 0x00007fe97bd9688d in clone () from /lib64/libc.so.6 Thread 20 (Thread 0x7fe8dffff700 (LWP 96572)): #0 0x00007fe97e235da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000000000997bcf in __gthread_cond_timedwait ( __abs_timeout=0x7fe8dffed960, __mutex=, __cond=0x7fe976d2a758) at /opt/gcc-10.2.0/include/c++/10.2.0/x86_64-pc-linux-gnu/bits/gthr-default.h:872 #2 __wait_until_impl > > (__lock=..., __lock=..., __atime=, this=0x7fe976d2a758) at /opt/gcc-10.2.0/include/c++/10.2.0/condition_variable:232 #3 wait_until > > (__atime=..., __lock=..., this=0x7fe976d2a758) at /opt/gcc-10.2.0/include/c++/10.2.0/condition_variable:141 #4 wait_for > (__rtime=..., __rtime=..., __lock=..., this=0x7fe976d2a758) at /opt/gcc-10.2.0/include/c++/10.2.0/condition_variable:172 #5 magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:98 #6 0x0000000000997d21 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:159 #7 0x0000000000aab9e9 in run (this=0x7fe96c005f80) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #8 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #9 0x00007fe97e231e65 in start_thread () from /lib64/libpthread.so.0 #10 0x00007fe97bd9688d in clone () from /lib64/libc.so.6 Thread 19 (Thread 0x7fe8df7fe700 (LWP 96573)): #0 0x00007fe97e235da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000000000997bcf in __gthread_cond_timedwait ( __abs_timeout=0x7fe8df7ec960, __mutex=, __cond=0x7fe976d2a758) at /opt/gcc-10.2.0/include/c++/10.2.0/x86_64-pc-linux-gnu/bits/gthr-default.h:872 #2 __wait_until_impl > > (__lock=..., __lock=..., __atime=, this=0x7fe976d2a758) at /opt/gcc-10.2.0/include/c++/10.2.0/condition_variable:232 #3 wait_until > > (__atime=..., __lock=..., this=0x7fe976d2a758) at /opt/gcc-10.2.0/include/c++/10.2.0/condition_variable:141 #4 wait_for > (__rtime=..., __rtime=..., __lock=..., this=0x7fe976d2a758) at /opt/gcc-10.2.0/include/c++/10.2.0/condition_variable:172 #5 magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:98 #6 0x0000000000997d21 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:159 #7 0x0000000000aab9e9 in run (this=0x7fe96c005fb0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #8 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #9 0x00007fe97e231e65 in start_thread () from /lib64/libpthread.so.0 #10 0x00007fe97bd9688d in clone () from /lib64/libc.so.6 Thread 18 (Thread 0x7fe8deffd700 (LWP 96574)): #0 0x00007fe97e2359f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fe97c6488bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x0000000000997938 in magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:100 #4 0x0000000000997d21 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:159 #5 0x0000000000aab9e9 in run (this=0x7fe96c005fe0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007fe97e231e65 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fe97bd9688d in clone () from /lib64/libc.so.6 Thread 17 (Thread 0x7fe8de7fc700 (LWP 96575)): #0 0x00007fe97e2359f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fe97c6488bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x0000000000997938 in magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:100 #4 0x0000000000997d21 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:159 #5 0x0000000000aab9e9 in run (this=0x7fe96c006010) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007fe97e231e65 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fe97bd9688d in clone () from /lib64/libc.so.6 Thread 16 (Thread 0x7fe8ddffb700 (LWP 96576)): #0 0x00007fe97e2359f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fe97c6488bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x0000000000997938 in magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:100 #4 0x0000000000997d21 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:159 #5 0x0000000000aab9e9 in run (this=0x7fe96c006040) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007fe97e231e65 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fe97bd9688d in clone () from /lib64/libc.so.6 Thread 15 (Thread 0x7fe8dd7fa700 (LWP 96577)): #0 0x00007fe97e2359f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fe97c6488bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x0000000000997938 in magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:100 #4 0x0000000000997d21 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:159 #5 0x0000000000aab9e9 in run (this=0x7fe96c006070) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007fe97e231e65 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fe97bd9688d in clone () from /lib64/libc.so.6 Thread 14 (Thread 0x7fe8d79ff700 (LWP 96579)): #0 0x00007fe97e2359f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fe97c6488bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x0000000000997938 in magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:100 #4 0x0000000000997d21 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:159 #5 0x0000000000aab9e9 in run (this=0x7fe97abe4030) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007fe97e231e65 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fe97bd9688d in clone () from /lib64/libc.so.6 Thread 13 (Thread 0x7fe8d71fe700 (LWP 96580)): #0 0x00007fe97e2359f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fe97c6488bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x0000000000997938 in magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:100 #4 0x0000000000997d21 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:159 #5 0x0000000000aab9e9 in run (this=0x7fe97abe41e0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007fe97e231e65 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fe97bd9688d in clone () from /lib64/libc.so.6 Thread 12 (Thread 0x7fe8d69fd700 (LWP 96581)): #0 0x00007fe97e2359f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fe97c6488bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x0000000000997938 in magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:100 #4 0x0000000000997d21 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:159 #5 0x0000000000aab9e9 in run (this=0x7fe97abe4210) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007fe97e231e65 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fe97bd9688d in clone () from /lib64/libc.so.6 Thread 11 (Thread 0x7fe8d61fc700 (LWP 96582)): #0 0x00007fe97e2359f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fe97c6488bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x0000000000997938 in magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:100 #4 0x0000000000997d21 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:159 #5 0x0000000000aab9e9 in run (this=0x7fe97abe6430) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007fe97e231e65 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fe97bd9688d in clone () from /lib64/libc.so.6 Thread 10 (Thread 0x7fe8d59fb700 (LWP 96583)): #0 0x00007fe97e2359f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fe97c6488bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x0000000000997938 in magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:100 #4 0x0000000000997d21 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:159 #5 0x0000000000aab9e9 in run (this=0x7fe97abe6580) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007fe97e231e65 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fe97bd9688d in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x7fe8d51fa700 (LWP 96584)): #0 0x00007fe97e2359f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fe97c6488bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x0000000000997938 in magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:100 #4 0x0000000000997d21 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:159 #5 0x0000000000aab9e9 in run (this=0x7fe97abe67f0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007fe97e231e65 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fe97bd9688d in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7fe8d49f9700 (LWP 96585)): #0 0x00007fe97e2359f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fe97c6488bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x0000000000997938 in magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:100 #4 0x0000000000997d21 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:159 #5 0x0000000000aab9e9 in run (this=0x7fe96c0048a0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007fe97e231e65 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fe97bd9688d in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7fe8d41f8700 (LWP 96586)): #0 0x00007fe97e2359f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fe97c6488bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x0000000000997938 in magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:100 #4 0x0000000000997d21 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:159 #5 0x0000000000aab9e9 in run (this=0x7fe96c0048d0) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007fe97e231e65 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fe97bd9688d in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7fe8d39f7700 (LWP 96587)): #0 0x00007fe97e2359f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fe97c6488bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x0000000000997938 in magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:100 #4 0x0000000000997d21 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:159 #5 0x0000000000aab9e9 in run (this=0x7fe96c004930) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007fe97e231e65 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fe97bd9688d in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7fe8d31f6700 (LWP 96588)): #0 0x00007fe97e2359f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fe97c6488bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x0000000000997938 in magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:100 #4 0x0000000000997d21 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:159 #5 0x0000000000aab9e9 in run (this=0x7fe96c004960) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007fe97e231e65 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fe97bd9688d in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7fe8d29f5700 (LWP 96589)): #0 0x00007fe97e2359f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fe97c6488bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x0000000000997938 in magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:100 #4 0x0000000000997d21 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:159 #5 0x0000000000aab9e9 in run (this=0x7fe96c004990) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007fe97e231e65 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fe97bd9688d in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7fe8d21f4700 (LWP 96590)): #0 0x00007fe97e2359f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fe97c6488bc in __gthread_cond_wait (__mutex=, __cond=) at /tmp/deploy/objdir/x86_64-pc-linux-gnu/libstdc++-v3/include/x86_64-pc-linux-gnu/bits/gthr-default.h:865 #2 std::condition_variable::wait (this=, __lock=...) at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/condition_variable.cc:53 #3 0x0000000000997938 in magma::TaskQueue::dequeue(std::atomic&) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:100 #4 0x0000000000997d21 in magma::TaskWorker::loop(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/magma/util/worker.cc:159 #5 0x0000000000aab9e9 in run (this=0x7fe96c004b70) at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:51 #6 platform_thread_wrap(void*) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/src/cb_pthreads.cc:64 #7 0x00007fe97e231e65 in start_thread () from /lib64/libpthread.so.0 #8 0x00007fe97bd9688d in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7fe8cda93700 (LWP 96600)): #0 0x00007fe97bd96e63 in epoll_wait () from /lib64/libc.so.6 #1 0x00007fe97d7393c5 in epoll_dispatch (base=0x7fe8cdaad000, tv=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/epoll.c:465 #2 0x00007fe97d730376 in event_base_loop (base=0x7fe8cdaad000, flags=1) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:1998 #3 0x0000000000b13a97 in folly::EventBase::loopBody ( this=this@entry=0x7fe976b57c00, flags=flags@entry=0, ignoreKeepAlive=ignoreKeepAlive@entry=false) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/io/async/EventBase.cpp:397 #4 0x0000000000b13f66 in folly::EventBase::loop ( this=this@entry=0x7fe976b57c00) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/io/async/EventBase.cpp:315 #5 0x0000000000b15926 in folly::EventBase::loopForever (this=0x7fe976b57c00) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/io/async/EventBase.cpp:538 #6 0x0000000000b01bd9 in folly::IOThreadPoolExecutor::threadRun ( this=0x7fe96c041c80, thread=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/executors/IOThreadPoolExecutor.cpp:195 #7 0x0000000000b0d369 in __invoke_impl), folly::ThreadPoolExecutor*&, std::shared_ptr&> ( __t=, __f=) at /usr/local/include/c++/7.3.0/bits/invoke.h:73 #8 __invoke), folly::ThreadPoolExecutor*&, std::shared_ptr&> (__fn=) at /usr/local/include/c++/7.3.0/bits/invoke.h:95 #9 __call (__args=, this=) at /usr/local/include/c++/7.3.0/functional:467 #10 operator()<> (this=) at /usr/local/include/c++/7.3.0/functional:551 #11 folly::detail::function::FunctionTraits::callBig))(std::shared_ptr)> >(folly::detail::function::Data&) (p=...) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/folly/folly-prefix/src/folly/folly/Function.h:401 #12 0x00007fe97c64dd40 in execute_native_thread_routine () at /tmp/deploy/objdir/../gcc-10.2.0/libstdc++-v3/src/c++11/thread.cc:80 #13 0x00007fe97e231e65 in start_thread () from /lib64/libpthread.so.0 #14 0x00007fe97bd9688d in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7fe97f0ddc40 (LWP 96529)): #0 0x00007fe97bd96e63 in epoll_wait () from /lib64/libc.so.6 #1 0x00007fe97d7393c5 in epoll_dispatch (base=0x7fe97abdf000, tv=) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/epoll.c:465 #2 0x00007fe97d730376 in event_base_loop (base=0x7fe97abdf000, flags=0) at /home/couchbase/jenkins/workspace/cbdeps-platform-build-old/deps/packages/build/libevent/libevent-prefix/src/libevent/event.c:1998 #3 0x000000000053bb26 in memcached_main(int, char**) () at /home/couchbase/jenkins/workspace/couchbase-server-unix/kv_engine/daemon/memcached.cc:1525 #4 0x00007fe97bcba505 in __libc_start_main () from /lib64/libc.so.6 #5 0x0000000000531d94 in _start () at /home/couchbase/jenkins/workspace/couchbase-server-unix/platform/include/platform/compression/allocator.h:68 Detaching from program: /opt/couchbase/bin/memcached, process 96529 2021-06-08 12:11:34,724 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: ls /opt/couchbase/var/lib/couchbase/logs/memcached.log.* 2021-06-08 12:11:34,740 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: grep -r ' CRITICAL ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.001369.txt | grep -v 'Rollback point not found' | grep -v 'No space left on device' | grep -v 'Permission denied' 2021-06-08 12:11:34,796 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: grep -r ' ERROR ' /opt/couchbase/var/lib/couchbase/logs/memcached.log.001369.txt | grep -v 'XERROR' | grep -v 'compaction failed for vb' | grep -v 'failed - Already exists' 2021-06-08 12:11:34,825 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: grep -r 'exception occurred in runloop' /opt/couchbase/var/lib/couchbase/logs/memcached.log.001369.txt 2021-06-08 12:11:34,845 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: grep -r 'Stream request failed because the snap start seqno' /opt/couchbase/var/lib/couchbase/logs/memcached.log.001369.txt 2021-06-08 12:11:34,865 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: ls /opt/couchbase/var/lib/couchbase/logs/sanitizers.log.* 2021-06-08 12:11:34,878 | test | DEBUG | MainThread | [basetestcase:check_coredump_exist:763] 172.23.106.155: No 'sanitizers.log.*' files found 2021-06-08 12:11:34,878 | infra | DEBUG | MainThread | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.155 2021-06-08 12:11:34,920 | test | INFO | MainThread | [cluster_ready_functions:trigger_cb_collect_on_cluster:1094] Running cbcollect on node ns_1@172.23.106.122,ns_1@172.23.106.125,ns_1@172.23.106.155,ns_1@172.23.106.156 2021-06-08 12:11:34,934 | test | DEBUG | MainThread | [rest_client:_http_request:729] {'date': 'Tue, 08 Jun 2021 19:11:34 GMT', 'content-length': '2', 'server': 'Couchbase Server', 'expires': 'Thu, 01 Jan 1970 00:00:00 GMT', 'x-frame-options': 'DENY', 'x-permitted-cross-domain-policies': 'none', 'pragma': 'no-cache', 'x-content-type-options': 'nosniff', 'x-xss-protection': '1; mode=block', 'connection': 'close', 'content-type': 'application/json', 'cache-control': 'no-cache,no-store,must-revalidate', 'status': '200'} 2021-06-08 12:11:34,936 | test | DEBUG | MainThread | [rest_client:_http_request:730] [] 2021-06-08 12:11:34,941 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from couchbase_utils/cluster_utils/cluster_ready_functions.py -> trigger_cb_collect_on_cluster():L1096 2021-06-08 12:11:34,941 | infra | INFO | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: Wait for CB collect to start 2021-06-08 12:11:44,943 | test | INFO | MainThread | [cluster_ready_functions:trigger_cb_collect_on_cluster:1097] ns_1@172.23.106.122,ns_1@172.23.106.125,ns_1@172.23.106.155,ns_1@172.23.106.156 - cbcollect status: True 2021-06-08 12:11:44,943 | test | INFO | MainThread | [cluster_ready_functions:wait_for_cb_collect_to_complete:1102] Polling active_tasks to check cbcollect status 2021-06-08 12:11:44,954 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_cb_collect_to_complete:1109] CBCollectInfo Iteration 0 - running 2021-06-08 12:11:44,963 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from couchbase_utils/cluster_utils/cluster_ready_functions.py -> wait_for_cb_collect_to_complete():L1119 2021-06-08 12:11:44,963 | infra | INFO | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: CB collect still running 2021-06-08 12:11:54,973 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_cb_collect_to_complete:1109] CBCollectInfo Iteration 1 - running 2021-06-08 12:11:54,980 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from couchbase_utils/cluster_utils/cluster_ready_functions.py -> wait_for_cb_collect_to_complete():L1119 2021-06-08 12:11:54,980 | infra | INFO | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: CB collect still running 2021-06-08 12:12:04,990 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_cb_collect_to_complete:1109] CBCollectInfo Iteration 2 - running 2021-06-08 12:12:04,996 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from couchbase_utils/cluster_utils/cluster_ready_functions.py -> wait_for_cb_collect_to_complete():L1119 2021-06-08 12:12:04,996 | infra | INFO | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: CB collect still running 2021-06-08 12:12:15,006 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_cb_collect_to_complete:1109] CBCollectInfo Iteration 3 - running 2021-06-08 12:12:15,013 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from couchbase_utils/cluster_utils/cluster_ready_functions.py -> wait_for_cb_collect_to_complete():L1119 2021-06-08 12:12:15,013 | infra | INFO | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: CB collect still running 2021-06-08 12:12:25,023 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_cb_collect_to_complete:1109] CBCollectInfo Iteration 4 - running 2021-06-08 12:12:25,029 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from couchbase_utils/cluster_utils/cluster_ready_functions.py -> wait_for_cb_collect_to_complete():L1119 2021-06-08 12:12:25,029 | infra | INFO | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: CB collect still running 2021-06-08 12:12:35,039 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_cb_collect_to_complete:1109] CBCollectInfo Iteration 5 - running 2021-06-08 12:12:35,046 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from couchbase_utils/cluster_utils/cluster_ready_functions.py -> wait_for_cb_collect_to_complete():L1119 2021-06-08 12:12:35,046 | infra | INFO | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: CB collect still running 2021-06-08 12:12:45,055 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_cb_collect_to_complete:1109] CBCollectInfo Iteration 6 - running 2021-06-08 12:12:45,062 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from couchbase_utils/cluster_utils/cluster_ready_functions.py -> wait_for_cb_collect_to_complete():L1119 2021-06-08 12:12:45,062 | infra | INFO | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: CB collect still running 2021-06-08 12:12:55,072 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_cb_collect_to_complete:1109] CBCollectInfo Iteration 7 - running 2021-06-08 12:12:55,078 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from couchbase_utils/cluster_utils/cluster_ready_functions.py -> wait_for_cb_collect_to_complete():L1119 2021-06-08 12:12:55,078 | infra | INFO | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: CB collect still running 2021-06-08 12:13:05,088 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_cb_collect_to_complete:1109] CBCollectInfo Iteration 8 - running 2021-06-08 12:13:05,095 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from couchbase_utils/cluster_utils/cluster_ready_functions.py -> wait_for_cb_collect_to_complete():L1119 2021-06-08 12:13:05,095 | infra | INFO | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: CB collect still running 2021-06-08 12:13:15,105 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_cb_collect_to_complete:1109] CBCollectInfo Iteration 9 - running 2021-06-08 12:13:15,112 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from couchbase_utils/cluster_utils/cluster_ready_functions.py -> wait_for_cb_collect_to_complete():L1119 2021-06-08 12:13:15,112 | infra | INFO | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: CB collect still running 2021-06-08 12:13:25,122 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_cb_collect_to_complete:1109] CBCollectInfo Iteration 10 - running 2021-06-08 12:13:25,128 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from couchbase_utils/cluster_utils/cluster_ready_functions.py -> wait_for_cb_collect_to_complete():L1119 2021-06-08 12:13:25,128 | infra | INFO | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: CB collect still running 2021-06-08 12:13:35,140 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_cb_collect_to_complete:1109] CBCollectInfo Iteration 11 - running 2021-06-08 12:13:35,145 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from couchbase_utils/cluster_utils/cluster_ready_functions.py -> wait_for_cb_collect_to_complete():L1119 2021-06-08 12:13:35,145 | infra | INFO | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: CB collect still running 2021-06-08 12:13:45,157 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_cb_collect_to_complete:1109] CBCollectInfo Iteration 12 - running 2021-06-08 12:13:45,163 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from couchbase_utils/cluster_utils/cluster_ready_functions.py -> wait_for_cb_collect_to_complete():L1119 2021-06-08 12:13:45,163 | infra | INFO | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: CB collect still running 2021-06-08 12:13:55,173 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_cb_collect_to_complete:1109] CBCollectInfo Iteration 13 - running 2021-06-08 12:13:55,178 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from couchbase_utils/cluster_utils/cluster_ready_functions.py -> wait_for_cb_collect_to_complete():L1119 2021-06-08 12:13:55,178 | infra | INFO | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: CB collect still running 2021-06-08 12:14:05,188 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_cb_collect_to_complete:1109] CBCollectInfo Iteration 14 - running 2021-06-08 12:14:05,194 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from couchbase_utils/cluster_utils/cluster_ready_functions.py -> wait_for_cb_collect_to_complete():L1119 2021-06-08 12:14:05,196 | infra | INFO | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: CB collect still running 2021-06-08 12:14:15,204 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_cb_collect_to_complete:1109] CBCollectInfo Iteration 15 - running 2021-06-08 12:14:15,213 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from couchbase_utils/cluster_utils/cluster_ready_functions.py -> wait_for_cb_collect_to_complete():L1119 2021-06-08 12:14:15,213 | infra | INFO | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: CB collect still running 2021-06-08 12:14:25,224 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_cb_collect_to_complete:1109] CBCollectInfo Iteration 16 - running 2021-06-08 12:14:25,230 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from couchbase_utils/cluster_utils/cluster_ready_functions.py -> wait_for_cb_collect_to_complete():L1119 2021-06-08 12:14:25,232 | infra | INFO | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: CB collect still running 2021-06-08 12:14:35,242 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_cb_collect_to_complete:1109] CBCollectInfo Iteration 17 - completed 2021-06-08 12:14:35,250 | test | DEBUG | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1125] {u'perNode': {u'ns_1@172.23.106.125': {u'path': u'/opt/couchbase/var/lib/couchbase/tmp/collectinfo-2021-06-08T191134-ns_1@172.23.106.125.zip', u'status': u'collected'}, u'ns_1@172.23.106.156': {u'path': u'/opt/couchbase/var/lib/couchbase/tmp/collectinfo-2021-06-08T191134-ns_1@172.23.106.156.zip', u'status': u'collected'}, u'ns_1@172.23.106.122': {u'path': u'/opt/couchbase/var/lib/couchbase/tmp/collectinfo-2021-06-08T191134-ns_1@172.23.106.122.zip', u'status': u'collected'}, u'ns_1@172.23.106.155': {u'path': u'/opt/couchbase/var/lib/couchbase/tmp/collectinfo-2021-06-08T191134-ns_1@172.23.106.155.zip', u'status': u'collected'}}, u'type': u'clusterLogsCollection', u'node': u'ns_1@172.23.106.156', u'progress': 100, u'ts': u'2021-06-08 19:11:34', u'status': u'completed'} 2021-06-08 12:14:35,252 | test | INFO | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1129] ns_1@172.23.106.122: Copying cbcollect ZIP file to Client 2021-06-08 12:14:35,253 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.122 with username: root, password: couchbase 2021-06-08 12:14:35,335 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 12:14:35,553 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: ls /etc/ 2021-06-08 12:14:35,612 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 12:14:35,612 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: ls /cygdrive/c/Windows 2021-06-08 12:14:35,628 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: uname -m 2021-06-08 12:14:35,642 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: cat /proc/cpuinfo 2021-06-08 12:14:35,658 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: df -Thl 2021-06-08 12:14:35,677 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: cat /proc/meminfo 2021-06-08 12:14:35,693 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: hostname 2021-06-08 12:14:35,704 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: hostname -d 2021-06-08 12:14:36,191 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.122: rm -f /opt/couchbase/var/lib/couchbase/tmp/collectinfo-2021-06-08T191134-ns_1@172.23.106.122.zip 2021-06-08 12:14:36,216 | infra | DEBUG | MainThread | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.122 2021-06-08 12:14:36,216 | test | INFO | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1129] ns_1@172.23.106.125: Copying cbcollect ZIP file to Client 2021-06-08 12:14:36,219 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.125 with username: root, password: couchbase 2021-06-08 12:14:36,305 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 12:14:36,520 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: ls /etc/ 2021-06-08 12:14:36,575 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 12:14:36,575 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: ls /cygdrive/c/Windows 2021-06-08 12:14:36,588 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: uname -m 2021-06-08 12:14:36,598 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: cat /proc/cpuinfo 2021-06-08 12:14:36,609 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: df -Thl 2021-06-08 12:14:36,624 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: cat /proc/meminfo 2021-06-08 12:14:36,634 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: hostname 2021-06-08 12:14:36,647 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: hostname -d 2021-06-08 12:14:37,118 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.125: rm -f /opt/couchbase/var/lib/couchbase/tmp/collectinfo-2021-06-08T191134-ns_1@172.23.106.125.zip 2021-06-08 12:14:37,138 | infra | DEBUG | MainThread | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.125 2021-06-08 12:14:37,140 | test | INFO | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1129] ns_1@172.23.106.155: Copying cbcollect ZIP file to Client 2021-06-08 12:14:37,141 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.155 with username: root, password: couchbase 2021-06-08 12:14:37,210 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 12:14:37,424 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: ls /etc/ 2021-06-08 12:14:37,479 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 12:14:37,479 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: ls /cygdrive/c/Windows 2021-06-08 12:14:37,490 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: uname -m 2021-06-08 12:14:37,503 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: cat /proc/cpuinfo 2021-06-08 12:14:37,513 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: df -Thl 2021-06-08 12:14:37,528 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: cat /proc/meminfo 2021-06-08 12:14:37,539 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: hostname 2021-06-08 12:14:37,549 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: hostname -d 2021-06-08 12:14:37,897 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.155: rm -f /opt/couchbase/var/lib/couchbase/tmp/collectinfo-2021-06-08T191134-ns_1@172.23.106.155.zip 2021-06-08 12:14:37,915 | infra | DEBUG | MainThread | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.155 2021-06-08 12:14:37,917 | test | INFO | MainThread | [cluster_ready_functions:copy_cb_collect_logs:1129] ns_1@172.23.106.156: Copying cbcollect ZIP file to Client 2021-06-08 12:14:37,920 | infra | DEBUG | MainThread | [remote_util:connect:206] Connecting to 172.23.106.156 with username: root, password: couchbase 2021-06-08 12:14:38,000 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: sw_vers | grep ProductVersion | awk '{ print $2 }' 2021-06-08 12:14:38,219 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: ls /etc/ 2021-06-08 12:14:38,279 | infra | DEBUG | MainThread | [remote_util:extract_remote_info:3472] os_distro: CentOS, os_version: centos 7, is_linux_distro: True 2021-06-08 12:14:38,279 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: ls /cygdrive/c/Windows 2021-06-08 12:14:38,292 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: uname -m 2021-06-08 12:14:38,305 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: cat /proc/cpuinfo 2021-06-08 12:14:38,318 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: df -Thl 2021-06-08 12:14:38,335 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: cat /proc/meminfo 2021-06-08 12:14:38,346 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: hostname 2021-06-08 12:14:38,359 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: hostname -d 2021-06-08 12:14:38,743 | infra | DEBUG | MainThread | [remote_util:execute_command_raw_jsch:3285] Running command on 172.23.106.156: rm -f /opt/couchbase/var/lib/couchbase/tmp/collectinfo-2021-06-08T191134-ns_1@172.23.106.156.zip 2021-06-08 12:14:38,766 | infra | DEBUG | MainThread | [remote_util:disconnect:219] Disconnecting ssh_client for 172.23.106.156 2021-06-08 12:14:38,778 | test | CRITICAL | MainThread | [basetestcase:tearDownEverything:493] Skipping get_trace !! 2021-06-08 12:14:38,805 | infra | WARNING | MainThread | [basetestcase:tearDownEverything:498] Alerts found: [{u'msg': u'Hard out-of-memory error: Bucket "default" on node 172.23.106.156 is full. No memory currently allocated to this bucket can be easily released.', u'disableUIPopUp': True, u'serverTime': u'2021-06-08T11:16:08.000Z'}, {u'msg': u'Approaching full disk warning. Usage of disk "/data" on node "172.23.106.156" is around 95%.', u'disableUIPopUp': False, u'serverTime': u'2021-06-08T11:31:08.000Z'}, {u'msg': u'Approaching full disk warning. Usage of disk "/data" on node "172.23.106.125" is around 97%.', u'disableUIPopUp': False, u'serverTime': u'2021-06-08T11:37:19.000Z'}, {u'msg': u'Approaching full disk warning. Usage of disk "/data" on node "172.23.106.122" is around 98%.', u'disableUIPopUp': False, u'serverTime': u'2021-06-08T11:38:17.000Z'}, {u'msg': u'Write Commit Failure. Disk write failed for item in Bucket "default" on node 172.23.106.155.', u'disableUIPopUp': True, u'serverTime': u'2021-06-08T11:39:24.000Z'}, {u'msg': u'Approaching full disk warning. Usage of disk "/data" on node "172.23.106.155" is around 100%.', u'disableUIPopUp': False, u'serverTime': u'2021-06-08T11:39:24.000Z'}, {u'msg': u'Write Commit Failure. Disk write failed for item in Bucket "default" on node 172.23.106.122.', u'disableUIPopUp': True, u'serverTime': u'2021-06-08T11:41:17.000Z'}, {u'msg': u'Write Commit Failure. Disk write failed for item in Bucket "default" on node 172.23.106.125.', u'disableUIPopUp': True, u'serverTime': u'2021-06-08T11:41:19.000Z'}, {u'msg': u'Hard out-of-memory error: Bucket "default" on node 172.23.106.122 is full. No memory currently allocated to this bucket can be easily released.', u'disableUIPopUp': True, u'serverTime': u'2021-06-08T11:47:17.000Z'}, {u'msg': u'Hard out-of-memory error: Bucket "default" on node 172.23.106.125 is full. No memory currently allocated to this bucket can be easily released.', u'disableUIPopUp': True, u'serverTime': u'2021-06-08T11:47:19.000Z'}, {u'msg': u'Hard out-of-memory error: Bucket "default" on node 172.23.106.155 is full. No memory currently allocated to this bucket can be easily released.', u'disableUIPopUp': True, u'serverTime': u'2021-06-08T11:47:24.000Z'}, {u'msg': u'Write Commit Failure. Disk write failed for item in Bucket "default" on node 172.23.106.156.', u'disableUIPopUp': True, u'serverTime': u'2021-06-08T12:02:08.000Z'}] 2021-06-08 12:14:38,805 | test | DEBUG | MainThread | [basetestcase:tearDownEverything:499] Cleaning up cluster 2021-06-08 12:14:38,855 | test | DEBUG | MainThread | [bucket_ready_functions:parse_get_bucket_json:3739] Stats: {u'opsPerSec': 0, u'diskFetches': 908, u'dataUsed': 21369020416L, u'itemCount': 4816001, u'quotaPercentUsed': 27.54363054286934, u'storageTotals': {u'hdd': {u'usedByData': 38902356326L, u'used': 40567184178L, u'quotaTotal': 386354135040L, u'total': 386354135040L, u'free': 328401014784L}, u'ram': {u'usedByData': 9646451040L, u'used': 31418322944L, u'quotaTotal': 35026632704L, u'total': 49176707072L, u'quotaUsed': 35022438400L, u'quotaUsedPerNode': 8755609600L, u'quotaTotalPerNode': 8756658176L}}, u'diskUsed': 38902356326L, u'vbActiveNumNonResident': 2220293, u'memUsed': 9646451040L} 2021-06-08 12:14:38,857 | test | DEBUG | MainThread | [bucket_ready_functions:delete_all_buckets:1571] Deleting existing buckets [u'default'] on 172.23.106.156 2021-06-08 12:14:38,857 | test | DEBUG | MainThread | [bucket_ready_functions:delete_all_buckets:1573] Remove bucket default ... 2021-06-08 12:14:38,857 | test | DEBUG | MainThread | [bucket_ready_functions:delete_bucket:1493] Deleting existing bucket default on ip:172.23.106.156 port:8091 ssh_username:root 2021-06-08 12:14:38,901 | test | DEBUG | MainThread | [bucket_ready_functions:parse_get_bucket_json:3739] Stats: {u'opsPerSec': 0, u'diskFetches': 908, u'dataUsed': 21369020416L, u'itemCount': 4816001, u'quotaPercentUsed': 27.54363054286934, u'storageTotals': {u'hdd': {u'usedByData': 38902356326L, u'used': 40567184178L, u'quotaTotal': 386354135040L, u'total': 386354135040L, u'free': 328401014784L}, u'ram': {u'usedByData': 9646451040L, u'used': 31418322944L, u'quotaTotal': 35026632704L, u'total': 49176707072L, u'quotaUsed': 35022438400L, u'quotaUsedPerNode': 8755609600L, u'quotaTotalPerNode': 8756658176L}}, u'diskUsed': 38902356326L, u'vbActiveNumNonResident': 2220293, u'memUsed': 9646451040L} 2021-06-08 12:14:52,900 | test | DEBUG | MainThread | [bucket_ready_functions:delete_bucket:1513] Deleted bucket: default from 172.23.106.156 2021-06-08 12:14:52,901 | test | DEBUG | MainThread | [bucket_ready_functions:wait_for_bucket_deletion:1533] Waiting for bucket default deletion to finish 2021-06-08 12:14:52,937 | test | DEBUG | MainThread | [bucket_ready_functions:delete_all_buckets:1571] Deleting existing buckets [] on 172.23.106.125 2021-06-08 12:14:52,956 | test | DEBUG | MainThread | [bucket_ready_functions:delete_all_buckets:1571] Deleting existing buckets [] on 172.23.106.122 2021-06-08 12:14:52,974 | test | DEBUG | MainThread | [bucket_ready_functions:delete_all_buckets:1571] Deleting existing buckets [] on 172.23.106.155 2021-06-08 12:14:53,009 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:46] Checking if ns_server is running with timeout 120secs 2021-06-08 12:14:53,036 | test | DEBUG | MainThread | [cluster_ready_functions:cleanup_cluster:176] Rebalancing all nodes in order to remove nodes 2021-06-08 12:14:53,043 | test | DEBUG | MainThread | [rest_client:remove_nodes:124] Ejecting nodes '[u'ns_1@172.23.106.155', u'ns_1@172.23.106.122', u'ns_1@172.23.106.125']' from cluster 2021-06-08 12:14:53,043 | test | DEBUG | MainThread | [rest_client:rebalance:1339] Rebalance params: {'password': 'password', 'knownNodes': u'ns_1@172.23.106.155,ns_1@172.23.106.156,ns_1@172.23.106.122,ns_1@172.23.106.125', 'ejectedNodes': u'ns_1@172.23.106.155,ns_1@172.23.106.122,ns_1@172.23.106.125', 'user': 'Administrator'} 2021-06-08 12:14:53,101 | test | DEBUG | MainThread | [rest_client:rebalance:1344] Rebalance operation started 2021-06-08 12:14:53,111 | test | DEBUG | MainThread | [rest_client:_rebalance_status_and_progress:1553] Rebalance percentage: 0.00 % 2021-06-08 12:14:53,118 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1475 2021-06-08 12:14:53,118 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-06-08 12:14:56,138 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> _rebalance_status_and_progress():L1557 2021-06-08 12:14:56,138 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 5 seconds. Reason: None 2021-06-08 12:15:01,154 | infra | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1475 2021-06-08 12:15:01,154 | infra | DEBUG | MainThread | [common_lib:sleep:24] Sleep 3 seconds. Reason: None 2021-06-08 12:15:04,164 | test | INFO | MainThread | [rest_client:monitorRebalance:1484] Rebalance done. Taken 11.0639998913 seconds to complete 2021-06-08 12:15:04,171 | test | DEBUG | MainThread | [common_lib:sleep:17] Sleep is called from lib/membase/api/rest_client.py -> monitorRebalance():L1487 2021-06-08 12:15:04,171 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete 2021-06-08 12:15:09,226 | test | DEBUG | MainThread | [cluster_ready_functions:cleanup_cluster:223] Removed all the nodes from cluster associated with ip:172.23.106.156 port:8091 ssh_username:root ? [(u'ns_1@172.23.106.155', 8091), (u'ns_1@172.23.106.122', 8091), (u'ns_1@172.23.106.125', 8091)] 2021-06-08 12:15:09,236 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:148] Waiting for ns_server @ 172.23.106.156:8091 2021-06-08 12:15:09,236 | test | DEBUG | MainThread | [rest_client:is_ns_server_running:46] Checking if ns_server is running with timeout 360secs 2021-06-08 12:15:09,246 | test | DEBUG | MainThread | [cluster_ready_functions:wait_for_ns_servers_or_assert:151] ns_server @ 172.23.106.156:8091 is running 2021-06-08 12:15:09,246 | infra | INFO | MainThread | [basetestcase:tearDownEverything:514] ========== tasks in thread pool ========== 2021-06-08 12:15:09,246 | infra | INFO | MainThread | [basetestcase:tearDownEverything:516] ========================================== ERROR ====================================================================== ERROR: test_disk_full_reduce_replica (magma.magma_disk_full.MagmaDiskFull) ---------------------------------------------------------------------- Traceback (most recent call last): File "pytests/magma/magma_disk_full.py", line 488, in test_disk_full_reduce_replica self.simulate_persistence_failure(self.cluster.nodes_in_cluster) File "pytests/magma/magma_disk_full.py", line 74, in simulate_persistence_failure raise Exception("Could not hit Write Commit Failures: ep_data_write_failed>0") Exception: Could not hit Write Commit Failures: ep_data_write_failed>0 ====================================================================== ERROR: test_disk_full_reduce_replica (magma.magma_disk_full.MagmaDiskFull) ---------------------------------------------------------------------- During the test, Remote Connections: 102, Disconnections: 95 SDK Connections: 20, Disconnections: 20 !!!!!! CRITICAL :: Shell disconnection mismatch !!!!! Traceback (most recent call last): File "pytests/magma/magma_disk_full.py", line 31, in tearDown MagmaBaseTest.tearDown(self) File "pytests/magma/magma_base.py", line 437, in tearDown super(MagmaBaseTest, self).tearDown() File "pytests/basetestcase.py", line 453, in tearDown self.assertFalse(result, msg="Cb_log file validation failed") AssertionError: Cb_log file validation failed ---------------------------------------------------------------------- Ran 1 test in 3733.844s FAILED (errors=2) summary so far suite magma.magma_disk_full.MagmaDiskFull , pass 2 , fail 1 failures so far... magma.magma_disk_full.MagmaDiskFull.test_disk_full_reduce_replica testrunner logs, diags and results are available under /data/workspace/centos-p0-magma-vset00-00-disk_full_7.0_P0/logs/testrunner-21-Jun-08_10-20-08/test_3 Logs will be stored at /data/workspace/centos-p0-magma-vset00-00-disk_full_7.0_P0/logs/testrunner-21-Jun-08_10-20-08/test_4