[2021-12-23 22:46:27,232] - [eventing_base] [140099887036160] - INFO - ============== EventingBaseTest setup has started ============== [2021-12-23 22:46:27,232] - [eventing_base] [140099887036160] - INFO - Starting Test: test_rebalance_in_with_different_topologies [2021-12-23 22:46:27,239] - [remote_util] [140099887036160] - INFO - SSH Connecting to 172.23.108.100 with username:root, attempt#1 of 5 [2021-12-23 22:46:27,404] - [remote_util] [140099887036160] - INFO - SSH Connected to 172.23.108.100 as root [2021-12-23 22:46:27,797] - [remote_util] [140099887036160] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-12-23 22:46:28,224] - [remote_util] [140099887036160] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-12-23 22:46:28,232] - [rest_client] [140099887036160] - INFO - -->is_ns_server_running? [2021-12-23 22:46:28,252] - [rest_client] [140099887036160] - INFO - Node version in cluster 7.1.0-1960-enterprise [2021-12-23 22:46:28,252] - [basetestcase] [140099887036160] - INFO - ============== basetestcase setup was started for test #5 test_rebalance_in_with_different_topologies============== [2021-12-23 22:46:28,253] - [basetestcase] [140099887036160] - INFO - initializing cluster [2021-12-23 22:46:28,256] - [remote_util] [140099887036160] - INFO - SSH Connecting to 172.23.108.100 with username:root, attempt#1 of 5 [2021-12-23 22:46:28,406] - [remote_util] [140099887036160] - INFO - SSH Connected to 172.23.108.100 as root [2021-12-23 22:46:28,787] - [remote_util] [140099887036160] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-12-23 22:46:29,216] - [remote_util] [140099887036160] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-12-23 22:46:29,234] - [remote_util] [140099887036160] - INFO - SSH Connecting to 172.23.108.100 with username:root, attempt#1 of 5 [2021-12-23 22:46:29,394] - [remote_util] [140099887036160] - INFO - SSH Connected to 172.23.108.100 as root [2021-12-23 22:46:29,752] - [remote_util] [140099887036160] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-12-23 22:46:30,180] - [remote_util] [140099887036160] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-12-23 22:46:30,263] - [remote_util] [140099887036160] - INFO - 172.23.108.100 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-12-23 22:46:30,392] - [remote_util] [140099887036160] - INFO - Running systemd command on this server [2021-12-23 22:46:30,392] - [remote_util] [140099887036160] - INFO - running command.raw on 172.23.108.100: systemctl stop couchbase-server.service [2021-12-23 22:46:32,648] - [remote_util] [140099887036160] - INFO - command executed successfully with root [2021-12-23 22:46:32,649] - [remote_util] [140099887036160] - INFO - Checking for process beam.smp on linux [2021-12-23 22:46:32,721] - [basetestcase] [140099887036160] - INFO - Couchbase stopped [2021-12-23 22:46:32,721] - [remote_util] [140099887036160] - INFO - running command.raw on 172.23.108.100: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-12-23 22:46:32,751] - [remote_util] [140099887036160] - INFO - command executed successfully with root [2021-12-23 22:46:32,751] - [remote_util] [140099887036160] - INFO - running command.raw on 172.23.108.100: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-12-23 22:46:32,824] - [remote_util] [140099887036160] - INFO - command executed successfully with root [2021-12-23 22:46:32,827] - [remote_util] [140099887036160] - INFO - SSH Connecting to 172.23.108.100 with username:root, attempt#1 of 5 [2021-12-23 22:46:32,986] - [remote_util] [140099887036160] - INFO - SSH Connected to 172.23.108.100 as root [2021-12-23 22:46:33,347] - [remote_util] [140099887036160] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-12-23 22:46:33,769] - [remote_util] [140099887036160] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-12-23 22:46:33,852] - [remote_util] [140099887036160] - INFO - 172.23.108.100 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-12-23 22:46:33,852] - [remote_util] [140099887036160] - INFO - Checking for process beam.smp on linux [2021-12-23 22:46:33,923] - [remote_util] [140099887036160] - INFO - Starting couchbase server [2021-12-23 22:46:34,047] - [remote_util] [140099887036160] - INFO - Running systemd command on this server [2021-12-23 22:46:34,048] - [remote_util] [140099887036160] - INFO - running command.raw on 172.23.108.100: systemctl start couchbase-server.service [2021-12-23 22:46:34,094] - [remote_util] [140099887036160] - INFO - command executed successfully with root [2021-12-23 22:46:34,095] - [remote_util] [140099887036160] - INFO - 172.23.108.100:sleep for 5 secs. waiting for couchbase server to come up ... [2021-12-23 22:46:39,100] - [remote_util] [140099887036160] - INFO - running command.raw on 172.23.108.100: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-12-23 22:46:39,153] - [remote_util] [140099887036160] - INFO - command executed successfully with root [2021-12-23 22:46:39,153] - [remote_util] [140099887036160] - INFO - Couchbase server status: [' Process: 87235 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-12-23 22:46:39,153] - [remote_util] [140099887036160] - INFO - Checking for process beam.smp on linux [2021-12-23 22:46:39,253] - [remote_util] [140099887036160] - INFO - process beam.smp is running on 172.23.108.100: with pid 87376 [2021-12-23 22:46:39,254] - [basetestcase] [140099887036160] - INFO - Couchbase started [2021-12-23 22:46:39,257] - [remote_util] [140099887036160] - INFO - SSH Connecting to 172.23.107.88 with username:root, attempt#1 of 5 [2021-12-23 22:46:39,425] - [remote_util] [140099887036160] - INFO - SSH Connected to 172.23.107.88 as root [2021-12-23 22:46:39,828] - [remote_util] [140099887036160] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-12-23 22:46:40,271] - [remote_util] [140099887036160] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-12-23 22:46:40,287] - [remote_util] [140099887036160] - INFO - SSH Connecting to 172.23.107.88 with username:root, attempt#1 of 5 [2021-12-23 22:46:40,450] - [remote_util] [140099887036160] - INFO - SSH Connected to 172.23.107.88 as root [2021-12-23 22:46:40,872] - [remote_util] [140099887036160] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-12-23 22:46:41,320] - [remote_util] [140099887036160] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-12-23 22:46:41,402] - [remote_util] [140099887036160] - INFO - 172.23.107.88 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-12-23 22:46:41,529] - [remote_util] [140099887036160] - INFO - Running systemd command on this server [2021-12-23 22:46:41,529] - [remote_util] [140099887036160] - INFO - running command.raw on 172.23.107.88: systemctl stop couchbase-server.service [2021-12-23 22:46:44,215] - [remote_util] [140099887036160] - INFO - command executed successfully with root [2021-12-23 22:46:44,215] - [remote_util] [140099887036160] - INFO - Checking for process beam.smp on linux [2021-12-23 22:46:44,330] - [basetestcase] [140099887036160] - INFO - Couchbase stopped [2021-12-23 22:46:44,331] - [remote_util] [140099887036160] - INFO - running command.raw on 172.23.107.88: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-12-23 22:46:44,365] - [remote_util] [140099887036160] - INFO - command executed successfully with root [2021-12-23 22:46:44,365] - [remote_util] [140099887036160] - INFO - running command.raw on 172.23.107.88: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-12-23 22:46:44,438] - [remote_util] [140099887036160] - INFO - command executed successfully with root [2021-12-23 22:46:44,441] - [remote_util] [140099887036160] - INFO - SSH Connecting to 172.23.107.88 with username:root, attempt#1 of 5 [2021-12-23 22:46:44,607] - [remote_util] [140099887036160] - INFO - SSH Connected to 172.23.107.88 as root [2021-12-23 22:46:44,989] - [remote_util] [140099887036160] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-12-23 22:46:45,407] - [remote_util] [140099887036160] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-12-23 22:46:45,495] - [remote_util] [140099887036160] - INFO - 172.23.107.88 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-12-23 22:46:45,495] - [remote_util] [140099887036160] - INFO - Checking for process beam.smp on linux [2021-12-23 22:46:45,572] - [remote_util] [140099887036160] - INFO - Starting couchbase server [2021-12-23 22:46:45,704] - [remote_util] [140099887036160] - INFO - Running systemd command on this server [2021-12-23 22:46:45,704] - [remote_util] [140099887036160] - INFO - running command.raw on 172.23.107.88: systemctl start couchbase-server.service [2021-12-23 22:46:45,759] - [remote_util] [140099887036160] - INFO - command executed successfully with root [2021-12-23 22:46:45,760] - [remote_util] [140099887036160] - INFO - 172.23.107.88:sleep for 5 secs. waiting for couchbase server to come up ... [2021-12-23 22:46:50,765] - [remote_util] [140099887036160] - INFO - running command.raw on 172.23.107.88: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-12-23 22:46:50,827] - [remote_util] [140099887036160] - INFO - command executed successfully with root [2021-12-23 22:46:50,828] - [remote_util] [140099887036160] - INFO - Couchbase server status: [' Process: 89260 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-12-23 22:46:50,828] - [remote_util] [140099887036160] - INFO - Checking for process beam.smp on linux [2021-12-23 22:46:50,913] - [remote_util] [140099887036160] - INFO - process beam.smp is running on 172.23.107.88: with pid 89401 [2021-12-23 22:46:50,914] - [basetestcase] [140099887036160] - INFO - Couchbase started [2021-12-23 22:46:50,917] - [remote_util] [140099887036160] - INFO - SSH Connecting to 172.23.108.0 with username:root, attempt#1 of 5 [2021-12-23 22:46:51,080] - [remote_util] [140099887036160] - INFO - SSH Connected to 172.23.108.0 as root [2021-12-23 22:46:51,486] - [remote_util] [140099887036160] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-12-23 22:46:51,921] - [remote_util] [140099887036160] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-12-23 22:46:51,931] - [remote_util] [140099887036160] - INFO - SSH Connecting to 172.23.108.0 with username:root, attempt#1 of 5 [2021-12-23 22:46:52,087] - [remote_util] [140099887036160] - INFO - SSH Connected to 172.23.108.0 as root [2021-12-23 22:46:52,488] - [remote_util] [140099887036160] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-12-23 22:46:52,924] - [remote_util] [140099887036160] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-12-23 22:46:53,005] - [remote_util] [140099887036160] - INFO - 172.23.108.0 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-12-23 22:46:53,140] - [remote_util] [140099887036160] - INFO - Running systemd command on this server [2021-12-23 22:46:53,140] - [remote_util] [140099887036160] - INFO - running command.raw on 172.23.108.0: systemctl stop couchbase-server.service [2021-12-23 22:46:55,195] - [remote_util] [140099887036160] - INFO - command executed successfully with root [2021-12-23 22:46:55,196] - [remote_util] [140099887036160] - INFO - Checking for process beam.smp on linux [2021-12-23 22:46:55,278] - [basetestcase] [140099887036160] - INFO - Couchbase stopped [2021-12-23 22:46:55,279] - [remote_util] [140099887036160] - INFO - running command.raw on 172.23.108.0: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-12-23 22:46:55,311] - [remote_util] [140099887036160] - INFO - command executed successfully with root [2021-12-23 22:46:55,312] - [remote_util] [140099887036160] - INFO - running command.raw on 172.23.108.0: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-12-23 22:46:55,385] - [remote_util] [140099887036160] - INFO - command executed successfully with root [2021-12-23 22:46:55,388] - [remote_util] [140099887036160] - INFO - SSH Connecting to 172.23.108.0 with username:root, attempt#1 of 5 [2021-12-23 22:46:55,544] - [remote_util] [140099887036160] - INFO - SSH Connected to 172.23.108.0 as root [2021-12-23 22:46:55,932] - [remote_util] [140099887036160] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-12-23 22:46:56,356] - [remote_util] [140099887036160] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-12-23 22:46:56,440] - [remote_util] [140099887036160] - INFO - 172.23.108.0 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-12-23 22:46:56,440] - [remote_util] [140099887036160] - INFO - Checking for process beam.smp on linux [2021-12-23 22:46:56,518] - [remote_util] [140099887036160] - INFO - Starting couchbase server [2021-12-23 22:46:56,686] - [remote_util] [140099887036160] - INFO - Running systemd command on this server [2021-12-23 22:46:56,686] - [remote_util] [140099887036160] - INFO - running command.raw on 172.23.108.0: systemctl start couchbase-server.service [2021-12-23 22:46:56,743] - [remote_util] [140099887036160] - INFO - command executed successfully with root [2021-12-23 22:46:56,744] - [remote_util] [140099887036160] - INFO - 172.23.108.0:sleep for 5 secs. waiting for couchbase server to come up ... [2021-12-23 22:47:01,749] - [remote_util] [140099887036160] - INFO - running command.raw on 172.23.108.0: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-12-23 22:47:01,791] - [remote_util] [140099887036160] - INFO - command executed successfully with root [2021-12-23 22:47:01,791] - [remote_util] [140099887036160] - INFO - Couchbase server status: [' Process: 78474 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-12-23 22:47:01,792] - [remote_util] [140099887036160] - INFO - Checking for process beam.smp on linux [2021-12-23 22:47:01,890] - [remote_util] [140099887036160] - INFO - process beam.smp is running on 172.23.108.0: with pid 78616 [2021-12-23 22:47:01,890] - [basetestcase] [140099887036160] - INFO - Couchbase started [2021-12-23 22:47:01,893] - [remote_util] [140099887036160] - INFO - SSH Connecting to 172.23.107.91 with username:root, attempt#1 of 5 [2021-12-23 22:47:02,060] - [remote_util] [140099887036160] - INFO - SSH Connected to 172.23.107.91 as root [2021-12-23 22:47:02,490] - [remote_util] [140099887036160] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-12-23 22:47:02,940] - [remote_util] [140099887036160] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-12-23 22:47:02,955] - [remote_util] [140099887036160] - INFO - SSH Connecting to 172.23.107.91 with username:root, attempt#1 of 5 [2021-12-23 22:47:03,107] - [remote_util] [140099887036160] - INFO - SSH Connected to 172.23.107.91 as root [2021-12-23 22:47:03,507] - [remote_util] [140099887036160] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-12-23 22:47:03,956] - [remote_util] [140099887036160] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-12-23 22:47:04,037] - [remote_util] [140099887036160] - INFO - 172.23.107.91 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-12-23 22:47:04,216] - [remote_util] [140099887036160] - INFO - Running systemd command on this server [2021-12-23 22:47:04,217] - [remote_util] [140099887036160] - INFO - running command.raw on 172.23.107.91: systemctl stop couchbase-server.service [2021-12-23 22:47:06,419] - [remote_util] [140099887036160] - INFO - command executed successfully with root [2021-12-23 22:47:06,420] - [remote_util] [140099887036160] - INFO - Checking for process beam.smp on linux [2021-12-23 22:47:06,490] - [basetestcase] [140099887036160] - INFO - Couchbase stopped [2021-12-23 22:47:06,490] - [remote_util] [140099887036160] - INFO - running command.raw on 172.23.107.91: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-12-23 22:47:06,517] - [remote_util] [140099887036160] - INFO - command executed successfully with root [2021-12-23 22:47:06,517] - [remote_util] [140099887036160] - INFO - running command.raw on 172.23.107.91: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-12-23 22:47:06,585] - [remote_util] [140099887036160] - INFO - command executed successfully with root [2021-12-23 22:47:06,588] - [remote_util] [140099887036160] - INFO - SSH Connecting to 172.23.107.91 with username:root, attempt#1 of 5 [2021-12-23 22:47:06,754] - [remote_util] [140099887036160] - INFO - SSH Connected to 172.23.107.91 as root [2021-12-23 22:47:07,132] - [remote_util] [140099887036160] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-12-23 22:47:07,577] - [remote_util] [140099887036160] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-12-23 22:47:07,659] - [remote_util] [140099887036160] - INFO - 172.23.107.91 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-12-23 22:47:07,659] - [remote_util] [140099887036160] - INFO - Checking for process beam.smp on linux [2021-12-23 22:47:07,727] - [remote_util] [140099887036160] - INFO - Starting couchbase server [2021-12-23 22:47:07,911] - [remote_util] [140099887036160] - INFO - Running systemd command on this server [2021-12-23 22:47:07,911] - [remote_util] [140099887036160] - INFO - running command.raw on 172.23.107.91: systemctl start couchbase-server.service [2021-12-23 22:47:07,967] - [remote_util] [140099887036160] - INFO - command executed successfully with root [2021-12-23 22:47:07,967] - [remote_util] [140099887036160] - INFO - 172.23.107.91:sleep for 5 secs. waiting for couchbase server to come up ... [2021-12-23 22:47:12,972] - [remote_util] [140099887036160] - INFO - running command.raw on 172.23.107.91: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-12-23 22:47:13,028] - [remote_util] [140099887036160] - INFO - command executed successfully with root [2021-12-23 22:47:13,028] - [remote_util] [140099887036160] - INFO - Couchbase server status: [' Process: 101290 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-12-23 22:47:13,028] - [remote_util] [140099887036160] - INFO - Checking for process beam.smp on linux [2021-12-23 22:47:13,126] - [remote_util] [140099887036160] - INFO - process beam.smp is running on 172.23.107.91: with pid 101431 [2021-12-23 22:47:13,126] - [basetestcase] [140099887036160] - INFO - Couchbase started [2021-12-23 22:47:13,129] - [remote_util] [140099887036160] - INFO - SSH Connecting to 172.23.108.102 with username:root, attempt#1 of 5 [2021-12-23 22:47:13,300] - [remote_util] [140099887036160] - INFO - SSH Connected to 172.23.108.102 as root [2021-12-23 22:47:13,709] - [remote_util] [140099887036160] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-12-23 22:47:14,142] - [remote_util] [140099887036160] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-12-23 22:47:14,157] - [remote_util] [140099887036160] - INFO - SSH Connecting to 172.23.108.102 with username:root, attempt#1 of 5 [2021-12-23 22:47:14,316] - [remote_util] [140099887036160] - INFO - SSH Connected to 172.23.108.102 as root [2021-12-23 22:47:14,682] - [remote_util] [140099887036160] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-12-23 22:47:15,115] - [remote_util] [140099887036160] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-12-23 22:47:15,198] - [remote_util] [140099887036160] - INFO - 172.23.108.102 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-12-23 22:47:15,324] - [remote_util] [140099887036160] - INFO - Running systemd command on this server [2021-12-23 22:47:15,324] - [remote_util] [140099887036160] - INFO - running command.raw on 172.23.108.102: systemctl stop couchbase-server.service [2021-12-23 22:47:18,170] - [remote_util] [140099887036160] - INFO - command executed successfully with root [2021-12-23 22:47:18,171] - [remote_util] [140099887036160] - INFO - Checking for process beam.smp on linux [2021-12-23 22:47:18,243] - [basetestcase] [140099887036160] - INFO - Couchbase stopped [2021-12-23 22:47:18,243] - [remote_util] [140099887036160] - INFO - running command.raw on 172.23.108.102: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-12-23 22:47:18,269] - [remote_util] [140099887036160] - INFO - command executed successfully with root [2021-12-23 22:47:18,269] - [remote_util] [140099887036160] - INFO - running command.raw on 172.23.108.102: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-12-23 22:47:18,341] - [remote_util] [140099887036160] - INFO - command executed successfully with root [2021-12-23 22:47:18,344] - [remote_util] [140099887036160] - INFO - SSH Connecting to 172.23.108.102 with username:root, attempt#1 of 5 [2021-12-23 22:47:18,508] - [remote_util] [140099887036160] - INFO - SSH Connected to 172.23.108.102 as root [2021-12-23 22:47:18,889] - [remote_util] [140099887036160] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-12-23 22:47:19,317] - [remote_util] [140099887036160] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-12-23 22:47:19,403] - [remote_util] [140099887036160] - INFO - 172.23.108.102 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-12-23 22:47:19,403] - [remote_util] [140099887036160] - INFO - Checking for process beam.smp on linux [2021-12-23 22:47:19,476] - [remote_util] [140099887036160] - INFO - Starting couchbase server [2021-12-23 22:47:19,644] - [remote_util] [140099887036160] - INFO - Running systemd command on this server [2021-12-23 22:47:19,644] - [remote_util] [140099887036160] - INFO - running command.raw on 172.23.108.102: systemctl start couchbase-server.service [2021-12-23 22:47:19,697] - [remote_util] [140099887036160] - INFO - command executed successfully with root [2021-12-23 22:47:19,698] - [remote_util] [140099887036160] - INFO - 172.23.108.102:sleep for 5 secs. waiting for couchbase server to come up ... [2021-12-23 22:47:24,703] - [remote_util] [140099887036160] - INFO - running command.raw on 172.23.108.102: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-12-23 22:47:24,757] - [remote_util] [140099887036160] - INFO - command executed successfully with root [2021-12-23 22:47:24,757] - [remote_util] [140099887036160] - INFO - Couchbase server status: [' Process: 72321 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-12-23 22:47:24,757] - [remote_util] [140099887036160] - INFO - Checking for process beam.smp on linux [2021-12-23 22:47:24,856] - [remote_util] [140099887036160] - INFO - process beam.smp is running on 172.23.108.102: with pid 72466 [2021-12-23 22:47:24,857] - [basetestcase] [140099887036160] - INFO - Couchbase started [2021-12-23 22:47:24,860] - [remote_util] [140099887036160] - INFO - SSH Connecting to 172.23.107.84 with username:root, attempt#1 of 5 [2021-12-23 22:47:25,027] - [remote_util] [140099887036160] - INFO - SSH Connected to 172.23.107.84 as root [2021-12-23 22:47:25,408] - [remote_util] [140099887036160] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-12-23 22:47:25,848] - [remote_util] [140099887036160] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-12-23 22:47:25,863] - [remote_util] [140099887036160] - INFO - SSH Connecting to 172.23.107.84 with username:root, attempt#1 of 5 [2021-12-23 22:47:26,028] - [remote_util] [140099887036160] - INFO - SSH Connected to 172.23.107.84 as root [2021-12-23 22:47:26,388] - [remote_util] [140099887036160] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-12-23 22:47:26,830] - [remote_util] [140099887036160] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-12-23 22:47:26,917] - [remote_util] [140099887036160] - INFO - 172.23.107.84 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-12-23 22:47:27,052] - [remote_util] [140099887036160] - INFO - Running systemd command on this server [2021-12-23 22:47:27,053] - [remote_util] [140099887036160] - INFO - running command.raw on 172.23.107.84: systemctl stop couchbase-server.service [2021-12-23 22:47:28,971] - [remote_util] [140099887036160] - INFO - command executed successfully with root [2021-12-23 22:47:28,971] - [remote_util] [140099887036160] - INFO - Checking for process beam.smp on linux [2021-12-23 22:47:29,051] - [basetestcase] [140099887036160] - INFO - Couchbase stopped [2021-12-23 22:47:29,051] - [remote_util] [140099887036160] - INFO - running command.raw on 172.23.107.84: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-12-23 22:47:29,084] - [remote_util] [140099887036160] - INFO - command executed successfully with root [2021-12-23 22:47:29,084] - [remote_util] [140099887036160] - INFO - running command.raw on 172.23.107.84: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-12-23 22:47:29,162] - [remote_util] [140099887036160] - INFO - command executed successfully with root [2021-12-23 22:47:29,165] - [remote_util] [140099887036160] - INFO - SSH Connecting to 172.23.107.84 with username:root, attempt#1 of 5 [2021-12-23 22:47:29,322] - [remote_util] [140099887036160] - INFO - SSH Connected to 172.23.107.84 as root [2021-12-23 22:47:29,735] - [remote_util] [140099887036160] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-12-23 22:47:30,174] - [remote_util] [140099887036160] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-12-23 22:47:30,258] - [remote_util] [140099887036160] - INFO - 172.23.107.84 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-12-23 22:47:30,259] - [remote_util] [140099887036160] - INFO - Checking for process beam.smp on linux [2021-12-23 22:47:30,339] - [remote_util] [140099887036160] - INFO - Starting couchbase server [2021-12-23 22:47:30,467] - [remote_util] [140099887036160] - INFO - Running systemd command on this server [2021-12-23 22:47:30,468] - [remote_util] [140099887036160] - INFO - running command.raw on 172.23.107.84: systemctl start couchbase-server.service [2021-12-23 22:47:30,522] - [remote_util] [140099887036160] - INFO - command executed successfully with root [2021-12-23 22:47:30,522] - [remote_util] [140099887036160] - INFO - 172.23.107.84:sleep for 5 secs. waiting for couchbase server to come up ... [2021-12-23 22:47:35,527] - [remote_util] [140099887036160] - INFO - running command.raw on 172.23.107.84: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-12-23 22:47:35,575] - [remote_util] [140099887036160] - INFO - command executed successfully with root [2021-12-23 22:47:35,575] - [remote_util] [140099887036160] - INFO - Couchbase server status: [' Process: 69272 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-12-23 22:47:35,575] - [remote_util] [140099887036160] - INFO - Checking for process beam.smp on linux [2021-12-23 22:47:35,674] - [remote_util] [140099887036160] - INFO - process beam.smp is running on 172.23.107.84: with pid 69413 [2021-12-23 22:47:35,674] - [basetestcase] [140099887036160] - INFO - Couchbase started [2021-12-23 22:47:35,677] - [remote_util] [140099887036160] - INFO - SSH Connecting to 172.23.107.85 with username:root, attempt#1 of 5 [2021-12-23 22:47:35,845] - [remote_util] [140099887036160] - INFO - SSH Connected to 172.23.107.85 as root [2021-12-23 22:47:36,233] - [remote_util] [140099887036160] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-12-23 22:47:36,682] - [remote_util] [140099887036160] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-12-23 22:47:36,698] - [remote_util] [140099887036160] - INFO - SSH Connecting to 172.23.107.85 with username:root, attempt#1 of 5 [2021-12-23 22:47:36,855] - [remote_util] [140099887036160] - INFO - SSH Connected to 172.23.107.85 as root [2021-12-23 22:47:37,237] - [remote_util] [140099887036160] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-12-23 22:47:37,687] - [remote_util] [140099887036160] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-12-23 22:47:37,732] - [remote_util] [140099887036160] - INFO - 172.23.107.85 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-12-23 22:47:37,875] - [remote_util] [140099887036160] - INFO - Running systemd command on this server [2021-12-23 22:47:37,875] - [remote_util] [140099887036160] - INFO - running command.raw on 172.23.107.85: systemctl stop couchbase-server.service [2021-12-23 22:47:40,378] - [remote_util] [140099887036160] - INFO - command executed successfully with root [2021-12-23 22:47:40,378] - [remote_util] [140099887036160] - INFO - Checking for process beam.smp on linux [2021-12-23 22:47:40,453] - [basetestcase] [140099887036160] - INFO - Couchbase stopped [2021-12-23 22:47:40,454] - [remote_util] [140099887036160] - INFO - running command.raw on 172.23.107.85: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-12-23 22:47:40,481] - [remote_util] [140099887036160] - INFO - command executed successfully with root [2021-12-23 22:47:40,481] - [remote_util] [140099887036160] - INFO - running command.raw on 172.23.107.85: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-12-23 22:47:40,556] - [remote_util] [140099887036160] - INFO - command executed successfully with root [2021-12-23 22:47:40,559] - [remote_util] [140099887036160] - INFO - SSH Connecting to 172.23.107.85 with username:root, attempt#1 of 5 [2021-12-23 22:47:40,723] - [remote_util] [140099887036160] - INFO - SSH Connected to 172.23.107.85 as root [2021-12-23 22:47:41,112] - [remote_util] [140099887036160] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-12-23 22:47:41,544] - [remote_util] [140099887036160] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-12-23 22:47:41,632] - [remote_util] [140099887036160] - INFO - 172.23.107.85 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-12-23 22:47:41,633] - [remote_util] [140099887036160] - INFO - Checking for process beam.smp on linux [2021-12-23 22:47:41,708] - [remote_util] [140099887036160] - INFO - Starting couchbase server [2021-12-23 22:47:41,883] - [remote_util] [140099887036160] - INFO - Running systemd command on this server [2021-12-23 22:47:41,883] - [remote_util] [140099887036160] - INFO - running command.raw on 172.23.107.85: systemctl start couchbase-server.service [2021-12-23 22:47:41,936] - [remote_util] [140099887036160] - INFO - command executed successfully with root [2021-12-23 22:47:41,937] - [remote_util] [140099887036160] - INFO - 172.23.107.85:sleep for 5 secs. waiting for couchbase server to come up ... [2021-12-23 22:47:46,942] - [remote_util] [140099887036160] - INFO - running command.raw on 172.23.107.85: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-12-23 22:47:47,008] - [remote_util] [140099887036160] - INFO - command executed successfully with root [2021-12-23 22:47:47,008] - [remote_util] [140099887036160] - INFO - Couchbase server status: [' Process: 20002 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-12-23 22:47:47,008] - [remote_util] [140099887036160] - INFO - Checking for process beam.smp on linux [2021-12-23 22:47:47,070] - [remote_util] [140099887036160] - INFO - process beam.smp is running on 172.23.107.85: with pid 20143 [2021-12-23 22:47:47,071] - [basetestcase] [140099887036160] - INFO - Couchbase started [2021-12-23 22:47:47,074] - [remote_util] [140099887036160] - INFO - SSH Connecting to 172.23.107.94 with username:root, attempt#1 of 5 [2021-12-23 22:47:47,252] - [remote_util] [140099887036160] - INFO - SSH Connected to 172.23.107.94 as root [2021-12-23 22:47:47,623] - [remote_util] [140099887036160] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-12-23 22:47:48,078] - [remote_util] [140099887036160] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-12-23 22:47:48,096] - [remote_util] [140099887036160] - INFO - SSH Connecting to 172.23.107.94 with username:root, attempt#1 of 5 [2021-12-23 22:47:48,262] - [remote_util] [140099887036160] - INFO - SSH Connected to 172.23.107.94 as root [2021-12-23 22:47:48,672] - [remote_util] [140099887036160] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-12-23 22:47:49,116] - [remote_util] [140099887036160] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-12-23 22:47:49,203] - [remote_util] [140099887036160] - INFO - 172.23.107.94 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-12-23 22:47:49,331] - [remote_util] [140099887036160] - INFO - Running systemd command on this server [2021-12-23 22:47:49,332] - [remote_util] [140099887036160] - INFO - running command.raw on 172.23.107.94: systemctl stop couchbase-server.service [2021-12-23 22:47:51,377] - [remote_util] [140099887036160] - INFO - command executed successfully with root [2021-12-23 22:47:51,377] - [remote_util] [140099887036160] - INFO - Checking for process beam.smp on linux [2021-12-23 22:47:51,445] - [basetestcase] [140099887036160] - INFO - Couchbase stopped [2021-12-23 22:47:51,445] - [remote_util] [140099887036160] - INFO - running command.raw on 172.23.107.94: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-12-23 22:47:51,475] - [remote_util] [140099887036160] - INFO - command executed successfully with root [2021-12-23 22:47:51,475] - [remote_util] [140099887036160] - INFO - running command.raw on 172.23.107.94: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-12-23 22:47:51,553] - [remote_util] [140099887036160] - INFO - command executed successfully with root [2021-12-23 22:47:51,557] - [remote_util] [140099887036160] - INFO - SSH Connecting to 172.23.107.94 with username:root, attempt#1 of 5 [2021-12-23 22:47:51,721] - [remote_util] [140099887036160] - INFO - SSH Connected to 172.23.107.94 as root [2021-12-23 22:47:52,104] - [remote_util] [140099887036160] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-12-23 22:47:52,555] - [remote_util] [140099887036160] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-12-23 22:47:52,643] - [remote_util] [140099887036160] - INFO - 172.23.107.94 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-12-23 22:47:52,643] - [remote_util] [140099887036160] - INFO - Checking for process beam.smp on linux [2021-12-23 22:47:52,718] - [remote_util] [140099887036160] - INFO - Starting couchbase server [2021-12-23 22:47:52,892] - [remote_util] [140099887036160] - INFO - Running systemd command on this server [2021-12-23 22:47:52,892] - [remote_util] [140099887036160] - INFO - running command.raw on 172.23.107.94: systemctl start couchbase-server.service [2021-12-23 22:47:52,951] - [remote_util] [140099887036160] - INFO - command executed successfully with root [2021-12-23 22:47:52,951] - [remote_util] [140099887036160] - INFO - 172.23.107.94:sleep for 5 secs. waiting for couchbase server to come up ... [2021-12-23 22:47:57,956] - [remote_util] [140099887036160] - INFO - running command.raw on 172.23.107.94: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-12-23 22:47:58,020] - [remote_util] [140099887036160] - INFO - command executed successfully with root [2021-12-23 22:47:58,021] - [remote_util] [140099887036160] - INFO - Couchbase server status: [' Process: 58769 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-12-23 22:47:58,021] - [remote_util] [140099887036160] - INFO - Checking for process beam.smp on linux [2021-12-23 22:47:58,086] - [remote_util] [140099887036160] - INFO - process beam.smp is running on 172.23.107.94: with pid 58909 [2021-12-23 22:47:58,086] - [basetestcase] [140099887036160] - INFO - Couchbase started [2021-12-23 22:47:58,235] - [rest_client] [140099887036160] - ERROR - socket error while connecting to http://172.23.107.94:8091/nodes/self error [Errno 111] Connection refused [2021-12-23 22:48:01,239] - [rest_client] [140099887036160] - ERROR - socket error while connecting to http://172.23.107.94:8091/nodes/self error [Errno 111] Connection refused [2021-12-23 22:48:07,348] - [task] [140100243371776] - INFO - server: ip:172.23.108.100 port:8091 ssh_username:root, nodes/self [2021-12-23 22:48:07,353] - [task] [140100243371776] - INFO - {'uptime': '88', 'memoryTotal': 3945775104, 'memoryFree': 3279806464, 'mcdMemoryReserved': 3010, 'mcdMemoryAllocated': 3010, 'status': 'healthy', 'hostname': '172.23.108.100:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-1960-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1591, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.108.100', 'ip': '172.23.108.100', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 3762, 'curr_items': 0} [2021-12-23 22:48:07,356] - [rest_client] [140100243371776] - ERROR - GET http://172.23.108.100:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-12-23 22:48:07,357] - [rest_client] [140100243371776] - INFO - pools/default params : memoryQuota=2016 [2021-12-23 22:48:07,368] - [rest_client] [140100243371776] - INFO - --> init_node_services(Administrator,password,172.23.108.100,8091,['kv']) [2021-12-23 22:48:07,368] - [rest_client] [140100243371776] - INFO - /node/controller/setupServices params on 172.23.108.100: 8091:hostname=172.23.108.100&user=Administrator&password=password&services=kv [2021-12-23 22:48:07,405] - [rest_client] [140100243371776] - INFO - --> in init_cluster...Administrator,password,8091 [2021-12-23 22:48:07,405] - [rest_client] [140100243371776] - INFO - settings/web params on 172.23.108.100:8091:port=8091&username=Administrator&password=password [2021-12-23 22:48:07,482] - [rest_client] [140100243371776] - INFO - --> status:True [2021-12-23 22:48:07,485] - [remote_util] [140100243371776] - INFO - SSH Connecting to 172.23.108.100 with username:root, attempt#1 of 5 [2021-12-23 22:48:07,650] - [remote_util] [140100243371776] - INFO - SSH Connected to 172.23.108.100 as root [2021-12-23 22:48:07,999] - [remote_util] [140100243371776] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-12-23 22:48:08,428] - [remote_util] [140100243371776] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-12-23 22:48:08,429] - [remote_util] [140100243371776] - INFO - running command.raw on 172.23.108.100: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-12-23 22:48:08,518] - [remote_util] [140100243371776] - INFO - command executed successfully with root [2021-12-23 22:48:08,518] - [remote_util] [140100243371776] - INFO - ['ok'] [2021-12-23 22:48:08,523] - [rest_client] [140100243371776] - INFO - /diag/eval status on 172.23.108.100:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2021-12-23 22:48:08,527] - [rest_client] [140100243371776] - INFO - /diag/eval status on 172.23.108.100:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2021-12-23 22:48:08,534] - [rest_client] [140100243371776] - INFO - settings/indexes params : storageMode=plasma [2021-12-23 22:48:08,566] - [task] [140100243371776] - INFO - server: ip:172.23.107.88 port:8091 ssh_username:root, nodes/self [2021-12-23 22:48:08,572] - [task] [140100243371776] - INFO - {'uptime': '78', 'memoryTotal': 3945775104, 'memoryFree': 3352158208, 'mcdMemoryReserved': 3010, 'mcdMemoryAllocated': 3010, 'status': 'healthy', 'hostname': '172.23.107.88:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-1960-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1591, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.107.88', 'ip': '172.23.107.88', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 3762, 'curr_items': 0} [2021-12-23 22:48:08,575] - [rest_client] [140100243371776] - ERROR - GET http://172.23.107.88:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-12-23 22:48:08,575] - [rest_client] [140100243371776] - INFO - pools/default params : memoryQuota=2016 [2021-12-23 22:48:08,590] - [rest_client] [140100243371776] - INFO - --> in init_cluster...Administrator,password,8091 [2021-12-23 22:48:08,591] - [rest_client] [140100243371776] - INFO - settings/web params on 172.23.107.88:8091:port=8091&username=Administrator&password=password [2021-12-23 22:48:08,670] - [rest_client] [140100243371776] - INFO - --> status:True [2021-12-23 22:48:08,672] - [remote_util] [140100243371776] - INFO - SSH Connecting to 172.23.107.88 with username:root, attempt#1 of 5 [2021-12-23 22:48:08,817] - [remote_util] [140100243371776] - INFO - SSH Connected to 172.23.107.88 as root [2021-12-23 22:48:09,258] - [remote_util] [140100243371776] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-12-23 22:48:09,693] - [remote_util] [140100243371776] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-12-23 22:48:09,694] - [remote_util] [140100243371776] - INFO - running command.raw on 172.23.107.88: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-12-23 22:48:09,788] - [remote_util] [140100243371776] - INFO - command executed successfully with root [2021-12-23 22:48:09,788] - [remote_util] [140100243371776] - INFO - ['ok'] [2021-12-23 22:48:09,793] - [rest_client] [140100243371776] - INFO - /diag/eval status on 172.23.107.88:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2021-12-23 22:48:09,797] - [rest_client] [140100243371776] - INFO - /diag/eval status on 172.23.107.88:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2021-12-23 22:48:09,802] - [rest_client] [140100243371776] - INFO - settings/indexes params : storageMode=plasma [2021-12-23 22:48:09,826] - [task] [140100243371776] - INFO - server: ip:172.23.108.0 port:8091 ssh_username:root, nodes/self [2021-12-23 22:48:09,832] - [task] [140100243371776] - INFO - {'uptime': '68', 'memoryTotal': 3945775104, 'memoryFree': 3364184064, 'mcdMemoryReserved': 3010, 'mcdMemoryAllocated': 3010, 'status': 'healthy', 'hostname': '172.23.108.0:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-1960-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1591, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.108.0', 'ip': '172.23.108.0', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 3762, 'curr_items': 0} [2021-12-23 22:48:09,835] - [rest_client] [140100243371776] - ERROR - GET http://172.23.108.0:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-12-23 22:48:09,835] - [rest_client] [140100243371776] - INFO - pools/default params : memoryQuota=2016 [2021-12-23 22:48:09,847] - [rest_client] [140100243371776] - INFO - --> in init_cluster...Administrator,password,8091 [2021-12-23 22:48:09,847] - [rest_client] [140100243371776] - INFO - settings/web params on 172.23.108.0:8091:port=8091&username=Administrator&password=password [2021-12-23 22:48:09,932] - [rest_client] [140100243371776] - INFO - --> status:True [2021-12-23 22:48:09,935] - [remote_util] [140100243371776] - INFO - SSH Connecting to 172.23.108.0 with username:root, attempt#1 of 5 [2021-12-23 22:48:10,065] - [remote_util] [140100243371776] - INFO - SSH Connected to 172.23.108.0 as root [2021-12-23 22:48:10,425] - [remote_util] [140100243371776] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-12-23 22:48:10,848] - [remote_util] [140100243371776] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-12-23 22:48:10,849] - [remote_util] [140100243371776] - INFO - running command.raw on 172.23.108.0: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-12-23 22:48:10,937] - [remote_util] [140100243371776] - INFO - command executed successfully with root [2021-12-23 22:48:10,937] - [remote_util] [140100243371776] - INFO - ['ok'] [2021-12-23 22:48:10,941] - [rest_client] [140100243371776] - INFO - /diag/eval status on 172.23.108.0:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2021-12-23 22:48:10,945] - [rest_client] [140100243371776] - INFO - /diag/eval status on 172.23.108.0:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2021-12-23 22:48:10,953] - [rest_client] [140100243371776] - INFO - settings/indexes params : storageMode=plasma [2021-12-23 22:48:10,983] - [task] [140100243371776] - INFO - server: ip:172.23.107.91 port:8091 ssh_username:root, nodes/self [2021-12-23 22:48:10,989] - [task] [140100243371776] - INFO - {'uptime': '58', 'memoryTotal': 3945775104, 'memoryFree': 3325300736, 'mcdMemoryReserved': 3010, 'mcdMemoryAllocated': 3010, 'status': 'healthy', 'hostname': '172.23.107.91:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-1960-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1591, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.107.91', 'ip': '172.23.107.91', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 3762, 'curr_items': 0} [2021-12-23 22:48:10,992] - [rest_client] [140100243371776] - ERROR - GET http://172.23.107.91:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-12-23 22:48:10,992] - [rest_client] [140100243371776] - INFO - pools/default params : memoryQuota=2016 [2021-12-23 22:48:11,007] - [rest_client] [140100243371776] - INFO - --> in init_cluster...Administrator,password,8091 [2021-12-23 22:48:11,008] - [rest_client] [140100243371776] - INFO - settings/web params on 172.23.107.91:8091:port=8091&username=Administrator&password=password [2021-12-23 22:48:11,096] - [rest_client] [140100243371776] - INFO - --> status:True [2021-12-23 22:48:11,099] - [remote_util] [140100243371776] - INFO - SSH Connecting to 172.23.107.91 with username:root, attempt#1 of 5 [2021-12-23 22:48:11,234] - [remote_util] [140100243371776] - INFO - SSH Connected to 172.23.107.91 as root [2021-12-23 22:48:11,629] - [remote_util] [140100243371776] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-12-23 22:48:12,056] - [remote_util] [140100243371776] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-12-23 22:48:12,057] - [remote_util] [140100243371776] - INFO - running command.raw on 172.23.107.91: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-12-23 22:48:12,142] - [remote_util] [140100243371776] - INFO - command executed successfully with root [2021-12-23 22:48:12,142] - [remote_util] [140100243371776] - INFO - ['ok'] [2021-12-23 22:48:12,147] - [rest_client] [140100243371776] - INFO - /diag/eval status on 172.23.107.91:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2021-12-23 22:48:12,150] - [rest_client] [140100243371776] - INFO - /diag/eval status on 172.23.107.91:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2021-12-23 22:48:12,158] - [rest_client] [140100243371776] - INFO - settings/indexes params : storageMode=plasma [2021-12-23 22:48:12,188] - [task] [140100243371776] - INFO - server: ip:172.23.108.102 port:8091 ssh_username:root, nodes/self [2021-12-23 22:48:12,194] - [task] [140100243371776] - INFO - {'uptime': '47', 'memoryTotal': 3945775104, 'memoryFree': 3365281792, 'mcdMemoryReserved': 3010, 'mcdMemoryAllocated': 3010, 'status': 'healthy', 'hostname': '172.23.108.102:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-1960-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1591, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.108.102', 'ip': '172.23.108.102', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 3762, 'curr_items': 0} [2021-12-23 22:48:12,197] - [rest_client] [140100243371776] - ERROR - GET http://172.23.108.102:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-12-23 22:48:12,197] - [rest_client] [140100243371776] - INFO - pools/default params : memoryQuota=2016 [2021-12-23 22:48:12,210] - [rest_client] [140100243371776] - INFO - --> in init_cluster...Administrator,password,8091 [2021-12-23 22:48:12,210] - [rest_client] [140100243371776] - INFO - settings/web params on 172.23.108.102:8091:port=8091&username=Administrator&password=password [2021-12-23 22:48:12,293] - [rest_client] [140100243371776] - INFO - --> status:True [2021-12-23 22:48:12,296] - [remote_util] [140100243371776] - INFO - SSH Connecting to 172.23.108.102 with username:root, attempt#1 of 5 [2021-12-23 22:48:12,458] - [remote_util] [140100243371776] - INFO - SSH Connected to 172.23.108.102 as root [2021-12-23 22:48:12,835] - [remote_util] [140100243371776] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-12-23 22:48:13,253] - [remote_util] [140100243371776] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-12-23 22:48:13,254] - [remote_util] [140100243371776] - INFO - running command.raw on 172.23.108.102: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-12-23 22:48:13,332] - [remote_util] [140100243371776] - INFO - command executed successfully with root [2021-12-23 22:48:13,332] - [remote_util] [140100243371776] - INFO - ['ok'] [2021-12-23 22:48:13,337] - [rest_client] [140100243371776] - INFO - /diag/eval status on 172.23.108.102:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2021-12-23 22:48:13,340] - [rest_client] [140100243371776] - INFO - /diag/eval status on 172.23.108.102:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2021-12-23 22:48:13,346] - [rest_client] [140100243371776] - INFO - settings/indexes params : storageMode=plasma [2021-12-23 22:48:13,377] - [task] [140100243371776] - INFO - server: ip:172.23.107.84 port:8091 ssh_username:root, nodes/self [2021-12-23 22:48:13,382] - [task] [140100243371776] - INFO - {'uptime': '38', 'memoryTotal': 3945775104, 'memoryFree': 3294547968, 'mcdMemoryReserved': 3010, 'mcdMemoryAllocated': 3010, 'status': 'healthy', 'hostname': '172.23.107.84:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-1960-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1591, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.107.84', 'ip': '172.23.107.84', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 3762, 'curr_items': 0} [2021-12-23 22:48:13,386] - [rest_client] [140100243371776] - ERROR - GET http://172.23.107.84:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-12-23 22:48:13,386] - [rest_client] [140100243371776] - INFO - pools/default params : memoryQuota=2016 [2021-12-23 22:48:13,400] - [rest_client] [140100243371776] - INFO - --> in init_cluster...Administrator,password,8091 [2021-12-23 22:48:13,401] - [rest_client] [140100243371776] - INFO - settings/web params on 172.23.107.84:8091:port=8091&username=Administrator&password=password [2021-12-23 22:48:13,475] - [rest_client] [140100243371776] - INFO - --> status:True [2021-12-23 22:48:13,478] - [remote_util] [140100243371776] - INFO - SSH Connecting to 172.23.107.84 with username:root, attempt#1 of 5 [2021-12-23 22:48:13,630] - [remote_util] [140100243371776] - INFO - SSH Connected to 172.23.107.84 as root [2021-12-23 22:48:14,008] - [remote_util] [140100243371776] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-12-23 22:48:14,461] - [remote_util] [140100243371776] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-12-23 22:48:14,462] - [remote_util] [140100243371776] - INFO - running command.raw on 172.23.107.84: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-12-23 22:48:14,557] - [remote_util] [140100243371776] - INFO - command executed successfully with root [2021-12-23 22:48:14,558] - [remote_util] [140100243371776] - INFO - ['ok'] [2021-12-23 22:48:14,563] - [rest_client] [140100243371776] - INFO - /diag/eval status on 172.23.107.84:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2021-12-23 22:48:14,566] - [rest_client] [140100243371776] - INFO - /diag/eval status on 172.23.107.84:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2021-12-23 22:48:14,574] - [rest_client] [140100243371776] - INFO - settings/indexes params : storageMode=plasma [2021-12-23 22:48:14,602] - [task] [140100243371776] - INFO - server: ip:172.23.107.85 port:8091 ssh_username:root, nodes/self [2021-12-23 22:48:14,608] - [task] [140100243371776] - INFO - {'uptime': '28', 'memoryTotal': 3945775104, 'memoryFree': 3283034112, 'mcdMemoryReserved': 3010, 'mcdMemoryAllocated': 3010, 'status': 'healthy', 'hostname': '172.23.107.85:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-1960-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1591, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.107.85', 'ip': '172.23.107.85', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 3762, 'curr_items': 0} [2021-12-23 22:48:14,611] - [rest_client] [140100243371776] - ERROR - GET http://172.23.107.85:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-12-23 22:48:14,612] - [rest_client] [140100243371776] - INFO - pools/default params : memoryQuota=2016 [2021-12-23 22:48:14,625] - [rest_client] [140100243371776] - INFO - --> in init_cluster...Administrator,password,8091 [2021-12-23 22:48:14,625] - [rest_client] [140100243371776] - INFO - settings/web params on 172.23.107.85:8091:port=8091&username=Administrator&password=password [2021-12-23 22:48:14,714] - [rest_client] [140100243371776] - INFO - --> status:True [2021-12-23 22:48:14,716] - [remote_util] [140100243371776] - INFO - SSH Connecting to 172.23.107.85 with username:root, attempt#1 of 5 [2021-12-23 22:48:14,867] - [remote_util] [140100243371776] - INFO - SSH Connected to 172.23.107.85 as root [2021-12-23 22:48:15,249] - [remote_util] [140100243371776] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-12-23 22:48:15,686] - [remote_util] [140100243371776] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-12-23 22:48:15,687] - [remote_util] [140100243371776] - INFO - running command.raw on 172.23.107.85: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-12-23 22:48:15,778] - [remote_util] [140100243371776] - INFO - command executed successfully with root [2021-12-23 22:48:15,779] - [remote_util] [140100243371776] - INFO - ['ok'] [2021-12-23 22:48:15,784] - [rest_client] [140100243371776] - INFO - /diag/eval status on 172.23.107.85:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2021-12-23 22:48:15,787] - [rest_client] [140100243371776] - INFO - /diag/eval status on 172.23.107.85:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2021-12-23 22:48:15,794] - [rest_client] [140100243371776] - INFO - settings/indexes params : storageMode=plasma [2021-12-23 22:48:15,826] - [task] [140100243371776] - INFO - server: ip:172.23.107.94 port:8091 ssh_username:root, nodes/self [2021-12-23 22:48:15,832] - [task] [140100243371776] - INFO - {'uptime': '17', 'memoryTotal': 3945779200, 'memoryFree': 3308093440, 'mcdMemoryReserved': 3010, 'mcdMemoryAllocated': 3010, 'status': 'healthy', 'hostname': '172.23.107.94:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-1960-enterprise', 'os': 'x86_64-pc-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1591, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.107.94', 'ip': '172.23.107.94', 'internal_ip': '', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 3762, 'curr_items': 0} [2021-12-23 22:48:15,835] - [rest_client] [140100243371776] - ERROR - GET http://172.23.107.94:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-12-23 22:48:15,836] - [rest_client] [140100243371776] - INFO - pools/default params : memoryQuota=2016 [2021-12-23 22:48:15,850] - [rest_client] [140100243371776] - INFO - --> in init_cluster...Administrator,password,8091 [2021-12-23 22:48:15,851] - [rest_client] [140100243371776] - INFO - settings/web params on 172.23.107.94:8091:port=8091&username=Administrator&password=password [2021-12-23 22:48:15,939] - [rest_client] [140100243371776] - INFO - --> status:True [2021-12-23 22:48:15,942] - [remote_util] [140100243371776] - INFO - SSH Connecting to 172.23.107.94 with username:root, attempt#1 of 5 [2021-12-23 22:48:16,102] - [remote_util] [140100243371776] - INFO - SSH Connected to 172.23.107.94 as root [2021-12-23 22:48:16,476] - [remote_util] [140100243371776] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-12-23 22:48:16,902] - [remote_util] [140100243371776] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-12-23 22:48:16,903] - [remote_util] [140100243371776] - INFO - running command.raw on 172.23.107.94: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-12-23 22:48:16,995] - [remote_util] [140100243371776] - INFO - command executed successfully with root [2021-12-23 22:48:16,995] - [remote_util] [140100243371776] - INFO - ['ok'] [2021-12-23 22:48:17,000] - [rest_client] [140100243371776] - INFO - /diag/eval status on 172.23.107.94:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2021-12-23 22:48:17,003] - [rest_client] [140100243371776] - INFO - /diag/eval status on 172.23.107.94:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2021-12-23 22:48:17,010] - [rest_client] [140100243371776] - INFO - settings/indexes params : storageMode=plasma [2021-12-23 22:48:17,036] - [basetestcase] [140099887036160] - INFO - **** add built-in 'cbadminbucket' user to node 172.23.108.100 **** [2021-12-23 22:48:17,059] - [rest_client] [140099887036160] - ERROR - DELETE http://172.23.108.100:8091/settings/rbac/users/local/cbadminbucket body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:password [2021-12-23 22:48:17,060] - [internal_user] [140099887036160] - INFO - Exception while deleting user. Exception is -b'"User was not found."' [2021-12-23 22:48:17,143] - [basetestcase] [140099887036160] - INFO - sleep for 5 secs. ... [2021-12-23 22:48:22,148] - [basetestcase] [140099887036160] - INFO - **** add 'admin' role to 'cbadminbucket' user **** [2021-12-23 22:48:22,169] - [basetestcase] [140099887036160] - INFO - done initializing cluster [2021-12-23 22:48:23,043] - [task] [140100243371776] - INFO - adding node 172.23.107.88:8091 to cluster [2021-12-23 22:48:23,053] - [rest_client] [140100243371776] - INFO - adding remote node @172.23.107.88:18091 to this cluster @172.23.108.100:8091 [2021-12-23 22:48:33,072] - [rest_client] [140100243371776] - INFO - rebalance progress took 10.02 seconds [2021-12-23 22:48:33,072] - [rest_client] [140100243371776] - INFO - sleep for 10 seconds after rebalance... [2021-12-23 22:48:47,493] - [task] [140100243371776] - INFO - adding node 172.23.108.0:8091 to cluster [2021-12-23 22:48:47,506] - [rest_client] [140100243371776] - INFO - adding remote node @172.23.108.0:18091 to this cluster @172.23.108.100:8091 [2021-12-23 22:48:57,524] - [rest_client] [140100243371776] - INFO - rebalance progress took 10.02 seconds [2021-12-23 22:48:57,524] - [rest_client] [140100243371776] - INFO - sleep for 10 seconds after rebalance... [2021-12-23 22:49:11,680] - [rest_client] [140100243371776] - INFO - Node 172.23.107.88 not part of cluster inactiveAdded [2021-12-23 22:49:11,681] - [rest_client] [140100243371776] - INFO - Node 172.23.108.0 not part of cluster inactiveAdded [2021-12-23 22:49:11,704] - [rest_client] [140100243371776] - INFO - rebalance params : {'knownNodes': 'ns_1@172.23.107.88,ns_1@172.23.108.0,ns_1@172.23.108.100', 'ejectedNodes': '', 'user': 'Administrator', 'password': 'password'} [2021-12-23 22:49:11,767] - [rest_client] [140100243371776] - INFO - rebalance operation started [2021-12-23 22:49:21,777] - [task] [140100243371776] - INFO - Rebalance - status: none, progress: 100.00% [2021-12-23 22:49:21,787] - [task] [140100243371776] - INFO - rebalancing was completed with progress: 100% in 10.019341945648193 sec [2021-12-23 22:49:21,789] - [remote_util] [140099887036160] - INFO - SSH Connecting to 172.23.108.100 with username:root, attempt#1 of 5 [2021-12-23 22:49:21,967] - [remote_util] [140099887036160] - INFO - SSH Connected to 172.23.108.100 as root [2021-12-23 22:49:22,330] - [remote_util] [140099887036160] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-12-23 22:49:22,755] - [remote_util] [140099887036160] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-12-23 22:49:22,755] - [remote_util] [140099887036160] - INFO - running command.raw on 172.23.108.100: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-12-23 22:49:22,806] - [remote_util] [140099887036160] - INFO - command executed successfully with root [2021-12-23 22:49:22,806] - [remote_util] [140099887036160] - INFO - ['ok'] [2021-12-23 22:49:22,807] - [basetestcase] [140099887036160] - INFO - Enabled diag/eval for non-local hosts from 172.23.108.100 [2021-12-23 22:49:22,807] - [basetestcase] [140099887036160] - INFO - ============== basetestcase setup was finished for test #5 test_rebalance_in_with_different_topologies ============== [2021-12-23 22:49:22,820] - [remote_util] [140099887036160] - INFO - SSH Connecting to 172.23.108.100 with username:root, attempt#1 of 5 [2021-12-23 22:49:22,982] - [remote_util] [140099887036160] - INFO - SSH Connected to 172.23.108.100 as root [2021-12-23 22:49:23,360] - [remote_util] [140099887036160] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-12-23 22:49:23,790] - [remote_util] [140099887036160] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-12-23 22:49:23,794] - [remote_util] [140099887036160] - INFO - SSH Connecting to 172.23.107.88 with username:root, attempt#1 of 5 [2021-12-23 22:49:23,964] - [remote_util] [140099887036160] - INFO - SSH Connected to 172.23.107.88 as root [2021-12-23 22:49:24,382] - [remote_util] [140099887036160] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-12-23 22:49:24,817] - [remote_util] [140099887036160] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-12-23 22:49:24,821] - [remote_util] [140099887036160] - INFO - SSH Connecting to 172.23.108.0 with username:root, attempt#1 of 5 [2021-12-23 22:49:24,990] - [remote_util] [140099887036160] - INFO - SSH Connected to 172.23.108.0 as root [2021-12-23 22:49:25,360] - [remote_util] [140099887036160] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-12-23 22:49:25,798] - [remote_util] [140099887036160] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-12-23 22:49:25,801] - [remote_util] [140099887036160] - INFO - SSH Connecting to 172.23.107.91 with username:root, attempt#1 of 5 [2021-12-23 22:49:25,968] - [remote_util] [140099887036160] - INFO - SSH Connected to 172.23.107.91 as root [2021-12-23 22:49:26,356] - [remote_util] [140099887036160] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-12-23 22:49:26,786] - [remote_util] [140099887036160] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-12-23 22:49:26,788] - [remote_util] [140099887036160] - INFO - SSH Connecting to 172.23.108.102 with username:root, attempt#1 of 5 [2021-12-23 22:49:26,960] - [remote_util] [140099887036160] - INFO - SSH Connected to 172.23.108.102 as root [2021-12-23 22:49:27,353] - [remote_util] [140099887036160] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-12-23 22:49:27,791] - [remote_util] [140099887036160] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-12-23 22:49:27,794] - [remote_util] [140099887036160] - INFO - SSH Connecting to 172.23.107.84 with username:root, attempt#1 of 5 [2021-12-23 22:49:27,964] - [remote_util] [140099887036160] - INFO - SSH Connected to 172.23.107.84 as root [2021-12-23 22:49:28,365] - [remote_util] [140099887036160] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-12-23 22:49:28,804] - [remote_util] [140099887036160] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-12-23 22:49:28,807] - [remote_util] [140099887036160] - INFO - SSH Connecting to 172.23.107.85 with username:root, attempt#1 of 5 [2021-12-23 22:49:28,975] - [remote_util] [140099887036160] - INFO - SSH Connected to 172.23.107.85 as root [2021-12-23 22:49:29,360] - [remote_util] [140099887036160] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-12-23 22:49:29,789] - [remote_util] [140099887036160] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-12-23 22:49:29,791] - [remote_util] [140099887036160] - INFO - SSH Connecting to 172.23.107.94 with username:root, attempt#1 of 5 [2021-12-23 22:49:29,953] - [remote_util] [140099887036160] - INFO - SSH Connected to 172.23.107.94 as root [2021-12-23 22:49:30,374] - [remote_util] [140099887036160] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-12-23 22:49:30,799] - [remote_util] [140099887036160] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-12-23 22:49:49,869] - [basetestcase] [140099887036160] - INFO - ------- Cluster statistics ------- [2021-12-23 22:49:49,870] - [basetestcase] [140099887036160] - INFO - 172.23.107.88:8091 => {'services': ['eventing', 'n1ql'], 'cpu_utilization': 2.070184296894724, 'mem_free': 3281350656, 'mem_total': 3945775104, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-12-23 22:49:49,870] - [basetestcase] [140099887036160] - INFO - 172.23.108.0:8091 => {'services': ['eventing', 'index', 'n1ql'], 'cpu_utilization': 13.04237503171784, 'mem_free': 3247849472, 'mem_total': 3945775104, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-12-23 22:49:49,870] - [basetestcase] [140099887036160] - INFO - 172.23.108.100:8091 => {'services': ['kv'], 'cpu_utilization': 1.931276649109606, 'mem_free': 3252047872, 'mem_total': 3945775104, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-12-23 22:49:49,870] - [basetestcase] [140099887036160] - INFO - --- End of cluster statistics --- [2021-12-23 22:49:49,870] - [basetestcase] [140099887036160] - INFO - Time to execute basesetup : 202.63782262802124 [2021-12-23 22:49:49,873] - [remote_util] [140099887036160] - INFO - SSH Connecting to 172.23.108.100 with username:root, attempt#1 of 5 [2021-12-23 22:49:50,028] - [remote_util] [140099887036160] - INFO - SSH Connected to 172.23.108.100 as root [2021-12-23 22:49:50,398] - [remote_util] [140099887036160] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-12-23 22:49:50,818] - [remote_util] [140099887036160] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-12-23 22:49:54,920] - [basetestcase] [140099887036160] - INFO - list of n1ql nodes in cluster: [ip:172.23.107.88 port:8091 ssh_username:root, ip:172.23.108.0 port:8091 ssh_username:root] [2021-12-23 22:49:54,929] - [basetestcase] [140099887036160] - INFO - list of eventing nodes in cluster: [ip:172.23.107.88 port:8091 ssh_username:root, ip:172.23.108.0 port:8091 ssh_username:root] [2021-12-23 22:49:54,938] - [rest_client] [140099887036160] - INFO - settings/indexes params : storageMode=plasma [2021-12-23 22:49:54,947] - [eventing_base] [140099887036160] - INFO - Setting the min possible memory quota so that adding mode nodes to the cluster wouldn't be a problem. [2021-12-23 22:49:54,947] - [rest_client] [140099887036160] - INFO - pools/default params : memoryQuota=330 [2021-12-23 22:49:54,959] - [rest_client] [140099887036160] - INFO - pools/default params : indexMemoryQuota=256 [2021-12-23 22:49:54,967] - [rest_client] [140099887036160] - INFO - pools/default params : eventingMemoryQuota=512 [2021-12-23 22:49:54,986] - [basetestcase] [140099887036160] - INFO - list of n1ql nodes in cluster: [ip:172.23.107.88 port:8091 ssh_username:root, ip:172.23.108.0 port:8091 ssh_username:root] [2021-12-23 22:49:54,990] - [eventing_base] [140099887036160] - INFO - ============== EventingBaseTest setup has completed ============== [2021-12-23 22:49:54,990] - [rest_client] [140099887036160] - INFO - pools/default params : memoryQuota=1200 [2021-12-23 22:49:54,995] - [eventing_rebalance] [140099887036160] - INFO - 200 [2021-12-23 22:49:55,915] - [rest_client] [140100243371776] - INFO - http://172.23.108.100:8091/pools/default/buckets with param: name=src_bucket&ramQuotaMB=200&replicaNumber=0&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore [2021-12-23 22:49:55,951] - [rest_client] [140100243371776] - INFO - 0.04 seconds to create bucket src_bucket [2021-12-23 22:49:55,952] - [bucket_helper] [140100243371776] - INFO - waiting for memcached bucket : src_bucket in 172.23.108.100 to accept set ops [2021-12-23 22:49:56,603] - [data_helper] [140100243371776] - INFO - creating direct client 172.23.108.100:11210 src_bucket [2021-12-23 22:49:56,717] - [data_helper] [140100243371776] - INFO - creating direct client 172.23.108.100:11210 src_bucket [2021-12-23 22:49:58,152] - [task] [140100243371776] - INFO - bucket 'src_bucket' was created with per node RAM quota: 200 [2021-12-23 22:49:59,188] - [rest_client] [140100243371776] - INFO - http://172.23.108.100:8091/pools/default/buckets with param: name=dst_bucket&ramQuotaMB=200&replicaNumber=0&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore [2021-12-23 22:49:59,225] - [rest_client] [140100243371776] - INFO - 0.04 seconds to create bucket dst_bucket [2021-12-23 22:49:59,226] - [bucket_helper] [140100243371776] - INFO - waiting for memcached bucket : dst_bucket in 172.23.108.100 to accept set ops [2021-12-23 22:49:59,860] - [data_helper] [140100243371776] - INFO - creating direct client 172.23.108.100:11210 dst_bucket [2021-12-23 22:50:00,098] - [data_helper] [140100243371776] - INFO - creating direct client 172.23.108.100:11210 dst_bucket [2021-12-23 22:50:01,621] - [task] [140100243371776] - INFO - bucket 'dst_bucket' was created with per node RAM quota: 200 [2021-12-23 22:50:02,655] - [rest_client] [140100243371776] - INFO - http://172.23.108.100:8091/pools/default/buckets with param: name=metadata&ramQuotaMB=400&replicaNumber=0&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore [2021-12-23 22:50:02,694] - [rest_client] [140100243371776] - INFO - 0.04 seconds to create bucket metadata [2021-12-23 22:50:02,694] - [bucket_helper] [140100243371776] - INFO - waiting for memcached bucket : metadata in 172.23.108.100 to accept set ops [2021-12-23 22:50:03,326] - [data_helper] [140100243371776] - INFO - creating direct client 172.23.108.100:11210 metadata [2021-12-23 22:50:03,421] - [data_helper] [140100243371776] - INFO - creating direct client 172.23.108.100:11210 metadata [2021-12-23 22:50:04,132] - [task] [140100243371776] - INFO - bucket 'metadata' was created with per node RAM quota: 400 [2021-12-23 22:50:04,302] - [basetestcase] [140099887036160] - INFO - list of n1ql nodes in cluster: [ip:172.23.107.88 port:8091 ssh_username:root, ip:172.23.108.0 port:8091 ssh_username:root] [2021-12-23 22:50:04,303] - [tuq_helper] [140099887036160] - INFO - Check if index existed in dst_bucket on server 172.23.107.88 [2021-12-23 22:50:04,313] - [tuq_helper] [140099887036160] - INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary' [2021-12-23 22:50:04,320] - [rest_client] [140099887036160] - INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27 [2021-12-23 22:50:04,599] - [tuq_helper] [140099887036160] - INFO - TOTAL ELAPSED TIME: 274.417496ms [2021-12-23 22:50:04,599] - [tuq_helper] [140099887036160] - ERROR - Fail to get index list. List output: {'requestID': 'ba4dfd8a-e026-4ac6-b212-31db0f7edf0d', 'signature': {'*': '*'}, 'results': [], 'status': 'success', 'metrics': {'elapsedTime': '274.417496ms', 'executionTime': '274.051015ms', 'resultCount': 0, 'resultSize': 0, 'serviceLoad': 6}} [2021-12-23 22:50:04,599] - [tuq_helper] [140099887036160] - INFO - Create primary index [2021-12-23 22:50:04,605] - [tuq_helper] [140099887036160] - INFO - RUN QUERY CREATE PRIMARY INDEX ON dst_bucket USING GSI [2021-12-23 22:50:04,609] - [rest_client] [140099887036160] - INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+dst_bucket++USING+GSI [2021-12-23 22:50:06,958] - [tuq_helper] [140099887036160] - INFO - TOTAL ELAPSED TIME: 2.343936415s [2021-12-23 22:50:06,958] - [tuq_helper] [140099887036160] - INFO - Check if index is online [2021-12-23 22:50:06,966] - [tuq_helper] [140099887036160] - INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary' [2021-12-23 22:50:06,973] - [rest_client] [140099887036160] - INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27 [2021-12-23 22:50:07,000] - [tuq_helper] [140099887036160] - INFO - TOTAL ELAPSED TIME: 24.479367ms [2021-12-23 22:50:07,001] - [tuq_helper] [140099887036160] - INFO - Check if index existed in metadata on server 172.23.107.88 [2021-12-23 22:50:07,008] - [tuq_helper] [140099887036160] - INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary' [2021-12-23 22:50:07,015] - [rest_client] [140099887036160] - INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27 [2021-12-23 22:50:07,021] - [tuq_helper] [140099887036160] - INFO - TOTAL ELAPSED TIME: 2.948218ms [2021-12-23 22:50:07,021] - [tuq_helper] [140099887036160] - INFO - Create primary index [2021-12-23 22:50:07,029] - [tuq_helper] [140099887036160] - INFO - RUN QUERY CREATE PRIMARY INDEX ON metadata USING GSI [2021-12-23 22:50:07,036] - [rest_client] [140099887036160] - INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+metadata++USING+GSI [2021-12-23 22:50:09,543] - [tuq_helper] [140099887036160] - INFO - TOTAL ELAPSED TIME: 2.504501366s [2021-12-23 22:50:09,544] - [tuq_helper] [140099887036160] - INFO - Check if index is online [2021-12-23 22:50:09,551] - [tuq_helper] [140099887036160] - INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary' [2021-12-23 22:50:09,562] - [rest_client] [140099887036160] - INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27 [2021-12-23 22:50:09,591] - [tuq_helper] [140099887036160] - INFO - TOTAL ELAPSED TIME: 25.852354ms [2021-12-23 22:50:09,591] - [tuq_helper] [140099887036160] - INFO - Check if index existed in src_bucket on server 172.23.107.88 [2021-12-23 22:50:09,596] - [tuq_helper] [140099887036160] - INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary' [2021-12-23 22:50:09,601] - [rest_client] [140099887036160] - INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27 [2021-12-23 22:50:09,607] - [tuq_helper] [140099887036160] - INFO - TOTAL ELAPSED TIME: 2.560665ms [2021-12-23 22:50:09,607] - [tuq_helper] [140099887036160] - INFO - Create primary index [2021-12-23 22:50:09,613] - [tuq_helper] [140099887036160] - INFO - RUN QUERY CREATE PRIMARY INDEX ON src_bucket USING GSI [2021-12-23 22:50:09,619] - [rest_client] [140099887036160] - INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+src_bucket++USING+GSI [2021-12-23 22:50:12,161] - [tuq_helper] [140099887036160] - INFO - TOTAL ELAPSED TIME: 2.539206014s [2021-12-23 22:50:12,161] - [tuq_helper] [140099887036160] - INFO - Check if index is online [2021-12-23 22:50:12,169] - [tuq_helper] [140099887036160] - INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary' [2021-12-23 22:50:12,176] - [rest_client] [140099887036160] - INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27 [2021-12-23 22:50:12,200] - [tuq_helper] [140099887036160] - INFO - TOTAL ELAPSED TIME: 21.181562ms [2021-12-23 22:50:13,908] - [eventing_base] [140099887036160] - INFO - saving function b'{\n "code": 0,\n "info": {\n "status": "Stored function: \'Function_618248006_test_rebalance_in_with_different_topologies\' in metakv",\n "warnings": null\n }\n}' [2021-12-23 22:50:13,908] - [eventing_base] [140099887036160] - INFO - Deploying the following handler code : Function_618248006_test_rebalance_in_with_different_topologies with bindings: {'buckets': [{'alias': 'dst_bucket', 'bucket_name': 'dst_bucket', 'access': 'rw'}], 'metadata_bucket': 'metadata', 'source_bucket': 'src_bucket', 'curl': []} and settings: {'checkpoint_interval': 20000, 'cleanup_timers': False, 'dcp_stream_boundary': 'everything', 'deployment_status': False, 'description': 'Sample Description', 'log_level': 'INFO', 'skip_timer_threshold': 86400, 'sock_batch_size': 1, 'tick_duration': 5000, 'timer_processing_tick_interval': 500, 'timer_worker_pool_size': 3, 'worker_count': 3, 'processing_status': False, 'cpp_worker_thread_count': 1, 'execution_timeout': 20, 'data_chan_size': 10000, 'worker_queue_cap': 100000, 'use_memory_manager': True, 'deadline_timeout': 21, 'timer_storage_chan_size': 10000, 'dcp_gen_chan_size': 10000, 'language_compatibility': '6.6.2'} [2021-12-23 22:50:13,908] - [eventing_base] [140099887036160] - INFO - function OnUpdate(doc, meta) { log("Doc created/updated", meta.id); var result1= couchbase.insert(dst_bucket,meta,doc); log(result1); } function OnDelete(meta, options) { log("Doc deleted/expired", meta.id); var doc={"id":meta.id} var result = couchbase.delete(dst_bucket,doc); log(result); } [2021-12-23 22:50:15,026] - [eventing_base] [140099887036160] - INFO - deploy Application : b'' [2021-12-23 22:50:15,027] - [basetestcase] [140099887036160] - INFO - sleep for 5 secs. Waiting for Function_618248006_test_rebalance_in_with_different_topologies to deployed... ... [2021-12-23 22:50:20,052] - [basetestcase] [140099887036160] - INFO - sleep for 5 secs. Waiting for Function_618248006_test_rebalance_in_with_different_topologies to deployed... ... [2021-12-23 22:50:25,076] - [basetestcase] [140099887036160] - INFO - sleep for 5 secs. Waiting for Function_618248006_test_rebalance_in_with_different_topologies to deployed... ... [2021-12-23 22:50:30,099] - [basetestcase] [140099887036160] - INFO - sleep for 5 secs. Waiting for Function_618248006_test_rebalance_in_with_different_topologies to deployed... ... [2021-12-23 22:50:35,197] - [data_helper] [140099887036160] - INFO - creating direct client 172.23.108.100:11210 src_bucket [2021-12-23 22:50:36,173] - [task] [140100243371776] - INFO - adding node 172.23.107.91:8091 to cluster [2021-12-23 22:50:36,193] - [rest_client] [140100243371776] - INFO - adding remote node @172.23.107.91:18091 to this cluster @172.23.108.100:8091 [2021-12-23 22:50:46,205] - [rest_client] [140100243371776] - INFO - rebalance progress took 10.01 seconds [2021-12-23 22:50:46,205] - [rest_client] [140100243371776] - INFO - sleep for 10 seconds after rebalance... [2021-12-23 22:51:00,310] - [rest_client] [140100243371776] - INFO - Node 172.23.107.91 not part of cluster inactiveAdded [2021-12-23 22:51:00,418] - [rest_client] [140100243371776] - INFO - rebalance params : {'knownNodes': 'ns_1@172.23.107.88,ns_1@172.23.107.91,ns_1@172.23.108.0,ns_1@172.23.108.100', 'ejectedNodes': '', 'user': 'Administrator', 'password': 'password'} [2021-12-23 22:51:00,496] - [rest_client] [140100243371776] - INFO - rebalance operation started [2021-12-23 22:51:10,502] - [rest_client] [140100243371776] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:51:10,502] - [task] [140100243371776] - INFO - Rebalance - status: running, progress: 33.00% [2021-12-23 22:51:27,027] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:51:30,035] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:51:30,536] - [rest_client] [140100243371776] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:51:30,536] - [task] [140100243371776] - INFO - Rebalance - status: running, progress: 33.00% [2021-12-23 22:51:33,043] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:51:36,051] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:51:39,060] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:51:42,068] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:51:45,076] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:51:48,085] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:51:50,573] - [rest_client] [140100243371776] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:51:50,573] - [task] [140100243371776] - INFO - Rebalance - status: running, progress: 33.00% [2021-12-23 22:51:51,092] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:51:54,100] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:51:57,108] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:52:00,117] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:52:03,125] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:52:06,136] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:52:09,144] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:52:10,610] - [rest_client] [140100243371776] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:52:10,610] - [task] [140100243371776] - INFO - Rebalance - status: running, progress: 33.00% [2021-12-23 22:52:12,152] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:52:15,161] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:52:18,169] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:52:21,177] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:52:24,185] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:52:27,192] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:52:30,201] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:52:30,646] - [rest_client] [140100243371776] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:52:30,646] - [task] [140100243371776] - INFO - Rebalance - status: running, progress: 33.00% [2021-12-23 22:52:33,209] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:52:36,218] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:52:39,226] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:52:42,234] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:52:45,242] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:52:48,249] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:52:50,683] - [rest_client] [140100243371776] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:52:50,683] - [task] [140100243371776] - INFO - Rebalance - status: running, progress: 33.00% [2021-12-23 22:52:51,257] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:52:54,265] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:52:57,274] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:53:00,282] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:53:03,290] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:53:06,298] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:53:09,307] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:53:10,723] - [rest_client] [140100243371776] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:53:10,723] - [task] [140100243371776] - INFO - Rebalance - status: running, progress: 33.00% [2021-12-23 22:53:12,315] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:53:15,323] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:53:18,331] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:53:21,340] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:53:24,346] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:53:27,354] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:53:30,362] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:53:30,760] - [rest_client] [140100243371776] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:53:30,760] - [task] [140100243371776] - INFO - Rebalance - status: running, progress: 33.00% [2021-12-23 22:53:33,371] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:53:36,379] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:53:39,388] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:53:42,399] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:53:45,408] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:53:48,416] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:53:50,796] - [rest_client] [140100243371776] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:53:50,796] - [task] [140100243371776] - INFO - Rebalance - status: running, progress: 33.00% [2021-12-23 22:53:51,428] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:53:54,436] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:53:57,444] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:54:00,456] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:54:03,464] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:54:06,472] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:54:09,481] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:54:10,838] - [rest_client] [140100243371776] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:54:10,838] - [task] [140100243371776] - INFO - Rebalance - status: running, progress: 33.00% [2021-12-23 22:54:12,489] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:54:15,498] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:54:18,506] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:54:21,514] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:54:24,523] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:54:27,532] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:54:30,540] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:54:30,875] - [rest_client] [140100243371776] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:54:30,876] - [task] [140100243371776] - INFO - Rebalance - status: running, progress: 33.00% [2021-12-23 22:54:33,549] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:54:36,558] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:54:39,566] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:54:42,575] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:54:45,585] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:54:48,595] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:54:50,916] - [rest_client] [140100243371776] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:54:50,916] - [task] [140100243371776] - INFO - Rebalance - status: running, progress: 33.00% [2021-12-23 22:54:51,603] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:54:54,612] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:54:57,621] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:55:00,626] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:55:03,635] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:55:06,643] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:55:09,652] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:55:10,963] - [rest_client] [140100243371776] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:55:10,963] - [task] [140100243371776] - INFO - Rebalance - status: running, progress: 33.00% [2021-12-23 22:55:12,660] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:55:15,668] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:55:18,677] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:55:21,685] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:55:24,694] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:55:27,705] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:55:30,714] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:55:31,004] - [rest_client] [140100243371776] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:55:31,005] - [task] [140100243371776] - INFO - Rebalance - status: running, progress: 33.00% [2021-12-23 22:55:33,725] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:55:36,733] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:55:39,742] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:55:42,750] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:55:45,759] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:55:48,767] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:55:51,045] - [rest_client] [140100243371776] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:55:51,045] - [task] [140100243371776] - INFO - Rebalance - status: running, progress: 33.00% [2021-12-23 22:55:51,775] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:55:54,785] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:55:57,793] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:56:00,802] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:56:03,810] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:56:06,818] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:56:09,827] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:56:11,083] - [rest_client] [140100243371776] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:56:11,083] - [task] [140100243371776] - INFO - Rebalance - status: running, progress: 33.00% [2021-12-23 22:56:12,836] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:56:15,845] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:56:18,855] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:56:21,863] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:56:24,872] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:56:27,878] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:56:30,887] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:56:31,119] - [rest_client] [140100243371776] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:56:31,119] - [task] [140100243371776] - INFO - Rebalance - status: running, progress: 33.00% [2021-12-23 22:56:33,895] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:56:36,904] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:56:39,914] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:56:42,928] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:56:45,938] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:56:48,947] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:56:51,157] - [rest_client] [140100243371776] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:56:51,157] - [task] [140100243371776] - INFO - Rebalance - status: running, progress: 33.00% [2021-12-23 22:56:51,956] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:56:54,964] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:56:57,972] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:57:00,981] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:57:03,990] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:57:06,998] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:57:10,007] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:57:11,196] - [rest_client] [140100243371776] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:57:11,196] - [task] [140100243371776] - INFO - Rebalance - status: running, progress: 33.00% [2021-12-23 22:57:13,016] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:57:16,023] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:57:19,031] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:57:22,041] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:57:25,050] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:57:28,059] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:57:31,067] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:57:31,224] - [rest_client] [140100243371776] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:57:31,225] - [task] [140100243371776] - INFO - Rebalance - status: running, progress: 33.00% [2021-12-23 22:57:34,076] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:57:37,085] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:57:40,093] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:57:43,101] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:57:46,109] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:57:49,118] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:57:51,255] - [rest_client] [140100243371776] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:57:51,255] - [task] [140100243371776] - INFO - Rebalance - status: running, progress: 33.00% [2021-12-23 22:57:52,126] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:57:55,135] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:57:58,143] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:58:01,151] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:58:04,160] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:58:07,168] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:58:10,177] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:58:11,293] - [rest_client] [140100243371776] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:58:11,294] - [task] [140100243371776] - INFO - Rebalance - status: running, progress: 33.00% [2021-12-23 22:58:13,185] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:58:16,194] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:58:19,202] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:58:22,211] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:58:25,219] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:58:28,228] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:58:31,235] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:58:31,331] - [rest_client] [140100243371776] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:58:31,331] - [task] [140100243371776] - INFO - Rebalance - status: running, progress: 33.00% [2021-12-23 22:58:34,244] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:58:37,252] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:58:40,261] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:58:43,270] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:58:46,278] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:58:49,284] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:58:51,365] - [rest_client] [140100243371776] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:58:51,365] - [task] [140100243371776] - INFO - Rebalance - status: running, progress: 33.00% [2021-12-23 22:58:52,289] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:58:55,298] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:58:58,304] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:59:01,313] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:59:04,322] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:59:07,330] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:59:10,339] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:59:11,391] - [rest_client] [140100243371776] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:59:11,391] - [task] [140100243371776] - INFO - Rebalance - status: running, progress: 33.00% [2021-12-23 22:59:13,348] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:59:16,357] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:59:19,365] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:59:22,373] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:59:25,381] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:59:28,390] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:59:31,396] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:59:31,430] - [rest_client] [140100243371776] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:59:31,430] - [task] [140100243371776] - INFO - Rebalance - status: running, progress: 33.00% [2021-12-23 22:59:34,404] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:59:37,413] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:59:40,423] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:59:43,431] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:59:46,440] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:59:49,449] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:59:51,468] - [rest_client] [140100243371776] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:59:51,468] - [task] [140100243371776] - INFO - Rebalance - status: running, progress: 33.00% [2021-12-23 22:59:52,457] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:59:55,466] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 22:59:58,474] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:00:01,484] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:00:04,494] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:00:07,503] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:00:10,511] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:00:11,506] - [rest_client] [140100243371776] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:00:11,506] - [task] [140100243371776] - INFO - Rebalance - status: running, progress: 33.00% [2021-12-23 23:00:13,520] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:00:16,529] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:00:19,537] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:00:22,546] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:00:25,552] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:00:28,559] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:00:31,542] - [rest_client] [140100243371776] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:00:31,542] - [task] [140100243371776] - INFO - Rebalance - status: running, progress: 33.00% [2021-12-23 23:00:31,565] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:00:34,573] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:00:37,581] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:00:40,590] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:00:43,598] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:00:46,607] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:00:49,615] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:00:51,584] - [rest_client] [140100243371776] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:00:51,585] - [task] [140100243371776] - INFO - Rebalance - status: running, progress: 33.00% [2021-12-23 23:00:52,624] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:00:55,632] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:00:58,638] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:01:01,646] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:01:04,654] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:01:07,660] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:01:10,668] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:01:11,623] - [rest_client] [140100243371776] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:01:11,623] - [task] [140100243371776] - INFO - Rebalance - status: running, progress: 33.00% [2021-12-23 23:01:13,677] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:01:16,685] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:01:19,693] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:01:22,702] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:01:25,710] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:01:28,718] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:01:31,661] - [rest_client] [140100243371776] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:01:31,661] - [task] [140100243371776] - INFO - Rebalance - status: running, progress: 33.00% [2021-12-23 23:01:31,726] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:01:34,734] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:01:37,742] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:01:40,750] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:01:43,758] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:01:46,767] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:01:49,775] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:01:51,698] - [rest_client] [140100243371776] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:01:51,698] - [task] [140100243371776] - INFO - Rebalance - status: running, progress: 33.00% [2021-12-23 23:01:52,783] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:01:55,792] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:01:58,799] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:02:01,807] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:02:04,815] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:02:07,823] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:02:10,832] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:02:11,732] - [rest_client] [140100243371776] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:02:11,732] - [task] [140100243371776] - INFO - Rebalance - status: running, progress: 33.00% [2021-12-23 23:02:13,841] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:02:16,848] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:02:19,856] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:02:22,864] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:02:25,872] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:02:28,879] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:02:31,772] - [rest_client] [140100243371776] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:02:31,772] - [task] [140100243371776] - INFO - Rebalance - status: running, progress: 33.00% [2021-12-23 23:02:31,887] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:02:34,895] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:02:37,903] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:02:40,911] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:02:43,919] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:02:46,925] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:02:49,933] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:02:51,808] - [rest_client] [140100243371776] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:02:51,809] - [task] [140100243371776] - INFO - Rebalance - status: running, progress: 33.00% [2021-12-23 23:02:52,941] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:02:55,950] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:02:58,956] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:03:01,965] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:03:04,973] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:03:07,982] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:03:10,991] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:03:11,843] - [rest_client] [140100243371776] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:03:11,843] - [task] [140100243371776] - INFO - Rebalance - status: running, progress: 33.00% [2021-12-23 23:03:13,999] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:03:17,005] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:03:20,013] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:03:23,022] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:03:26,030] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:03:29,039] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:03:31,877] - [rest_client] [140100243371776] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:03:31,877] - [task] [140100243371776] - INFO - Rebalance - status: running, progress: 33.00% [2021-12-23 23:03:32,047] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:03:35,055] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:03:38,064] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:03:41,072] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:03:44,081] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:03:47,089] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:03:50,097] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:03:51,913] - [rest_client] [140100243371776] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:03:51,913] - [task] [140100243371776] - INFO - Rebalance - status: running, progress: 33.00% [2021-12-23 23:03:53,104] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:03:56,116] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:03:59,124] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:04:02,129] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:04:05,137] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:04:08,145] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:04:11,153] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:04:11,945] - [rest_client] [140100243371776] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:04:11,946] - [task] [140100243371776] - INFO - Rebalance - status: running, progress: 33.00% [2021-12-23 23:04:14,160] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:04:17,168] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:04:20,176] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:04:23,184] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:04:26,192] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:04:29,201] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:04:31,983] - [rest_client] [140100243371776] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:04:31,984] - [task] [140100243371776] - INFO - Rebalance - status: running, progress: 33.00% [2021-12-23 23:04:32,208] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:04:35,217] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:04:38,225] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:04:41,234] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:04:44,241] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:04:47,250] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:04:50,258] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:04:52,013] - [rest_client] [140100243371776] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:04:52,014] - [task] [140100243371776] - INFO - Rebalance - status: running, progress: 33.00% [2021-12-23 23:04:53,266] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:04:56,274] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:04:59,282] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:05:02,291] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:05:05,299] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:05:08,307] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:05:11,316] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:05:12,047] - [rest_client] [140100243371776] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:05:12,047] - [task] [140100243371776] - INFO - Rebalance - status: running, progress: 33.00% [2021-12-23 23:05:14,324] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:05:17,333] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:05:20,341] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:05:23,349] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:05:26,358] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:05:29,366] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:05:32,084] - [rest_client] [140100243371776] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:05:32,084] - [task] [140100243371776] - INFO - Rebalance - status: running, progress: 33.00% [2021-12-23 23:05:32,374] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:05:35,382] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:05:38,390] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:05:41,398] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:05:44,405] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:05:47,413] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:05:50,421] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:05:52,113] - [rest_client] [140100243371776] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:05:52,113] - [task] [140100243371776] - INFO - Rebalance - status: running, progress: 33.00% [2021-12-23 23:05:53,429] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:05:56,435] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:05:59,444] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:06:02,453] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:06:05,462] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:06:08,471] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:06:11,479] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:06:12,149] - [rest_client] [140100243371776] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:06:12,150] - [task] [140100243371776] - INFO - Rebalance - status: running, progress: 33.00% [2021-12-23 23:06:14,488] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:06:17,496] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:06:20,504] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:06:23,512] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:06:26,521] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:06:29,529] - [rest_client] [140099887036160] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:06:32,186] - [rest_client] [140100243371776] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:06:32,186] - [task] [140100243371776] - INFO - Rebalance - status: running, progress: 33.00% [2021-12-23 23:06:32,532] - [rest_client] [140099887036160] - ERROR - rebalance stuck on 33.0% [2021-12-23 23:06:32,547] - [basetestcase] [140099887036160] - INFO - list of eventing nodes in cluster: [ip:172.23.107.88 port:8091 ssh_username:root, ip:172.23.107.91 port:8091 ssh_username:root, ip:172.23.108.0 port:8091 ssh_username:root] [2021-12-23 23:06:32,574] - [eventing_base] [140099887036160] - INFO - Stats for Node 172.23.107.88 is [ { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 14, "agg_messages_sent_to_worker": 33545, "agg_queue_memory": 0, "agg_queue_memory_cap": 106954752, "agg_queue_size": 0, "agg_queue_size_cap": 300000, "agg_timer_feedback_queue_cap": 1500, "dcp_mutation": 17647, "dcp_mutation_sent_to_worker": 17647, "dcp_snapshot": 17515, "dcp_stream_req_counter": 854, "dcp_streamend": 512, "dcp_streamreq": 854, "execution_stats": 2901, "failure_stats": 2901, "latency_stats": 2901, "lcb_exception_stats": 2901, "log_level": 3, "num_processed_events": 17647, "processed_events_size": 12800816, "thr_count": 3, "thr_map": 3, "v8_init": 3, "v8_load": 3 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 17647, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 17647, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 33533, "no_op_counter": 0, "num_processed_events": 17647, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 17647, "processed_events_size": 12800816, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "91872": "2021-12-24T07:06:32Z", "91874": "2021-12-24T07:06:32Z", "91890": "2021-12-24T07:06:32Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_cache_overflow_count": 0, "bucket_op_cache_miss_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "91872": "2021-12-24T07:06:32Z", "91874": "2021-12-24T07:06:32Z", "91890": "2021-12-24T07:06:32Z" }, "v8worker_events_lost": 0 }, "function_id": 2997765224, "function_name": "Function_618248006_test_rebalance_in_with_different_topologies", "gocb_creds_request_counter": 1, "internal_vb_distribution_stats": { "worker_Function_618248006_test_rebalance_in_with_different_topologies_0": "[0-113]", "worker_Function_618248006_test_rebalance_in_with_different_topologies_1": "[114-227]", "worker_Function_618248006_test_rebalance_in_with_different_topologies_2": "[228-341]" }, "latency_percentile_stats": { "100": 21400, "50": 1100, "80": 1300, "90": 1500, "95": 1900, "99": 5100 }, "lcb_creds_request_counter": 3, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.107.88:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.107.91:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.108.0:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.107.88:8096": { "worker_Function_618248006_test_rebalance_in_with_different_topologies_0": "[0-113]", "worker_Function_618248006_test_rebalance_in_with_different_topologies_1": "[114-227]", "worker_Function_618248006_test_rebalance_in_with_different_topologies_2": "[228-341]" }, "172.23.107.91:8096": { "worker_Function_618248006_test_rebalance_in_with_different_topologies_0": "[342-455]", "worker_Function_618248006_test_rebalance_in_with_different_topologies_1": "[456-569]", "worker_Function_618248006_test_rebalance_in_with_different_topologies_2": "[570-682]" } }, "worker_pids": { "worker_Function_618248006_test_rebalance_in_with_different_topologies_0": 91872, "worker_Function_618248006_test_rebalance_in_with_different_topologies_1": 91874, "worker_Function_618248006_test_rebalance_in_with_different_topologies_2": 91890 } } ] [2021-12-23 23:06:32,606] - [eventing_base] [140099887036160] - INFO - Stats for Node 172.23.107.91 is [ { "dcp_feed_boundary": "everything", "event_processing_stats": { "agg_messages_sent_to_worker": 13827, "agg_queue_memory": 0, "agg_queue_memory_cap": 106954752, "agg_queue_size": 0, "agg_queue_size_cap": 300000, "agg_timer_feedback_queue_cap": 1500, "execution_stats": 2760, "failure_stats": 2760, "is_rebalance_ongoing": 3, "latency_stats": 2760, "lcb_exception_stats": 2760, "log_level": 3, "num_processed_events": 0, "processed_events_size": 0, "reb_vb_remaining_to_own": 341, "reb_vb_remaining_to_stream_req": 341, "thr_count": 3, "thr_map": 3, "v8_init": 3, "v8_load": 3, "vbs_state_update_running": 3 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 0, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 0, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 13815, "no_op_counter": 0, "num_processed_events": 0, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 0, "processed_events_size": 0, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "103740": "2021-12-24T07:06:32Z", "103749": "2021-12-24T07:06:32Z", "103759": "2021-12-24T07:06:32Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_cache_overflow_count": 0, "bucket_op_cache_miss_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "103740": "2021-12-24T07:06:32Z", "103749": "2021-12-24T07:06:32Z", "103759": "2021-12-24T07:06:32Z" }, "v8worker_events_lost": 0 }, "function_id": 2997765224, "function_name": "Function_618248006_test_rebalance_in_with_different_topologies", "gocb_creds_request_counter": 1, "internal_vb_distribution_stats": { "worker_Function_618248006_test_rebalance_in_with_different_topologies_0": "[]", "worker_Function_618248006_test_rebalance_in_with_different_topologies_1": "[]", "worker_Function_618248006_test_rebalance_in_with_different_topologies_2": "[]" }, "latency_percentile_stats": { "100": 0, "50": 0, "80": 0, "90": 0, "95": 0, "99": 0 }, "lcb_creds_request_counter": 3, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.107.88:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.107.91:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.108.0:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.107.88:8096": { "worker_Function_618248006_test_rebalance_in_with_different_topologies_0": "[0-113]", "worker_Function_618248006_test_rebalance_in_with_different_topologies_1": "[114-227]", "worker_Function_618248006_test_rebalance_in_with_different_topologies_2": "[228-341]" }, "172.23.107.91:8096": { "worker_Function_618248006_test_rebalance_in_with_different_topologies_0": "[342-455]", "worker_Function_618248006_test_rebalance_in_with_different_topologies_1": "[456-569]", "worker_Function_618248006_test_rebalance_in_with_different_topologies_2": "[570-682]" } }, "worker_pids": { "worker_Function_618248006_test_rebalance_in_with_different_topologies_0": 103740, "worker_Function_618248006_test_rebalance_in_with_different_topologies_1": 103749, "worker_Function_618248006_test_rebalance_in_with_different_topologies_2": 103759 } } ] [2021-12-23 23:06:32,635] - [eventing_base] [140099887036160] - INFO - Stats for Node 172.23.108.0 is [ { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 13, "agg_messages_sent_to_worker": 33373, "agg_queue_memory": 0, "agg_queue_memory_cap": 106954752, "agg_queue_size": 0, "agg_queue_size_cap": 300000, "agg_timer_feedback_queue_cap": 1500, "dcp_mutation": 17476, "dcp_mutation_sent_to_worker": 17476, "dcp_snapshot": 17376, "dcp_stream_req_counter": 853, "dcp_streamend": 512, "dcp_streamreq": 853, "execution_stats": 2901, "failure_stats": 2901, "latency_stats": 2901, "lcb_exception_stats": 2901, "log_level": 3, "num_processed_events": 17476, "processed_events_size": 12683492, "thr_count": 3, "thr_map": 3, "v8_init": 3, "v8_load": 3 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 17476, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 17476, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 33361, "no_op_counter": 0, "num_processed_events": 17476, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 17476, "processed_events_size": 12683492, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "81014": "2021-12-24T07:06:32Z", "81020": "2021-12-24T07:06:32Z", "81033": "2021-12-24T07:06:32Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_cache_overflow_count": 0, "bucket_op_cache_miss_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "81014": "2021-12-24T07:06:32Z", "81020": "2021-12-24T07:06:32Z", "81033": "2021-12-24T07:06:32Z" }, "v8worker_events_lost": 0 }, "function_id": 2997765224, "function_name": "Function_618248006_test_rebalance_in_with_different_topologies", "gocb_creds_request_counter": 1, "internal_vb_distribution_stats": { "worker_Function_618248006_test_rebalance_in_with_different_topologies_0": "[342-455]", "worker_Function_618248006_test_rebalance_in_with_different_topologies_1": "[456-569]", "worker_Function_618248006_test_rebalance_in_with_different_topologies_2": "[570-682]" }, "latency_percentile_stats": { "100": 40500, "50": 1300, "80": 1700, "90": 2300, "95": 3200, "99": 7500 }, "lcb_creds_request_counter": 3, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.107.88:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.107.91:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.108.0:8096", "start_vb": 683, "vb_count": 341 } ], "rebalance_stats": { "is_leader": true, "node_level_stats": { "172.23.107.88:8096": { "close_stream_vbs_len": 0, "stream_req_vbs_len": 0, "vbs_owned_per_plan": 0, "vbs_remaining_to_shuffle": 0 }, "172.23.107.91:8096": { "close_stream_vbs_len": 0, "stream_req_vbs_len": 341, "vbs_owned_per_plan": 341, "vbs_remaining_to_shuffle": 341 }, "172.23.108.0:8096": { "close_stream_vbs_len": 0, "stream_req_vbs_len": 0, "vbs_owned_per_plan": 0, "vbs_remaining_to_shuffle": 0 } }, "rebalance_progress": 0.29239766081871066, "rebalance_progress_counter": 304, "rebalance_start_ts": "2021-12-23 22:51:01.745232112 -0800 PST m=+109.679760444", "total_vbs_to_shuffle": 342, "vbs_remaining_to_shuffle": 341 }, "vb_distribution_stats_from_metadata": { "172.23.107.88:8096": { "worker_Function_618248006_test_rebalance_in_with_different_topologies_0": "[0-113]", "worker_Function_618248006_test_rebalance_in_with_different_topologies_1": "[114-227]", "worker_Function_618248006_test_rebalance_in_with_different_topologies_2": "[228-341]" }, "172.23.107.91:8096": { "worker_Function_618248006_test_rebalance_in_with_different_topologies_0": "[342-455]", "worker_Function_618248006_test_rebalance_in_with_different_topologies_1": "[456-569]", "worker_Function_618248006_test_rebalance_in_with_different_topologies_2": "[570-682]" } }, "worker_pids": { "worker_Function_618248006_test_rebalance_in_with_different_topologies_0": 81014, "worker_Function_618248006_test_rebalance_in_with_different_topologies_1": 81020, "worker_Function_618248006_test_rebalance_in_with_different_topologies_2": 81033 } } ] [2021-12-23 23:06:32,635] - [eventing_base] [140099887036160] - INFO - ============== EventingBaseTest tearDown has started ============== [2021-12-23 23:06:32,660] - [basetestcase] [140099887036160] - INFO - list of eventing nodes in cluster: [ip:172.23.107.88 port:8091 ssh_username:root, ip:172.23.107.91 port:8091 ssh_username:root, ip:172.23.108.0 port:8091 ssh_username:root] [2021-12-23 23:06:32,664] - [remote_util] [140099887036160] - INFO - SSH Connecting to 172.23.107.88 with username:root, attempt#1 of 5 [2021-12-23 23:06:32,839] - [remote_util] [140099887036160] - INFO - SSH Connected to 172.23.107.88 as root [2021-12-23 23:06:33,229] - [remote_util] [140099887036160] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-12-23 23:06:33,669] - [remote_util] [140099887036160] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-12-23 23:06:33,681] - [rest_client] [140099887036160] - INFO - /diag/eval status on 172.23.107.88:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). [2021-12-23 23:06:33,682] - [remote_util] [140099887036160] - INFO - running command.raw on 172.23.107.88: zgrep "panic" "/opt/couchbase/var/lib/couchbase/logs"/eventing.log* | wc -l [2021-12-23 23:06:33,835] - [remote_util] [140099887036160] - INFO - command executed successfully with root [2021-12-23 23:06:33,835] - [remote_util] [140099887036160] - INFO - running command.raw on 172.23.107.88: ls "/opt/couchbase/var/lib/couchbase/logs"/../crash/| wc -l [2021-12-23 23:06:33,909] - [remote_util] [140099887036160] - INFO - command executed successfully with root [2021-12-23 23:06:33,912] - [remote_util] [140099887036160] - INFO - SSH Connecting to 172.23.107.91 with username:root, attempt#1 of 5 [2021-12-23 23:06:34,095] - [remote_util] [140099887036160] - INFO - SSH Connected to 172.23.107.91 as root [2021-12-23 23:06:34,497] - [remote_util] [140099887036160] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-12-23 23:06:34,938] - [remote_util] [140099887036160] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-12-23 23:06:34,953] - [rest_client] [140099887036160] - INFO - /diag/eval status on 172.23.107.91:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). [2021-12-23 23:06:34,953] - [remote_util] [140099887036160] - INFO - running command.raw on 172.23.107.91: zgrep "panic" "/opt/couchbase/var/lib/couchbase/logs"/eventing.log* | wc -l [2021-12-23 23:06:35,063] - [remote_util] [140099887036160] - INFO - command executed successfully with root [2021-12-23 23:06:35,063] - [remote_util] [140099887036160] - INFO - running command.raw on 172.23.107.91: ls "/opt/couchbase/var/lib/couchbase/logs"/../crash/| wc -l [2021-12-23 23:06:35,140] - [remote_util] [140099887036160] - INFO - command executed successfully with root [2021-12-23 23:06:35,143] - [remote_util] [140099887036160] - INFO - SSH Connecting to 172.23.108.0 with username:root, attempt#1 of 5 [2021-12-23 23:06:35,303] - [remote_util] [140099887036160] - INFO - SSH Connected to 172.23.108.0 as root [2021-12-23 23:06:35,680] - [remote_util] [140099887036160] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-12-23 23:06:36,112] - [remote_util] [140099887036160] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-12-23 23:06:36,126] - [rest_client] [140099887036160] - INFO - /diag/eval status on 172.23.108.0:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). [2021-12-23 23:06:36,127] - [remote_util] [140099887036160] - INFO - running command.raw on 172.23.108.0: zgrep "panic" "/opt/couchbase/var/lib/couchbase/logs"/eventing.log* | wc -l [2021-12-23 23:06:36,301] - [remote_util] [140099887036160] - INFO - command executed successfully with root [2021-12-23 23:06:36,301] - [remote_util] [140099887036160] - INFO - running command.raw on 172.23.108.0: ls "/opt/couchbase/var/lib/couchbase/logs"/../crash/| wc -l [2021-12-23 23:06:36,374] - [remote_util] [140099887036160] - INFO - command executed successfully with root [2021-12-23 23:06:36,635] - [basetestcase] [140099887036160] - INFO - list of n1ql nodes in cluster: [ip:172.23.107.88 port:8091 ssh_username:root, ip:172.23.107.91 port:8091 ssh_username:root, ip:172.23.108.0 port:8091 ssh_username:root] [2021-12-23 23:06:36,645] - [rest_client] [140099887036160] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+metadata [2021-12-23 23:06:36,668] - [eventing_base] [140099887036160] - INFO - number of documents in metadata bucket 1024 [2021-12-23 23:06:36,710] - [remote_util] [140098637113088] - INFO - SSH Connecting to 172.23.108.0 with username:root, attempt#1 of 5 [2021-12-23 23:06:36,712] - [remote_util] [140098628720384] - INFO - SSH Connecting to 172.23.107.88 with username:root, attempt#1 of 5 [2021-12-23 23:06:36,716] - [remote_util] [140098679076608] - INFO - SSH Connecting to 172.23.108.102 with username:root, attempt#1 of 5 [2021-12-23 23:06:36,718] - [remote_util] [140099887036160] - INFO - SSH Connecting to 172.23.108.100 with username:root, attempt#1 of 5 [2021-12-23 23:06:36,722] - [remote_util] [140098645505792] - INFO - SSH Connecting to 172.23.107.91 with username:root, attempt#1 of 5 [2021-12-23 23:06:36,725] - [remote_util] [140098670683904] - INFO - SSH Connecting to 172.23.107.84 with username:root, attempt#1 of 5 [2021-12-23 23:06:36,728] - [remote_util] [140098653898496] - INFO - SSH Connecting to 172.23.107.85 with username:root, attempt#1 of 5 [2021-12-23 23:06:36,731] - [remote_util] [140098326746880] - INFO - SSH Connecting to 172.23.107.94 with username:root, attempt#1 of 5 [2021-12-23 23:06:36,879] - [remote_util] [140098637113088] - INFO - SSH Connected to 172.23.108.0 as root [2021-12-23 23:06:36,894] - [remote_util] [140098628720384] - INFO - SSH Connected to 172.23.107.88 as root [2021-12-23 23:06:36,895] - [remote_util] [140099887036160] - INFO - SSH Connected to 172.23.108.100 as root [2021-12-23 23:06:36,899] - [remote_util] [140098679076608] - INFO - SSH Connected to 172.23.108.102 as root [2021-12-23 23:06:36,902] - [remote_util] [140098645505792] - INFO - SSH Connected to 172.23.107.91 as root [2021-12-23 23:06:36,902] - [remote_util] [140098326746880] - INFO - SSH Connected to 172.23.107.94 as root [2021-12-23 23:06:36,915] - [remote_util] [140098670683904] - INFO - SSH Connected to 172.23.107.84 as root [2021-12-23 23:06:36,919] - [remote_util] [140098653898496] - INFO - SSH Connected to 172.23.107.85 as root [2021-12-23 23:06:37,345] - [remote_util] [140099887036160] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-12-23 23:06:37,346] - [remote_util] [140098326746880] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-12-23 23:06:37,351] - [remote_util] [140098637113088] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-12-23 23:06:37,357] - [remote_util] [140098679076608] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-12-23 23:06:37,368] - [remote_util] [140098645505792] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-12-23 23:06:37,372] - [remote_util] [140098670683904] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-12-23 23:06:37,383] - [remote_util] [140098628720384] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-12-23 23:06:37,384] - [remote_util] [140098653898496] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-12-23 23:06:37,780] - [remote_util] [140099887036160] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-12-23 23:06:37,781] - [remote_util] [140099887036160] - INFO - running command.raw on 172.23.108.100: /opt/couchbase/bin/cbcollect_info 172.23.108.100-20211223-2306-diag.zip [2021-12-23 23:06:37,793] - [remote_util] [140098637113088] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-12-23 23:06:37,793] - [remote_util] [140098637113088] - INFO - running command.raw on 172.23.108.0: /opt/couchbase/bin/cbcollect_info 172.23.108.0-20211223-2306-diag.zip [2021-12-23 23:06:37,796] - [remote_util] [140098326746880] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-12-23 23:06:37,798] - [remote_util] [140098326746880] - INFO - running command.raw on 172.23.107.94: /opt/couchbase/bin/cbcollect_info 172.23.107.94-20211223-2306-diag.zip [2021-12-23 23:06:37,802] - [remote_util] [140098679076608] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-12-23 23:06:37,803] - [remote_util] [140098679076608] - INFO - running command.raw on 172.23.108.102: /opt/couchbase/bin/cbcollect_info 172.23.108.102-20211223-2306-diag.zip [2021-12-23 23:06:37,813] - [remote_util] [140098645505792] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-12-23 23:06:37,814] - [remote_util] [140098645505792] - INFO - running command.raw on 172.23.107.91: /opt/couchbase/bin/cbcollect_info 172.23.107.91-20211223-2306-diag.zip [2021-12-23 23:06:37,829] - [remote_util] [140098670683904] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-12-23 23:06:37,830] - [remote_util] [140098670683904] - INFO - running command.raw on 172.23.107.84: /opt/couchbase/bin/cbcollect_info 172.23.107.84-20211223-2306-diag.zip [2021-12-23 23:06:37,832] - [remote_util] [140098628720384] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-12-23 23:06:37,833] - [remote_util] [140098628720384] - INFO - running command.raw on 172.23.107.88: /opt/couchbase/bin/cbcollect_info 172.23.107.88-20211223-2306-diag.zip [2021-12-23 23:06:37,837] - [remote_util] [140098653898496] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-12-23 23:06:37,837] - [remote_util] [140098653898496] - INFO - running command.raw on 172.23.107.85: /opt/couchbase/bin/cbcollect_info 172.23.107.85-20211223-2306-diag.zip [2021-12-23 23:06:52,218] - [rest_client] [140100243371776] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:06:52,219] - [task] [140100243371776] - INFO - Rebalance - status: running, progress: 33.00% [2021-12-23 23:07:12,248] - [rest_client] [140100243371776] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:07:12,248] - [task] [140100243371776] - INFO - Rebalance - status: running, progress: 33.00% [2021-12-23 23:07:32,278] - [rest_client] [140100243371776] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:07:32,278] - [task] [140100243371776] - INFO - Rebalance - status: running, progress: 33.00% [2021-12-23 23:07:52,316] - [rest_client] [140100243371776] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:07:52,317] - [task] [140100243371776] - INFO - Rebalance - status: running, progress: 33.00% [2021-12-23 23:08:12,357] - [rest_client] [140100243371776] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:08:12,358] - [task] [140100243371776] - INFO - Rebalance - status: running, progress: 33.00% [2021-12-23 23:08:32,394] - [rest_client] [140100243371776] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:08:32,395] - [task] [140100243371776] - INFO - Rebalance - status: running, progress: 33.00% [2021-12-23 23:08:52,428] - [rest_client] [140100243371776] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:08:52,428] - [task] [140100243371776] - INFO - Rebalance - status: running, progress: 33.00% [2021-12-23 23:09:07,954] - [remote_util] [140098653898496] - INFO - command executed successfully with root [2021-12-23 23:09:08,263] - [remote_util] [140098653898496] - INFO - running command.raw on 172.23.107.85: rm -f /root/172.23.107.85-20211223-2306-diag.zip [2021-12-23 23:09:08,299] - [remote_util] [140098653898496] - INFO - command executed successfully with root [2021-12-23 23:09:08,442] - [remote_util] [140098326746880] - INFO - command executed successfully with root [2021-12-23 23:09:08,722] - [remote_util] [140098670683904] - INFO - command executed successfully with root [2021-12-23 23:09:08,770] - [remote_util] [140098326746880] - INFO - running command.raw on 172.23.107.94: rm -f /root/172.23.107.94-20211223-2306-diag.zip [2021-12-23 23:09:08,806] - [remote_util] [140098326746880] - INFO - command executed successfully with root [2021-12-23 23:09:09,036] - [remote_util] [140098670683904] - INFO - running command.raw on 172.23.107.84: rm -f /root/172.23.107.84-20211223-2306-diag.zip [2021-12-23 23:09:09,075] - [remote_util] [140098670683904] - INFO - command executed successfully with root [2021-12-23 23:09:10,571] - [remote_util] [140098679076608] - INFO - command executed successfully with root [2021-12-23 23:09:11,008] - [remote_util] [140098679076608] - INFO - running command.raw on 172.23.108.102: rm -f /root/172.23.108.102-20211223-2306-diag.zip [2021-12-23 23:09:11,046] - [remote_util] [140098679076608] - INFO - command executed successfully with root [2021-12-23 23:09:12,458] - [rest_client] [140100243371776] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:09:12,459] - [task] [140100243371776] - INFO - Rebalance - status: running, progress: 33.00% [2021-12-23 23:09:32,493] - [rest_client] [140100243371776] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:09:32,493] - [task] [140100243371776] - INFO - Rebalance - status: running, progress: 33.00% [2021-12-23 23:09:45,928] - [remote_util] [140098645505792] - INFO - command executed successfully with root [2021-12-23 23:09:46,354] - [remote_util] [140098645505792] - INFO - running command.raw on 172.23.107.91: rm -f /root/172.23.107.91-20211223-2306-diag.zip [2021-12-23 23:09:46,392] - [remote_util] [140098645505792] - INFO - command executed successfully with root [2021-12-23 23:09:46,550] - [remote_util] [140098628720384] - INFO - command executed successfully with root [2021-12-23 23:09:47,007] - [remote_util] [140098628720384] - INFO - running command.raw on 172.23.107.88: rm -f /root/172.23.107.88-20211223-2306-diag.zip [2021-12-23 23:09:47,047] - [remote_util] [140098628720384] - INFO - command executed successfully with root [2021-12-23 23:09:52,533] - [rest_client] [140100243371776] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:09:52,533] - [task] [140100243371776] - INFO - Rebalance - status: running, progress: 33.00% [2021-12-23 23:10:12,567] - [rest_client] [140100243371776] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:10:12,568] - [task] [140100243371776] - INFO - Rebalance - status: running, progress: 33.00% [2021-12-23 23:10:17,240] - [remote_util] [140098637113088] - INFO - command executed successfully with root [2021-12-23 23:10:17,644] - [remote_util] [140098637113088] - INFO - running command.raw on 172.23.108.0: rm -f /root/172.23.108.0-20211223-2306-diag.zip [2021-12-23 23:10:17,681] - [remote_util] [140098637113088] - INFO - command executed successfully with root [2021-12-23 23:10:32,602] - [rest_client] [140100243371776] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:10:32,602] - [task] [140100243371776] - INFO - Rebalance - status: running, progress: 33.00% [2021-12-23 23:10:52,634] - [rest_client] [140100243371776] - INFO - rebalance percentage : 33.00 % [2021-12-23 23:10:52,634] - [task] [140100243371776] - INFO - Rebalance - status: running, progress: 33.00% [2021-12-23 23:11:11,150] - [remote_util] [140099887036160] - INFO - command executed successfully with root [2021-12-23 23:11:12,038] - [remote_util] [140099887036160] - INFO - running command.raw on 172.23.108.100: rm -f /root/172.23.108.100-20211223-2306-diag.zip [2021-12-23 23:11:12,082] - [remote_util] [140099887036160] - INFO - command executed successfully with root