[2021-02-22 17:23:47,373] - [eventing_base] [139837055153920] - INFO - Starting Test: test_eventing_rebalance_with_multiple_kv_nodes [2021-02-22 17:23:47,379] - [remote_util] [139837055153920] - INFO - SSH Connecting to 172.23.123.42 with username:root, attempt#1 of 5 [2021-02-22 17:23:47,489] - [remote_util] [139837055153920] - INFO - SSH Connected to 172.23.123.42 as root [2021-02-22 17:23:47,773] - [remote_util] [139837055153920] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 17:23:48,092] - [remote_util] [139837055153920] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 17:23:48,099] - [rest_client] [139837055153920] - INFO - -->is_ns_server_running? [2021-02-22 17:23:48,112] - [rest_client] [139837055153920] - INFO - Node version in cluster 7.0.0-4514-enterprise [2021-02-22 17:23:48,120] - [rest_client] [139837055153920] - ERROR - GET http://172.23.123.42:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-02-22 17:23:48,121] - [rest_client] [139837055153920] - INFO - Node versions in cluster [] [2021-02-22 17:23:48,122] - [basetestcase] [139837055153920] - INFO - ============== basetestcase setup was started for test #8 test_eventing_rebalance_with_multiple_kv_nodes============== [2021-02-22 17:23:48,129] - [rest_client] [139837055153920] - ERROR - GET http://172.23.123.42:8091/pools/default/ body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-02-22 17:23:48,147] - [bucket_helper] [139837055153920] - INFO - Could not find any buckets for node 172.23.123.42, nothing to delete [2021-02-22 17:23:48,162] - [bucket_helper] [139837055153920] - INFO - Could not find any buckets for node 172.23.123.44, nothing to delete [2021-02-22 17:23:48,176] - [bucket_helper] [139837055153920] - INFO - Could not find any buckets for node 172.23.122.132, nothing to delete [2021-02-22 17:23:48,191] - [bucket_helper] [139837055153920] - INFO - Could not find any buckets for node 172.23.122.133, nothing to delete [2021-02-22 17:23:48,205] - [bucket_helper] [139837055153920] - INFO - Could not find any buckets for node 172.23.122.134, nothing to delete [2021-02-22 17:23:48,221] - [bucket_helper] [139837055153920] - INFO - Could not find any buckets for node 172.23.122.131, nothing to delete [2021-02-22 17:23:48,235] - [bucket_helper] [139837055153920] - INFO - Could not find any buckets for node 172.23.122.180, nothing to delete [2021-02-22 17:23:48,250] - [bucket_helper] [139837055153920] - INFO - Could not find any buckets for node 172.23.122.130, nothing to delete [2021-02-22 17:23:48,255] - [rest_client] [139837055153920] - INFO - -->is_ns_server_running? [2021-02-22 17:23:48,280] - [cluster_helper] [139837055153920] - INFO - waiting for ns_server @ 172.23.123.42:8091 [2021-02-22 17:23:48,281] - [rest_client] [139837055153920] - INFO - -->is_ns_server_running? [2021-02-22 17:23:48,286] - [cluster_helper] [139837055153920] - INFO - ns_server @ 172.23.123.42:8091 is running [2021-02-22 17:23:48,291] - [cluster_helper] [139837055153920] - INFO - waiting for ns_server @ 172.23.123.44:8091 [2021-02-22 17:23:48,292] - [rest_client] [139837055153920] - INFO - -->is_ns_server_running? [2021-02-22 17:23:48,299] - [cluster_helper] [139837055153920] - INFO - ns_server @ 172.23.123.44:8091 is running [2021-02-22 17:23:48,304] - [cluster_helper] [139837055153920] - INFO - waiting for ns_server @ 172.23.122.132:8091 [2021-02-22 17:23:48,305] - [rest_client] [139837055153920] - INFO - -->is_ns_server_running? [2021-02-22 17:23:48,309] - [cluster_helper] [139837055153920] - INFO - ns_server @ 172.23.122.132:8091 is running [2021-02-22 17:23:48,313] - [cluster_helper] [139837055153920] - INFO - waiting for ns_server @ 172.23.122.133:8091 [2021-02-22 17:23:48,314] - [rest_client] [139837055153920] - INFO - -->is_ns_server_running? [2021-02-22 17:23:48,318] - [cluster_helper] [139837055153920] - INFO - ns_server @ 172.23.122.133:8091 is running [2021-02-22 17:23:48,322] - [cluster_helper] [139837055153920] - INFO - waiting for ns_server @ 172.23.122.134:8091 [2021-02-22 17:23:48,322] - [rest_client] [139837055153920] - INFO - -->is_ns_server_running? [2021-02-22 17:23:48,326] - [cluster_helper] [139837055153920] - INFO - ns_server @ 172.23.122.134:8091 is running [2021-02-22 17:23:48,330] - [cluster_helper] [139837055153920] - INFO - waiting for ns_server @ 172.23.122.131:8091 [2021-02-22 17:23:48,331] - [rest_client] [139837055153920] - INFO - -->is_ns_server_running? [2021-02-22 17:23:48,335] - [cluster_helper] [139837055153920] - INFO - ns_server @ 172.23.122.131:8091 is running [2021-02-22 17:23:48,339] - [cluster_helper] [139837055153920] - INFO - waiting for ns_server @ 172.23.122.180:8091 [2021-02-22 17:23:48,339] - [rest_client] [139837055153920] - INFO - -->is_ns_server_running? [2021-02-22 17:23:48,344] - [cluster_helper] [139837055153920] - INFO - ns_server @ 172.23.122.180:8091 is running [2021-02-22 17:23:48,348] - [cluster_helper] [139837055153920] - INFO - waiting for ns_server @ 172.23.122.130:8091 [2021-02-22 17:23:48,348] - [rest_client] [139837055153920] - INFO - -->is_ns_server_running? [2021-02-22 17:23:48,352] - [cluster_helper] [139837055153920] - INFO - ns_server @ 172.23.122.130:8091 is running [2021-02-22 17:23:48,353] - [basetestcase] [139837055153920] - INFO - initializing cluster [2021-02-22 17:23:48,356] - [remote_util] [139837055153920] - INFO - SSH Connecting to 172.23.123.42 with username:root, attempt#1 of 5 [2021-02-22 17:23:48,463] - [remote_util] [139837055153920] - INFO - SSH Connected to 172.23.123.42 as root [2021-02-22 17:23:48,731] - [remote_util] [139837055153920] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 17:23:49,056] - [remote_util] [139837055153920] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 17:23:49,068] - [remote_util] [139837055153920] - INFO - SSH Connecting to 172.23.123.42 with username:root, attempt#1 of 5 [2021-02-22 17:23:49,176] - [remote_util] [139837055153920] - INFO - SSH Connected to 172.23.123.42 as root [2021-02-22 17:23:49,456] - [remote_util] [139837055153920] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 17:23:49,776] - [remote_util] [139837055153920] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 17:23:49,840] - [remote_util] [139837055153920] - INFO - 172.23.123.42 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-22 17:23:49,951] - [remote_util] [139837055153920] - INFO - Running systemd command on this server [2021-02-22 17:23:49,951] - [remote_util] [139837055153920] - INFO - running command.raw on 172.23.123.42: systemctl stop couchbase-server.service [2021-02-22 17:23:51,711] - [remote_util] [139837055153920] - INFO - command executed successfully with root [2021-02-22 17:23:51,712] - [basetestcase] [139837055153920] - INFO - Couchbase stopped [2021-02-22 17:23:51,713] - [basetestcase] [139837055153920] - INFO - sleep for 5 secs. ... [2021-02-22 17:23:56,718] - [remote_util] [139837055153920] - INFO - running command.raw on 172.23.123.42: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-22 17:23:56,736] - [remote_util] [139837055153920] - INFO - command executed successfully with root [2021-02-22 17:23:56,736] - [remote_util] [139837055153920] - INFO - running command.raw on 172.23.123.42: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-22 17:23:56,794] - [remote_util] [139837055153920] - INFO - command executed successfully with root [2021-02-22 17:23:56,797] - [remote_util] [139837055153920] - INFO - SSH Connecting to 172.23.123.42 with username:root, attempt#1 of 5 [2021-02-22 17:23:56,909] - [remote_util] [139837055153920] - INFO - SSH Connected to 172.23.123.42 as root [2021-02-22 17:23:57,180] - [remote_util] [139837055153920] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 17:23:57,506] - [remote_util] [139837055153920] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 17:23:57,573] - [remote_util] [139837055153920] - INFO - 172.23.123.42 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-22 17:23:57,573] - [remote_util] [139837055153920] - INFO - Checking for process beam.smp on linux [2021-02-22 17:23:57,656] - [remote_util] [139837055153920] - INFO - Starting couchbase server [2021-02-22 17:23:57,813] - [remote_util] [139837055153920] - INFO - Running systemd command on this server [2021-02-22 17:23:57,813] - [remote_util] [139837055153920] - INFO - running command.raw on 172.23.123.42: systemctl start couchbase-server.service [2021-02-22 17:23:57,843] - [remote_util] [139837055153920] - INFO - command executed successfully with root [2021-02-22 17:23:57,844] - [remote_util] [139837055153920] - INFO - 172.23.123.42:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-22 17:24:02,849] - [remote_util] [139837055153920] - INFO - running command.raw on 172.23.123.42: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-22 17:24:02,882] - [remote_util] [139837055153920] - INFO - command executed successfully with root [2021-02-22 17:24:02,882] - [remote_util] [139837055153920] - INFO - Couchbase server status: [' Process: 40993 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-22 17:24:02,882] - [remote_util] [139837055153920] - INFO - Checking for process beam.smp on linux [2021-02-22 17:24:02,955] - [remote_util] [139837055153920] - INFO - process beam.smp is running on 172.23.123.42: with pid 41136 [2021-02-22 17:24:02,956] - [basetestcase] [139837055153920] - INFO - Couchbase started [2021-02-22 17:24:02,959] - [remote_util] [139837055153920] - INFO - SSH Connecting to 172.23.123.44 with username:root, attempt#1 of 5 [2021-02-22 17:24:03,071] - [remote_util] [139837055153920] - INFO - SSH Connected to 172.23.123.44 as root [2021-02-22 17:24:03,379] - [remote_util] [139837055153920] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 17:24:03,698] - [remote_util] [139837055153920] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 17:24:03,711] - [remote_util] [139837055153920] - INFO - SSH Connecting to 172.23.123.44 with username:root, attempt#1 of 5 [2021-02-22 17:24:03,815] - [remote_util] [139837055153920] - INFO - SSH Connected to 172.23.123.44 as root [2021-02-22 17:24:04,097] - [remote_util] [139837055153920] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 17:24:04,417] - [remote_util] [139837055153920] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 17:24:04,485] - [remote_util] [139837055153920] - INFO - 172.23.123.44 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-22 17:24:04,602] - [remote_util] [139837055153920] - INFO - Running systemd command on this server [2021-02-22 17:24:04,602] - [remote_util] [139837055153920] - INFO - running command.raw on 172.23.123.44: systemctl stop couchbase-server.service [2021-02-22 17:24:06,274] - [remote_util] [139837055153920] - INFO - command executed successfully with root [2021-02-22 17:24:06,275] - [basetestcase] [139837055153920] - INFO - Couchbase stopped [2021-02-22 17:24:06,276] - [basetestcase] [139837055153920] - INFO - sleep for 5 secs. ... [2021-02-22 17:24:11,281] - [remote_util] [139837055153920] - INFO - running command.raw on 172.23.123.44: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-22 17:24:11,299] - [remote_util] [139837055153920] - INFO - command executed successfully with root [2021-02-22 17:24:11,299] - [remote_util] [139837055153920] - INFO - running command.raw on 172.23.123.44: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-22 17:24:11,354] - [remote_util] [139837055153920] - INFO - command executed successfully with root [2021-02-22 17:24:11,357] - [remote_util] [139837055153920] - INFO - SSH Connecting to 172.23.123.44 with username:root, attempt#1 of 5 [2021-02-22 17:24:11,468] - [remote_util] [139837055153920] - INFO - SSH Connected to 172.23.123.44 as root [2021-02-22 17:24:11,766] - [remote_util] [139837055153920] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 17:24:12,089] - [remote_util] [139837055153920] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 17:24:12,153] - [remote_util] [139837055153920] - INFO - 172.23.123.44 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-22 17:24:12,153] - [remote_util] [139837055153920] - INFO - Checking for process beam.smp on linux [2021-02-22 17:24:12,237] - [remote_util] [139837055153920] - INFO - Starting couchbase server [2021-02-22 17:24:12,353] - [remote_util] [139837055153920] - INFO - Running systemd command on this server [2021-02-22 17:24:12,353] - [remote_util] [139837055153920] - INFO - running command.raw on 172.23.123.44: systemctl start couchbase-server.service [2021-02-22 17:24:12,384] - [remote_util] [139837055153920] - INFO - command executed successfully with root [2021-02-22 17:24:12,384] - [remote_util] [139837055153920] - INFO - 172.23.123.44:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-22 17:24:17,389] - [remote_util] [139837055153920] - INFO - running command.raw on 172.23.123.44: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-22 17:24:17,414] - [remote_util] [139837055153920] - INFO - command executed successfully with root [2021-02-22 17:24:17,414] - [remote_util] [139837055153920] - INFO - Couchbase server status: [' Process: 6405 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-22 17:24:17,415] - [remote_util] [139837055153920] - INFO - Checking for process beam.smp on linux [2021-02-22 17:24:17,484] - [remote_util] [139837055153920] - INFO - process beam.smp is running on 172.23.123.44: with pid 6547 [2021-02-22 17:24:17,485] - [basetestcase] [139837055153920] - INFO - Couchbase started [2021-02-22 17:24:17,488] - [remote_util] [139837055153920] - INFO - SSH Connecting to 172.23.122.132 with username:root, attempt#1 of 5 [2021-02-22 17:24:17,592] - [remote_util] [139837055153920] - INFO - SSH Connected to 172.23.122.132 as root [2021-02-22 17:24:17,860] - [remote_util] [139837055153920] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 17:24:18,165] - [remote_util] [139837055153920] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 17:24:18,177] - [remote_util] [139837055153920] - INFO - SSH Connecting to 172.23.122.132 with username:root, attempt#1 of 5 [2021-02-22 17:24:18,278] - [remote_util] [139837055153920] - INFO - SSH Connected to 172.23.122.132 as root [2021-02-22 17:24:18,539] - [remote_util] [139837055153920] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 17:24:18,847] - [remote_util] [139837055153920] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 17:24:18,911] - [remote_util] [139837055153920] - INFO - 172.23.122.132 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-22 17:24:19,094] - [remote_util] [139837055153920] - INFO - Running systemd command on this server [2021-02-22 17:24:19,094] - [remote_util] [139837055153920] - INFO - running command.raw on 172.23.122.132: systemctl stop couchbase-server.service [2021-02-22 17:24:20,789] - [remote_util] [139837055153920] - INFO - command executed successfully with root [2021-02-22 17:24:20,790] - [basetestcase] [139837055153920] - INFO - Couchbase stopped [2021-02-22 17:24:20,790] - [basetestcase] [139837055153920] - INFO - sleep for 5 secs. ... [2021-02-22 17:24:25,796] - [remote_util] [139837055153920] - INFO - running command.raw on 172.23.122.132: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-22 17:24:25,812] - [remote_util] [139837055153920] - INFO - command executed successfully with root [2021-02-22 17:24:25,812] - [remote_util] [139837055153920] - INFO - running command.raw on 172.23.122.132: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-22 17:24:25,867] - [remote_util] [139837055153920] - INFO - command executed successfully with root [2021-02-22 17:24:25,871] - [remote_util] [139837055153920] - INFO - SSH Connecting to 172.23.122.132 with username:root, attempt#1 of 5 [2021-02-22 17:24:25,975] - [remote_util] [139837055153920] - INFO - SSH Connected to 172.23.122.132 as root [2021-02-22 17:24:26,239] - [remote_util] [139837055153920] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 17:24:26,541] - [remote_util] [139837055153920] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 17:24:26,605] - [remote_util] [139837055153920] - INFO - 172.23.122.132 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-22 17:24:26,605] - [remote_util] [139837055153920] - INFO - Checking for process beam.smp on linux [2021-02-22 17:24:26,682] - [remote_util] [139837055153920] - INFO - Starting couchbase server [2021-02-22 17:24:26,787] - [remote_util] [139837055153920] - INFO - Running systemd command on this server [2021-02-22 17:24:26,787] - [remote_util] [139837055153920] - INFO - running command.raw on 172.23.122.132: systemctl start couchbase-server.service [2021-02-22 17:24:26,809] - [remote_util] [139837055153920] - INFO - command executed successfully with root [2021-02-22 17:24:26,809] - [remote_util] [139837055153920] - INFO - 172.23.122.132:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-22 17:24:31,815] - [remote_util] [139837055153920] - INFO - running command.raw on 172.23.122.132: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-22 17:24:31,842] - [remote_util] [139837055153920] - INFO - command executed successfully with root [2021-02-22 17:24:31,842] - [remote_util] [139837055153920] - INFO - Couchbase server status: [' Process: 37320 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-22 17:24:31,843] - [remote_util] [139837055153920] - INFO - Checking for process beam.smp on linux [2021-02-22 17:24:31,914] - [remote_util] [139837055153920] - INFO - process beam.smp is running on 172.23.122.132: with pid 37464 [2021-02-22 17:24:31,914] - [basetestcase] [139837055153920] - INFO - Couchbase started [2021-02-22 17:24:31,919] - [remote_util] [139837055153920] - INFO - SSH Connecting to 172.23.122.133 with username:root, attempt#1 of 5 [2021-02-22 17:24:32,027] - [remote_util] [139837055153920] - INFO - SSH Connected to 172.23.122.133 as root [2021-02-22 17:24:32,314] - [remote_util] [139837055153920] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 17:24:32,619] - [remote_util] [139837055153920] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 17:24:32,633] - [remote_util] [139837055153920] - INFO - SSH Connecting to 172.23.122.133 with username:root, attempt#1 of 5 [2021-02-22 17:24:32,738] - [remote_util] [139837055153920] - INFO - SSH Connected to 172.23.122.133 as root [2021-02-22 17:24:33,015] - [remote_util] [139837055153920] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 17:24:33,324] - [remote_util] [139837055153920] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 17:24:33,386] - [remote_util] [139837055153920] - INFO - 172.23.122.133 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-22 17:24:33,528] - [remote_util] [139837055153920] - INFO - Running systemd command on this server [2021-02-22 17:24:33,528] - [remote_util] [139837055153920] - INFO - running command.raw on 172.23.122.133: systemctl stop couchbase-server.service [2021-02-22 17:24:35,319] - [remote_util] [139837055153920] - INFO - command executed successfully with root [2021-02-22 17:24:35,320] - [basetestcase] [139837055153920] - INFO - Couchbase stopped [2021-02-22 17:24:35,321] - [basetestcase] [139837055153920] - INFO - sleep for 5 secs. ... [2021-02-22 17:24:40,326] - [remote_util] [139837055153920] - INFO - running command.raw on 172.23.122.133: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-22 17:24:40,345] - [remote_util] [139837055153920] - INFO - command executed successfully with root [2021-02-22 17:24:40,345] - [remote_util] [139837055153920] - INFO - running command.raw on 172.23.122.133: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-22 17:24:40,399] - [remote_util] [139837055153920] - INFO - command executed successfully with root [2021-02-22 17:24:40,403] - [remote_util] [139837055153920] - INFO - SSH Connecting to 172.23.122.133 with username:root, attempt#1 of 5 [2021-02-22 17:24:40,504] - [remote_util] [139837055153920] - INFO - SSH Connected to 172.23.122.133 as root [2021-02-22 17:24:40,789] - [remote_util] [139837055153920] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 17:24:41,100] - [remote_util] [139837055153920] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 17:24:41,163] - [remote_util] [139837055153920] - INFO - 172.23.122.133 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-22 17:24:41,163] - [remote_util] [139837055153920] - INFO - Checking for process beam.smp on linux [2021-02-22 17:24:41,240] - [remote_util] [139837055153920] - INFO - Starting couchbase server [2021-02-22 17:24:41,345] - [remote_util] [139837055153920] - INFO - Running systemd command on this server [2021-02-22 17:24:41,345] - [remote_util] [139837055153920] - INFO - running command.raw on 172.23.122.133: systemctl start couchbase-server.service [2021-02-22 17:24:41,370] - [remote_util] [139837055153920] - INFO - command executed successfully with root [2021-02-22 17:24:41,371] - [remote_util] [139837055153920] - INFO - 172.23.122.133:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-22 17:24:46,377] - [remote_util] [139837055153920] - INFO - running command.raw on 172.23.122.133: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-22 17:24:46,404] - [remote_util] [139837055153920] - INFO - command executed successfully with root [2021-02-22 17:24:46,404] - [remote_util] [139837055153920] - INFO - Couchbase server status: [' Process: 40014 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-22 17:24:46,405] - [remote_util] [139837055153920] - INFO - Checking for process beam.smp on linux [2021-02-22 17:24:46,474] - [remote_util] [139837055153920] - INFO - process beam.smp is running on 172.23.122.133: with pid 40159 [2021-02-22 17:24:46,474] - [basetestcase] [139837055153920] - INFO - Couchbase started [2021-02-22 17:24:46,479] - [remote_util] [139837055153920] - INFO - SSH Connecting to 172.23.122.134 with username:root, attempt#1 of 5 [2021-02-22 17:24:46,586] - [remote_util] [139837055153920] - INFO - SSH Connected to 172.23.122.134 as root [2021-02-22 17:24:46,866] - [remote_util] [139837055153920] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 17:24:47,168] - [remote_util] [139837055153920] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 17:24:47,181] - [remote_util] [139837055153920] - INFO - SSH Connecting to 172.23.122.134 with username:root, attempt#1 of 5 [2021-02-22 17:24:47,279] - [remote_util] [139837055153920] - INFO - SSH Connected to 172.23.122.134 as root [2021-02-22 17:24:47,553] - [remote_util] [139837055153920] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 17:24:47,861] - [remote_util] [139837055153920] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 17:24:47,923] - [remote_util] [139837055153920] - INFO - 172.23.122.134 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-22 17:24:48,022] - [remote_util] [139837055153920] - INFO - Running systemd command on this server [2021-02-22 17:24:48,023] - [remote_util] [139837055153920] - INFO - running command.raw on 172.23.122.134: systemctl stop couchbase-server.service [2021-02-22 17:24:49,635] - [remote_util] [139837055153920] - INFO - command executed successfully with root [2021-02-22 17:24:49,635] - [basetestcase] [139837055153920] - INFO - Couchbase stopped [2021-02-22 17:24:49,636] - [basetestcase] [139837055153920] - INFO - sleep for 5 secs. ... [2021-02-22 17:24:54,640] - [remote_util] [139837055153920] - INFO - running command.raw on 172.23.122.134: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-22 17:24:54,657] - [remote_util] [139837055153920] - INFO - command executed successfully with root [2021-02-22 17:24:54,658] - [remote_util] [139837055153920] - INFO - running command.raw on 172.23.122.134: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-22 17:24:54,711] - [remote_util] [139837055153920] - INFO - command executed successfully with root [2021-02-22 17:24:54,715] - [remote_util] [139837055153920] - INFO - SSH Connecting to 172.23.122.134 with username:root, attempt#1 of 5 [2021-02-22 17:24:54,816] - [remote_util] [139837055153920] - INFO - SSH Connected to 172.23.122.134 as root [2021-02-22 17:24:55,094] - [remote_util] [139837055153920] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 17:24:55,400] - [remote_util] [139837055153920] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 17:24:55,463] - [remote_util] [139837055153920] - INFO - 172.23.122.134 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-22 17:24:55,463] - [remote_util] [139837055153920] - INFO - Checking for process beam.smp on linux [2021-02-22 17:24:55,544] - [remote_util] [139837055153920] - INFO - Starting couchbase server [2021-02-22 17:24:55,692] - [remote_util] [139837055153920] - INFO - Running systemd command on this server [2021-02-22 17:24:55,692] - [remote_util] [139837055153920] - INFO - running command.raw on 172.23.122.134: systemctl start couchbase-server.service [2021-02-22 17:24:55,716] - [remote_util] [139837055153920] - INFO - command executed successfully with root [2021-02-22 17:24:55,716] - [remote_util] [139837055153920] - INFO - 172.23.122.134:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-22 17:25:00,717] - [remote_util] [139837055153920] - INFO - running command.raw on 172.23.122.134: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-22 17:25:00,744] - [remote_util] [139837055153920] - INFO - command executed successfully with root [2021-02-22 17:25:00,744] - [remote_util] [139837055153920] - INFO - Couchbase server status: [' Process: 123230 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-22 17:25:00,744] - [remote_util] [139837055153920] - INFO - Checking for process beam.smp on linux [2021-02-22 17:25:00,811] - [remote_util] [139837055153920] - INFO - process beam.smp is running on 172.23.122.134: with pid 123375 [2021-02-22 17:25:00,812] - [basetestcase] [139837055153920] - INFO - Couchbase started [2021-02-22 17:25:00,816] - [remote_util] [139837055153920] - INFO - SSH Connecting to 172.23.122.131 with username:root, attempt#1 of 5 [2021-02-22 17:25:00,923] - [remote_util] [139837055153920] - INFO - SSH Connected to 172.23.122.131 as root [2021-02-22 17:25:01,218] - [remote_util] [139837055153920] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 17:25:01,571] - [remote_util] [139837055153920] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 17:25:01,590] - [remote_util] [139837055153920] - INFO - SSH Connecting to 172.23.122.131 with username:root, attempt#1 of 5 [2021-02-22 17:25:01,691] - [remote_util] [139837055153920] - INFO - SSH Connected to 172.23.122.131 as root [2021-02-22 17:25:01,989] - [remote_util] [139837055153920] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 17:25:02,344] - [remote_util] [139837055153920] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 17:25:02,408] - [remote_util] [139837055153920] - INFO - 172.23.122.131 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-22 17:25:02,560] - [remote_util] [139837055153920] - INFO - Running systemd command on this server [2021-02-22 17:25:02,560] - [remote_util] [139837055153920] - INFO - running command.raw on 172.23.122.131: systemctl stop couchbase-server.service [2021-02-22 17:25:04,309] - [remote_util] [139837055153920] - INFO - command executed successfully with root [2021-02-22 17:25:04,309] - [basetestcase] [139837055153920] - INFO - Couchbase stopped [2021-02-22 17:25:04,309] - [basetestcase] [139837055153920] - INFO - sleep for 5 secs. ... [2021-02-22 17:25:09,315] - [remote_util] [139837055153920] - INFO - running command.raw on 172.23.122.131: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-22 17:25:09,334] - [remote_util] [139837055153920] - INFO - command executed successfully with root [2021-02-22 17:25:09,335] - [remote_util] [139837055153920] - INFO - running command.raw on 172.23.122.131: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-22 17:25:09,390] - [remote_util] [139837055153920] - INFO - command executed successfully with root [2021-02-22 17:25:09,394] - [remote_util] [139837055153920] - INFO - SSH Connecting to 172.23.122.131 with username:root, attempt#1 of 5 [2021-02-22 17:25:09,499] - [remote_util] [139837055153920] - INFO - SSH Connected to 172.23.122.131 as root [2021-02-22 17:25:09,795] - [remote_util] [139837055153920] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 17:25:10,109] - [remote_util] [139837055153920] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 17:25:10,174] - [remote_util] [139837055153920] - INFO - 172.23.122.131 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-22 17:25:10,175] - [remote_util] [139837055153920] - INFO - Checking for process beam.smp on linux [2021-02-22 17:25:10,258] - [remote_util] [139837055153920] - INFO - Starting couchbase server [2021-02-22 17:25:10,410] - [remote_util] [139837055153920] - INFO - Running systemd command on this server [2021-02-22 17:25:10,411] - [remote_util] [139837055153920] - INFO - running command.raw on 172.23.122.131: systemctl start couchbase-server.service [2021-02-22 17:25:10,438] - [remote_util] [139837055153920] - INFO - command executed successfully with root [2021-02-22 17:25:10,438] - [remote_util] [139837055153920] - INFO - 172.23.122.131:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-22 17:25:15,443] - [remote_util] [139837055153920] - INFO - running command.raw on 172.23.122.131: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-22 17:25:15,468] - [remote_util] [139837055153920] - INFO - command executed successfully with root [2021-02-22 17:25:15,470] - [remote_util] [139837055153920] - INFO - Couchbase server status: [' Process: 90615 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-22 17:25:15,470] - [remote_util] [139837055153920] - INFO - Checking for process beam.smp on linux [2021-02-22 17:25:15,541] - [remote_util] [139837055153920] - INFO - process beam.smp is running on 172.23.122.131: with pid 90759 [2021-02-22 17:25:15,542] - [basetestcase] [139837055153920] - INFO - Couchbase started [2021-02-22 17:25:15,545] - [remote_util] [139837055153920] - INFO - SSH Connecting to 172.23.122.180 with username:root, attempt#1 of 5 [2021-02-22 17:25:15,652] - [remote_util] [139837055153920] - INFO - SSH Connected to 172.23.122.180 as root [2021-02-22 17:25:15,937] - [remote_util] [139837055153920] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 17:25:16,248] - [remote_util] [139837055153920] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 17:25:16,262] - [remote_util] [139837055153920] - INFO - SSH Connecting to 172.23.122.180 with username:root, attempt#1 of 5 [2021-02-22 17:25:16,366] - [remote_util] [139837055153920] - INFO - SSH Connected to 172.23.122.180 as root [2021-02-22 17:25:16,650] - [remote_util] [139837055153920] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 17:25:16,971] - [remote_util] [139837055153920] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 17:25:17,035] - [remote_util] [139837055153920] - INFO - 172.23.122.180 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-22 17:25:17,147] - [remote_util] [139837055153920] - INFO - Running systemd command on this server [2021-02-22 17:25:17,147] - [remote_util] [139837055153920] - INFO - running command.raw on 172.23.122.180: systemctl stop couchbase-server.service [2021-02-22 17:25:18,876] - [remote_util] [139837055153920] - INFO - command executed successfully with root [2021-02-22 17:25:18,877] - [basetestcase] [139837055153920] - INFO - Couchbase stopped [2021-02-22 17:25:18,878] - [basetestcase] [139837055153920] - INFO - sleep for 5 secs. ... [2021-02-22 17:25:23,883] - [remote_util] [139837055153920] - INFO - running command.raw on 172.23.122.180: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-22 17:25:23,900] - [remote_util] [139837055153920] - INFO - command executed successfully with root [2021-02-22 17:25:23,900] - [remote_util] [139837055153920] - INFO - running command.raw on 172.23.122.180: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-22 17:25:23,955] - [remote_util] [139837055153920] - INFO - command executed successfully with root [2021-02-22 17:25:23,958] - [remote_util] [139837055153920] - INFO - SSH Connecting to 172.23.122.180 with username:root, attempt#1 of 5 [2021-02-22 17:25:24,059] - [remote_util] [139837055153920] - INFO - SSH Connected to 172.23.122.180 as root [2021-02-22 17:25:24,330] - [remote_util] [139837055153920] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 17:25:24,632] - [remote_util] [139837055153920] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 17:25:24,692] - [remote_util] [139837055153920] - INFO - 172.23.122.180 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-22 17:25:24,693] - [remote_util] [139837055153920] - INFO - Checking for process beam.smp on linux [2021-02-22 17:25:24,771] - [remote_util] [139837055153920] - INFO - Starting couchbase server [2021-02-22 17:25:24,913] - [remote_util] [139837055153920] - INFO - Running systemd command on this server [2021-02-22 17:25:24,913] - [remote_util] [139837055153920] - INFO - running command.raw on 172.23.122.180: systemctl start couchbase-server.service [2021-02-22 17:25:24,936] - [remote_util] [139837055153920] - INFO - command executed successfully with root [2021-02-22 17:25:24,936] - [remote_util] [139837055153920] - INFO - 172.23.122.180:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-22 17:25:29,938] - [remote_util] [139837055153920] - INFO - running command.raw on 172.23.122.180: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-22 17:25:29,967] - [remote_util] [139837055153920] - INFO - command executed successfully with root [2021-02-22 17:25:29,967] - [remote_util] [139837055153920] - INFO - Couchbase server status: [' Process: 100752 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-22 17:25:29,968] - [remote_util] [139837055153920] - INFO - Checking for process beam.smp on linux [2021-02-22 17:25:30,039] - [remote_util] [139837055153920] - INFO - process beam.smp is running on 172.23.122.180: with pid 100895 [2021-02-22 17:25:30,040] - [basetestcase] [139837055153920] - INFO - Couchbase started [2021-02-22 17:25:30,043] - [remote_util] [139837055153920] - INFO - SSH Connecting to 172.23.122.130 with username:root, attempt#1 of 5 [2021-02-22 17:25:30,152] - [remote_util] [139837055153920] - INFO - SSH Connected to 172.23.122.130 as root [2021-02-22 17:25:30,433] - [remote_util] [139837055153920] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 17:25:30,755] - [remote_util] [139837055153920] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 17:25:30,768] - [remote_util] [139837055153920] - INFO - SSH Connecting to 172.23.122.130 with username:root, attempt#1 of 5 [2021-02-22 17:25:30,873] - [remote_util] [139837055153920] - INFO - SSH Connected to 172.23.122.130 as root [2021-02-22 17:25:31,153] - [remote_util] [139837055153920] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 17:25:31,471] - [remote_util] [139837055153920] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 17:25:31,536] - [remote_util] [139837055153920] - INFO - 172.23.122.130 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-22 17:25:31,646] - [remote_util] [139837055153920] - INFO - Running systemd command on this server [2021-02-22 17:25:31,646] - [remote_util] [139837055153920] - INFO - running command.raw on 172.23.122.130: systemctl stop couchbase-server.service [2021-02-22 17:25:33,279] - [remote_util] [139837055153920] - INFO - command executed successfully with root [2021-02-22 17:25:33,279] - [basetestcase] [139837055153920] - INFO - Couchbase stopped [2021-02-22 17:25:33,280] - [basetestcase] [139837055153920] - INFO - sleep for 5 secs. ... [2021-02-22 17:25:38,286] - [remote_util] [139837055153920] - INFO - running command.raw on 172.23.122.130: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-22 17:25:38,305] - [remote_util] [139837055153920] - INFO - command executed successfully with root [2021-02-22 17:25:38,305] - [remote_util] [139837055153920] - INFO - running command.raw on 172.23.122.130: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-22 17:25:38,361] - [remote_util] [139837055153920] - INFO - command executed successfully with root [2021-02-22 17:25:38,365] - [remote_util] [139837055153920] - INFO - SSH Connecting to 172.23.122.130 with username:root, attempt#1 of 5 [2021-02-22 17:25:38,477] - [remote_util] [139837055153920] - INFO - SSH Connected to 172.23.122.130 as root [2021-02-22 17:25:38,748] - [remote_util] [139837055153920] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 17:25:39,062] - [remote_util] [139837055153920] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 17:25:39,126] - [remote_util] [139837055153920] - INFO - 172.23.122.130 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-22 17:25:39,127] - [remote_util] [139837055153920] - INFO - Checking for process beam.smp on linux [2021-02-22 17:25:39,208] - [remote_util] [139837055153920] - INFO - Starting couchbase server [2021-02-22 17:25:39,313] - [remote_util] [139837055153920] - INFO - Running systemd command on this server [2021-02-22 17:25:39,314] - [remote_util] [139837055153920] - INFO - running command.raw on 172.23.122.130: systemctl start couchbase-server.service [2021-02-22 17:25:39,339] - [remote_util] [139837055153920] - INFO - command executed successfully with root [2021-02-22 17:25:39,339] - [remote_util] [139837055153920] - INFO - 172.23.122.130:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-22 17:25:44,345] - [remote_util] [139837055153920] - INFO - running command.raw on 172.23.122.130: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-22 17:25:44,368] - [remote_util] [139837055153920] - INFO - command executed successfully with root [2021-02-22 17:25:44,369] - [remote_util] [139837055153920] - INFO - Couchbase server status: [' Process: 125192 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-22 17:25:44,369] - [remote_util] [139837055153920] - INFO - Checking for process beam.smp on linux [2021-02-22 17:25:44,439] - [remote_util] [139837055153920] - INFO - process beam.smp is running on 172.23.122.130: with pid 125367 [2021-02-22 17:25:44,439] - [basetestcase] [139837055153920] - INFO - Couchbase started [2021-02-22 17:25:44,440] - [basetestcase] [139837055153920] - INFO - sleep for 10 secs. ... [2021-02-22 17:25:54,518] - [task] [139837071939328] - INFO - server: ip:172.23.123.42 port:8091 ssh_username:root, nodes/self [2021-02-22 17:25:54,524] - [task] [139837071939328] - INFO - {'uptime': '110', 'memoryTotal': 4201672704, 'memoryFree': 3524513792, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.123.42:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4514-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.123.42', 'ip': '172.23.123.42', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} [2021-02-22 17:25:54,527] - [rest_client] [139837071939328] - ERROR - GET http://172.23.123.42:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-02-22 17:25:54,528] - [rest_client] [139837071939328] - INFO - pools/default params : memoryQuota=2147 [2021-02-22 17:25:54,535] - [rest_client] [139837071939328] - INFO - --> init_node_services(Administrator,password,172.23.123.42,8091,['kv']) [2021-02-22 17:25:54,536] - [rest_client] [139837071939328] - INFO - /node/controller/setupServices params on 172.23.123.42: 8091:hostname=172.23.123.42&user=Administrator&password=password&services=kv [2021-02-22 17:25:54,565] - [rest_client] [139837071939328] - INFO - --> in init_cluster...Administrator,password,8091 [2021-02-22 17:25:54,566] - [rest_client] [139837071939328] - INFO - settings/web params on 172.23.123.42:8091:port=8091&username=Administrator&password=password [2021-02-22 17:25:54,626] - [rest_client] [139837071939328] - INFO - --> status:True [2021-02-22 17:25:54,629] - [remote_util] [139837071939328] - INFO - SSH Connecting to 172.23.123.42 with username:root, attempt#1 of 5 [2021-02-22 17:25:54,736] - [remote_util] [139837071939328] - INFO - SSH Connected to 172.23.123.42 as root [2021-02-22 17:25:55,012] - [remote_util] [139837071939328] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 17:25:55,323] - [remote_util] [139837071939328] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 17:25:55,336] - [remote_util] [139837071939328] - INFO - running command.raw on 172.23.123.42: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-02-22 17:25:55,391] - [remote_util] [139837071939328] - INFO - command executed successfully with root [2021-02-22 17:25:55,393] - [remote_util] [139837071939328] - INFO - ['ok'] [2021-02-22 17:25:55,398] - [rest_client] [139837071939328] - INFO - /diag/eval status on 172.23.123.42:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-22 17:25:55,401] - [rest_client] [139837071939328] - INFO - /diag/eval status on 172.23.123.42:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-22 17:25:55,406] - [rest_client] [139837071939328] - INFO - settings/indexes params : storageMode=plasma [2021-02-22 17:25:55,428] - [task] [139837071939328] - INFO - server: ip:172.23.123.44 port:8091 ssh_username:root, nodes/self [2021-02-22 17:25:55,432] - [task] [139837071939328] - INFO - {'uptime': '100', 'memoryTotal': 4201676800, 'memoryFree': 3583569920, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.123.44:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4514-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.123.44', 'ip': '172.23.123.44', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} [2021-02-22 17:25:55,435] - [rest_client] [139837071939328] - ERROR - GET http://172.23.123.44:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-02-22 17:25:55,436] - [rest_client] [139837071939328] - INFO - pools/default params : memoryQuota=2147 [2021-02-22 17:25:55,443] - [rest_client] [139837071939328] - INFO - --> in init_cluster...Administrator,password,8091 [2021-02-22 17:25:55,443] - [rest_client] [139837071939328] - INFO - settings/web params on 172.23.123.44:8091:port=8091&username=Administrator&password=password [2021-02-22 17:25:55,502] - [rest_client] [139837071939328] - INFO - --> status:True [2021-02-22 17:25:55,506] - [remote_util] [139837071939328] - INFO - SSH Connecting to 172.23.123.44 with username:root, attempt#1 of 5 [2021-02-22 17:25:55,617] - [remote_util] [139837071939328] - INFO - SSH Connected to 172.23.123.44 as root [2021-02-22 17:25:55,895] - [remote_util] [139837071939328] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 17:25:56,201] - [remote_util] [139837071939328] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 17:25:56,212] - [remote_util] [139837071939328] - INFO - running command.raw on 172.23.123.44: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-02-22 17:25:56,266] - [remote_util] [139837071939328] - INFO - command executed successfully with root [2021-02-22 17:25:56,267] - [remote_util] [139837071939328] - INFO - ['ok'] [2021-02-22 17:25:56,271] - [rest_client] [139837071939328] - INFO - /diag/eval status on 172.23.123.44:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-22 17:25:56,274] - [rest_client] [139837071939328] - INFO - /diag/eval status on 172.23.123.44:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-22 17:25:56,280] - [rest_client] [139837071939328] - INFO - settings/indexes params : storageMode=plasma [2021-02-22 17:25:56,301] - [task] [139837071939328] - INFO - server: ip:172.23.122.132 port:8091 ssh_username:root, nodes/self [2021-02-22 17:25:56,305] - [task] [139837071939328] - INFO - {'uptime': '84', 'memoryTotal': 4201684992, 'memoryFree': 3599241216, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.122.132:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4514-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.122.132', 'ip': '172.23.122.132', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} [2021-02-22 17:25:56,308] - [rest_client] [139837071939328] - ERROR - GET http://172.23.122.132:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-02-22 17:25:56,309] - [rest_client] [139837071939328] - INFO - pools/default params : memoryQuota=2147 [2021-02-22 17:25:56,316] - [rest_client] [139837071939328] - INFO - --> in init_cluster...Administrator,password,8091 [2021-02-22 17:25:56,316] - [rest_client] [139837071939328] - INFO - settings/web params on 172.23.122.132:8091:port=8091&username=Administrator&password=password [2021-02-22 17:25:56,372] - [rest_client] [139837071939328] - INFO - --> status:True [2021-02-22 17:25:56,376] - [remote_util] [139837071939328] - INFO - SSH Connecting to 172.23.122.132 with username:root, attempt#1 of 5 [2021-02-22 17:25:56,485] - [remote_util] [139837071939328] - INFO - SSH Connected to 172.23.122.132 as root [2021-02-22 17:25:56,767] - [remote_util] [139837071939328] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 17:25:57,077] - [remote_util] [139837071939328] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 17:25:57,089] - [remote_util] [139837071939328] - INFO - running command.raw on 172.23.122.132: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-02-22 17:25:57,145] - [remote_util] [139837071939328] - INFO - command executed successfully with root [2021-02-22 17:25:57,145] - [remote_util] [139837071939328] - INFO - ['ok'] [2021-02-22 17:25:57,149] - [rest_client] [139837071939328] - INFO - /diag/eval status on 172.23.122.132:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-22 17:25:57,152] - [rest_client] [139837071939328] - INFO - /diag/eval status on 172.23.122.132:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-22 17:25:57,160] - [rest_client] [139837071939328] - INFO - settings/indexes params : storageMode=plasma [2021-02-22 17:25:57,183] - [task] [139837071939328] - INFO - server: ip:172.23.122.133 port:8091 ssh_username:root, nodes/self [2021-02-22 17:25:57,186] - [task] [139837071939328] - INFO - {'uptime': '69', 'memoryTotal': 4201684992, 'memoryFree': 3585945600, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.122.133:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4514-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.122.133', 'ip': '172.23.122.133', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} [2021-02-22 17:25:57,189] - [rest_client] [139837071939328] - ERROR - GET http://172.23.122.133:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-02-22 17:25:57,190] - [rest_client] [139837071939328] - INFO - pools/default params : memoryQuota=2147 [2021-02-22 17:25:57,197] - [rest_client] [139837071939328] - INFO - --> in init_cluster...Administrator,password,8091 [2021-02-22 17:25:57,197] - [rest_client] [139837071939328] - INFO - settings/web params on 172.23.122.133:8091:port=8091&username=Administrator&password=password [2021-02-22 17:25:57,253] - [rest_client] [139837071939328] - INFO - --> status:True [2021-02-22 17:25:57,256] - [remote_util] [139837071939328] - INFO - SSH Connecting to 172.23.122.133 with username:root, attempt#1 of 5 [2021-02-22 17:25:57,365] - [remote_util] [139837071939328] - INFO - SSH Connected to 172.23.122.133 as root [2021-02-22 17:25:57,663] - [remote_util] [139837071939328] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 17:25:57,977] - [remote_util] [139837071939328] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 17:25:57,988] - [remote_util] [139837071939328] - INFO - running command.raw on 172.23.122.133: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-02-22 17:25:58,044] - [remote_util] [139837071939328] - INFO - command executed successfully with root [2021-02-22 17:25:58,044] - [remote_util] [139837071939328] - INFO - ['ok'] [2021-02-22 17:25:58,048] - [rest_client] [139837071939328] - INFO - /diag/eval status on 172.23.122.133:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-22 17:25:58,051] - [rest_client] [139837071939328] - INFO - /diag/eval status on 172.23.122.133:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-22 17:25:58,057] - [rest_client] [139837071939328] - INFO - settings/indexes params : storageMode=plasma [2021-02-22 17:25:58,082] - [task] [139837071939328] - INFO - server: ip:172.23.122.134 port:8091 ssh_username:root, nodes/self [2021-02-22 17:25:58,086] - [task] [139837071939328] - INFO - {'uptime': '59', 'memoryTotal': 4201684992, 'memoryFree': 3584094208, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.122.134:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4514-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.122.134', 'ip': '172.23.122.134', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} [2021-02-22 17:25:58,089] - [rest_client] [139837071939328] - ERROR - GET http://172.23.122.134:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-02-22 17:25:58,090] - [rest_client] [139837071939328] - INFO - pools/default params : memoryQuota=2147 [2021-02-22 17:25:58,097] - [rest_client] [139837071939328] - INFO - --> in init_cluster...Administrator,password,8091 [2021-02-22 17:25:58,097] - [rest_client] [139837071939328] - INFO - settings/web params on 172.23.122.134:8091:port=8091&username=Administrator&password=password [2021-02-22 17:25:58,155] - [rest_client] [139837071939328] - INFO - --> status:True [2021-02-22 17:25:58,158] - [remote_util] [139837071939328] - INFO - SSH Connecting to 172.23.122.134 with username:root, attempt#1 of 5 [2021-02-22 17:25:58,268] - [remote_util] [139837071939328] - INFO - SSH Connected to 172.23.122.134 as root [2021-02-22 17:25:58,565] - [remote_util] [139837071939328] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 17:25:58,880] - [remote_util] [139837071939328] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 17:25:58,893] - [remote_util] [139837071939328] - INFO - running command.raw on 172.23.122.134: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-02-22 17:25:58,946] - [remote_util] [139837071939328] - INFO - command executed successfully with root [2021-02-22 17:25:58,947] - [remote_util] [139837071939328] - INFO - ['ok'] [2021-02-22 17:25:58,950] - [rest_client] [139837071939328] - INFO - /diag/eval status on 172.23.122.134:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-22 17:25:58,953] - [rest_client] [139837071939328] - INFO - /diag/eval status on 172.23.122.134:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-22 17:25:58,958] - [rest_client] [139837071939328] - INFO - settings/indexes params : storageMode=plasma [2021-02-22 17:25:58,980] - [task] [139837071939328] - INFO - server: ip:172.23.122.131 port:8091 ssh_username:root, nodes/self [2021-02-22 17:25:58,985] - [task] [139837071939328] - INFO - {'uptime': '44', 'memoryTotal': 4201684992, 'memoryFree': 3586797568, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.122.131:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4514-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.122.131', 'ip': '172.23.122.131', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} [2021-02-22 17:25:58,988] - [rest_client] [139837071939328] - ERROR - GET http://172.23.122.131:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-02-22 17:25:58,989] - [rest_client] [139837071939328] - INFO - pools/default params : memoryQuota=2147 [2021-02-22 17:25:58,996] - [rest_client] [139837071939328] - INFO - --> in init_cluster...Administrator,password,8091 [2021-02-22 17:25:58,997] - [rest_client] [139837071939328] - INFO - settings/web params on 172.23.122.131:8091:port=8091&username=Administrator&password=password [2021-02-22 17:25:59,053] - [rest_client] [139837071939328] - INFO - --> status:True [2021-02-22 17:25:59,056] - [remote_util] [139837071939328] - INFO - SSH Connecting to 172.23.122.131 with username:root, attempt#1 of 5 [2021-02-22 17:25:59,170] - [remote_util] [139837071939328] - INFO - SSH Connected to 172.23.122.131 as root [2021-02-22 17:25:59,475] - [remote_util] [139837071939328] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 17:25:59,786] - [remote_util] [139837071939328] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 17:25:59,798] - [remote_util] [139837071939328] - INFO - running command.raw on 172.23.122.131: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-02-22 17:25:59,853] - [remote_util] [139837071939328] - INFO - command executed successfully with root [2021-02-22 17:25:59,854] - [remote_util] [139837071939328] - INFO - ['ok'] [2021-02-22 17:25:59,858] - [rest_client] [139837071939328] - INFO - /diag/eval status on 172.23.122.131:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-22 17:25:59,861] - [rest_client] [139837071939328] - INFO - /diag/eval status on 172.23.122.131:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-22 17:25:59,867] - [rest_client] [139837071939328] - INFO - settings/indexes params : storageMode=plasma [2021-02-22 17:25:59,890] - [task] [139837071939328] - INFO - server: ip:172.23.122.180 port:8091 ssh_username:root, nodes/self [2021-02-22 17:25:59,895] - [task] [139837071939328] - INFO - {'uptime': '30', 'memoryTotal': 4201684992, 'memoryFree': 3528818688, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.122.180:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4514-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.122.180', 'ip': '172.23.122.180', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} [2021-02-22 17:25:59,898] - [rest_client] [139837071939328] - ERROR - GET http://172.23.122.180:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-02-22 17:25:59,898] - [rest_client] [139837071939328] - INFO - pools/default params : memoryQuota=2147 [2021-02-22 17:25:59,905] - [rest_client] [139837071939328] - INFO - --> in init_cluster...Administrator,password,8091 [2021-02-22 17:25:59,905] - [rest_client] [139837071939328] - INFO - settings/web params on 172.23.122.180:8091:port=8091&username=Administrator&password=password [2021-02-22 17:25:59,963] - [rest_client] [139837071939328] - INFO - --> status:True [2021-02-22 17:25:59,966] - [remote_util] [139837071939328] - INFO - SSH Connecting to 172.23.122.180 with username:root, attempt#1 of 5 [2021-02-22 17:26:00,075] - [remote_util] [139837071939328] - INFO - SSH Connected to 172.23.122.180 as root [2021-02-22 17:26:00,371] - [remote_util] [139837071939328] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 17:26:00,678] - [remote_util] [139837071939328] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 17:26:00,690] - [remote_util] [139837071939328] - INFO - running command.raw on 172.23.122.180: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-02-22 17:26:00,744] - [remote_util] [139837071939328] - INFO - command executed successfully with root [2021-02-22 17:26:00,745] - [remote_util] [139837071939328] - INFO - ['ok'] [2021-02-22 17:26:00,748] - [rest_client] [139837071939328] - INFO - /diag/eval status on 172.23.122.180:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-22 17:26:00,751] - [rest_client] [139837071939328] - INFO - /diag/eval status on 172.23.122.180:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-22 17:26:00,757] - [rest_client] [139837071939328] - INFO - settings/indexes params : storageMode=plasma [2021-02-22 17:26:00,790] - [task] [139837071939328] - INFO - server: ip:172.23.122.130 port:8091 ssh_username:root, nodes/self [2021-02-22 17:26:00,795] - [task] [139837071939328] - INFO - {'uptime': '14', 'memoryTotal': 4201689088, 'memoryFree': 3558993920, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.122.130:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4514-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@cb.local', 'ip': '172.23.122.130', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} [2021-02-22 17:26:00,798] - [rest_client] [139837071939328] - ERROR - GET http://172.23.122.130:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-02-22 17:26:00,799] - [rest_client] [139837071939328] - INFO - pools/default params : memoryQuota=2147 [2021-02-22 17:26:00,806] - [rest_client] [139837071939328] - INFO - --> in init_cluster...Administrator,password,8091 [2021-02-22 17:26:00,806] - [rest_client] [139837071939328] - INFO - settings/web params on 172.23.122.130:8091:port=8091&username=Administrator&password=password [2021-02-22 17:26:00,864] - [rest_client] [139837071939328] - INFO - --> status:True [2021-02-22 17:26:00,868] - [remote_util] [139837071939328] - INFO - SSH Connecting to 172.23.122.130 with username:root, attempt#1 of 5 [2021-02-22 17:26:00,978] - [remote_util] [139837071939328] - INFO - SSH Connected to 172.23.122.130 as root [2021-02-22 17:26:01,265] - [remote_util] [139837071939328] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 17:26:01,580] - [remote_util] [139837071939328] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 17:26:01,593] - [remote_util] [139837071939328] - INFO - running command.raw on 172.23.122.130: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-02-22 17:26:01,648] - [remote_util] [139837071939328] - INFO - command executed successfully with root [2021-02-22 17:26:01,648] - [remote_util] [139837071939328] - INFO - ['ok'] [2021-02-22 17:26:01,652] - [rest_client] [139837071939328] - INFO - /diag/eval status on 172.23.122.130:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-22 17:26:01,655] - [rest_client] [139837071939328] - INFO - /diag/eval status on 172.23.122.130:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-22 17:26:01,662] - [rest_client] [139837071939328] - INFO - settings/indexes params : storageMode=plasma [2021-02-22 17:26:01,693] - [basetestcase] [139837055153920] - INFO - **** add built-in 'cbadminbucket' user to node 172.23.123.42 **** [2021-02-22 17:26:01,707] - [rest_client] [139837055153920] - ERROR - DELETE http://172.23.123.42:8091/settings/rbac/users/local/cbadminbucket body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:password [2021-02-22 17:26:01,708] - [internal_user] [139837055153920] - INFO - Exception while deleting user. Exception is -b'"User was not found."' [2021-02-22 17:26:01,772] - [basetestcase] [139837055153920] - INFO - sleep for 5 secs. ... [2021-02-22 17:26:06,777] - [basetestcase] [139837055153920] - INFO - **** add 'admin' role to 'cbadminbucket' user **** [2021-02-22 17:26:06,793] - [basetestcase] [139837055153920] - INFO - done initializing cluster [2021-02-22 17:26:07,686] - [task] [139837071939328] - INFO - adding node 172.23.123.44:8091 to cluster [2021-02-22 17:26:07,687] - [rest_client] [139837071939328] - INFO - adding remote node @172.23.123.44:8091 to this cluster @172.23.123.42:8091 [2021-02-22 17:26:17,705] - [rest_client] [139837071939328] - INFO - rebalance progress took 10.02 seconds [2021-02-22 17:26:17,706] - [rest_client] [139837071939328] - INFO - sleep for 10 seconds after rebalance... [2021-02-22 17:26:29,741] - [task] [139837071939328] - INFO - adding node 172.23.122.132:8091 to cluster [2021-02-22 17:26:29,742] - [rest_client] [139837071939328] - INFO - adding remote node @172.23.122.132:8091 to this cluster @172.23.123.42:8091 [2021-02-22 17:26:39,753] - [rest_client] [139837071939328] - INFO - rebalance progress took 10.01 seconds [2021-02-22 17:26:39,754] - [rest_client] [139837071939328] - INFO - sleep for 10 seconds after rebalance... [2021-02-22 17:26:52,172] - [task] [139837071939328] - INFO - adding node 172.23.122.133:8091 to cluster [2021-02-22 17:26:52,173] - [rest_client] [139837071939328] - INFO - adding remote node @172.23.122.133:8091 to this cluster @172.23.123.42:8091 [2021-02-22 17:27:02,190] - [rest_client] [139837071939328] - INFO - rebalance progress took 10.02 seconds [2021-02-22 17:27:02,190] - [rest_client] [139837071939328] - INFO - sleep for 10 seconds after rebalance... [2021-02-22 17:27:14,905] - [task] [139837071939328] - INFO - adding node 172.23.122.134:8091 to cluster [2021-02-22 17:27:14,905] - [rest_client] [139837071939328] - INFO - adding remote node @172.23.122.134:8091 to this cluster @172.23.123.42:8091 [2021-02-22 17:27:24,923] - [rest_client] [139837071939328] - INFO - rebalance progress took 10.02 seconds [2021-02-22 17:27:24,924] - [rest_client] [139837071939328] - INFO - sleep for 10 seconds after rebalance... [2021-02-22 17:27:37,258] - [rest_client] [139837071939328] - INFO - rebalance params : {'knownNodes': 'ns_1@172.23.122.132,ns_1@172.23.122.133,ns_1@172.23.122.134,ns_1@172.23.123.42,ns_1@172.23.123.44', 'ejectedNodes': '', 'user': 'Administrator', 'password': 'password'} [2021-02-22 17:27:37,286] - [rest_client] [139837071939328] - INFO - rebalance operation started [2021-02-22 17:27:37,290] - [rest_client] [139837071939328] - INFO - rebalance percentage : 0.00 % [2021-02-22 17:27:37,290] - [task] [139837071939328] - INFO - Rebalance - status: running, progress: 0.00% [2021-02-22 17:27:47,314] - [task] [139837071939328] - INFO - Rebalance - status: none, progress: 100.00% [2021-02-22 17:27:47,323] - [task] [139837071939328] - INFO - rebalancing was completed with progress: 100% in 10.035813808441162 sec [2021-02-22 17:27:47,326] - [remote_util] [139837055153920] - INFO - SSH Connecting to 172.23.123.42 with username:root, attempt#1 of 5 [2021-02-22 17:27:47,432] - [remote_util] [139837055153920] - INFO - SSH Connected to 172.23.123.42 as root [2021-02-22 17:27:47,699] - [remote_util] [139837055153920] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 17:27:48,007] - [remote_util] [139837055153920] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 17:27:48,019] - [remote_util] [139837055153920] - INFO - running command.raw on 172.23.123.42: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-02-22 17:27:48,074] - [remote_util] [139837055153920] - INFO - command executed successfully with root [2021-02-22 17:27:48,074] - [remote_util] [139837055153920] - INFO - ['ok'] [2021-02-22 17:27:48,075] - [basetestcase] [139837055153920] - INFO - Enabled diag/eval for non-local hosts from 172.23.123.42 [2021-02-22 17:27:48,075] - [basetestcase] [139837055153920] - INFO - ============== basetestcase setup was finished for test #8 test_eventing_rebalance_with_multiple_kv_nodes ============== [2021-02-22 17:27:48,088] - [remote_util] [139837055153920] - INFO - SSH Connecting to 172.23.123.42 with username:root, attempt#1 of 5 [2021-02-22 17:27:48,192] - [remote_util] [139837055153920] - INFO - SSH Connected to 172.23.123.42 as root [2021-02-22 17:27:48,449] - [remote_util] [139837055153920] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 17:27:48,764] - [remote_util] [139837055153920] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 17:27:48,768] - [remote_util] [139837055153920] - INFO - SSH Connecting to 172.23.123.44 with username:root, attempt#1 of 5 [2021-02-22 17:27:48,875] - [remote_util] [139837055153920] - INFO - SSH Connected to 172.23.123.44 as root [2021-02-22 17:27:49,150] - [remote_util] [139837055153920] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 17:27:49,459] - [remote_util] [139837055153920] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 17:27:49,463] - [remote_util] [139837055153920] - INFO - SSH Connecting to 172.23.122.132 with username:root, attempt#1 of 5 [2021-02-22 17:27:49,570] - [remote_util] [139837055153920] - INFO - SSH Connected to 172.23.122.132 as root [2021-02-22 17:27:49,852] - [remote_util] [139837055153920] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 17:27:50,149] - [remote_util] [139837055153920] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 17:27:50,154] - [remote_util] [139837055153920] - INFO - SSH Connecting to 172.23.122.133 with username:root, attempt#1 of 5 [2021-02-22 17:27:50,266] - [remote_util] [139837055153920] - INFO - SSH Connected to 172.23.122.133 as root [2021-02-22 17:27:50,553] - [remote_util] [139837055153920] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 17:27:50,861] - [remote_util] [139837055153920] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 17:27:50,866] - [remote_util] [139837055153920] - INFO - SSH Connecting to 172.23.122.134 with username:root, attempt#1 of 5 [2021-02-22 17:27:50,979] - [remote_util] [139837055153920] - INFO - SSH Connected to 172.23.122.134 as root [2021-02-22 17:27:51,277] - [remote_util] [139837055153920] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 17:27:51,590] - [remote_util] [139837055153920] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 17:27:51,596] - [remote_util] [139837055153920] - INFO - SSH Connecting to 172.23.122.131 with username:root, attempt#1 of 5 [2021-02-22 17:27:51,702] - [remote_util] [139837055153920] - INFO - SSH Connected to 172.23.122.131 as root [2021-02-22 17:27:52,007] - [remote_util] [139837055153920] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 17:27:52,325] - [remote_util] [139837055153920] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 17:27:52,331] - [remote_util] [139837055153920] - INFO - SSH Connecting to 172.23.122.180 with username:root, attempt#1 of 5 [2021-02-22 17:27:52,434] - [remote_util] [139837055153920] - INFO - SSH Connected to 172.23.122.180 as root [2021-02-22 17:27:52,718] - [remote_util] [139837055153920] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 17:27:53,026] - [remote_util] [139837055153920] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 17:27:53,031] - [remote_util] [139837055153920] - INFO - SSH Connecting to 172.23.122.130 with username:root, attempt#1 of 5 [2021-02-22 17:27:53,136] - [remote_util] [139837055153920] - INFO - SSH Connected to 172.23.122.130 as root [2021-02-22 17:27:53,410] - [remote_util] [139837055153920] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 17:27:53,720] - [remote_util] [139837055153920] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 17:28:44,111] - [basetestcase] [139837055153920] - INFO - ------- Cluster statistics ------- [2021-02-22 17:28:44,111] - [basetestcase] [139837055153920] - INFO - 172.23.122.132:8091 => {'services': ['kv'], 'cpu_utilization': 0.6042296072507553, 'mem_free': 3571826688, 'mem_total': 4201684992, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-02-22 17:28:44,112] - [basetestcase] [139837055153920] - INFO - 172.23.122.133:8091 => {'services': ['eventing'], 'cpu_utilization': 0.5790533736153072, 'mem_free': 3547951104, 'mem_total': 4201684992, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-02-22 17:28:44,112] - [basetestcase] [139837055153920] - INFO - 172.23.122.134:8091 => {'services': ['index', 'n1ql'], 'cpu_utilization': 8.839918946301925, 'mem_free': 3510525952, 'mem_total': 4201684992, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-02-22 17:28:44,112] - [basetestcase] [139837055153920] - INFO - 172.23.123.42:8091 => {'services': ['kv'], 'cpu_utilization': 0.5784708249496981, 'mem_free': 3492556800, 'mem_total': 4201672704, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-02-22 17:28:44,113] - [basetestcase] [139837055153920] - INFO - 172.23.123.44:8091 => {'services': ['kv'], 'cpu_utilization': 0.6037735849056604, 'mem_free': 3556638720, 'mem_total': 4201676800, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-02-22 17:28:44,113] - [basetestcase] [139837055153920] - INFO - --- End of cluster statistics --- [2021-02-22 17:28:44,116] - [remote_util] [139837055153920] - INFO - SSH Connecting to 172.23.123.42 with username:root, attempt#1 of 5 [2021-02-22 17:28:44,231] - [remote_util] [139837055153920] - INFO - SSH Connected to 172.23.123.42 as root [2021-02-22 17:28:44,523] - [remote_util] [139837055153920] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 17:28:44,843] - [remote_util] [139837055153920] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 17:28:47,139] - [basetestcase] [139837055153920] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.134 port:8091 ssh_username:root] [2021-02-22 17:28:47,150] - [basetestcase] [139837055153920] - INFO - list of eventing nodes in cluster: [ip:172.23.122.133 port:8091 ssh_username:root] [2021-02-22 17:28:47,159] - [rest_client] [139837055153920] - INFO - settings/indexes params : storageMode=plasma [2021-02-22 17:28:47,165] - [eventing_base] [139837055153920] - INFO - Setting the min possible memory quota so that adding mode nodes to the cluster wouldn't be a problem. [2021-02-22 17:28:47,165] - [rest_client] [139837055153920] - INFO - pools/default params : memoryQuota=330 [2021-02-22 17:28:47,171] - [rest_client] [139837055153920] - INFO - pools/default params : indexMemoryQuota=256 [2021-02-22 17:28:47,179] - [rest_client] [139837055153920] - INFO - pools/default params : eventingMemoryQuota=512 [2021-02-22 17:28:47,205] - [basetestcase] [139837055153920] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.134 port:8091 ssh_username:root] [2021-02-22 17:28:47,212] - [rest_client] [139837055153920] - INFO - pools/default params : memoryQuota=700 [2021-02-22 17:28:47,218] - [eventing_rebalance] [139837055153920] - INFO - 100 [2021-02-22 17:28:48,169] - [rest_client] [139837071939328] - INFO - http://172.23.123.42:8091/pools/default/buckets with param: name=src_bucket&ramQuotaMB=100&authType=none&replicaNumber=0&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore [2021-02-22 17:28:48,202] - [rest_client] [139837071939328] - INFO - 0.03 seconds to create bucket src_bucket [2021-02-22 17:28:48,203] - [bucket_helper] [139837071939328] - INFO - waiting for memcached bucket : src_bucket in 172.23.123.42 to accept set ops [2021-02-22 17:28:48,945] - [data_helper] [139837071939328] - INFO - creating direct client 172.23.122.132:11210 src_bucket [2021-02-22 17:28:49,217] - [data_helper] [139837071939328] - INFO - creating direct client 172.23.123.42:11210 src_bucket [2021-02-22 17:28:49,347] - [data_helper] [139837071939328] - INFO - creating direct client 172.23.123.44:11210 src_bucket [2021-02-22 17:28:49,681] - [data_helper] [139837071939328] - INFO - creating direct client 172.23.122.132:11210 src_bucket [2021-02-22 17:28:49,845] - [data_helper] [139837071939328] - INFO - creating direct client 172.23.123.42:11210 src_bucket [2021-02-22 17:28:49,955] - [data_helper] [139837071939328] - INFO - creating direct client 172.23.123.44:11210 src_bucket [2021-02-22 17:28:50,187] - [task] [139837071939328] - WARNING - vbucket map not ready after try 0 [2021-02-22 17:28:50,187] - [bucket_helper] [139837071939328] - INFO - waiting for memcached bucket : src_bucket in 172.23.123.42 to accept set ops [2021-02-22 17:28:50,365] - [data_helper] [139837071939328] - INFO - creating direct client 172.23.122.132:11210 src_bucket [2021-02-22 17:28:50,584] - [data_helper] [139837071939328] - INFO - creating direct client 172.23.123.42:11210 src_bucket [2021-02-22 17:28:50,691] - [data_helper] [139837071939328] - INFO - creating direct client 172.23.123.44:11210 src_bucket [2021-02-22 17:28:50,956] - [data_helper] [139837071939328] - INFO - creating direct client 172.23.122.132:11210 src_bucket [2021-02-22 17:28:51,070] - [data_helper] [139837071939328] - INFO - creating direct client 172.23.123.42:11210 src_bucket [2021-02-22 17:28:51,177] - [data_helper] [139837071939328] - INFO - creating direct client 172.23.123.44:11210 src_bucket [2021-02-22 17:28:51,381] - [task] [139837071939328] - WARNING - vbucket map not ready after try 1 [2021-02-22 17:28:51,382] - [bucket_helper] [139837071939328] - INFO - waiting for memcached bucket : src_bucket in 172.23.123.42 to accept set ops [2021-02-22 17:28:51,533] - [data_helper] [139837071939328] - INFO - creating direct client 172.23.122.132:11210 src_bucket [2021-02-22 17:28:51,638] - [data_helper] [139837071939328] - INFO - creating direct client 172.23.123.42:11210 src_bucket [2021-02-22 17:28:51,851] - [data_helper] [139837071939328] - INFO - creating direct client 172.23.123.44:11210 src_bucket [2021-02-22 17:28:52,000] - [data_helper] [139837071939328] - INFO - creating direct client 172.23.122.132:11210 src_bucket [2021-02-22 17:28:52,224] - [data_helper] [139837071939328] - INFO - creating direct client 172.23.123.42:11210 src_bucket [2021-02-22 17:28:52,325] - [data_helper] [139837071939328] - INFO - creating direct client 172.23.123.44:11210 src_bucket [2021-02-22 17:28:52,415] - [task] [139837071939328] - WARNING - vbucket map not ready after try 2 [2021-02-22 17:28:52,416] - [bucket_helper] [139837071939328] - INFO - waiting for memcached bucket : src_bucket in 172.23.123.42 to accept set ops [2021-02-22 17:28:52,675] - [data_helper] [139837071939328] - INFO - creating direct client 172.23.122.132:11210 src_bucket [2021-02-22 17:28:52,783] - [data_helper] [139837071939328] - INFO - creating direct client 172.23.123.42:11210 src_bucket [2021-02-22 17:28:53,003] - [data_helper] [139837071939328] - INFO - creating direct client 172.23.123.44:11210 src_bucket [2021-02-22 17:28:53,155] - [data_helper] [139837071939328] - INFO - creating direct client 172.23.122.132:11210 src_bucket [2021-02-22 17:28:53,387] - [data_helper] [139837071939328] - INFO - creating direct client 172.23.123.42:11210 src_bucket [2021-02-22 17:28:53,491] - [data_helper] [139837071939328] - INFO - creating direct client 172.23.123.44:11210 src_bucket [2021-02-22 17:28:53,582] - [task] [139837071939328] - WARNING - vbucket map not ready after try 3 [2021-02-22 17:28:53,582] - [bucket_helper] [139837071939328] - INFO - waiting for memcached bucket : src_bucket in 172.23.123.42 to accept set ops [2021-02-22 17:28:53,845] - [data_helper] [139837071939328] - INFO - creating direct client 172.23.122.132:11210 src_bucket [2021-02-22 17:28:53,950] - [data_helper] [139837071939328] - INFO - creating direct client 172.23.123.42:11210 src_bucket [2021-02-22 17:28:54,050] - [data_helper] [139837071939328] - INFO - creating direct client 172.23.123.44:11210 src_bucket [2021-02-22 17:28:54,320] - [data_helper] [139837071939328] - INFO - creating direct client 172.23.122.132:11210 src_bucket [2021-02-22 17:28:54,433] - [data_helper] [139837071939328] - INFO - creating direct client 172.23.123.42:11210 src_bucket [2021-02-22 17:28:54,646] - [data_helper] [139837071939328] - INFO - creating direct client 172.23.123.44:11210 src_bucket [2021-02-22 17:28:54,739] - [task] [139837071939328] - WARNING - vbucket map not ready after try 4 [2021-02-22 17:28:54,739] - [bucket_helper] [139837071939328] - INFO - waiting for memcached bucket : src_bucket in 172.23.123.42 to accept set ops [2021-02-22 17:28:54,958] - [data_helper] [139837071939328] - INFO - creating direct client 172.23.122.132:11210 src_bucket [2021-02-22 17:28:55,030] - [data_helper] [139837071939328] - INFO - creating direct client 172.23.123.42:11210 src_bucket [2021-02-22 17:28:55,103] - [data_helper] [139837071939328] - INFO - creating direct client 172.23.123.44:11210 src_bucket [2021-02-22 17:28:55,320] - [data_helper] [139837071939328] - INFO - creating direct client 172.23.122.132:11210 src_bucket [2021-02-22 17:28:55,395] - [data_helper] [139837071939328] - INFO - creating direct client 172.23.123.42:11210 src_bucket [2021-02-22 17:28:55,473] - [data_helper] [139837071939328] - INFO - creating direct client 172.23.123.44:11210 src_bucket [2021-02-22 17:28:55,644] - [task] [139837071939328] - WARNING - vbucket map not ready after try 5 [2021-02-22 17:28:56,675] - [rest_client] [139837071939328] - INFO - http://172.23.123.42:8091/pools/default/buckets with param: name=dst_bucket&ramQuotaMB=100&authType=none&replicaNumber=0&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore [2021-02-22 17:28:56,706] - [rest_client] [139837071939328] - INFO - 0.03 seconds to create bucket dst_bucket [2021-02-22 17:28:56,706] - [bucket_helper] [139837071939328] - INFO - waiting for memcached bucket : dst_bucket in 172.23.123.42 to accept set ops [2021-02-22 17:28:57,512] - [data_helper] [139837071939328] - INFO - creating direct client 172.23.122.132:11210 dst_bucket [2021-02-22 17:28:57,639] - [data_helper] [139837071939328] - INFO - creating direct client 172.23.123.42:11210 dst_bucket [2021-02-22 17:28:57,745] - [data_helper] [139837071939328] - INFO - creating direct client 172.23.123.44:11210 dst_bucket [2021-02-22 17:28:58,031] - [data_helper] [139837071939328] - INFO - creating direct client 172.23.122.132:11210 dst_bucket [2021-02-22 17:28:58,135] - [data_helper] [139837071939328] - INFO - creating direct client 172.23.123.42:11210 dst_bucket [2021-02-22 17:28:58,249] - [data_helper] [139837071939328] - INFO - creating direct client 172.23.123.44:11210 dst_bucket [2021-02-22 17:28:58,456] - [task] [139837071939328] - WARNING - vbucket map not ready after try 0 [2021-02-22 17:28:58,456] - [bucket_helper] [139837071939328] - INFO - waiting for memcached bucket : dst_bucket in 172.23.123.42 to accept set ops [2021-02-22 17:28:58,610] - [data_helper] [139837071939328] - INFO - creating direct client 172.23.122.132:11210 dst_bucket [2021-02-22 17:28:58,843] - [data_helper] [139837071939328] - INFO - creating direct client 172.23.123.42:11210 dst_bucket [2021-02-22 17:28:58,950] - [data_helper] [139837071939328] - INFO - creating direct client 172.23.123.44:11210 dst_bucket [2021-02-22 17:28:59,226] - [data_helper] [139837071939328] - INFO - creating direct client 172.23.122.132:11210 dst_bucket [2021-02-22 17:28:59,328] - [data_helper] [139837071939328] - INFO - creating direct client 172.23.123.42:11210 dst_bucket [2021-02-22 17:28:59,431] - [data_helper] [139837071939328] - INFO - creating direct client 172.23.123.44:11210 dst_bucket [2021-02-22 17:28:59,639] - [task] [139837071939328] - WARNING - vbucket map not ready after try 1 [2021-02-22 17:28:59,640] - [bucket_helper] [139837071939328] - INFO - waiting for memcached bucket : dst_bucket in 172.23.123.42 to accept set ops [2021-02-22 17:28:59,802] - [data_helper] [139837071939328] - INFO - creating direct client 172.23.122.132:11210 dst_bucket [2021-02-22 17:29:00,027] - [data_helper] [139837071939328] - INFO - creating direct client 172.23.123.42:11210 dst_bucket [2021-02-22 17:29:00,132] - [data_helper] [139837071939328] - INFO - creating direct client 172.23.123.44:11210 dst_bucket [2021-02-22 17:29:00,294] - [data_helper] [139837071939328] - INFO - creating direct client 172.23.122.132:11210 dst_bucket [2021-02-22 17:29:00,516] - [data_helper] [139837071939328] - INFO - creating direct client 172.23.123.42:11210 dst_bucket [2021-02-22 17:29:00,625] - [data_helper] [139837071939328] - INFO - creating direct client 172.23.123.44:11210 dst_bucket [2021-02-22 17:29:00,721] - [task] [139837071939328] - WARNING - vbucket map not ready after try 2 [2021-02-22 17:29:00,721] - [bucket_helper] [139837071939328] - INFO - waiting for memcached bucket : dst_bucket in 172.23.123.42 to accept set ops [2021-02-22 17:29:00,995] - [data_helper] [139837071939328] - INFO - creating direct client 172.23.122.132:11210 dst_bucket [2021-02-22 17:29:01,115] - [data_helper] [139837071939328] - INFO - creating direct client 172.23.123.42:11210 dst_bucket [2021-02-22 17:29:01,340] - [data_helper] [139837071939328] - INFO - creating direct client 172.23.123.44:11210 dst_bucket [2021-02-22 17:29:01,509] - [data_helper] [139837071939328] - INFO - creating direct client 172.23.122.132:11210 dst_bucket [2021-02-22 17:29:01,742] - [data_helper] [139837071939328] - INFO - creating direct client 172.23.123.42:11210 dst_bucket [2021-02-22 17:29:01,861] - [data_helper] [139837071939328] - INFO - creating direct client 172.23.123.44:11210 dst_bucket [2021-02-22 17:29:01,964] - [task] [139837071939328] - WARNING - vbucket map not ready after try 3 [2021-02-22 17:29:01,965] - [bucket_helper] [139837071939328] - INFO - waiting for memcached bucket : dst_bucket in 172.23.123.42 to accept set ops [2021-02-22 17:29:02,240] - [data_helper] [139837071939328] - INFO - creating direct client 172.23.122.132:11210 dst_bucket [2021-02-22 17:29:02,360] - [data_helper] [139837071939328] - INFO - creating direct client 172.23.123.42:11210 dst_bucket [2021-02-22 17:29:02,592] - [data_helper] [139837071939328] - INFO - creating direct client 172.23.123.44:11210 dst_bucket [2021-02-22 17:29:02,698] - [data_helper] [139837071939328] - INFO - creating direct client 172.23.122.132:11210 dst_bucket [2021-02-22 17:29:02,776] - [data_helper] [139837071939328] - INFO - creating direct client 172.23.123.42:11210 dst_bucket [2021-02-22 17:29:02,974] - [data_helper] [139837071939328] - INFO - creating direct client 172.23.123.44:11210 dst_bucket [2021-02-22 17:29:03,032] - [task] [139837071939328] - WARNING - vbucket map not ready after try 4 [2021-02-22 17:29:03,032] - [bucket_helper] [139837071939328] - INFO - waiting for memcached bucket : dst_bucket in 172.23.123.42 to accept set ops [2021-02-22 17:29:03,249] - [data_helper] [139837071939328] - INFO - creating direct client 172.23.122.132:11210 dst_bucket [2021-02-22 17:29:03,323] - [data_helper] [139837071939328] - INFO - creating direct client 172.23.123.42:11210 dst_bucket [2021-02-22 17:29:03,395] - [data_helper] [139837071939328] - INFO - creating direct client 172.23.123.44:11210 dst_bucket [2021-02-22 17:29:03,621] - [data_helper] [139837071939328] - INFO - creating direct client 172.23.122.132:11210 dst_bucket [2021-02-22 17:29:03,696] - [data_helper] [139837071939328] - INFO - creating direct client 172.23.123.42:11210 dst_bucket [2021-02-22 17:29:03,906] - [data_helper] [139837071939328] - INFO - creating direct client 172.23.123.44:11210 dst_bucket [2021-02-22 17:29:03,968] - [task] [139837071939328] - WARNING - vbucket map not ready after try 5 [2021-02-22 17:29:05,001] - [rest_client] [139837071939328] - INFO - http://172.23.123.42:8091/pools/default/buckets with param: name=metadata&ramQuotaMB=400&authType=none&replicaNumber=0&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore [2021-02-22 17:29:05,031] - [rest_client] [139837071939328] - INFO - 0.03 seconds to create bucket metadata [2021-02-22 17:29:05,032] - [bucket_helper] [139837071939328] - INFO - waiting for memcached bucket : metadata in 172.23.123.42 to accept set ops [2021-02-22 17:29:05,716] - [data_helper] [139837071939328] - INFO - creating direct client 172.23.122.132:11210 metadata [2021-02-22 17:29:05,949] - [data_helper] [139837071939328] - INFO - creating direct client 172.23.123.42:11210 metadata [2021-02-22 17:29:06,055] - [data_helper] [139837071939328] - INFO - creating direct client 172.23.123.44:11210 metadata [2021-02-22 17:29:06,344] - [data_helper] [139837071939328] - INFO - creating direct client 172.23.122.132:11210 metadata [2021-02-22 17:29:06,452] - [data_helper] [139837071939328] - INFO - creating direct client 172.23.123.42:11210 metadata [2021-02-22 17:29:06,558] - [data_helper] [139837071939328] - INFO - creating direct client 172.23.123.44:11210 metadata [2021-02-22 17:29:06,780] - [task] [139837071939328] - WARNING - vbucket map not ready after try 0 [2021-02-22 17:29:06,781] - [bucket_helper] [139837071939328] - INFO - waiting for memcached bucket : metadata in 172.23.123.42 to accept set ops [2021-02-22 17:29:06,943] - [data_helper] [139837071939328] - INFO - creating direct client 172.23.122.132:11210 metadata [2021-02-22 17:29:07,167] - [data_helper] [139837071939328] - INFO - creating direct client 172.23.123.42:11210 metadata [2021-02-22 17:29:07,277] - [data_helper] [139837071939328] - INFO - creating direct client 172.23.123.44:11210 metadata [2021-02-22 17:29:07,552] - [data_helper] [139837071939328] - INFO - creating direct client 172.23.122.132:11210 metadata [2021-02-22 17:29:07,663] - [data_helper] [139837071939328] - INFO - creating direct client 172.23.123.42:11210 metadata [2021-02-22 17:29:07,773] - [data_helper] [139837071939328] - INFO - creating direct client 172.23.123.44:11210 metadata [2021-02-22 17:29:07,988] - [task] [139837071939328] - WARNING - vbucket map not ready after try 1 [2021-02-22 17:29:07,988] - [bucket_helper] [139837071939328] - INFO - waiting for memcached bucket : metadata in 172.23.123.42 to accept set ops [2021-02-22 17:29:08,141] - [data_helper] [139837071939328] - INFO - creating direct client 172.23.122.132:11210 metadata [2021-02-22 17:29:08,241] - [data_helper] [139837071939328] - INFO - creating direct client 172.23.123.42:11210 metadata [2021-02-22 17:29:08,464] - [data_helper] [139837071939328] - INFO - creating direct client 172.23.123.44:11210 metadata [2021-02-22 17:29:08,631] - [data_helper] [139837071939328] - INFO - creating direct client 172.23.122.132:11210 metadata [2021-02-22 17:29:08,859] - [data_helper] [139837071939328] - INFO - creating direct client 172.23.123.42:11210 metadata [2021-02-22 17:29:08,961] - [data_helper] [139837071939328] - INFO - creating direct client 172.23.123.44:11210 metadata [2021-02-22 17:29:09,056] - [task] [139837071939328] - WARNING - vbucket map not ready after try 2 [2021-02-22 17:29:09,057] - [bucket_helper] [139837071939328] - INFO - waiting for memcached bucket : metadata in 172.23.123.42 to accept set ops [2021-02-22 17:29:09,324] - [data_helper] [139837071939328] - INFO - creating direct client 172.23.122.132:11210 metadata [2021-02-22 17:29:09,429] - [data_helper] [139837071939328] - INFO - creating direct client 172.23.123.42:11210 metadata [2021-02-22 17:29:09,668] - [data_helper] [139837071939328] - INFO - creating direct client 172.23.123.44:11210 metadata [2021-02-22 17:29:09,828] - [data_helper] [139837071939328] - INFO - creating direct client 172.23.122.132:11210 metadata [2021-02-22 17:29:10,056] - [data_helper] [139837071939328] - INFO - creating direct client 172.23.123.42:11210 metadata [2021-02-22 17:29:10,166] - [data_helper] [139837071939328] - INFO - creating direct client 172.23.123.44:11210 metadata [2021-02-22 17:29:10,265] - [task] [139837071939328] - WARNING - vbucket map not ready after try 3 [2021-02-22 17:29:10,266] - [bucket_helper] [139837071939328] - INFO - waiting for memcached bucket : metadata in 172.23.123.42 to accept set ops [2021-02-22 17:29:10,533] - [data_helper] [139837071939328] - INFO - creating direct client 172.23.122.132:11210 metadata [2021-02-22 17:29:10,642] - [data_helper] [139837071939328] - INFO - creating direct client 172.23.123.42:11210 metadata [2021-02-22 17:29:10,757] - [data_helper] [139837071939328] - INFO - creating direct client 172.23.123.44:11210 metadata [2021-02-22 17:29:11,029] - [data_helper] [139837071939328] - INFO - creating direct client 172.23.122.132:11210 metadata [2021-02-22 17:29:11,142] - [data_helper] [139837071939328] - INFO - creating direct client 172.23.123.42:11210 metadata [2021-02-22 17:29:11,371] - [data_helper] [139837071939328] - INFO - creating direct client 172.23.123.44:11210 metadata [2021-02-22 17:29:11,469] - [task] [139837071939328] - WARNING - vbucket map not ready after try 4 [2021-02-22 17:29:11,470] - [bucket_helper] [139837071939328] - INFO - waiting for memcached bucket : metadata in 172.23.123.42 to accept set ops [2021-02-22 17:29:11,733] - [data_helper] [139837071939328] - INFO - creating direct client 172.23.122.132:11210 metadata [2021-02-22 17:29:11,822] - [data_helper] [139837071939328] - INFO - creating direct client 172.23.123.42:11210 metadata [2021-02-22 17:29:11,892] - [data_helper] [139837071939328] - INFO - creating direct client 172.23.123.44:11210 metadata [2021-02-22 17:29:12,111] - [data_helper] [139837071939328] - INFO - creating direct client 172.23.122.132:11210 metadata [2021-02-22 17:29:12,190] - [data_helper] [139837071939328] - INFO - creating direct client 172.23.123.42:11210 metadata [2021-02-22 17:29:12,264] - [data_helper] [139837071939328] - INFO - creating direct client 172.23.123.44:11210 metadata [2021-02-22 17:29:12,437] - [task] [139837071939328] - WARNING - vbucket map not ready after try 5 [2021-02-22 17:29:12,729] - [basetestcase] [139837055153920] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.134 port:8091 ssh_username:root] [2021-02-22 17:29:12,736] - [tuq_helper] [139837055153920] - INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary' [2021-02-22 17:29:12,742] - [rest_client] [139837055153920] - INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27 [2021-02-22 17:29:13,138] - [tuq_helper] [139837055153920] - INFO - TOTAL ELAPSED TIME: 392.986726ms [2021-02-22 17:29:13,144] - [tuq_helper] [139837055153920] - INFO - RUN QUERY CREATE PRIMARY INDEX ON dst_bucket USING GSI [2021-02-22 17:29:13,149] - [rest_client] [139837055153920] - INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+dst_bucket++USING+GSI [2021-02-22 17:29:14,936] - [tuq_helper] [139837055153920] - INFO - TOTAL ELAPSED TIME: 1.784296571s [2021-02-22 17:29:14,944] - [tuq_helper] [139837055153920] - INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary' [2021-02-22 17:29:14,950] - [rest_client] [139837055153920] - INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27 [2021-02-22 17:29:14,973] - [tuq_helper] [139837055153920] - INFO - TOTAL ELAPSED TIME: 20.415703ms [2021-02-22 17:29:14,979] - [tuq_helper] [139837055153920] - INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary' [2021-02-22 17:29:14,989] - [rest_client] [139837055153920] - INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27 [2021-02-22 17:29:14,993] - [tuq_helper] [139837055153920] - INFO - TOTAL ELAPSED TIME: 1.054174ms [2021-02-22 17:29:14,998] - [tuq_helper] [139837055153920] - INFO - RUN QUERY CREATE PRIMARY INDEX ON metadata USING GSI [2021-02-22 17:29:15,004] - [rest_client] [139837055153920] - INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+metadata++USING+GSI [2021-02-22 17:29:16,920] - [tuq_helper] [139837055153920] - INFO - TOTAL ELAPSED TIME: 1.912821805s [2021-02-22 17:29:16,926] - [tuq_helper] [139837055153920] - INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary' [2021-02-22 17:29:16,937] - [rest_client] [139837055153920] - INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27 [2021-02-22 17:29:16,959] - [tuq_helper] [139837055153920] - INFO - TOTAL ELAPSED TIME: 18.669546ms [2021-02-22 17:29:16,965] - [tuq_helper] [139837055153920] - INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary' [2021-02-22 17:29:16,970] - [rest_client] [139837055153920] - INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27 [2021-02-22 17:29:16,975] - [tuq_helper] [139837055153920] - INFO - TOTAL ELAPSED TIME: 1.431394ms [2021-02-22 17:29:16,981] - [tuq_helper] [139837055153920] - INFO - RUN QUERY CREATE PRIMARY INDEX ON src_bucket USING GSI [2021-02-22 17:29:16,987] - [rest_client] [139837055153920] - INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+src_bucket++USING+GSI [2021-02-22 17:29:18,819] - [tuq_helper] [139837055153920] - INFO - TOTAL ELAPSED TIME: 1.828949128s [2021-02-22 17:29:18,825] - [tuq_helper] [139837055153920] - INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary' [2021-02-22 17:29:18,832] - [rest_client] [139837055153920] - INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27 [2021-02-22 17:29:18,852] - [tuq_helper] [139837055153920] - INFO - TOTAL ELAPSED TIME: 17.319015ms [2021-02-22 17:29:20,727] - [eventing_base] [139837055153920] - INFO - saving function b'{\n "code": 0,\n "info": {\n "status": "Stored function: \'Function_431426416_test_eventing_rebalance_with_multiple_kv_nodes\' in metakv",\n "warnings": null\n }\n}' [2021-02-22 17:29:20,727] - [eventing_base] [139837055153920] - INFO - Deploying the following handler code : Function_431426416_test_eventing_rebalance_with_multiple_kv_nodes with bindings: {'buckets': [{'alias': 'dst_bucket', 'bucket_name': 'dst_bucket', 'access': 'rw'}], 'metadata_bucket': 'metadata', 'source_bucket': 'src_bucket', 'curl': []} and settings: {'checkpoint_interval': 20000, 'cleanup_timers': False, 'dcp_stream_boundary': 'everything', 'deployment_status': False, 'description': 'Sample Description', 'log_level': 'INFO', 'skip_timer_threshold': 86400, 'sock_batch_size': 1, 'tick_duration': 5000, 'timer_processing_tick_interval': 500, 'timer_worker_pool_size': 3, 'worker_count': 3, 'processing_status': False, 'cpp_worker_thread_count': 1, 'execution_timeout': 20, 'data_chan_size': 10000, 'worker_queue_cap': 100000, 'use_memory_manager': True, 'deadline_timeout': 21, 'timer_storage_chan_size': 10000, 'dcp_gen_chan_size': 10000, 'language_compatibility': '6.6.2'} [2021-02-22 17:29:20,727] - [eventing_base] [139837055153920] - INFO - function OnUpdate(doc, meta) { var expiry = new Date(); expiry.setSeconds(expiry.getSeconds() + 30); var context = {docID : meta.id, random_text : "e6cZZGHuh0R7Aumoe6cZZGHuh0R7Aumoe6cZZGHuh0R7Aumoe6cZZGHuh0R7Aumoe6cZZGHuh0R7Aumoe6cZZGHuh0R7Aumoe6cZZGHuh0R7Aumoe6cZZGHuh0R7Aumoe6cZZGHuh0R7Aumoe6cZZGHuh0R7Aumoe6cZZGHuh0R7Aumoe6cZZGHuh0R7Aumoe6cZZGHuh0R7Aumoe6cZZGHuh07Aumoe6cZZGHuh07Aumoe6cZZGHuh07Aumoe6"}; createTimer(timerCallback, expiry, meta.id, context); } function OnDelete(meta) { var expiry = new Date(); expiry.setSeconds(expiry.getSeconds() + 30); var context = {docID : meta.id}; createTimer(NDtimerCallback, expiry, meta.id, context); } function NDtimerCallback(context) { delete dst_bucket[context.docID]; } function timerCallback(context) { dst_bucket[context.docID] = context.random_text; } [2021-02-22 17:29:21,932] - [eventing_base] [139837055153920] - INFO - deploy Application : b'' [2021-02-22 17:29:21,932] - [basetestcase] [139837055153920] - INFO - sleep for 5 secs. Waiting for Function_431426416_test_eventing_rebalance_with_multiple_kv_nodes to deployed... ... [2021-02-22 17:29:26,944] - [basetestcase] [139837055153920] - INFO - sleep for 5 secs. Waiting for Function_431426416_test_eventing_rebalance_with_multiple_kv_nodes to deployed... ... [2021-02-22 17:29:31,963] - [basetestcase] [139837055153920] - INFO - sleep for 5 secs. Waiting for Function_431426416_test_eventing_rebalance_with_multiple_kv_nodes to deployed... ... [2021-02-22 17:29:36,982] - [basetestcase] [139837055153920] - INFO - sleep for 5 secs. Waiting for Function_431426416_test_eventing_rebalance_with_multiple_kv_nodes to deployed... ... [2021-02-22 17:29:42,215] - [data_helper] [139837055153920] - INFO - creating direct client 172.23.122.132:11210 src_bucket [2021-02-22 17:29:42,303] - [data_helper] [139837055153920] - INFO - creating direct client 172.23.123.42:11210 src_bucket [2021-02-22 17:29:42,383] - [data_helper] [139837055153920] - INFO - creating direct client 172.23.123.44:11210 src_bucket [2021-02-22 17:29:43,481] - [task] [139837071939328] - INFO - adding node 172.23.122.131:8091 to cluster [2021-02-22 17:29:43,482] - [rest_client] [139837071939328] - INFO - adding remote node @172.23.122.131:8091 to this cluster @172.23.123.42:8091 [2021-02-22 17:29:45,512] - [rest_client] [139837055153920] - INFO - rebalance reached >100% in 3.0052173137664795 seconds [2021-02-22 17:29:53,507] - [rest_client] [139837071939328] - INFO - rebalance progress took 10.02 seconds [2021-02-22 17:29:53,509] - [rest_client] [139837071939328] - INFO - sleep for 10 seconds after rebalance... [2021-02-22 17:30:05,965] - [task] [139837071939328] - INFO - adding node 172.23.122.180:8091 to cluster [2021-02-22 17:30:05,966] - [rest_client] [139837071939328] - INFO - adding remote node @172.23.122.180:8091 to this cluster @172.23.123.42:8091 [2021-02-22 17:30:15,986] - [rest_client] [139837071939328] - INFO - rebalance progress took 10.02 seconds [2021-02-22 17:30:15,986] - [rest_client] [139837071939328] - INFO - sleep for 10 seconds after rebalance... [2021-02-22 17:30:28,706] - [rest_client] [139837071939328] - INFO - rebalance params : {'knownNodes': 'ns_1@172.23.122.131,ns_1@172.23.122.132,ns_1@172.23.122.133,ns_1@172.23.122.134,ns_1@172.23.122.180,ns_1@172.23.123.42,ns_1@172.23.123.44', 'ejectedNodes': '', 'user': 'Administrator', 'password': 'password'} [2021-02-22 17:30:28,737] - [rest_client] [139837071939328] - INFO - rebalance operation started [2021-02-22 17:30:28,741] - [rest_client] [139837071939328] - INFO - rebalance percentage : 0.00 % [2021-02-22 17:30:28,741] - [task] [139837071939328] - INFO - Rebalance - status: running, progress: 0.00% [2021-02-22 17:30:38,770] - [rest_client] [139837071939328] - INFO - rebalance percentage : 44.00 % [2021-02-22 17:30:38,771] - [task] [139837071939328] - INFO - Rebalance - status: running, progress: 44.00% [2021-02-22 17:30:48,801] - [rest_client] [139837071939328] - INFO - rebalance percentage : 78.00 % [2021-02-22 17:30:48,802] - [task] [139837071939328] - INFO - Rebalance - status: running, progress: 78.00% [2021-02-22 17:30:58,831] - [rest_client] [139837071939328] - INFO - rebalance percentage : 78.00 % [2021-02-22 17:30:58,831] - [task] [139837071939328] - INFO - Rebalance - status: running, progress: 78.00% [2021-02-22 17:31:08,864] - [task] [139837071939328] - INFO - Rebalance - status: none, progress: 100.00% [2021-02-22 17:31:08,876] - [task] [139837071939328] - INFO - rebalancing was completed with progress: 100% in 40.138943672180176 sec [2021-02-22 17:31:08,895] - [basetestcase] [139837055153920] - INFO - list of eventing nodes in cluster: [ip:172.23.122.133 port:8091 ssh_username:root] [2021-02-22 17:31:08,914] - [basetestcase] [139837055153920] - INFO - list of eventing nodes in cluster: [ip:172.23.122.133 port:8091 ssh_username:root] [2021-02-22 17:31:10,092] - [eventing_base] [139837055153920] - INFO - Documents in source bucket : 10080 [2021-02-22 17:31:10,092] - [eventing_base] [139837055153920] - INFO - Final docs count... Current : 10080 Expected : 10080 [2021-02-22 17:31:11,343] - [eventing_base] [139837055153920] - INFO - Stats for Node 172.23.122.133 is [ { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 9, "agg_messages_sent_to_worker": 13900, "agg_queue_memory": 0, "agg_queue_memory_cap": 106954752, "agg_queue_size": 0, "agg_queue_size_cap": 300000, "agg_timer_feedback_queue_cap": 1500, "dcp_mutation": 10080, "dcp_mutation_sent_to_worker": 10080, "dcp_snapshot": 10080, "dcp_stream_req_counter": 4494, "dcp_stream_req_err_counter": 2696, "dcp_streamend": 702, "dcp_streamreq": 1798, "execution_stats": 291, "failure_stats": 291, "latency_stats": 291, "lcb_exception_stats": 291, "log_level": 3, "num_processed_events": 10080, "processed_events_size": 7254832, "thr_count": 3, "thr_map": 3, "timer_events": 10080, "timer_responses_received": 10080, "v8_init": 3, "v8_load": 3 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 10080, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 10080, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 13893, "no_op_counter": 0, "num_processed_events": 10080, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 10080, "processed_events_size": 7254832, "timer_callback_failure": 0, "timer_callback_success": 10080, "timer_cancel_counter": 0, "timer_create_counter": 10080, "timer_create_failure": 0, "timer_msg_counter": 10080, "timer_responses_sent": 0, "timestamp": { "43365": "2021-02-23T01:31:10Z", "43373": "2021-02-23T01:31:09Z", "43382": "2021-02-23T01:31:09Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "43365": "2021-02-23T01:31:10Z", "43373": "2021-02-23T01:31:09Z", "43382": "2021-02-23T01:31:09Z" }, "v8worker_events_lost": 0 }, "function_id": 2575840121, "function_name": "Function_431426416_test_eventing_rebalance_with_multiple_kv_nodes", "gocb_creds_request_counter": 33, "internal_vb_distribution_stats": { "worker_Function_431426416_test_eventing_rebalance_with_multiple_kv_nodes_0": "[0-341]", "worker_Function_431426416_test_eventing_rebalance_with_multiple_kv_nodes_1": "[342-682]", "worker_Function_431426416_test_eventing_rebalance_with_multiple_kv_nodes_2": "[683-1023]" }, "latency_percentile_stats": { "100": 9400, "50": 1000, "80": 1200, "90": 1300, "95": 1500, "99": 2200 }, "lcb_creds_request_counter": 66, "lcb_exception_stats": {}, "metastore_stats": { "metastore_delete_err": 0, "metastore_deletes": 0, "metastore_not_found": 0, "metastore_scan": 0, "metastore_scan_due": 0, "metastore_scan_err": 0, "metastore_set": 0, "metastore_set_err": 0 }, "planner_stats": [ { "host_name": "172.23.122.133:8096", "start_vb": 0, "vb_count": 1024 } ], "vb_distribution_stats_from_metadata": { "172.23.122.133:8096": { "worker_Function_431426416_test_eventing_rebalance_with_multiple_kv_nodes_0": "[0-341]", "worker_Function_431426416_test_eventing_rebalance_with_multiple_kv_nodes_1": "[342-682]", "worker_Function_431426416_test_eventing_rebalance_with_multiple_kv_nodes_2": "[683-1023]" } }, "worker_pids": { "worker_Function_431426416_test_eventing_rebalance_with_multiple_kv_nodes_0": 43365, "worker_Function_431426416_test_eventing_rebalance_with_multiple_kv_nodes_1": 43373, "worker_Function_431426416_test_eventing_rebalance_with_multiple_kv_nodes_2": 43382 } } ] [2021-02-22 17:31:11,831] - [data_helper] [139837055153920] - INFO - creating direct client 172.23.122.132:11210 src_bucket [2021-02-22 17:31:11,925] - [data_helper] [139837055153920] - INFO - creating direct client 172.23.122.131:11210 src_bucket [2021-02-22 17:31:12,023] - [data_helper] [139837055153920] - INFO - creating direct client 172.23.123.42:11210 src_bucket [2021-02-22 17:31:12,259] - [data_helper] [139837055153920] - INFO - creating direct client 172.23.122.180:11210 src_bucket [2021-02-22 17:31:12,352] - [data_helper] [139837055153920] - INFO - creating direct client 172.23.123.44:11210 src_bucket [2021-02-22 17:31:12,907] - [rest_client] [139837071939328] - INFO - rebalance params : {'knownNodes': 'ns_1@172.23.122.131,ns_1@172.23.122.132,ns_1@172.23.122.133,ns_1@172.23.122.134,ns_1@172.23.122.180,ns_1@172.23.123.42,ns_1@172.23.123.44', 'ejectedNodes': 'ns_1@172.23.122.131,ns_1@172.23.122.180', 'user': 'Administrator', 'password': 'password'} [2021-02-22 17:31:12,940] - [rest_client] [139837071939328] - INFO - rebalance operation started [2021-02-22 17:31:12,945] - [rest_client] [139837071939328] - INFO - rebalance percentage : 0.00 % [2021-02-22 17:31:12,946] - [task] [139837071939328] - INFO - Rebalance - status: running, progress: 0.00% [2021-02-22 17:31:15,496] - [rest_client] [139837055153920] - INFO - rebalance reached >100% in 3.006530284881592 seconds [2021-02-22 17:31:22,977] - [rest_client] [139837071939328] - INFO - rebalance percentage : 23.00 % [2021-02-22 17:31:22,978] - [task] [139837071939328] - INFO - Rebalance - status: running, progress: 23.00% [2021-02-22 17:31:33,007] - [rest_client] [139837071939328] - INFO - rebalance percentage : 47.00 % [2021-02-22 17:31:33,007] - [task] [139837071939328] - INFO - Rebalance - status: running, progress: 47.00% [2021-02-22 17:31:43,036] - [rest_client] [139837071939328] - INFO - rebalance percentage : 55.00 % [2021-02-22 17:31:43,036] - [task] [139837071939328] - INFO - Rebalance - status: running, progress: 55.00% [2021-02-22 17:31:53,073] - [rest_client] [139837071939328] - INFO - rebalance percentage : 71.00 % [2021-02-22 17:31:53,074] - [task] [139837071939328] - INFO - Rebalance - status: running, progress: 71.00% [2021-02-22 17:32:03,103] - [rest_client] [139837071939328] - INFO - rebalance percentage : 78.00 % [2021-02-22 17:32:03,103] - [task] [139837071939328] - INFO - Rebalance - status: running, progress: 78.00% [2021-02-22 17:32:13,133] - [task] [139837071939328] - INFO - Rebalance - status: none, progress: 100.00% [2021-02-22 17:32:13,146] - [rest_client] [139837071939328] - ERROR - GET http://172.23.122.131:8091/nodes/self body: headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"Node is unknown to this cluster."' auth: Administrator:password [2021-02-22 17:32:13,147] - [rest_client] [139837071939328] - ERROR - Error Node is unknown to this cluster. was gotten, 5 seconds sleep before retry [2021-02-22 17:32:18,156] - [rest_client] [139837071939328] - ERROR - GET http://172.23.122.131:8091/nodes/self body: headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"Node is unknown to this cluster."' auth: Administrator:password [2021-02-22 17:32:18,157] - [rest_client] [139837071939328] - ERROR - Error Node is unknown to this cluster. was gotten, 5 seconds sleep before retry [2021-02-22 17:32:23,166] - [rest_client] [139837071939328] - ERROR - GET http://172.23.122.131:8091/nodes/self body: headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"Node is unknown to this cluster."' auth: Administrator:password [2021-02-22 17:32:23,167] - [rest_client] [139837071939328] - ERROR - Error Node is unknown to this cluster. was gotten, 5 seconds sleep before retry [2021-02-22 17:32:28,173] - [rest_client] [139837071939328] - ERROR - node 172.23.122.131:8091 is in a broken state! [2021-02-22 17:32:28,173] - [task] [139837071939328] - ERROR - unable to reach the host @ 172.23.122.131 [2021-02-22 17:32:28,185] - [task] [139837071939328] - ERROR - node 172.23.122.131:8091 was not cleaned after removing from cluster [2021-02-22 17:32:28,185] - [task] [139837071939328] - INFO - rebalancing was completed with progress: 100% in 75.24464440345764 sec [2021-02-22 17:32:28,210] - [basetestcase] [139837055153920] - INFO - list of eventing nodes in cluster: [ip:172.23.122.133 port:8091 ssh_username:root] [2021-02-22 17:32:28,227] - [basetestcase] [139837055153920] - INFO - list of eventing nodes in cluster: [ip:172.23.122.133 port:8091 ssh_username:root] [2021-02-22 17:32:28,830] - [basetestcase] [139837055153920] - INFO - sleep for 30 secs. Waiting for handler code Function_431426416_test_eventing_rebalance_with_multiple_kv_nodes to complete bucket operations... Current : 143 Expected : 0 ... [2021-02-22 17:33:00,723] - [eventing_base] [139837055153920] - INFO - Documents in source bucket : 0 [2021-02-22 17:33:00,723] - [eventing_base] [139837055153920] - INFO - Final docs count... Current : 0 Expected : 0 [2021-02-22 17:33:01,796] - [eventing_base] [139837055153920] - INFO - Stats for Node 172.23.122.133 is [ { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 16, "agg_messages_sent_to_worker": 26468, "agg_queue_memory": 0, "agg_queue_memory_cap": 106954752, "agg_queue_size": 0, "agg_queue_size_cap": 300000, "agg_timer_feedback_queue_cap": 1500, "dcp_deletion": 10080, "dcp_deletion_sent_to_worker": 10080, "dcp_mutation": 10080, "dcp_mutation_sent_to_worker": 10080, "dcp_snapshot": 20154, "dcp_stream_req_counter": 29307, "dcp_stream_req_err_counter": 26892, "dcp_streamend": 1112, "dcp_streamreq": 2415, "execution_stats": 624, "failure_stats": 624, "latency_stats": 624, "lcb_exception_stats": 624, "log_level": 3, "num_processed_events": 20160, "processed_events_size": 10559340, "thr_count": 3, "thr_map": 3, "timer_events": 20160, "timer_responses_received": 20160, "v8_init": 3, "v8_load": 3 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 10080, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 10080, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 10080, "enqueued_dcp_mutation_msg_counter": 10080, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 26456, "no_op_counter": 0, "num_processed_events": 20160, "on_delete_failure": 0, "on_delete_success": 10080, "on_update_failure": 0, "on_update_success": 10080, "processed_events_size": 10559340, "timer_callback_failure": 0, "timer_callback_success": 20160, "timer_cancel_counter": 0, "timer_create_counter": 20160, "timer_create_failure": 0, "timer_msg_counter": 20160, "timer_responses_sent": 0, "timestamp": { "43365": "2021-02-23T01:33:00Z", "43373": "2021-02-23T01:33:00Z", "43382": "2021-02-23T01:33:00Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "43365": "2021-02-23T01:33:00Z", "43373": "2021-02-23T01:33:00Z", "43382": "2021-02-23T01:33:00Z" }, "v8worker_events_lost": 0 }, "function_id": 2575840121, "function_name": "Function_431426416_test_eventing_rebalance_with_multiple_kv_nodes", "gocb_creds_request_counter": 50, "internal_vb_distribution_stats": { "worker_Function_431426416_test_eventing_rebalance_with_multiple_kv_nodes_0": "[0-341]", "worker_Function_431426416_test_eventing_rebalance_with_multiple_kv_nodes_1": "[342-682]", "worker_Function_431426416_test_eventing_rebalance_with_multiple_kv_nodes_2": "[683-1023]" }, "latency_percentile_stats": { "100": 17500, "50": 1100, "80": 1300, "90": 1700, "95": 2400, "99": 6200 }, "lcb_creds_request_counter": 74, "lcb_exception_stats": {}, "metastore_stats": { "metastore_delete_err": 0, "metastore_deletes": 0, "metastore_not_found": 0, "metastore_scan": 0, "metastore_scan_due": 0, "metastore_scan_err": 0, "metastore_set": 0, "metastore_set_err": 0 }, "planner_stats": [ { "host_name": "172.23.122.133:8096", "start_vb": 0, "vb_count": 1024 } ], "vb_distribution_stats_from_metadata": { "172.23.122.133:8096": { "worker_Function_431426416_test_eventing_rebalance_with_multiple_kv_nodes_0": "[0-341]", "worker_Function_431426416_test_eventing_rebalance_with_multiple_kv_nodes_1": "[342-682]", "worker_Function_431426416_test_eventing_rebalance_with_multiple_kv_nodes_2": "[683-1023]" } }, "worker_pids": { "worker_Function_431426416_test_eventing_rebalance_with_multiple_kv_nodes_0": 43365, "worker_Function_431426416_test_eventing_rebalance_with_multiple_kv_nodes_1": 43373, "worker_Function_431426416_test_eventing_rebalance_with_multiple_kv_nodes_2": 43382 } } ] [2021-02-22 17:33:02,291] - [basetestcase] [139837055153920] - INFO - list of kv nodes in cluster: [ip:172.23.122.132 port:8091 ssh_username:root, ip:172.23.123.42 port:8091 ssh_username:root, ip:172.23.123.44 port:8091 ssh_username:root] [2021-02-22 17:33:02,292] - [rest_client] [139837055153920] - INFO - adding remote node @172.23.122.131:8091 to this cluster @172.23.122.133:8091 [2021-02-22 17:33:12,308] - [rest_client] [139837055153920] - INFO - rebalance progress took 10.02 seconds [2021-02-22 17:33:12,309] - [rest_client] [139837055153920] - INFO - sleep for 10 seconds after rebalance... [2021-02-22 17:33:25,363] - [rest_client] [139837055153920] - INFO - adding remote node @172.23.122.180:8091 to this cluster @172.23.122.133:8091 [2021-02-22 17:33:35,380] - [rest_client] [139837055153920] - INFO - rebalance progress took 10.02 seconds [2021-02-22 17:33:35,381] - [rest_client] [139837055153920] - INFO - sleep for 10 seconds after rebalance... [2021-02-22 17:33:48,338] - [rest_client] [139837055153920] - INFO - Node 172.23.122.131 not part of cluster inactiveAdded [2021-02-22 17:33:48,338] - [rest_client] [139837055153920] - INFO - Node 172.23.122.180 not part of cluster inactiveAdded [2021-02-22 17:33:48,353] - [data_helper] [139837055153920] - INFO - creating direct client 172.23.122.132:11210 src_bucket [2021-02-22 17:33:48,428] - [rest_client] [139837055153920] - INFO - Node 172.23.122.131 not part of cluster inactiveAdded [2021-02-22 17:33:48,428] - [rest_client] [139837055153920] - INFO - Node 172.23.122.180 not part of cluster inactiveAdded [2021-02-22 17:33:48,436] - [rest_client] [139837055153920] - INFO - Node 172.23.122.131 not part of cluster inactiveAdded [2021-02-22 17:33:48,436] - [rest_client] [139837055153920] - INFO - Node 172.23.122.180 not part of cluster inactiveAdded [2021-02-22 17:33:48,449] - [data_helper] [139837055153920] - INFO - creating direct client 172.23.123.42:11210 src_bucket [2021-02-22 17:33:48,712] - [rest_client] [139837055153920] - INFO - Node 172.23.122.131 not part of cluster inactiveAdded [2021-02-22 17:33:48,712] - [rest_client] [139837055153920] - INFO - Node 172.23.122.180 not part of cluster inactiveAdded [2021-02-22 17:33:48,720] - [rest_client] [139837055153920] - INFO - Node 172.23.122.131 not part of cluster inactiveAdded [2021-02-22 17:33:48,720] - [rest_client] [139837055153920] - INFO - Node 172.23.122.180 not part of cluster inactiveAdded [2021-02-22 17:33:48,734] - [data_helper] [139837055153920] - INFO - creating direct client 172.23.123.44:11210 src_bucket [2021-02-22 17:33:48,801] - [rest_client] [139837055153920] - INFO - Node 172.23.122.131 not part of cluster inactiveAdded [2021-02-22 17:33:48,801] - [rest_client] [139837055153920] - INFO - Node 172.23.122.180 not part of cluster inactiveAdded [2021-02-22 17:33:49,300] - [rest_client] [139837071939328] - INFO - rebalance params : {'knownNodes': 'ns_1@172.23.122.131,ns_1@172.23.122.132,ns_1@172.23.122.133,ns_1@172.23.122.134,ns_1@172.23.122.180,ns_1@172.23.123.42,ns_1@172.23.123.44', 'ejectedNodes': 'ns_1@172.23.123.42,ns_1@172.23.123.44', 'user': 'Administrator', 'password': 'password'} [2021-02-22 17:33:49,334] - [rest_client] [139837071939328] - INFO - rebalance operation started [2021-02-22 17:33:49,342] - [rest_client] [139837071939328] - INFO - rebalance percentage : 0.00 % [2021-02-22 17:33:49,342] - [task] [139837071939328] - INFO - Rebalance - status: running, progress: 0.00% [2021-02-22 17:33:51,876] - [rest_client] [139837055153920] - INFO - rebalance reached >100% in 3.006530284881592 seconds [2021-02-22 17:33:59,373] - [rest_client] [139837071939328] - INFO - rebalance percentage : 19.00 % [2021-02-22 17:33:59,373] - [task] [139837071939328] - INFO - Rebalance - status: running, progress: 19.00% [2021-02-22 17:34:09,404] - [rest_client] [139837071939328] - INFO - rebalance percentage : 23.00 % [2021-02-22 17:34:09,404] - [task] [139837071939328] - INFO - Rebalance - status: running, progress: 23.00% [2021-02-22 17:34:19,439] - [rest_client] [139837071939328] - INFO - rebalance percentage : 42.00 % [2021-02-22 17:34:19,439] - [task] [139837071939328] - INFO - Rebalance - status: running, progress: 42.00% [2021-02-22 17:34:29,468] - [rest_client] [139837071939328] - INFO - rebalance percentage : 53.00 % [2021-02-22 17:34:29,469] - [task] [139837071939328] - INFO - Rebalance - status: running, progress: 53.00% [2021-02-22 17:34:39,498] - [rest_client] [139837071939328] - INFO - rebalance percentage : 66.00 % [2021-02-22 17:34:39,499] - [task] [139837071939328] - INFO - Rebalance - status: running, progress: 66.00% [2021-02-22 17:34:49,527] - [rest_client] [139837071939328] - INFO - rebalance percentage : 78.00 % [2021-02-22 17:34:49,528] - [task] [139837071939328] - INFO - Rebalance - status: running, progress: 78.00% [2021-02-22 17:34:59,795] - [rest_client] [139837071939328] - INFO - rebalance percentage : 92.00 % [2021-02-22 17:34:59,796] - [task] [139837071939328] - INFO - Rebalance - status: running, progress: 92.00% [2021-02-22 17:35:09,822] - [task] [139837071939328] - INFO - Rebalance - status: none, progress: 100.00% [2021-02-22 17:35:09,837] - [rest_client] [139837071939328] - ERROR - GET http://172.23.123.42:8091/nodes/self body: headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"Node is unknown to this cluster."' auth: Administrator:password [2021-02-22 17:35:09,838] - [rest_client] [139837071939328] - ERROR - Error Node is unknown to this cluster. was gotten, 5 seconds sleep before retry [2021-02-22 17:35:14,847] - [rest_client] [139837071939328] - ERROR - GET http://172.23.123.42:8091/nodes/self body: headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"Node is unknown to this cluster."' auth: Administrator:password [2021-02-22 17:35:14,848] - [rest_client] [139837071939328] - ERROR - Error Node is unknown to this cluster. was gotten, 5 seconds sleep before retry [2021-02-22 17:35:19,857] - [rest_client] [139837071939328] - ERROR - GET http://172.23.123.42:8091/nodes/self body: headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"Node is unknown to this cluster."' auth: Administrator:password [2021-02-22 17:35:19,858] - [rest_client] [139837071939328] - ERROR - Error Node is unknown to this cluster. was gotten, 5 seconds sleep before retry [2021-02-22 17:35:24,864] - [rest_client] [139837071939328] - ERROR - node 172.23.123.42:8091 is in a broken state! [2021-02-22 17:35:24,864] - [task] [139837071939328] - ERROR - unable to reach the host @ 172.23.123.42 [2021-02-22 17:35:24,875] - [task] [139837071939328] - ERROR - node 172.23.123.42:8091 was not cleaned after removing from cluster [2021-02-22 17:35:24,875] - [task] [139837071939328] - INFO - rebalancing was completed with progress: 100% in 95.54093885421753 sec [2021-02-22 17:35:24,905] - [basetestcase] [139837055153920] - INFO - list of eventing nodes in cluster: [ip:172.23.122.133 port:8091 ssh_username:root] [2021-02-22 17:35:24,924] - [basetestcase] [139837055153920] - INFO - list of eventing nodes in cluster: [ip:172.23.122.133 port:8091 ssh_username:root] [2021-02-22 17:35:25,597] - [basetestcase] [139837055153920] - INFO - sleep for 30 secs. Waiting for handler code Function_431426416_test_eventing_rebalance_with_multiple_kv_nodes to complete bucket operations... Current : 9789 Expected : 10080 ... [2021-02-22 17:35:56,879] - [basetestcase] [139837055153920] - INFO - sleep for 30 secs. Waiting for handler code Function_431426416_test_eventing_rebalance_with_multiple_kv_nodes to complete bucket operations... Current : 10079 Expected : 10080 ... [2021-02-22 17:36:28,124] - [basetestcase] [139837055153920] - INFO - sleep for 30 secs. Waiting for handler code Function_431426416_test_eventing_rebalance_with_multiple_kv_nodes to complete bucket operations... Current : 10079 Expected : 10080 ... [2021-02-22 17:36:59,322] - [basetestcase] [139837055153920] - INFO - sleep for 30 secs. Waiting for handler code Function_431426416_test_eventing_rebalance_with_multiple_kv_nodes to complete bucket operations... Current : 10079 Expected : 10080 ... [2021-02-22 17:37:30,510] - [basetestcase] [139837055153920] - INFO - sleep for 30 secs. Waiting for handler code Function_431426416_test_eventing_rebalance_with_multiple_kv_nodes to complete bucket operations... Current : 10079 Expected : 10080 ... [2021-02-22 17:38:01,718] - [basetestcase] [139837055153920] - INFO - sleep for 30 secs. Waiting for handler code Function_431426416_test_eventing_rebalance_with_multiple_kv_nodes to complete bucket operations... Current : 10079 Expected : 10080 ... [2021-02-22 17:38:32,965] - [basetestcase] [139837055153920] - INFO - sleep for 30 secs. Waiting for handler code Function_431426416_test_eventing_rebalance_with_multiple_kv_nodes to complete bucket operations... Current : 10079 Expected : 10080 ... [2021-02-22 17:39:04,147] - [basetestcase] [139837055153920] - INFO - sleep for 30 secs. Waiting for handler code Function_431426416_test_eventing_rebalance_with_multiple_kv_nodes to complete bucket operations... Current : 10079 Expected : 10080 ... [2021-02-22 17:39:35,385] - [basetestcase] [139837055153920] - INFO - sleep for 30 secs. Waiting for handler code Function_431426416_test_eventing_rebalance_with_multiple_kv_nodes to complete bucket operations... Current : 10079 Expected : 10080 ... [2021-02-22 17:40:06,611] - [basetestcase] [139837055153920] - INFO - sleep for 30 secs. Waiting for handler code Function_431426416_test_eventing_rebalance_with_multiple_kv_nodes to complete bucket operations... Current : 10079 Expected : 10080 ... [2021-02-22 17:40:37,861] - [basetestcase] [139837055153920] - INFO - sleep for 30 secs. Waiting for handler code Function_431426416_test_eventing_rebalance_with_multiple_kv_nodes to complete bucket operations... Current : 10079 Expected : 10080 ... [2021-02-22 17:41:09,156] - [basetestcase] [139837055153920] - INFO - sleep for 30 secs. Waiting for handler code Function_431426416_test_eventing_rebalance_with_multiple_kv_nodes to complete bucket operations... Current : 10079 Expected : 10080 ... [2021-02-22 17:41:39,759] - [eventing_base] [139837055153920] - INFO - Compacting bucket : dst_bucket [2021-02-22 17:41:39,768] - [rest_client] [139837055153920] - INFO - bucket compaction successful [2021-02-22 17:41:39,768] - [eventing_base] [139837055153920] - INFO - Compacting bucket : metadata [2021-02-22 17:41:39,773] - [rest_client] [139837055153920] - INFO - bucket compaction successful [2021-02-22 17:41:39,774] - [eventing_base] [139837055153920] - INFO - Compacting bucket : src_bucket [2021-02-22 17:41:39,779] - [rest_client] [139837055153920] - INFO - bucket compaction successful [2021-02-22 17:41:40,405] - [basetestcase] [139837055153920] - INFO - sleep for 30 secs. Waiting for handler code Function_431426416_test_eventing_rebalance_with_multiple_kv_nodes to complete bucket operations... Current : 10079 Expected : 10080 ... [2021-02-22 17:42:11,629] - [basetestcase] [139837055153920] - INFO - sleep for 30 secs. Waiting for handler code Function_431426416_test_eventing_rebalance_with_multiple_kv_nodes to complete bucket operations... Current : 10079 Expected : 10080 ... [2021-02-22 17:42:42,844] - [basetestcase] [139837055153920] - INFO - sleep for 30 secs. Waiting for handler code Function_431426416_test_eventing_rebalance_with_multiple_kv_nodes to complete bucket operations... Current : 10079 Expected : 10080 ... [2021-02-22 17:43:14,071] - [basetestcase] [139837055153920] - INFO - sleep for 30 secs. Waiting for handler code Function_431426416_test_eventing_rebalance_with_multiple_kv_nodes to complete bucket operations... Current : 10079 Expected : 10080 ... [2021-02-22 17:43:45,273] - [basetestcase] [139837055153920] - INFO - sleep for 30 secs. Waiting for handler code Function_431426416_test_eventing_rebalance_with_multiple_kv_nodes to complete bucket operations... Current : 10079 Expected : 10080 ... [2021-02-22 17:44:16,492] - [basetestcase] [139837055153920] - INFO - sleep for 30 secs. Waiting for handler code Function_431426416_test_eventing_rebalance_with_multiple_kv_nodes to complete bucket operations... Current : 10079 Expected : 10080 ... [2021-02-22 17:44:47,696] - [basetestcase] [139837055153920] - INFO - sleep for 30 secs. Waiting for handler code Function_431426416_test_eventing_rebalance_with_multiple_kv_nodes to complete bucket operations... Current : 10079 Expected : 10080 ... [2021-02-22 17:45:18,920] - [basetestcase] [139837055153920] - INFO - sleep for 30 secs. Waiting for handler code Function_431426416_test_eventing_rebalance_with_multiple_kv_nodes to complete bucket operations... Current : 10079 Expected : 10080 ... [2021-02-22 17:45:50,152] - [basetestcase] [139837055153920] - INFO - sleep for 30 secs. Waiting for handler code Function_431426416_test_eventing_rebalance_with_multiple_kv_nodes to complete bucket operations... Current : 10079 Expected : 10080 ... [2021-02-22 17:46:22,013] - [eventing_base] [139837055153920] - INFO - Documents in source bucket : 10080 [2021-02-22 17:46:23,041] - [eventing_base] [139837055153920] - INFO - Stats for Node 172.23.122.133 is [ { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 24, "agg_messages_sent_to_worker": 49930, "agg_queue_memory": 0, "agg_queue_memory_cap": 106954752, "agg_queue_size": 0, "agg_queue_size_cap": 300000, "agg_timer_feedback_queue_cap": 1500, "dcp_deletion": 10080, "dcp_deletion_sent_to_worker": 10080, "dcp_mutation": 20160, "dcp_mutation_sent_to_worker": 20160, "dcp_snapshot": 30233, "dcp_stream_req_counter": 66340, "dcp_stream_req_err_counter": 62940, "dcp_streamend": 1794, "dcp_streamreq": 3400, "execution_stats": 3027, "failure_stats": 3027, "latency_stats": 3027, "lcb_exception_stats": 3027, "log_level": 3, "num_processed_events": 30240, "processed_events_size": 17822664, "thr_count": 3, "thr_map": 3, "timer_events": 30240, "timer_responses_received": 30240, "v8_init": 3, "v8_load": 3 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 10080, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 20160, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 10080, "enqueued_dcp_mutation_msg_counter": 20160, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 49918, "no_op_counter": 0, "num_processed_events": 30240, "on_delete_failure": 0, "on_delete_success": 10080, "on_update_failure": 0, "on_update_success": 20160, "processed_events_size": 17822664, "timer_callback_failure": 1, "timer_callback_success": 30239, "timer_cancel_counter": 0, "timer_create_counter": 30240, "timer_create_failure": 0, "timer_msg_counter": 30240, "timer_responses_sent": 0, "timestamp": { "43365": "2021-02-23T01:46:21Z", "43373": "2021-02-23T01:46:21Z", "43382": "2021-02-23T01:46:21Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 1, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "43365": "2021-02-23T01:46:21Z", "43373": "2021-02-23T01:46:21Z", "43382": "2021-02-23T01:46:21Z" }, "v8worker_events_lost": 0 }, "function_id": 2575840121, "function_name": "Function_431426416_test_eventing_rebalance_with_multiple_kv_nodes", "gocb_creds_request_counter": 82, "internal_vb_distribution_stats": { "worker_Function_431426416_test_eventing_rebalance_with_multiple_kv_nodes_0": "[0-341]", "worker_Function_431426416_test_eventing_rebalance_with_multiple_kv_nodes_1": "[342-682]", "worker_Function_431426416_test_eventing_rebalance_with_multiple_kv_nodes_2": "[683-1023]" }, "latency_percentile_stats": { "100": 19900, "50": 1100, "80": 1400, "90": 1900, "95": 2900, "99": 6800 }, "lcb_creds_request_counter": 106, "lcb_exception_stats": { "210": 1 }, "metastore_stats": { "metastore_delete_err": 0, "metastore_deletes": 0, "metastore_not_found": 0, "metastore_scan": 0, "metastore_scan_due": 0, "metastore_scan_err": 0, "metastore_set": 0, "metastore_set_err": 0 }, "planner_stats": [ { "host_name": "172.23.122.133:8096", "start_vb": 0, "vb_count": 1024 } ], "vb_distribution_stats_from_metadata": { "172.23.122.133:8096": { "worker_Function_431426416_test_eventing_rebalance_with_multiple_kv_nodes_0": "[0-341]", "worker_Function_431426416_test_eventing_rebalance_with_multiple_kv_nodes_1": "[342-682]", "worker_Function_431426416_test_eventing_rebalance_with_multiple_kv_nodes_2": "[683-1023]" } }, "worker_pids": { "worker_Function_431426416_test_eventing_rebalance_with_multiple_kv_nodes_0": 43365, "worker_Function_431426416_test_eventing_rebalance_with_multiple_kv_nodes_1": 43373, "worker_Function_431426416_test_eventing_rebalance_with_multiple_kv_nodes_2": 43382 } } ] [2021-02-22 17:46:23,626] - [tuq_helper] [139837055153920] - INFO - RUN QUERY select RAW count(*) from dst_bucket [2021-02-22 17:46:23,632] - [rest_client] [139837055153920] - INFO - query params : statement=select+RAW+count%28%2A%29+from+dst_bucket [2021-02-22 17:46:23,647] - [tuq_helper] [139837055153920] - INFO - TOTAL ELAPSED TIME: 12.647846ms [2021-02-22 17:46:23,647] - [eventing_base] [139837055153920] - INFO - ================== number of doc in dst_bucket via index =========================== [2021-02-22 17:46:23,647] - [eventing_base] [139837055153920] - INFO - [10079] [2021-02-22 17:46:23,647] - [eventing_base] [139837055153920] - INFO - =============================================================================================== [2021-02-22 17:46:23,653] - [tuq_helper] [139837055153920] - INFO - RUN QUERY select RAW count(0) from metadata where meta().id like 'eventing:%:al%' [2021-02-22 17:46:23,659] - [rest_client] [139837055153920] - INFO - query params : statement=select+RAW+count%280%29+from+metadata+where+meta%28%29.id+like+%27eventing%3A%25%3Aal%25%27 [2021-02-22 17:46:23,677] - [tuq_helper] [139837055153920] - INFO - TOTAL ELAPSED TIME: 15.395168ms [2021-02-22 17:46:23,677] - [eventing_base] [139837055153920] - INFO - ================== alarm documents in metadata ================================================ [2021-02-22 17:46:23,678] - [eventing_base] [139837055153920] - INFO - {'requestID': 'cde33268-1804-489c-9614-2765f6e02cfb', 'signature': 'number', 'results': [0], 'status': 'success', 'metrics': {'elapsedTime': '15.395168ms', 'executionTime': '15.325902ms', 'resultCount': 1, 'resultSize': 1, 'serviceLoad': 6}} [2021-02-22 17:46:23,678] - [eventing_base] [139837055153920] - INFO - =============================================================================================== [2021-02-22 17:46:23,684] - [tuq_helper] [139837055153920] - INFO - RUN QUERY select RAW count(0) from metadata where meta().id like 'eventing:%:cx%' [2021-02-22 17:46:23,690] - [rest_client] [139837055153920] - INFO - query params : statement=select+RAW+count%280%29+from+metadata+where+meta%28%29.id+like+%27eventing%3A%25%3Acx%25%27 [2021-02-22 17:46:23,706] - [tuq_helper] [139837055153920] - INFO - TOTAL ELAPSED TIME: 13.7022ms [2021-02-22 17:46:23,706] - [eventing_base] [139837055153920] - INFO - ================== context documents in metadata ============================================= [2021-02-22 17:46:23,706] - [eventing_base] [139837055153920] - INFO - {'requestID': 'a356263e-6bba-4970-995b-43d8b9e15405', 'signature': 'number', 'results': [0], 'status': 'success', 'metrics': {'elapsedTime': '13.7022ms', 'executionTime': '13.652239ms', 'resultCount': 1, 'resultSize': 1, 'serviceLoad': 6}} [2021-02-22 17:46:23,707] - [eventing_base] [139837055153920] - INFO - =============================================================================================== [2021-02-22 17:46:23,727] - [basetestcase] [139837055153920] - INFO - list of eventing nodes in cluster: [ip:172.23.122.133 port:8091 ssh_username:root] [2021-02-22 17:46:23,748] - [eventing_base] [139837055153920] - INFO - Stats for Node 172.23.122.133 is [ { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 24, "agg_messages_sent_to_worker": 49960, "agg_queue_memory": 0, "agg_queue_memory_cap": 106954752, "agg_queue_size": 0, "agg_queue_size_cap": 300000, "agg_timer_feedback_queue_cap": 1500, "dcp_deletion": 10080, "dcp_deletion_sent_to_worker": 10080, "dcp_mutation": 20160, "dcp_mutation_sent_to_worker": 20160, "dcp_snapshot": 30233, "dcp_stream_req_counter": 66340, "dcp_stream_req_err_counter": 62940, "dcp_streamend": 1794, "dcp_streamreq": 3400, "execution_stats": 3033, "failure_stats": 3033, "latency_stats": 3033, "lcb_exception_stats": 3033, "log_level": 3, "num_processed_events": 30240, "processed_events_size": 17822664, "thr_count": 3, "thr_map": 3, "timer_events": 30240, "timer_responses_received": 30240, "v8_init": 3, "v8_load": 3 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 10080, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 20160, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 10080, "enqueued_dcp_mutation_msg_counter": 20160, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 49948, "no_op_counter": 0, "num_processed_events": 30240, "on_delete_failure": 0, "on_delete_success": 10080, "on_update_failure": 0, "on_update_success": 20160, "processed_events_size": 17822664, "timer_callback_failure": 1, "timer_callback_success": 30239, "timer_cancel_counter": 0, "timer_create_counter": 30240, "timer_create_failure": 0, "timer_msg_counter": 30240, "timer_responses_sent": 0, "timestamp": { "43365": "2021-02-23T01:46:23Z", "43373": "2021-02-23T01:46:23Z", "43382": "2021-02-23T01:46:23Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 1, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "43365": "2021-02-23T01:46:23Z", "43373": "2021-02-23T01:46:23Z", "43382": "2021-02-23T01:46:23Z" }, "v8worker_events_lost": 0 }, "function_id": 2575840121, "function_name": "Function_431426416_test_eventing_rebalance_with_multiple_kv_nodes", "gocb_creds_request_counter": 82, "internal_vb_distribution_stats": { "worker_Function_431426416_test_eventing_rebalance_with_multiple_kv_nodes_0": "[0-341]", "worker_Function_431426416_test_eventing_rebalance_with_multiple_kv_nodes_1": "[342-682]", "worker_Function_431426416_test_eventing_rebalance_with_multiple_kv_nodes_2": "[683-1023]" }, "latency_percentile_stats": { "100": 19900, "50": 1100, "80": 1400, "90": 1900, "95": 2900, "99": 6800 }, "lcb_creds_request_counter": 106, "lcb_exception_stats": { "210": 1 }, "metastore_stats": { "metastore_delete_err": 0, "metastore_deletes": 0, "metastore_not_found": 0, "metastore_scan": 0, "metastore_scan_due": 0, "metastore_scan_err": 0, "metastore_set": 0, "metastore_set_err": 0 }, "planner_stats": [ { "host_name": "172.23.122.133:8096", "start_vb": 0, "vb_count": 1024 } ], "vb_distribution_stats_from_metadata": { "172.23.122.133:8096": { "worker_Function_431426416_test_eventing_rebalance_with_multiple_kv_nodes_0": "[0-341]", "worker_Function_431426416_test_eventing_rebalance_with_multiple_kv_nodes_1": "[342-682]", "worker_Function_431426416_test_eventing_rebalance_with_multiple_kv_nodes_2": "[683-1023]" } }, "worker_pids": { "worker_Function_431426416_test_eventing_rebalance_with_multiple_kv_nodes_0": 43365, "worker_Function_431426416_test_eventing_rebalance_with_multiple_kv_nodes_1": 43373, "worker_Function_431426416_test_eventing_rebalance_with_multiple_kv_nodes_2": 43382 } } ] [2021-02-22 17:46:23,767] - [basetestcase] [139837055153920] - INFO - list of eventing nodes in cluster: [ip:172.23.122.133 port:8091 ssh_username:root] [2021-02-22 17:46:23,773] - [remote_util] [139837055153920] - INFO - SSH Connecting to 172.23.122.133 with username:root, attempt#1 of 5 [2021-02-22 17:46:23,879] - [remote_util] [139837055153920] - INFO - SSH Connected to 172.23.122.133 as root [2021-02-22 17:46:24,161] - [remote_util] [139837055153920] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 17:46:24,467] - [remote_util] [139837055153920] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 17:46:24,480] - [rest_client] [139837055153920] - INFO - /diag/eval status on 172.23.122.133:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). [2021-02-22 17:46:24,480] - [remote_util] [139837055153920] - INFO - running command.raw on 172.23.122.133: zgrep "panic" "/opt/couchbase/var/lib/couchbase/logs"/eventing.log* | wc -l [2021-02-22 17:46:25,654] - [remote_util] [139837055153920] - INFO - command executed successfully with root [2021-02-22 17:46:25,655] - [remote_util] [139837055153920] - INFO - running command.raw on 172.23.122.133: ls "/opt/couchbase/var/lib/couchbase/logs"/../crash/| wc -l [2021-02-22 17:46:25,670] - [remote_util] [139837055153920] - INFO - command executed successfully with root [2021-02-22 17:46:25,741] - [basetestcase] [139837055153920] - INFO - ------- Cluster statistics ------- [2021-02-22 17:46:25,742] - [basetestcase] [139837055153920] - INFO - 172.23.122.131:8091 => {'services': ['kv'], 'cpu_utilization': 2.218860312657589, 'mem_free': 3076509696, 'mem_total': 4201684992, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-02-22 17:46:25,742] - [basetestcase] [139837055153920] - INFO - 172.23.122.132:8091 => {'services': ['kv'], 'cpu_utilization': 3.055555555555555, 'mem_free': 2905526272, 'mem_total': 4201684992, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-02-22 17:46:25,742] - [basetestcase] [139837055153920] - INFO - 172.23.122.133:8091 => {'services': ['eventing'], 'cpu_utilization': 2.279057989364396, 'mem_free': 3250655232, 'mem_total': 4201684992, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-02-22 17:46:25,742] - [basetestcase] [139837055153920] - INFO - 172.23.122.134:8091 => {'services': ['index', 'n1ql'], 'cpu_utilization': 2.30438085591289, 'mem_free': 3155976192, 'mem_total': 4201684992, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-02-22 17:46:25,743] - [basetestcase] [139837055153920] - INFO - 172.23.122.180:8091 => {'services': ['kv'], 'cpu_utilization': 4.02327935222672, 'mem_free': 2976002048, 'mem_total': 4201684992, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-02-22 17:46:25,743] - [basetestcase] [139837055153920] - INFO - --- End of cluster statistics --- [2021-02-22 17:46:25,746] - [remote_util] [139837055153920] - INFO - SSH Connecting to 172.23.123.42 with username:root, attempt#1 of 5 [2021-02-22 17:46:25,853] - [remote_util] [139837055153920] - INFO - SSH Connected to 172.23.123.42 as root [2021-02-22 17:46:26,124] - [remote_util] [139837055153920] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 17:46:26,442] - [remote_util] [139837055153920] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 17:46:26,446] - [remote_util] [139837055153920] - INFO - SSH Connecting to 172.23.123.44 with username:root, attempt#1 of 5 [2021-02-22 17:46:26,554] - [remote_util] [139837055153920] - INFO - SSH Connected to 172.23.123.44 as root [2021-02-22 17:46:26,842] - [remote_util] [139837055153920] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 17:46:27,155] - [remote_util] [139837055153920] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 17:46:27,159] - [remote_util] [139837055153920] - INFO - SSH Connecting to 172.23.122.132 with username:root, attempt#1 of 5 [2021-02-22 17:46:27,264] - [remote_util] [139837055153920] - INFO - SSH Connected to 172.23.122.132 as root [2021-02-22 17:46:27,535] - [remote_util] [139837055153920] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 17:46:27,843] - [remote_util] [139837055153920] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 17:46:27,847] - [remote_util] [139837055153920] - INFO - SSH Connecting to 172.23.122.133 with username:root, attempt#1 of 5 [2021-02-22 17:46:27,950] - [remote_util] [139837055153920] - INFO - SSH Connected to 172.23.122.133 as root [2021-02-22 17:46:28,228] - [remote_util] [139837055153920] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 17:46:28,536] - [remote_util] [139837055153920] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 17:46:28,542] - [remote_util] [139837055153920] - INFO - SSH Connecting to 172.23.122.134 with username:root, attempt#1 of 5 [2021-02-22 17:46:28,647] - [remote_util] [139837055153920] - INFO - SSH Connected to 172.23.122.134 as root [2021-02-22 17:46:28,929] - [remote_util] [139837055153920] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 17:46:29,231] - [remote_util] [139837055153920] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 17:46:29,236] - [remote_util] [139837055153920] - INFO - SSH Connecting to 172.23.122.131 with username:root, attempt#1 of 5 [2021-02-22 17:46:29,343] - [remote_util] [139837055153920] - INFO - SSH Connected to 172.23.122.131 as root [2021-02-22 17:46:29,645] - [remote_util] [139837055153920] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 17:46:29,955] - [remote_util] [139837055153920] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 17:46:29,959] - [remote_util] [139837055153920] - INFO - SSH Connecting to 172.23.122.180 with username:root, attempt#1 of 5 [2021-02-22 17:46:30,064] - [remote_util] [139837055153920] - INFO - SSH Connected to 172.23.122.180 as root [2021-02-22 17:46:30,346] - [remote_util] [139837055153920] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 17:46:30,652] - [remote_util] [139837055153920] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 17:46:30,656] - [remote_util] [139837055153920] - INFO - SSH Connecting to 172.23.122.130 with username:root, attempt#1 of 5 [2021-02-22 17:46:30,763] - [remote_util] [139837055153920] - INFO - SSH Connected to 172.23.122.130 as root [2021-02-22 17:46:31,028] - [remote_util] [139837055153920] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 17:46:31,332] - [remote_util] [139837055153920] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 17:47:45,316] - [basetestcase] [139837055153920] - INFO - ============== basetestcase cleanup was started for test #8 test_eventing_rebalance_with_multiple_kv_nodes ============== [2021-02-22 17:47:45,347] - [bucket_helper] [139837055153920] - INFO - Could not find any buckets for node 172.23.123.42, nothing to delete [2021-02-22 17:47:45,354] - [bucket_helper] [139837055153920] - INFO - Could not find any buckets for node 172.23.123.44, nothing to delete [2021-02-22 17:47:45,627] - [bucket_helper] [139837055153920] - INFO - deleting existing buckets ['dst_bucket', 'metadata', 'src_bucket'] on 172.23.122.132 [2021-02-22 17:47:46,673] - [bucket_helper] [139837055153920] - INFO - waiting for bucket deletion to complete.... [2021-02-22 17:47:46,703] - [rest_client] [139837055153920] - INFO - node 172.23.122.132 existing buckets : ['metadata', 'src_bucket'] [2021-02-22 17:47:46,705] - [bucket_helper] [139837055153920] - INFO - deleted bucket : dst_bucket from 172.23.122.132 [2021-02-22 17:47:48,033] - [bucket_helper] [139837055153920] - INFO - waiting for bucket deletion to complete.... [2021-02-22 17:47:48,052] - [rest_client] [139837055153920] - INFO - node 172.23.122.132 existing buckets : ['src_bucket'] [2021-02-22 17:47:48,054] - [bucket_helper] [139837055153920] - INFO - deleted bucket : metadata from 172.23.122.132 [2021-02-22 17:47:49,129] - [bucket_helper] [139837055153920] - INFO - waiting for bucket deletion to complete.... [2021-02-22 17:47:49,136] - [rest_client] [139837055153920] - INFO - node 172.23.122.132 existing buckets : [] [2021-02-22 17:47:49,136] - [bucket_helper] [139837055153920] - INFO - deleted bucket : src_bucket from 172.23.122.132 [2021-02-22 17:47:49,147] - [bucket_helper] [139837055153920] - INFO - Could not find any buckets for node 172.23.122.133, nothing to delete [2021-02-22 17:47:49,170] - [bucket_helper] [139837055153920] - INFO - Could not find any buckets for node 172.23.122.134, nothing to delete [2021-02-22 17:47:49,180] - [bucket_helper] [139837055153920] - INFO - Could not find any buckets for node 172.23.122.131, nothing to delete [2021-02-22 17:47:49,189] - [bucket_helper] [139837055153920] - INFO - Could not find any buckets for node 172.23.122.180, nothing to delete [2021-02-22 17:47:49,196] - [bucket_helper] [139837055153920] - INFO - Could not find any buckets for node 172.23.122.130, nothing to delete [2021-02-22 17:47:49,203] - [rest_client] [139837055153920] - INFO - -->is_ns_server_running? [2021-02-22 17:47:49,218] - [cluster_helper] [139837055153920] - INFO - rebalancing all nodes in order to remove nodes [2021-02-22 17:47:49,222] - [rest_client] [139837055153920] - INFO - rebalance params : {'knownNodes': 'ns_1@172.23.122.131,ns_1@172.23.122.132,ns_1@172.23.122.133,ns_1@172.23.122.134,ns_1@172.23.122.180', 'ejectedNodes': 'ns_1@172.23.122.131,ns_1@172.23.122.133,ns_1@172.23.122.134,ns_1@172.23.122.180', 'user': 'Administrator', 'password': 'password'} [2021-02-22 17:47:49,256] - [rest_client] [139837055153920] - INFO - rebalance operation started [2021-02-22 17:47:49,260] - [rest_client] [139837055153920] - INFO - rebalance percentage : 0.00 % [2021-02-22 17:47:59,272] - [rest_client] [139837055153920] - INFO - rebalance percentage : 70.00 % [2021-02-22 17:48:09,287] - [rest_client] [139837055153920] - INFO - rebalance percentage : 90.00 % [2021-02-22 17:48:29,300] - [rest_client] [139837055153920] - INFO - rebalance progress took 40.04 seconds [2021-02-22 17:48:29,300] - [rest_client] [139837055153920] - INFO - sleep for 10 seconds after rebalance... [2021-02-22 17:48:39,343] - [cluster_helper] [139837055153920] - INFO - removed all the nodes from cluster associated with ip:172.23.123.42 port:8091 ssh_username:root ? [('ns_1@172.23.122.131', 8091), ('ns_1@172.23.122.133', 8091), ('ns_1@172.23.122.134', 8091), ('ns_1@172.23.122.180', 8091)] [2021-02-22 17:48:39,349] - [cluster_helper] [139837055153920] - INFO - waiting for ns_server @ 172.23.123.42:8091 [2021-02-22 17:48:39,349] - [rest_client] [139837055153920] - INFO - -->is_ns_server_running? [2021-02-22 17:48:39,353] - [cluster_helper] [139837055153920] - INFO - ns_server @ 172.23.123.42:8091 is running [2021-02-22 17:48:39,358] - [cluster_helper] [139837055153920] - INFO - waiting for ns_server @ 172.23.123.44:8091 [2021-02-22 17:48:39,358] - [rest_client] [139837055153920] - INFO - -->is_ns_server_running? [2021-02-22 17:48:39,363] - [cluster_helper] [139837055153920] - INFO - ns_server @ 172.23.123.44:8091 is running [2021-02-22 17:48:39,369] - [cluster_helper] [139837055153920] - INFO - waiting for ns_server @ 172.23.122.132:8091 [2021-02-22 17:48:39,370] - [rest_client] [139837055153920] - INFO - -->is_ns_server_running? [2021-02-22 17:48:39,375] - [cluster_helper] [139837055153920] - INFO - ns_server @ 172.23.122.132:8091 is running [2021-02-22 17:48:39,379] - [cluster_helper] [139837055153920] - INFO - waiting for ns_server @ 172.23.122.133:8091 [2021-02-22 17:48:39,379] - [rest_client] [139837055153920] - INFO - -->is_ns_server_running? [2021-02-22 17:48:39,383] - [cluster_helper] [139837055153920] - INFO - ns_server @ 172.23.122.133:8091 is running [2021-02-22 17:48:39,387] - [cluster_helper] [139837055153920] - INFO - waiting for ns_server @ 172.23.122.134:8091 [2021-02-22 17:48:39,387] - [rest_client] [139837055153920] - INFO - -->is_ns_server_running? [2021-02-22 17:48:39,391] - [cluster_helper] [139837055153920] - INFO - ns_server @ 172.23.122.134:8091 is running [2021-02-22 17:48:39,396] - [cluster_helper] [139837055153920] - INFO - waiting for ns_server @ 172.23.122.131:8091 [2021-02-22 17:48:39,396] - [rest_client] [139837055153920] - INFO - -->is_ns_server_running? [2021-02-22 17:48:39,400] - [cluster_helper] [139837055153920] - INFO - ns_server @ 172.23.122.131:8091 is running [2021-02-22 17:48:39,405] - [cluster_helper] [139837055153920] - INFO - waiting for ns_server @ 172.23.122.180:8091 [2021-02-22 17:48:39,405] - [rest_client] [139837055153920] - INFO - -->is_ns_server_running? [2021-02-22 17:48:39,409] - [cluster_helper] [139837055153920] - INFO - ns_server @ 172.23.122.180:8091 is running [2021-02-22 17:48:39,414] - [cluster_helper] [139837055153920] - INFO - waiting for ns_server @ 172.23.122.130:8091 [2021-02-22 17:48:39,415] - [rest_client] [139837055153920] - INFO - -->is_ns_server_running? [2021-02-22 17:48:39,419] - [cluster_helper] [139837055153920] - INFO - ns_server @ 172.23.122.130:8091 is running [2021-02-22 17:48:39,419] - [basetestcase] [139837055153920] - INFO - ============== basetestcase cleanup was finished for test #8 test_eventing_rebalance_with_multiple_kv_nodes ============== [2021-02-22 17:48:39,420] - [basetestcase] [139837055153920] - INFO - closing all ssh connections [2021-02-22 17:48:39,423] - [basetestcase] [139837055153920] - INFO - closing all memcached connections [2021-02-22 17:48:39,426] - [remote_util] [139837055153920] - INFO - SSH Connecting to 172.23.123.42 with username:root, attempt#1 of 5 [2021-02-22 17:48:39,532] - [remote_util] [139837055153920] - INFO - SSH Connected to 172.23.123.42 as root [2021-02-22 17:48:39,798] - [remote_util] [139837055153920] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 17:48:40,112] - [remote_util] [139837055153920] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 17:48:40,125] - [remote_util] [139837055153920] - INFO - SSH Connecting to 172.23.123.42 with username:root, attempt#1 of 5 [2021-02-22 17:48:40,232] - [remote_util] [139837055153920] - INFO - SSH Connected to 172.23.123.42 as root [2021-02-22 17:48:40,496] - [remote_util] [139837055153920] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 17:48:40,818] - [remote_util] [139837055153920] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 17:48:40,880] - [remote_util] [139837055153920] - INFO - 172.23.123.42 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-22 17:48:40,993] - [remote_util] [139837055153920] - INFO - Running systemd command on this server [2021-02-22 17:48:40,994] - [remote_util] [139837055153920] - INFO - running command.raw on 172.23.123.42: systemctl stop couchbase-server.service [2021-02-22 17:48:42,726] - [remote_util] [139837055153920] - INFO - command executed successfully with root [2021-02-22 17:48:42,726] - [basetestcase] [139837055153920] - INFO - Couchbase stopped [2021-02-22 17:48:42,727] - [basetestcase] [139837055153920] - INFO - sleep for 5 secs. ... [2021-02-22 17:48:47,728] - [remote_util] [139837055153920] - INFO - running command.raw on 172.23.123.42: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-22 17:48:47,751] - [remote_util] [139837055153920] - INFO - command executed successfully with root [2021-02-22 17:48:47,752] - [remote_util] [139837055153920] - INFO - running command.raw on 172.23.123.42: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-22 17:48:47,807] - [remote_util] [139837055153920] - INFO - command executed successfully with root [2021-02-22 17:48:47,810] - [remote_util] [139837055153920] - INFO - SSH Connecting to 172.23.123.42 with username:root, attempt#1 of 5 [2021-02-22 17:48:47,920] - [remote_util] [139837055153920] - INFO - SSH Connected to 172.23.123.42 as root [2021-02-22 17:48:48,192] - [remote_util] [139837055153920] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 17:48:48,511] - [remote_util] [139837055153920] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 17:48:48,575] - [remote_util] [139837055153920] - INFO - 172.23.123.42 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-22 17:48:48,575] - [remote_util] [139837055153920] - INFO - Checking for process beam.smp on linux [2021-02-22 17:48:48,655] - [remote_util] [139837055153920] - INFO - Starting couchbase server [2021-02-22 17:48:48,766] - [remote_util] [139837055153920] - INFO - Running systemd command on this server [2021-02-22 17:48:48,766] - [remote_util] [139837055153920] - INFO - running command.raw on 172.23.123.42: systemctl start couchbase-server.service [2021-02-22 17:48:48,791] - [remote_util] [139837055153920] - INFO - command executed successfully with root [2021-02-22 17:48:48,791] - [remote_util] [139837055153920] - INFO - 172.23.123.42:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-22 17:48:53,792] - [remote_util] [139837055153920] - INFO - running command.raw on 172.23.123.42: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-22 17:48:53,821] - [remote_util] [139837055153920] - INFO - command executed successfully with root [2021-02-22 17:48:53,821] - [remote_util] [139837055153920] - INFO - Couchbase server status: [' Process: 50428 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-22 17:48:53,822] - [remote_util] [139837055153920] - INFO - Checking for process beam.smp on linux [2021-02-22 17:48:53,893] - [remote_util] [139837055153920] - INFO - process beam.smp is running on 172.23.123.42: with pid 50575 [2021-02-22 17:48:53,893] - [basetestcase] [139837055153920] - INFO - Couchbase started [2021-02-22 17:48:53,897] - [remote_util] [139837055153920] - INFO - SSH Connecting to 172.23.123.44 with username:root, attempt#1 of 5 [2021-02-22 17:48:54,005] - [remote_util] [139837055153920] - INFO - SSH Connected to 172.23.123.44 as root [2021-02-22 17:48:54,290] - [remote_util] [139837055153920] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 17:48:54,610] - [remote_util] [139837055153920] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 17:48:54,623] - [remote_util] [139837055153920] - INFO - SSH Connecting to 172.23.123.44 with username:root, attempt#1 of 5 [2021-02-22 17:48:54,726] - [remote_util] [139837055153920] - INFO - SSH Connected to 172.23.123.44 as root [2021-02-22 17:48:55,022] - [remote_util] [139837055153920] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 17:48:55,343] - [remote_util] [139837055153920] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 17:48:55,406] - [remote_util] [139837055153920] - INFO - 172.23.123.44 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-22 17:48:55,516] - [remote_util] [139837055153920] - INFO - Running systemd command on this server [2021-02-22 17:48:55,516] - [remote_util] [139837055153920] - INFO - running command.raw on 172.23.123.44: systemctl stop couchbase-server.service [2021-02-22 17:48:57,087] - [remote_util] [139837055153920] - INFO - command executed successfully with root [2021-02-22 17:48:57,087] - [basetestcase] [139837055153920] - INFO - Couchbase stopped [2021-02-22 17:48:57,088] - [basetestcase] [139837055153920] - INFO - sleep for 5 secs. ... [2021-02-22 17:49:02,093] - [remote_util] [139837055153920] - INFO - running command.raw on 172.23.123.44: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-22 17:49:02,114] - [remote_util] [139837055153920] - INFO - command executed successfully with root [2021-02-22 17:49:02,115] - [remote_util] [139837055153920] - INFO - running command.raw on 172.23.123.44: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-22 17:49:02,169] - [remote_util] [139837055153920] - INFO - command executed successfully with root [2021-02-22 17:49:02,173] - [remote_util] [139837055153920] - INFO - SSH Connecting to 172.23.123.44 with username:root, attempt#1 of 5 [2021-02-22 17:49:02,282] - [remote_util] [139837055153920] - INFO - SSH Connected to 172.23.123.44 as root [2021-02-22 17:49:02,551] - [remote_util] [139837055153920] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 17:49:02,867] - [remote_util] [139837055153920] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 17:49:02,927] - [remote_util] [139837055153920] - INFO - 172.23.123.44 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-22 17:49:02,928] - [remote_util] [139837055153920] - INFO - Checking for process beam.smp on linux [2021-02-22 17:49:03,007] - [remote_util] [139837055153920] - INFO - Starting couchbase server [2021-02-22 17:49:03,113] - [remote_util] [139837055153920] - INFO - Running systemd command on this server [2021-02-22 17:49:03,113] - [remote_util] [139837055153920] - INFO - running command.raw on 172.23.123.44: systemctl start couchbase-server.service [2021-02-22 17:49:03,139] - [remote_util] [139837055153920] - INFO - command executed successfully with root [2021-02-22 17:49:03,139] - [remote_util] [139837055153920] - INFO - 172.23.123.44:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-22 17:49:08,144] - [remote_util] [139837055153920] - INFO - running command.raw on 172.23.123.44: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-22 17:49:08,168] - [remote_util] [139837055153920] - INFO - command executed successfully with root [2021-02-22 17:49:08,169] - [remote_util] [139837055153920] - INFO - Couchbase server status: [' Process: 14305 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-22 17:49:08,169] - [remote_util] [139837055153920] - INFO - Checking for process beam.smp on linux [2021-02-22 17:49:08,240] - [remote_util] [139837055153920] - INFO - process beam.smp is running on 172.23.123.44: with pid 14446 [2021-02-22 17:49:08,240] - [basetestcase] [139837055153920] - INFO - Couchbase started [2021-02-22 17:49:08,244] - [remote_util] [139837055153920] - INFO - SSH Connecting to 172.23.122.132 with username:root, attempt#1 of 5 [2021-02-22 17:49:08,344] - [remote_util] [139837055153920] - INFO - SSH Connected to 172.23.122.132 as root [2021-02-22 17:49:08,612] - [remote_util] [139837055153920] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 17:49:08,915] - [remote_util] [139837055153920] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 17:49:08,931] - [remote_util] [139837055153920] - INFO - SSH Connecting to 172.23.122.132 with username:root, attempt#1 of 5 [2021-02-22 17:49:09,033] - [remote_util] [139837055153920] - INFO - SSH Connected to 172.23.122.132 as root [2021-02-22 17:49:09,290] - [remote_util] [139837055153920] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 17:49:09,599] - [remote_util] [139837055153920] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 17:49:09,664] - [remote_util] [139837055153920] - INFO - 172.23.122.132 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-22 17:49:09,764] - [remote_util] [139837055153920] - INFO - Running systemd command on this server [2021-02-22 17:49:09,765] - [remote_util] [139837055153920] - INFO - running command.raw on 172.23.122.132: systemctl stop couchbase-server.service [2021-02-22 17:49:11,598] - [remote_util] [139837055153920] - INFO - command executed successfully with root [2021-02-22 17:49:11,599] - [basetestcase] [139837055153920] - INFO - Couchbase stopped [2021-02-22 17:49:11,599] - [basetestcase] [139837055153920] - INFO - sleep for 5 secs. ... [2021-02-22 17:49:16,605] - [remote_util] [139837055153920] - INFO - running command.raw on 172.23.122.132: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-22 17:49:16,621] - [remote_util] [139837055153920] - INFO - command executed successfully with root [2021-02-22 17:49:16,621] - [remote_util] [139837055153920] - INFO - running command.raw on 172.23.122.132: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-22 17:49:16,681] - [remote_util] [139837055153920] - INFO - command executed successfully with root [2021-02-22 17:49:16,685] - [remote_util] [139837055153920] - INFO - SSH Connecting to 172.23.122.132 with username:root, attempt#1 of 5 [2021-02-22 17:49:16,786] - [remote_util] [139837055153920] - INFO - SSH Connected to 172.23.122.132 as root [2021-02-22 17:49:17,052] - [remote_util] [139837055153920] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 17:49:17,354] - [remote_util] [139837055153920] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 17:49:17,416] - [remote_util] [139837055153920] - INFO - 172.23.122.132 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-22 17:49:17,417] - [remote_util] [139837055153920] - INFO - Checking for process beam.smp on linux [2021-02-22 17:49:17,491] - [remote_util] [139837055153920] - INFO - Starting couchbase server [2021-02-22 17:49:17,591] - [remote_util] [139837055153920] - INFO - Running systemd command on this server [2021-02-22 17:49:17,592] - [remote_util] [139837055153920] - INFO - running command.raw on 172.23.122.132: systemctl start couchbase-server.service [2021-02-22 17:49:17,613] - [remote_util] [139837055153920] - INFO - command executed successfully with root [2021-02-22 17:49:17,613] - [remote_util] [139837055153920] - INFO - 172.23.122.132:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-22 17:49:22,618] - [remote_util] [139837055153920] - INFO - running command.raw on 172.23.122.132: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-22 17:49:22,642] - [remote_util] [139837055153920] - INFO - command executed successfully with root [2021-02-22 17:49:22,643] - [remote_util] [139837055153920] - INFO - Couchbase server status: [' Process: 44146 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-22 17:49:22,643] - [remote_util] [139837055153920] - INFO - Checking for process beam.smp on linux [2021-02-22 17:49:22,709] - [remote_util] [139837055153920] - INFO - process beam.smp is running on 172.23.122.132: with pid 44291 [2021-02-22 17:49:22,709] - [basetestcase] [139837055153920] - INFO - Couchbase started [2021-02-22 17:49:22,713] - [remote_util] [139837055153920] - INFO - SSH Connecting to 172.23.122.133 with username:root, attempt#1 of 5 [2021-02-22 17:49:22,817] - [remote_util] [139837055153920] - INFO - SSH Connected to 172.23.122.133 as root [2021-02-22 17:49:23,092] - [remote_util] [139837055153920] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 17:49:23,430] - [remote_util] [139837055153920] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 17:49:23,444] - [remote_util] [139837055153920] - INFO - SSH Connecting to 172.23.122.133 with username:root, attempt#1 of 5 [2021-02-22 17:49:23,540] - [remote_util] [139837055153920] - INFO - SSH Connected to 172.23.122.133 as root [2021-02-22 17:49:23,805] - [remote_util] [139837055153920] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 17:49:24,104] - [remote_util] [139837055153920] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 17:49:24,165] - [remote_util] [139837055153920] - INFO - 172.23.122.133 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-22 17:49:24,302] - [remote_util] [139837055153920] - INFO - Running systemd command on this server [2021-02-22 17:49:24,302] - [remote_util] [139837055153920] - INFO - running command.raw on 172.23.122.133: systemctl stop couchbase-server.service [2021-02-22 17:49:25,913] - [remote_util] [139837055153920] - INFO - command executed successfully with root [2021-02-22 17:49:25,913] - [basetestcase] [139837055153920] - INFO - Couchbase stopped [2021-02-22 17:49:25,915] - [basetestcase] [139837055153920] - INFO - sleep for 5 secs. ... [2021-02-22 17:49:30,920] - [remote_util] [139837055153920] - INFO - running command.raw on 172.23.122.133: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-22 17:49:30,937] - [remote_util] [139837055153920] - INFO - command executed successfully with root [2021-02-22 17:49:30,937] - [remote_util] [139837055153920] - INFO - running command.raw on 172.23.122.133: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-22 17:49:30,992] - [remote_util] [139837055153920] - INFO - command executed successfully with root [2021-02-22 17:49:30,996] - [remote_util] [139837055153920] - INFO - SSH Connecting to 172.23.122.133 with username:root, attempt#1 of 5 [2021-02-22 17:49:31,100] - [remote_util] [139837055153920] - INFO - SSH Connected to 172.23.122.133 as root [2021-02-22 17:49:31,381] - [remote_util] [139837055153920] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 17:49:31,686] - [remote_util] [139837055153920] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 17:49:31,748] - [remote_util] [139837055153920] - INFO - 172.23.122.133 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-22 17:49:31,748] - [remote_util] [139837055153920] - INFO - Checking for process beam.smp on linux [2021-02-22 17:49:31,825] - [remote_util] [139837055153920] - INFO - Starting couchbase server [2021-02-22 17:49:31,938] - [remote_util] [139837055153920] - INFO - Running systemd command on this server [2021-02-22 17:49:31,938] - [remote_util] [139837055153920] - INFO - running command.raw on 172.23.122.133: systemctl start couchbase-server.service [2021-02-22 17:49:31,962] - [remote_util] [139837055153920] - INFO - command executed successfully with root [2021-02-22 17:49:31,962] - [remote_util] [139837055153920] - INFO - 172.23.122.133:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-22 17:49:36,967] - [remote_util] [139837055153920] - INFO - running command.raw on 172.23.122.133: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-22 17:49:36,995] - [remote_util] [139837055153920] - INFO - command executed successfully with root [2021-02-22 17:49:36,995] - [remote_util] [139837055153920] - INFO - Couchbase server status: [' Process: 48106 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-22 17:49:36,995] - [remote_util] [139837055153920] - INFO - Checking for process beam.smp on linux [2021-02-22 17:49:37,064] - [remote_util] [139837055153920] - INFO - process beam.smp is running on 172.23.122.133: with pid 48250 [2021-02-22 17:49:37,064] - [basetestcase] [139837055153920] - INFO - Couchbase started [2021-02-22 17:49:37,067] - [remote_util] [139837055153920] - INFO - SSH Connecting to 172.23.122.134 with username:root, attempt#1 of 5 [2021-02-22 17:49:37,174] - [remote_util] [139837055153920] - INFO - SSH Connected to 172.23.122.134 as root [2021-02-22 17:49:37,459] - [remote_util] [139837055153920] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 17:49:37,767] - [remote_util] [139837055153920] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 17:49:37,782] - [remote_util] [139837055153920] - INFO - SSH Connecting to 172.23.122.134 with username:root, attempt#1 of 5 [2021-02-22 17:49:37,882] - [remote_util] [139837055153920] - INFO - SSH Connected to 172.23.122.134 as root [2021-02-22 17:49:38,155] - [remote_util] [139837055153920] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 17:49:38,502] - [remote_util] [139837055153920] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 17:49:38,565] - [remote_util] [139837055153920] - INFO - 172.23.122.134 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-22 17:49:38,708] - [remote_util] [139837055153920] - INFO - Running systemd command on this server [2021-02-22 17:49:38,708] - [remote_util] [139837055153920] - INFO - running command.raw on 172.23.122.134: systemctl stop couchbase-server.service [2021-02-22 17:49:40,472] - [remote_util] [139837055153920] - INFO - command executed successfully with root [2021-02-22 17:49:40,473] - [basetestcase] [139837055153920] - INFO - Couchbase stopped [2021-02-22 17:49:40,473] - [basetestcase] [139837055153920] - INFO - sleep for 5 secs. ... [2021-02-22 17:49:45,479] - [remote_util] [139837055153920] - INFO - running command.raw on 172.23.122.134: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-22 17:49:45,497] - [remote_util] [139837055153920] - INFO - command executed successfully with root [2021-02-22 17:49:45,498] - [remote_util] [139837055153920] - INFO - running command.raw on 172.23.122.134: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-22 17:49:45,551] - [remote_util] [139837055153920] - INFO - command executed successfully with root [2021-02-22 17:49:45,554] - [remote_util] [139837055153920] - INFO - SSH Connecting to 172.23.122.134 with username:root, attempt#1 of 5 [2021-02-22 17:49:45,660] - [remote_util] [139837055153920] - INFO - SSH Connected to 172.23.122.134 as root [2021-02-22 17:49:45,944] - [remote_util] [139837055153920] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 17:49:46,250] - [remote_util] [139837055153920] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 17:49:46,313] - [remote_util] [139837055153920] - INFO - 172.23.122.134 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-22 17:49:46,313] - [remote_util] [139837055153920] - INFO - Checking for process beam.smp on linux [2021-02-22 17:49:46,390] - [remote_util] [139837055153920] - INFO - Starting couchbase server [2021-02-22 17:49:46,537] - [remote_util] [139837055153920] - INFO - Running systemd command on this server [2021-02-22 17:49:46,538] - [remote_util] [139837055153920] - INFO - running command.raw on 172.23.122.134: systemctl start couchbase-server.service [2021-02-22 17:49:46,562] - [remote_util] [139837055153920] - INFO - command executed successfully with root [2021-02-22 17:49:46,562] - [remote_util] [139837055153920] - INFO - 172.23.122.134:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-22 17:49:51,568] - [remote_util] [139837055153920] - INFO - running command.raw on 172.23.122.134: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-22 17:49:51,594] - [remote_util] [139837055153920] - INFO - command executed successfully with root [2021-02-22 17:49:51,595] - [remote_util] [139837055153920] - INFO - Couchbase server status: [' Process: 678 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-22 17:49:51,596] - [remote_util] [139837055153920] - INFO - Checking for process beam.smp on linux [2021-02-22 17:49:51,666] - [remote_util] [139837055153920] - INFO - process beam.smp is running on 172.23.122.134: with pid 835 [2021-02-22 17:49:51,666] - [basetestcase] [139837055153920] - INFO - Couchbase started [2021-02-22 17:49:51,670] - [remote_util] [139837055153920] - INFO - SSH Connecting to 172.23.122.131 with username:root, attempt#1 of 5 [2021-02-22 17:49:51,774] - [remote_util] [139837055153920] - INFO - SSH Connected to 172.23.122.131 as root [2021-02-22 17:49:52,069] - [remote_util] [139837055153920] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 17:49:52,378] - [remote_util] [139837055153920] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 17:49:52,392] - [remote_util] [139837055153920] - INFO - SSH Connecting to 172.23.122.131 with username:root, attempt#1 of 5 [2021-02-22 17:49:52,496] - [remote_util] [139837055153920] - INFO - SSH Connected to 172.23.122.131 as root [2021-02-22 17:49:52,796] - [remote_util] [139837055153920] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 17:49:53,113] - [remote_util] [139837055153920] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 17:49:53,177] - [remote_util] [139837055153920] - INFO - 172.23.122.131 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-22 17:49:53,363] - [remote_util] [139837055153920] - INFO - Running systemd command on this server [2021-02-22 17:49:53,363] - [remote_util] [139837055153920] - INFO - running command.raw on 172.23.122.131: systemctl stop couchbase-server.service [2021-02-22 17:49:55,128] - [remote_util] [139837055153920] - INFO - command executed successfully with root [2021-02-22 17:49:55,129] - [basetestcase] [139837055153920] - INFO - Couchbase stopped [2021-02-22 17:49:55,130] - [basetestcase] [139837055153920] - INFO - sleep for 5 secs. ... [2021-02-22 17:50:00,131] - [remote_util] [139837055153920] - INFO - running command.raw on 172.23.122.131: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-22 17:50:00,149] - [remote_util] [139837055153920] - INFO - command executed successfully with root [2021-02-22 17:50:00,150] - [remote_util] [139837055153920] - INFO - running command.raw on 172.23.122.131: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-22 17:50:00,205] - [remote_util] [139837055153920] - INFO - command executed successfully with root [2021-02-22 17:50:00,210] - [remote_util] [139837055153920] - INFO - SSH Connecting to 172.23.122.131 with username:root, attempt#1 of 5 [2021-02-22 17:50:00,315] - [remote_util] [139837055153920] - INFO - SSH Connected to 172.23.122.131 as root [2021-02-22 17:50:00,621] - [remote_util] [139837055153920] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 17:50:00,932] - [remote_util] [139837055153920] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 17:50:00,997] - [remote_util] [139837055153920] - INFO - 172.23.122.131 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-22 17:50:00,997] - [remote_util] [139837055153920] - INFO - Checking for process beam.smp on linux [2021-02-22 17:50:01,081] - [remote_util] [139837055153920] - INFO - Starting couchbase server [2021-02-22 17:50:01,224] - [remote_util] [139837055153920] - INFO - Running systemd command on this server [2021-02-22 17:50:01,224] - [remote_util] [139837055153920] - INFO - running command.raw on 172.23.122.131: systemctl start couchbase-server.service [2021-02-22 17:50:01,249] - [remote_util] [139837055153920] - INFO - command executed successfully with root [2021-02-22 17:50:01,250] - [remote_util] [139837055153920] - INFO - 172.23.122.131:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-22 17:50:06,255] - [remote_util] [139837055153920] - INFO - running command.raw on 172.23.122.131: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-22 17:50:06,282] - [remote_util] [139837055153920] - INFO - command executed successfully with root [2021-02-22 17:50:06,283] - [remote_util] [139837055153920] - INFO - Couchbase server status: [' Process: 98397 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-22 17:50:06,284] - [remote_util] [139837055153920] - INFO - Checking for process beam.smp on linux [2021-02-22 17:50:06,353] - [remote_util] [139837055153920] - INFO - process beam.smp is running on 172.23.122.131: with pid 98542 [2021-02-22 17:50:06,354] - [basetestcase] [139837055153920] - INFO - Couchbase started [2021-02-22 17:50:06,357] - [remote_util] [139837055153920] - INFO - SSH Connecting to 172.23.122.180 with username:root, attempt#1 of 5 [2021-02-22 17:50:06,467] - [remote_util] [139837055153920] - INFO - SSH Connected to 172.23.122.180 as root [2021-02-22 17:50:06,750] - [remote_util] [139837055153920] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 17:50:07,056] - [remote_util] [139837055153920] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 17:50:07,070] - [remote_util] [139837055153920] - INFO - SSH Connecting to 172.23.122.180 with username:root, attempt#1 of 5 [2021-02-22 17:50:07,172] - [remote_util] [139837055153920] - INFO - SSH Connected to 172.23.122.180 as root [2021-02-22 17:50:07,442] - [remote_util] [139837055153920] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 17:50:07,750] - [remote_util] [139837055153920] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 17:50:07,812] - [remote_util] [139837055153920] - INFO - 172.23.122.180 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-22 17:50:07,924] - [remote_util] [139837055153920] - INFO - Running systemd command on this server [2021-02-22 17:50:07,924] - [remote_util] [139837055153920] - INFO - running command.raw on 172.23.122.180: systemctl stop couchbase-server.service [2021-02-22 17:50:09,687] - [remote_util] [139837055153920] - INFO - command executed successfully with root [2021-02-22 17:50:09,688] - [basetestcase] [139837055153920] - INFO - Couchbase stopped [2021-02-22 17:50:09,688] - [basetestcase] [139837055153920] - INFO - sleep for 5 secs. ... [2021-02-22 17:50:14,694] - [remote_util] [139837055153920] - INFO - running command.raw on 172.23.122.180: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-22 17:50:14,711] - [remote_util] [139837055153920] - INFO - command executed successfully with root [2021-02-22 17:50:14,712] - [remote_util] [139837055153920] - INFO - running command.raw on 172.23.122.180: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-22 17:50:14,766] - [remote_util] [139837055153920] - INFO - command executed successfully with root [2021-02-22 17:50:14,770] - [remote_util] [139837055153920] - INFO - SSH Connecting to 172.23.122.180 with username:root, attempt#1 of 5 [2021-02-22 17:50:14,874] - [remote_util] [139837055153920] - INFO - SSH Connected to 172.23.122.180 as root [2021-02-22 17:50:15,142] - [remote_util] [139837055153920] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 17:50:15,447] - [remote_util] [139837055153920] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 17:50:15,509] - [remote_util] [139837055153920] - INFO - 172.23.122.180 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-22 17:50:15,510] - [remote_util] [139837055153920] - INFO - Checking for process beam.smp on linux [2021-02-22 17:50:15,588] - [remote_util] [139837055153920] - INFO - Starting couchbase server [2021-02-22 17:50:15,692] - [remote_util] [139837055153920] - INFO - Running systemd command on this server [2021-02-22 17:50:15,692] - [remote_util] [139837055153920] - INFO - running command.raw on 172.23.122.180: systemctl start couchbase-server.service [2021-02-22 17:50:15,714] - [remote_util] [139837055153920] - INFO - command executed successfully with root [2021-02-22 17:50:15,714] - [remote_util] [139837055153920] - INFO - 172.23.122.180:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-22 17:50:20,720] - [remote_util] [139837055153920] - INFO - running command.raw on 172.23.122.180: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-22 17:50:20,747] - [remote_util] [139837055153920] - INFO - command executed successfully with root [2021-02-22 17:50:20,748] - [remote_util] [139837055153920] - INFO - Couchbase server status: [' Process: 109213 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-22 17:50:20,748] - [remote_util] [139837055153920] - INFO - Checking for process beam.smp on linux [2021-02-22 17:50:20,818] - [remote_util] [139837055153920] - INFO - process beam.smp is running on 172.23.122.180: with pid 109357 [2021-02-22 17:50:20,819] - [basetestcase] [139837055153920] - INFO - Couchbase started [2021-02-22 17:50:20,823] - [remote_util] [139837055153920] - INFO - SSH Connecting to 172.23.122.130 with username:root, attempt#1 of 5 [2021-02-22 17:50:20,927] - [remote_util] [139837055153920] - INFO - SSH Connected to 172.23.122.130 as root [2021-02-22 17:50:21,191] - [remote_util] [139837055153920] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 17:50:21,492] - [remote_util] [139837055153920] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 17:50:21,505] - [remote_util] [139837055153920] - INFO - SSH Connecting to 172.23.122.130 with username:root, attempt#1 of 5 [2021-02-22 17:50:21,604] - [remote_util] [139837055153920] - INFO - SSH Connected to 172.23.122.130 as root [2021-02-22 17:50:21,865] - [remote_util] [139837055153920] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 17:50:22,169] - [remote_util] [139837055153920] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 17:50:22,233] - [remote_util] [139837055153920] - INFO - 172.23.122.130 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-22 17:50:22,378] - [remote_util] [139837055153920] - INFO - Running systemd command on this server [2021-02-22 17:50:22,378] - [remote_util] [139837055153920] - INFO - running command.raw on 172.23.122.130: systemctl stop couchbase-server.service [2021-02-22 17:50:24,370] - [remote_util] [139837055153920] - INFO - command executed successfully with root [2021-02-22 17:50:24,371] - [basetestcase] [139837055153920] - INFO - Couchbase stopped [2021-02-22 17:50:24,372] - [basetestcase] [139837055153920] - INFO - sleep for 5 secs. ... [2021-02-22 17:50:29,377] - [remote_util] [139837055153920] - INFO - running command.raw on 172.23.122.130: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-22 17:50:29,394] - [remote_util] [139837055153920] - INFO - command executed successfully with root [2021-02-22 17:50:29,394] - [remote_util] [139837055153920] - INFO - running command.raw on 172.23.122.130: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-22 17:50:29,448] - [remote_util] [139837055153920] - INFO - command executed successfully with root [2021-02-22 17:50:29,453] - [remote_util] [139837055153920] - INFO - SSH Connecting to 172.23.122.130 with username:root, attempt#1 of 5 [2021-02-22 17:50:29,558] - [remote_util] [139837055153920] - INFO - SSH Connected to 172.23.122.130 as root [2021-02-22 17:50:29,830] - [remote_util] [139837055153920] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 17:50:30,136] - [remote_util] [139837055153920] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 17:50:30,198] - [remote_util] [139837055153920] - INFO - 172.23.122.130 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-22 17:50:30,199] - [remote_util] [139837055153920] - INFO - Checking for process beam.smp on linux [2021-02-22 17:50:30,275] - [remote_util] [139837055153920] - INFO - Starting couchbase server [2021-02-22 17:50:30,419] - [remote_util] [139837055153920] - INFO - Running systemd command on this server [2021-02-22 17:50:30,419] - [remote_util] [139837055153920] - INFO - running command.raw on 172.23.122.130: systemctl start couchbase-server.service [2021-02-22 17:50:30,441] - [remote_util] [139837055153920] - INFO - command executed successfully with root [2021-02-22 17:50:30,442] - [remote_util] [139837055153920] - INFO - 172.23.122.130:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-22 17:50:35,448] - [remote_util] [139837055153920] - INFO - running command.raw on 172.23.122.130: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-22 17:50:35,473] - [remote_util] [139837055153920] - INFO - command executed successfully with root [2021-02-22 17:50:35,474] - [remote_util] [139837055153920] - INFO - Couchbase server status: [' Process: 964 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-22 17:50:35,474] - [remote_util] [139837055153920] - INFO - Checking for process beam.smp on linux [2021-02-22 17:50:35,544] - [remote_util] [139837055153920] - INFO - process beam.smp is running on 172.23.122.130: with pid 1110 [2021-02-22 17:50:35,545] - [basetestcase] [139837055153920] - INFO - Couchbase started [2021-02-22 17:50:35,546] - [basetestcase] [139837055153920] - INFO - sleep for 10 secs. ... [2021-02-22 17:50:45,686] - [remote_util] [139837055153920] - INFO - SSH Connecting to 172.23.123.42 with username:root, attempt#1 of 5 [2021-02-22 17:50:45,689] - [remote_util] [139837088724736] - INFO - SSH Connecting to 172.23.123.44 with username:root, attempt#1 of 5 [2021-02-22 17:50:45,699] - [remote_util] [139837063546624] - INFO - SSH Connecting to 172.23.122.132 with username:root, attempt#1 of 5 [2021-02-22 17:50:45,703] - [remote_util] [139837046761216] - INFO - SSH Connecting to 172.23.122.133 with username:root, attempt#1 of 5 [2021-02-22 17:50:45,707] - [remote_util] [139836451174144] - INFO - SSH Connecting to 172.23.122.131 with username:root, attempt#1 of 5 [2021-02-22 17:50:45,709] - [remote_util] [139837038368512] - INFO - SSH Connecting to 172.23.122.134 with username:root, attempt#1 of 5 [2021-02-22 17:50:45,719] - [remote_util] [139835075458816] - INFO - SSH Connecting to 172.23.122.180 with username:root, attempt#1 of 5 [2021-02-22 17:50:45,727] - [remote_util] [139835050280704] - INFO - SSH Connecting to 172.23.122.130 with username:root, attempt#1 of 5 [2021-02-22 17:50:45,818] - [remote_util] [139836451174144] - INFO - SSH Connected to 172.23.122.131 as root [2021-02-22 17:50:45,830] - [remote_util] [139835075458816] - INFO - SSH Connected to 172.23.122.180 as root [2021-02-22 17:50:45,837] - [remote_util] [139835050280704] - INFO - SSH Connected to 172.23.122.130 as root [2021-02-22 17:50:45,841] - [remote_util] [139837055153920] - INFO - SSH Connected to 172.23.123.42 as root [2021-02-22 17:50:45,842] - [remote_util] [139837088724736] - INFO - SSH Connected to 172.23.123.44 as root [2021-02-22 17:50:45,846] - [remote_util] [139837063546624] - INFO - SSH Connected to 172.23.122.132 as root [2021-02-22 17:50:45,852] - [remote_util] [139837046761216] - INFO - SSH Connected to 172.23.122.133 as root [2021-02-22 17:50:45,854] - [remote_util] [139837038368512] - INFO - SSH Connected to 172.23.122.134 as root [2021-02-22 17:50:46,226] - [remote_util] [139835050280704] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 17:50:46,228] - [remote_util] [139837055153920] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 17:50:46,230] - [remote_util] [139835075458816] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 17:50:46,252] - [remote_util] [139837063546624] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 17:50:46,255] - [remote_util] [139836451174144] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 17:50:46,275] - [remote_util] [139837046761216] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 17:50:46,279] - [remote_util] [139837088724736] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 17:50:46,297] - [remote_util] [139837038368512] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 17:50:46,604] - [remote_util] [139837088724736] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 17:50:46,605] - [remote_util] [139837088724736] - INFO - running command.raw on 172.23.123.44: /opt/couchbase/bin/cbcollect_info 172.23.123.44-20210222-1750-diag.zip [2021-02-22 17:50:46,609] - [remote_util] [139837055153920] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 17:50:46,612] - [remote_util] [139837055153920] - INFO - running command.raw on 172.23.123.42: /opt/couchbase/bin/cbcollect_info 172.23.123.42-20210222-1750-diag.zip [2021-02-22 17:50:46,613] - [remote_util] [139837038368512] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 17:50:46,615] - [remote_util] [139837038368512] - INFO - running command.raw on 172.23.122.134: /opt/couchbase/bin/cbcollect_info 172.23.122.134-20210222-1750-diag.zip [2021-02-22 17:50:46,620] - [remote_util] [139835050280704] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 17:50:46,622] - [remote_util] [139835050280704] - INFO - running command.raw on 172.23.122.130: /opt/couchbase/bin/cbcollect_info 172.23.122.130-20210222-1750-diag.zip [2021-02-22 17:50:46,630] - [remote_util] [139835075458816] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 17:50:46,632] - [remote_util] [139835075458816] - INFO - running command.raw on 172.23.122.180: /opt/couchbase/bin/cbcollect_info 172.23.122.180-20210222-1750-diag.zip [2021-02-22 17:50:46,635] - [remote_util] [139837063546624] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 17:50:46,636] - [remote_util] [139837063546624] - INFO - running command.raw on 172.23.122.132: /opt/couchbase/bin/cbcollect_info 172.23.122.132-20210222-1750-diag.zip [2021-02-22 17:50:46,640] - [remote_util] [139836451174144] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 17:50:46,644] - [remote_util] [139836451174144] - INFO - running command.raw on 172.23.122.131: /opt/couchbase/bin/cbcollect_info 172.23.122.131-20210222-1750-diag.zip [2021-02-22 17:50:46,645] - [remote_util] [139837046761216] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 17:50:46,649] - [remote_util] [139837046761216] - INFO - running command.raw on 172.23.122.133: /opt/couchbase/bin/cbcollect_info 172.23.122.133-20210222-1750-diag.zip [2021-02-22 17:52:12,393] - [remote_util] [139835050280704] - INFO - command executed successfully with root [2021-02-22 17:52:13,262] - [remote_util] [139835050280704] - INFO - running command.raw on 172.23.122.130: rm -f /root/172.23.122.130-20210222-1750-diag.zip [2021-02-22 17:52:13,318] - [remote_util] [139835050280704] - INFO - command executed successfully with root [2021-02-22 17:52:15,229] - [remote_util] [139837088724736] - INFO - command executed successfully with root [2021-02-22 17:52:16,873] - [remote_util] [139837088724736] - INFO - running command.raw on 172.23.123.44: rm -f /root/172.23.123.44-20210222-1750-diag.zip [2021-02-22 17:52:16,933] - [remote_util] [139837088724736] - INFO - command executed successfully with root [2021-02-22 17:52:19,018] - [remote_util] [139837038368512] - INFO - command executed successfully with root [2021-02-22 17:52:20,233] - [remote_util] [139837038368512] - INFO - running command.raw on 172.23.122.134: rm -f /root/172.23.122.134-20210222-1750-diag.zip [2021-02-22 17:52:20,290] - [remote_util] [139837038368512] - INFO - command executed successfully with root [2021-02-22 17:52:20,376] - [remote_util] [139837063546624] - INFO - command executed successfully with root [2021-02-22 17:52:21,873] - [remote_util] [139837046761216] - INFO - command executed successfully with root [2021-02-22 17:52:21,963] - [remote_util] [139837063546624] - INFO - running command.raw on 172.23.122.132: rm -f /root/172.23.122.132-20210222-1750-diag.zip [2021-02-22 17:52:22,020] - [remote_util] [139837063546624] - INFO - command executed successfully with root [2021-02-22 17:52:23,603] - [remote_util] [139837046761216] - INFO - running command.raw on 172.23.122.133: rm -f /root/172.23.122.133-20210222-1750-diag.zip [2021-02-22 17:52:23,662] - [remote_util] [139837046761216] - INFO - command executed successfully with root [2021-02-22 17:52:25,629] - [remote_util] [139837055153920] - INFO - command executed successfully with root [2021-02-22 17:52:26,698] - [remote_util] [139835075458816] - INFO - command executed successfully with root [2021-02-22 17:52:28,081] - [remote_util] [139836451174144] - INFO - command executed successfully with root [2021-02-22 17:52:30,164] - [remote_util] [139837055153920] - INFO - running command.raw on 172.23.123.42: rm -f /root/172.23.123.42-20210222-1750-diag.zip [2021-02-22 17:52:30,227] - [remote_util] [139837055153920] - INFO - command executed successfully with root [2021-02-22 17:52:30,646] - [remote_util] [139835075458816] - INFO - running command.raw on 172.23.122.180: rm -f /root/172.23.122.180-20210222-1750-diag.zip [2021-02-22 17:52:30,706] - [remote_util] [139835075458816] - INFO - command executed successfully with root [2021-02-22 17:52:30,838] - [remote_util] [139836451174144] - INFO - running command.raw on 172.23.122.131: rm -f /root/172.23.122.131-20210222-1750-diag.zip [2021-02-22 17:52:30,895] - [remote_util] [139836451174144] - INFO - command executed successfully with root [2021-02-22 17:52:31,053] - [eventing_base] [139837458200384] - INFO - Starting Test: suite_tearDown [2021-02-22 17:52:31,057] - [remote_util] [139837458200384] - INFO - SSH Connecting to 172.23.123.42 with username:root, attempt#1 of 5 [2021-02-22 17:52:31,167] - [remote_util] [139837458200384] - INFO - SSH Connected to 172.23.123.42 as root [2021-02-22 17:52:31,437] - [remote_util] [139837458200384] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 17:52:31,748] - [remote_util] [139837458200384] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 17:52:31,754] - [rest_client] [139837458200384] - INFO - -->is_ns_server_running? [2021-02-22 17:52:31,767] - [rest_client] [139837458200384] - INFO - Node version in cluster 7.0.0-4514-enterprise [2021-02-22 17:52:31,775] - [rest_client] [139837458200384] - ERROR - GET http://172.23.123.42:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-02-22 17:52:31,776] - [rest_client] [139837458200384] - INFO - Node versions in cluster [] [2021-02-22 17:52:31,776] - [basetestcase] [139837458200384] - INFO - ============== basetestcase setup was started for test #8 suite_tearDown============== [2021-02-22 17:52:31,783] - [rest_client] [139837458200384] - ERROR - GET http://172.23.123.42:8091/pools/default/ body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-02-22 17:52:31,801] - [bucket_helper] [139837458200384] - INFO - Could not find any buckets for node 172.23.123.42, nothing to delete [2021-02-22 17:52:31,813] - [bucket_helper] [139837458200384] - INFO - Could not find any buckets for node 172.23.123.44, nothing to delete [2021-02-22 17:52:31,825] - [bucket_helper] [139837458200384] - INFO - Could not find any buckets for node 172.23.122.132, nothing to delete [2021-02-22 17:52:31,836] - [bucket_helper] [139837458200384] - INFO - Could not find any buckets for node 172.23.122.133, nothing to delete [2021-02-22 17:52:31,848] - [bucket_helper] [139837458200384] - INFO - Could not find any buckets for node 172.23.122.134, nothing to delete [2021-02-22 17:52:31,860] - [bucket_helper] [139837458200384] - INFO - Could not find any buckets for node 172.23.122.131, nothing to delete [2021-02-22 17:52:31,872] - [bucket_helper] [139837458200384] - INFO - Could not find any buckets for node 172.23.122.180, nothing to delete [2021-02-22 17:52:31,885] - [bucket_helper] [139837458200384] - INFO - Could not find any buckets for node 172.23.122.130, nothing to delete [2021-02-22 17:52:31,889] - [rest_client] [139837458200384] - INFO - -->is_ns_server_running? [2021-02-22 17:52:31,903] - [cluster_helper] [139837458200384] - INFO - waiting for ns_server @ 172.23.123.42:8091 [2021-02-22 17:52:31,903] - [rest_client] [139837458200384] - INFO - -->is_ns_server_running? [2021-02-22 17:52:31,907] - [cluster_helper] [139837458200384] - INFO - ns_server @ 172.23.123.42:8091 is running [2021-02-22 17:52:31,912] - [cluster_helper] [139837458200384] - INFO - waiting for ns_server @ 172.23.123.44:8091 [2021-02-22 17:52:31,912] - [rest_client] [139837458200384] - INFO - -->is_ns_server_running? [2021-02-22 17:52:31,916] - [cluster_helper] [139837458200384] - INFO - ns_server @ 172.23.123.44:8091 is running [2021-02-22 17:52:31,920] - [cluster_helper] [139837458200384] - INFO - waiting for ns_server @ 172.23.122.132:8091 [2021-02-22 17:52:31,921] - [rest_client] [139837458200384] - INFO - -->is_ns_server_running? [2021-02-22 17:52:31,925] - [cluster_helper] [139837458200384] - INFO - ns_server @ 172.23.122.132:8091 is running [2021-02-22 17:52:31,929] - [cluster_helper] [139837458200384] - INFO - waiting for ns_server @ 172.23.122.133:8091 [2021-02-22 17:52:31,929] - [rest_client] [139837458200384] - INFO - -->is_ns_server_running? [2021-02-22 17:52:31,933] - [cluster_helper] [139837458200384] - INFO - ns_server @ 172.23.122.133:8091 is running [2021-02-22 17:52:31,937] - [cluster_helper] [139837458200384] - INFO - waiting for ns_server @ 172.23.122.134:8091 [2021-02-22 17:52:31,938] - [rest_client] [139837458200384] - INFO - -->is_ns_server_running? [2021-02-22 17:52:31,942] - [cluster_helper] [139837458200384] - INFO - ns_server @ 172.23.122.134:8091 is running [2021-02-22 17:52:31,946] - [cluster_helper] [139837458200384] - INFO - waiting for ns_server @ 172.23.122.131:8091 [2021-02-22 17:52:31,946] - [rest_client] [139837458200384] - INFO - -->is_ns_server_running? [2021-02-22 17:52:31,950] - [cluster_helper] [139837458200384] - INFO - ns_server @ 172.23.122.131:8091 is running [2021-02-22 17:52:31,955] - [cluster_helper] [139837458200384] - INFO - waiting for ns_server @ 172.23.122.180:8091 [2021-02-22 17:52:31,955] - [rest_client] [139837458200384] - INFO - -->is_ns_server_running? [2021-02-22 17:52:31,959] - [cluster_helper] [139837458200384] - INFO - ns_server @ 172.23.122.180:8091 is running [2021-02-22 17:52:31,963] - [cluster_helper] [139837458200384] - INFO - waiting for ns_server @ 172.23.122.130:8091 [2021-02-22 17:52:31,963] - [rest_client] [139837458200384] - INFO - -->is_ns_server_running? [2021-02-22 17:52:31,967] - [cluster_helper] [139837458200384] - INFO - ns_server @ 172.23.122.130:8091 is running [2021-02-22 17:52:31,968] - [basetestcase] [139837458200384] - INFO - initializing cluster [2021-02-22 17:52:31,971] - [remote_util] [139837458200384] - INFO - SSH Connecting to 172.23.123.42 with username:root, attempt#1 of 5 [2021-02-22 17:52:32,077] - [remote_util] [139837458200384] - INFO - SSH Connected to 172.23.123.42 as root [2021-02-22 17:52:32,344] - [remote_util] [139837458200384] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 17:52:32,657] - [remote_util] [139837458200384] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 17:52:32,670] - [remote_util] [139837458200384] - INFO - SSH Connecting to 172.23.123.42 with username:root, attempt#1 of 5 [2021-02-22 17:52:32,772] - [remote_util] [139837458200384] - INFO - SSH Connected to 172.23.123.42 as root [2021-02-22 17:52:33,039] - [remote_util] [139837458200384] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 17:52:33,355] - [remote_util] [139837458200384] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 17:52:33,418] - [remote_util] [139837458200384] - INFO - 172.23.123.42 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-22 17:52:33,524] - [remote_util] [139837458200384] - INFO - Running systemd command on this server [2021-02-22 17:52:33,525] - [remote_util] [139837458200384] - INFO - running command.raw on 172.23.123.42: systemctl stop couchbase-server.service [2021-02-22 17:52:35,277] - [remote_util] [139837458200384] - INFO - command executed successfully with root [2021-02-22 17:52:35,277] - [basetestcase] [139837458200384] - INFO - Couchbase stopped [2021-02-22 17:52:35,278] - [basetestcase] [139837458200384] - INFO - sleep for 5 secs. ... [2021-02-22 17:52:40,284] - [remote_util] [139837458200384] - INFO - running command.raw on 172.23.123.42: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-22 17:52:40,304] - [remote_util] [139837458200384] - INFO - command executed successfully with root [2021-02-22 17:52:40,305] - [remote_util] [139837458200384] - INFO - running command.raw on 172.23.123.42: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-22 17:52:40,359] - [remote_util] [139837458200384] - INFO - command executed successfully with root [2021-02-22 17:52:40,363] - [remote_util] [139837458200384] - INFO - SSH Connecting to 172.23.123.42 with username:root, attempt#1 of 5 [2021-02-22 17:52:40,467] - [remote_util] [139837458200384] - INFO - SSH Connected to 172.23.123.42 as root [2021-02-22 17:52:40,731] - [remote_util] [139837458200384] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 17:52:41,042] - [remote_util] [139837458200384] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 17:52:41,106] - [remote_util] [139837458200384] - INFO - 172.23.123.42 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-22 17:52:41,106] - [remote_util] [139837458200384] - INFO - Checking for process beam.smp on linux [2021-02-22 17:52:41,185] - [remote_util] [139837458200384] - INFO - Starting couchbase server [2021-02-22 17:52:41,299] - [remote_util] [139837458200384] - INFO - Running systemd command on this server [2021-02-22 17:52:41,299] - [remote_util] [139837458200384] - INFO - running command.raw on 172.23.123.42: systemctl start couchbase-server.service [2021-02-22 17:52:41,325] - [remote_util] [139837458200384] - INFO - command executed successfully with root [2021-02-22 17:52:41,325] - [remote_util] [139837458200384] - INFO - 172.23.123.42:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-22 17:52:46,331] - [remote_util] [139837458200384] - INFO - running command.raw on 172.23.123.42: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-22 17:52:46,359] - [remote_util] [139837458200384] - INFO - command executed successfully with root [2021-02-22 17:52:46,359] - [remote_util] [139837458200384] - INFO - Couchbase server status: [' Process: 53373 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-22 17:52:46,360] - [remote_util] [139837458200384] - INFO - Checking for process beam.smp on linux [2021-02-22 17:52:46,434] - [remote_util] [139837458200384] - INFO - process beam.smp is running on 172.23.123.42: with pid 53547 [2021-02-22 17:52:46,434] - [basetestcase] [139837458200384] - INFO - Couchbase started [2021-02-22 17:52:46,440] - [remote_util] [139837458200384] - INFO - SSH Connecting to 172.23.123.44 with username:root, attempt#1 of 5 [2021-02-22 17:52:46,554] - [remote_util] [139837458200384] - INFO - SSH Connected to 172.23.123.44 as root [2021-02-22 17:52:46,850] - [remote_util] [139837458200384] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 17:52:47,157] - [remote_util] [139837458200384] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 17:52:47,170] - [remote_util] [139837458200384] - INFO - SSH Connecting to 172.23.123.44 with username:root, attempt#1 of 5 [2021-02-22 17:52:47,271] - [remote_util] [139837458200384] - INFO - SSH Connected to 172.23.123.44 as root [2021-02-22 17:52:47,556] - [remote_util] [139837458200384] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 17:52:47,873] - [remote_util] [139837458200384] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 17:52:47,938] - [remote_util] [139837458200384] - INFO - 172.23.123.44 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-22 17:52:48,005] - [remote_util] [139837458200384] - INFO - Running systemd command on this server [2021-02-22 17:52:48,005] - [remote_util] [139837458200384] - INFO - running command.raw on 172.23.123.44: systemctl stop couchbase-server.service [2021-02-22 17:52:49,680] - [remote_util] [139837458200384] - INFO - command executed successfully with root [2021-02-22 17:52:49,680] - [basetestcase] [139837458200384] - INFO - Couchbase stopped [2021-02-22 17:52:49,682] - [basetestcase] [139837458200384] - INFO - sleep for 5 secs. ... [2021-02-22 17:52:54,687] - [remote_util] [139837458200384] - INFO - running command.raw on 172.23.123.44: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-22 17:52:54,705] - [remote_util] [139837458200384] - INFO - command executed successfully with root [2021-02-22 17:52:54,706] - [remote_util] [139837458200384] - INFO - running command.raw on 172.23.123.44: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-22 17:52:54,761] - [remote_util] [139837458200384] - INFO - command executed successfully with root [2021-02-22 17:52:54,765] - [remote_util] [139837458200384] - INFO - SSH Connecting to 172.23.123.44 with username:root, attempt#1 of 5 [2021-02-22 17:52:54,870] - [remote_util] [139837458200384] - INFO - SSH Connected to 172.23.123.44 as root [2021-02-22 17:52:55,165] - [remote_util] [139837458200384] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 17:52:55,473] - [remote_util] [139837458200384] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 17:52:55,538] - [remote_util] [139837458200384] - INFO - 172.23.123.44 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-22 17:52:55,538] - [remote_util] [139837458200384] - INFO - Checking for process beam.smp on linux [2021-02-22 17:52:55,620] - [remote_util] [139837458200384] - INFO - Starting couchbase server [2021-02-22 17:52:55,726] - [remote_util] [139837458200384] - INFO - Running systemd command on this server [2021-02-22 17:52:55,726] - [remote_util] [139837458200384] - INFO - running command.raw on 172.23.123.44: systemctl start couchbase-server.service [2021-02-22 17:52:55,751] - [remote_util] [139837458200384] - INFO - command executed successfully with root [2021-02-22 17:52:55,751] - [remote_util] [139837458200384] - INFO - 172.23.123.44:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-22 17:53:00,756] - [remote_util] [139837458200384] - INFO - running command.raw on 172.23.123.44: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-22 17:53:00,782] - [remote_util] [139837458200384] - INFO - command executed successfully with root [2021-02-22 17:53:00,783] - [remote_util] [139837458200384] - INFO - Couchbase server status: [' Process: 17194 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-22 17:53:00,784] - [remote_util] [139837458200384] - INFO - Checking for process beam.smp on linux [2021-02-22 17:53:00,854] - [remote_util] [139837458200384] - INFO - process beam.smp is running on 172.23.123.44: with pid 17336 [2021-02-22 17:53:00,854] - [basetestcase] [139837458200384] - INFO - Couchbase started [2021-02-22 17:53:00,858] - [remote_util] [139837458200384] - INFO - SSH Connecting to 172.23.122.132 with username:root, attempt#1 of 5 [2021-02-22 17:53:00,963] - [remote_util] [139837458200384] - INFO - SSH Connected to 172.23.122.132 as root [2021-02-22 17:53:01,230] - [remote_util] [139837458200384] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 17:53:01,533] - [remote_util] [139837458200384] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 17:53:01,548] - [remote_util] [139837458200384] - INFO - SSH Connecting to 172.23.122.132 with username:root, attempt#1 of 5 [2021-02-22 17:53:01,650] - [remote_util] [139837458200384] - INFO - SSH Connected to 172.23.122.132 as root [2021-02-22 17:53:01,913] - [remote_util] [139837458200384] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 17:53:02,217] - [remote_util] [139837458200384] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 17:53:02,281] - [remote_util] [139837458200384] - INFO - 172.23.122.132 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-22 17:53:02,461] - [remote_util] [139837458200384] - INFO - Running systemd command on this server [2021-02-22 17:53:02,461] - [remote_util] [139837458200384] - INFO - running command.raw on 172.23.122.132: systemctl stop couchbase-server.service [2021-02-22 17:53:04,116] - [remote_util] [139837458200384] - INFO - command executed successfully with root [2021-02-22 17:53:04,117] - [basetestcase] [139837458200384] - INFO - Couchbase stopped [2021-02-22 17:53:04,117] - [basetestcase] [139837458200384] - INFO - sleep for 5 secs. ... [2021-02-22 17:53:09,123] - [remote_util] [139837458200384] - INFO - running command.raw on 172.23.122.132: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-22 17:53:09,139] - [remote_util] [139837458200384] - INFO - command executed successfully with root [2021-02-22 17:53:09,140] - [remote_util] [139837458200384] - INFO - running command.raw on 172.23.122.132: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-22 17:53:09,194] - [remote_util] [139837458200384] - INFO - command executed successfully with root [2021-02-22 17:53:09,198] - [remote_util] [139837458200384] - INFO - SSH Connecting to 172.23.122.132 with username:root, attempt#1 of 5 [2021-02-22 17:53:09,299] - [remote_util] [139837458200384] - INFO - SSH Connected to 172.23.122.132 as root [2021-02-22 17:53:09,570] - [remote_util] [139837458200384] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 17:53:09,871] - [remote_util] [139837458200384] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 17:53:09,934] - [remote_util] [139837458200384] - INFO - 172.23.122.132 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-22 17:53:09,935] - [remote_util] [139837458200384] - INFO - Checking for process beam.smp on linux [2021-02-22 17:53:10,012] - [remote_util] [139837458200384] - INFO - Starting couchbase server [2021-02-22 17:53:10,156] - [remote_util] [139837458200384] - INFO - Running systemd command on this server [2021-02-22 17:53:10,156] - [remote_util] [139837458200384] - INFO - running command.raw on 172.23.122.132: systemctl start couchbase-server.service [2021-02-22 17:53:10,182] - [remote_util] [139837458200384] - INFO - command executed successfully with root [2021-02-22 17:53:10,182] - [remote_util] [139837458200384] - INFO - 172.23.122.132:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-22 17:53:15,188] - [remote_util] [139837458200384] - INFO - running command.raw on 172.23.122.132: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-22 17:53:15,215] - [remote_util] [139837458200384] - INFO - command executed successfully with root [2021-02-22 17:53:15,216] - [remote_util] [139837458200384] - INFO - Couchbase server status: [' Process: 47111 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-22 17:53:15,216] - [remote_util] [139837458200384] - INFO - Checking for process beam.smp on linux [2021-02-22 17:53:15,285] - [remote_util] [139837458200384] - INFO - process beam.smp is running on 172.23.122.132: with pid 47256 [2021-02-22 17:53:15,286] - [basetestcase] [139837458200384] - INFO - Couchbase started [2021-02-22 17:53:15,290] - [remote_util] [139837458200384] - INFO - SSH Connecting to 172.23.122.133 with username:root, attempt#1 of 5 [2021-02-22 17:53:15,397] - [remote_util] [139837458200384] - INFO - SSH Connected to 172.23.122.133 as root [2021-02-22 17:53:15,684] - [remote_util] [139837458200384] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 17:53:15,994] - [remote_util] [139837458200384] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 17:53:16,008] - [remote_util] [139837458200384] - INFO - SSH Connecting to 172.23.122.133 with username:root, attempt#1 of 5 [2021-02-22 17:53:16,112] - [remote_util] [139837458200384] - INFO - SSH Connected to 172.23.122.133 as root [2021-02-22 17:53:16,392] - [remote_util] [139837458200384] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 17:53:16,699] - [remote_util] [139837458200384] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 17:53:16,762] - [remote_util] [139837458200384] - INFO - 172.23.122.133 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-22 17:53:16,944] - [remote_util] [139837458200384] - INFO - Running systemd command on this server [2021-02-22 17:53:16,944] - [remote_util] [139837458200384] - INFO - running command.raw on 172.23.122.133: systemctl stop couchbase-server.service [2021-02-22 17:53:18,672] - [remote_util] [139837458200384] - INFO - command executed successfully with root [2021-02-22 17:53:18,673] - [basetestcase] [139837458200384] - INFO - Couchbase stopped [2021-02-22 17:53:18,674] - [basetestcase] [139837458200384] - INFO - sleep for 5 secs. ... [2021-02-22 17:53:23,679] - [remote_util] [139837458200384] - INFO - running command.raw on 172.23.122.133: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-22 17:53:23,695] - [remote_util] [139837458200384] - INFO - command executed successfully with root [2021-02-22 17:53:23,695] - [remote_util] [139837458200384] - INFO - running command.raw on 172.23.122.133: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-22 17:53:23,748] - [remote_util] [139837458200384] - INFO - command executed successfully with root [2021-02-22 17:53:23,752] - [remote_util] [139837458200384] - INFO - SSH Connecting to 172.23.122.133 with username:root, attempt#1 of 5 [2021-02-22 17:53:23,853] - [remote_util] [139837458200384] - INFO - SSH Connected to 172.23.122.133 as root [2021-02-22 17:53:24,133] - [remote_util] [139837458200384] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 17:53:24,481] - [remote_util] [139837458200384] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 17:53:24,545] - [remote_util] [139837458200384] - INFO - 172.23.122.133 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-22 17:53:24,545] - [remote_util] [139837458200384] - INFO - Checking for process beam.smp on linux [2021-02-22 17:53:24,623] - [remote_util] [139837458200384] - INFO - Starting couchbase server [2021-02-22 17:53:24,767] - [remote_util] [139837458200384] - INFO - Running systemd command on this server [2021-02-22 17:53:24,768] - [remote_util] [139837458200384] - INFO - running command.raw on 172.23.122.133: systemctl start couchbase-server.service [2021-02-22 17:53:24,791] - [remote_util] [139837458200384] - INFO - command executed successfully with root [2021-02-22 17:53:24,792] - [remote_util] [139837458200384] - INFO - 172.23.122.133:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-22 17:53:29,797] - [remote_util] [139837458200384] - INFO - running command.raw on 172.23.122.133: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-22 17:53:29,821] - [remote_util] [139837458200384] - INFO - command executed successfully with root [2021-02-22 17:53:29,821] - [remote_util] [139837458200384] - INFO - Couchbase server status: [' Process: 51071 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-22 17:53:29,822] - [remote_util] [139837458200384] - INFO - Checking for process beam.smp on linux [2021-02-22 17:53:29,890] - [remote_util] [139837458200384] - INFO - process beam.smp is running on 172.23.122.133: with pid 51216 [2021-02-22 17:53:29,890] - [basetestcase] [139837458200384] - INFO - Couchbase started [2021-02-22 17:53:29,894] - [remote_util] [139837458200384] - INFO - SSH Connecting to 172.23.122.134 with username:root, attempt#1 of 5 [2021-02-22 17:53:29,996] - [remote_util] [139837458200384] - INFO - SSH Connected to 172.23.122.134 as root [2021-02-22 17:53:30,276] - [remote_util] [139837458200384] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 17:53:30,581] - [remote_util] [139837458200384] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 17:53:30,594] - [remote_util] [139837458200384] - INFO - SSH Connecting to 172.23.122.134 with username:root, attempt#1 of 5 [2021-02-22 17:53:30,696] - [remote_util] [139837458200384] - INFO - SSH Connected to 172.23.122.134 as root [2021-02-22 17:53:30,965] - [remote_util] [139837458200384] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 17:53:31,268] - [remote_util] [139837458200384] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 17:53:31,331] - [remote_util] [139837458200384] - INFO - 172.23.122.134 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-22 17:53:31,474] - [remote_util] [139837458200384] - INFO - Running systemd command on this server [2021-02-22 17:53:31,474] - [remote_util] [139837458200384] - INFO - running command.raw on 172.23.122.134: systemctl stop couchbase-server.service [2021-02-22 17:53:33,317] - [remote_util] [139837458200384] - INFO - command executed successfully with root [2021-02-22 17:53:33,317] - [basetestcase] [139837458200384] - INFO - Couchbase stopped [2021-02-22 17:53:33,318] - [basetestcase] [139837458200384] - INFO - sleep for 5 secs. ... [2021-02-22 17:53:38,322] - [remote_util] [139837458200384] - INFO - running command.raw on 172.23.122.134: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-22 17:53:38,339] - [remote_util] [139837458200384] - INFO - command executed successfully with root [2021-02-22 17:53:38,339] - [remote_util] [139837458200384] - INFO - running command.raw on 172.23.122.134: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-22 17:53:38,392] - [remote_util] [139837458200384] - INFO - command executed successfully with root [2021-02-22 17:53:38,396] - [remote_util] [139837458200384] - INFO - SSH Connecting to 172.23.122.134 with username:root, attempt#1 of 5 [2021-02-22 17:53:38,495] - [remote_util] [139837458200384] - INFO - SSH Connected to 172.23.122.134 as root [2021-02-22 17:53:38,769] - [remote_util] [139837458200384] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 17:53:39,069] - [remote_util] [139837458200384] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 17:53:39,133] - [remote_util] [139837458200384] - INFO - 172.23.122.134 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-22 17:53:39,133] - [remote_util] [139837458200384] - INFO - Checking for process beam.smp on linux [2021-02-22 17:53:39,209] - [remote_util] [139837458200384] - INFO - Starting couchbase server [2021-02-22 17:53:39,351] - [remote_util] [139837458200384] - INFO - Running systemd command on this server [2021-02-22 17:53:39,352] - [remote_util] [139837458200384] - INFO - running command.raw on 172.23.122.134: systemctl start couchbase-server.service [2021-02-22 17:53:39,374] - [remote_util] [139837458200384] - INFO - command executed successfully with root [2021-02-22 17:53:39,375] - [remote_util] [139837458200384] - INFO - 172.23.122.134:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-22 17:53:44,379] - [remote_util] [139837458200384] - INFO - running command.raw on 172.23.122.134: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-22 17:53:44,407] - [remote_util] [139837458200384] - INFO - command executed successfully with root [2021-02-22 17:53:44,407] - [remote_util] [139837458200384] - INFO - Couchbase server status: [' Process: 3686 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-22 17:53:44,408] - [remote_util] [139837458200384] - INFO - Checking for process beam.smp on linux [2021-02-22 17:53:44,477] - [remote_util] [139837458200384] - INFO - process beam.smp is running on 172.23.122.134: with pid 3832 [2021-02-22 17:53:44,478] - [basetestcase] [139837458200384] - INFO - Couchbase started [2021-02-22 17:53:44,481] - [remote_util] [139837458200384] - INFO - SSH Connecting to 172.23.122.131 with username:root, attempt#1 of 5 [2021-02-22 17:53:44,588] - [remote_util] [139837458200384] - INFO - SSH Connected to 172.23.122.131 as root [2021-02-22 17:53:44,886] - [remote_util] [139837458200384] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 17:53:45,227] - [remote_util] [139837458200384] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 17:53:45,240] - [remote_util] [139837458200384] - INFO - SSH Connecting to 172.23.122.131 with username:root, attempt#1 of 5 [2021-02-22 17:53:45,339] - [remote_util] [139837458200384] - INFO - SSH Connected to 172.23.122.131 as root [2021-02-22 17:53:45,629] - [remote_util] [139837458200384] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 17:53:45,933] - [remote_util] [139837458200384] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 17:53:45,996] - [remote_util] [139837458200384] - INFO - 172.23.122.131 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-22 17:53:46,101] - [remote_util] [139837458200384] - INFO - Running systemd command on this server [2021-02-22 17:53:46,101] - [remote_util] [139837458200384] - INFO - running command.raw on 172.23.122.131: systemctl stop couchbase-server.service [2021-02-22 17:53:47,719] - [remote_util] [139837458200384] - INFO - command executed successfully with root [2021-02-22 17:53:47,719] - [basetestcase] [139837458200384] - INFO - Couchbase stopped [2021-02-22 17:53:47,721] - [basetestcase] [139837458200384] - INFO - sleep for 5 secs. ... [2021-02-22 17:53:52,726] - [remote_util] [139837458200384] - INFO - running command.raw on 172.23.122.131: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-22 17:53:52,744] - [remote_util] [139837458200384] - INFO - command executed successfully with root [2021-02-22 17:53:52,745] - [remote_util] [139837458200384] - INFO - running command.raw on 172.23.122.131: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-22 17:53:52,800] - [remote_util] [139837458200384] - INFO - command executed successfully with root [2021-02-22 17:53:52,804] - [remote_util] [139837458200384] - INFO - SSH Connecting to 172.23.122.131 with username:root, attempt#1 of 5 [2021-02-22 17:53:52,909] - [remote_util] [139837458200384] - INFO - SSH Connected to 172.23.122.131 as root [2021-02-22 17:53:53,210] - [remote_util] [139837458200384] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 17:53:53,512] - [remote_util] [139837458200384] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 17:53:53,574] - [remote_util] [139837458200384] - INFO - 172.23.122.131 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-22 17:53:53,574] - [remote_util] [139837458200384] - INFO - Checking for process beam.smp on linux [2021-02-22 17:53:53,654] - [remote_util] [139837458200384] - INFO - Starting couchbase server [2021-02-22 17:53:53,758] - [remote_util] [139837458200384] - INFO - Running systemd command on this server [2021-02-22 17:53:53,759] - [remote_util] [139837458200384] - INFO - running command.raw on 172.23.122.131: systemctl start couchbase-server.service [2021-02-22 17:53:53,782] - [remote_util] [139837458200384] - INFO - command executed successfully with root [2021-02-22 17:53:53,782] - [remote_util] [139837458200384] - INFO - 172.23.122.131:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-22 17:53:58,784] - [remote_util] [139837458200384] - INFO - running command.raw on 172.23.122.131: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-22 17:53:58,812] - [remote_util] [139837458200384] - INFO - command executed successfully with root [2021-02-22 17:53:58,812] - [remote_util] [139837458200384] - INFO - Couchbase server status: [' Process: 101364 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-22 17:53:58,813] - [remote_util] [139837458200384] - INFO - Checking for process beam.smp on linux [2021-02-22 17:53:58,883] - [remote_util] [139837458200384] - INFO - process beam.smp is running on 172.23.122.131: with pid 101509 [2021-02-22 17:53:58,884] - [basetestcase] [139837458200384] - INFO - Couchbase started [2021-02-22 17:53:58,887] - [remote_util] [139837458200384] - INFO - SSH Connecting to 172.23.122.180 with username:root, attempt#1 of 5 [2021-02-22 17:53:58,994] - [remote_util] [139837458200384] - INFO - SSH Connected to 172.23.122.180 as root [2021-02-22 17:53:59,281] - [remote_util] [139837458200384] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 17:53:59,600] - [remote_util] [139837458200384] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 17:53:59,617] - [remote_util] [139837458200384] - INFO - SSH Connecting to 172.23.122.180 with username:root, attempt#1 of 5 [2021-02-22 17:53:59,734] - [remote_util] [139837458200384] - INFO - SSH Connected to 172.23.122.180 as root [2021-02-22 17:54:00,085] - [remote_util] [139837458200384] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 17:54:00,487] - [remote_util] [139837458200384] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 17:54:00,584] - [remote_util] [139837458200384] - INFO - 172.23.122.180 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-22 17:54:00,756] - [remote_util] [139837458200384] - INFO - Running systemd command on this server [2021-02-22 17:54:00,757] - [remote_util] [139837458200384] - INFO - running command.raw on 172.23.122.180: systemctl stop couchbase-server.service [2021-02-22 17:54:02,451] - [remote_util] [139837458200384] - INFO - command executed successfully with root [2021-02-22 17:54:02,451] - [basetestcase] [139837458200384] - INFO - Couchbase stopped [2021-02-22 17:54:02,452] - [basetestcase] [139837458200384] - INFO - sleep for 5 secs. ... [2021-02-22 17:54:07,458] - [remote_util] [139837458200384] - INFO - running command.raw on 172.23.122.180: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-22 17:54:07,476] - [remote_util] [139837458200384] - INFO - command executed successfully with root [2021-02-22 17:54:07,476] - [remote_util] [139837458200384] - INFO - running command.raw on 172.23.122.180: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-22 17:54:07,532] - [remote_util] [139837458200384] - INFO - command executed successfully with root [2021-02-22 17:54:07,536] - [remote_util] [139837458200384] - INFO - SSH Connecting to 172.23.122.180 with username:root, attempt#1 of 5 [2021-02-22 17:54:07,644] - [remote_util] [139837458200384] - INFO - SSH Connected to 172.23.122.180 as root [2021-02-22 17:54:07,932] - [remote_util] [139837458200384] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 17:54:08,253] - [remote_util] [139837458200384] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 17:54:08,318] - [remote_util] [139837458200384] - INFO - 172.23.122.180 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-22 17:54:08,318] - [remote_util] [139837458200384] - INFO - Checking for process beam.smp on linux [2021-02-22 17:54:08,399] - [remote_util] [139837458200384] - INFO - Starting couchbase server [2021-02-22 17:54:08,506] - [remote_util] [139837458200384] - INFO - Running systemd command on this server [2021-02-22 17:54:08,507] - [remote_util] [139837458200384] - INFO - running command.raw on 172.23.122.180: systemctl start couchbase-server.service [2021-02-22 17:54:08,532] - [remote_util] [139837458200384] - INFO - command executed successfully with root [2021-02-22 17:54:08,532] - [remote_util] [139837458200384] - INFO - 172.23.122.180:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-22 17:54:13,537] - [remote_util] [139837458200384] - INFO - running command.raw on 172.23.122.180: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-22 17:54:13,563] - [remote_util] [139837458200384] - INFO - command executed successfully with root [2021-02-22 17:54:13,564] - [remote_util] [139837458200384] - INFO - Couchbase server status: [' Process: 112161 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-22 17:54:13,564] - [remote_util] [139837458200384] - INFO - Checking for process beam.smp on linux [2021-02-22 17:54:13,636] - [remote_util] [139837458200384] - INFO - process beam.smp is running on 172.23.122.180: with pid 112306 [2021-02-22 17:54:13,636] - [basetestcase] [139837458200384] - INFO - Couchbase started [2021-02-22 17:54:13,641] - [remote_util] [139837458200384] - INFO - SSH Connecting to 172.23.122.130 with username:root, attempt#1 of 5 [2021-02-22 17:54:13,745] - [remote_util] [139837458200384] - INFO - SSH Connected to 172.23.122.130 as root [2021-02-22 17:54:14,006] - [remote_util] [139837458200384] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 17:54:14,347] - [remote_util] [139837458200384] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 17:54:14,363] - [remote_util] [139837458200384] - INFO - SSH Connecting to 172.23.122.130 with username:root, attempt#1 of 5 [2021-02-22 17:54:14,464] - [remote_util] [139837458200384] - INFO - SSH Connected to 172.23.122.130 as root [2021-02-22 17:54:14,722] - [remote_util] [139837458200384] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 17:54:15,025] - [remote_util] [139837458200384] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 17:54:15,087] - [remote_util] [139837458200384] - INFO - 172.23.122.130 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-22 17:54:15,189] - [remote_util] [139837458200384] - INFO - Running systemd command on this server [2021-02-22 17:54:15,189] - [remote_util] [139837458200384] - INFO - running command.raw on 172.23.122.130: systemctl stop couchbase-server.service [2021-02-22 17:54:16,913] - [remote_util] [139837458200384] - INFO - command executed successfully with root [2021-02-22 17:54:16,914] - [basetestcase] [139837458200384] - INFO - Couchbase stopped [2021-02-22 17:54:16,914] - [basetestcase] [139837458200384] - INFO - sleep for 5 secs. ... [2021-02-22 17:54:21,919] - [remote_util] [139837458200384] - INFO - running command.raw on 172.23.122.130: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-22 17:54:21,934] - [remote_util] [139837458200384] - INFO - command executed successfully with root [2021-02-22 17:54:21,934] - [remote_util] [139837458200384] - INFO - running command.raw on 172.23.122.130: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-22 17:54:21,988] - [remote_util] [139837458200384] - INFO - command executed successfully with root [2021-02-22 17:54:21,992] - [remote_util] [139837458200384] - INFO - SSH Connecting to 172.23.122.130 with username:root, attempt#1 of 5 [2021-02-22 17:54:22,088] - [remote_util] [139837458200384] - INFO - SSH Connected to 172.23.122.130 as root [2021-02-22 17:54:22,346] - [remote_util] [139837458200384] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 17:54:22,646] - [remote_util] [139837458200384] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 17:54:22,708] - [remote_util] [139837458200384] - INFO - 172.23.122.130 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-22 17:54:22,708] - [remote_util] [139837458200384] - INFO - Checking for process beam.smp on linux [2021-02-22 17:54:22,783] - [remote_util] [139837458200384] - INFO - Starting couchbase server [2021-02-22 17:54:22,883] - [remote_util] [139837458200384] - INFO - Running systemd command on this server [2021-02-22 17:54:22,884] - [remote_util] [139837458200384] - INFO - running command.raw on 172.23.122.130: systemctl start couchbase-server.service [2021-02-22 17:54:22,905] - [remote_util] [139837458200384] - INFO - command executed successfully with root [2021-02-22 17:54:22,905] - [remote_util] [139837458200384] - INFO - 172.23.122.130:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-22 17:54:27,909] - [remote_util] [139837458200384] - INFO - running command.raw on 172.23.122.130: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-22 17:54:27,934] - [remote_util] [139837458200384] - INFO - command executed successfully with root [2021-02-22 17:54:27,935] - [remote_util] [139837458200384] - INFO - Couchbase server status: [' Process: 3931 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-22 17:54:27,935] - [remote_util] [139837458200384] - INFO - Checking for process beam.smp on linux [2021-02-22 17:54:28,004] - [remote_util] [139837458200384] - INFO - process beam.smp is running on 172.23.122.130: with pid 4076 [2021-02-22 17:54:28,005] - [basetestcase] [139837458200384] - INFO - Couchbase started [2021-02-22 17:54:28,006] - [basetestcase] [139837458200384] - INFO - sleep for 10 secs. ... [2021-02-22 17:54:38,202] - [task] [139835234313984] - INFO - server: ip:172.23.123.42 port:8091 ssh_username:root, nodes/self [2021-02-22 17:54:38,207] - [task] [139835234313984] - INFO - {'uptime': '110', 'memoryTotal': 4201672704, 'memoryFree': 3560022016, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.123.42:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4514-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.123.42', 'ip': '172.23.123.42', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} [2021-02-22 17:54:38,211] - [rest_client] [139835234313984] - ERROR - GET http://172.23.123.42:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-02-22 17:54:38,212] - [rest_client] [139835234313984] - INFO - pools/default params : memoryQuota=2147 [2021-02-22 17:54:38,220] - [rest_client] [139835234313984] - INFO - --> init_node_services(Administrator,password,172.23.123.42,8091,['kv']) [2021-02-22 17:54:38,220] - [rest_client] [139835234313984] - INFO - /node/controller/setupServices params on 172.23.123.42: 8091:hostname=172.23.123.42&user=Administrator&password=password&services=kv [2021-02-22 17:54:38,250] - [rest_client] [139835234313984] - INFO - --> in init_cluster...Administrator,password,8091 [2021-02-22 17:54:38,250] - [rest_client] [139835234313984] - INFO - settings/web params on 172.23.123.42:8091:port=8091&username=Administrator&password=password [2021-02-22 17:54:38,308] - [rest_client] [139835234313984] - INFO - --> status:True [2021-02-22 17:54:38,311] - [remote_util] [139835234313984] - INFO - SSH Connecting to 172.23.123.42 with username:root, attempt#1 of 5 [2021-02-22 17:54:38,423] - [remote_util] [139835234313984] - INFO - SSH Connected to 172.23.123.42 as root [2021-02-22 17:54:38,698] - [remote_util] [139835234313984] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 17:54:39,014] - [remote_util] [139835234313984] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 17:54:39,026] - [remote_util] [139835234313984] - INFO - running command.raw on 172.23.123.42: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-02-22 17:54:39,084] - [remote_util] [139835234313984] - INFO - command executed successfully with root [2021-02-22 17:54:39,085] - [remote_util] [139835234313984] - INFO - ['ok'] [2021-02-22 17:54:39,088] - [rest_client] [139835234313984] - INFO - /diag/eval status on 172.23.123.42:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-22 17:54:39,093] - [rest_client] [139835234313984] - INFO - /diag/eval status on 172.23.123.42:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-22 17:54:39,099] - [rest_client] [139835234313984] - INFO - settings/indexes params : storageMode=plasma [2021-02-22 17:54:39,123] - [task] [139835234313984] - INFO - server: ip:172.23.123.44 port:8091 ssh_username:root, nodes/self [2021-02-22 17:54:39,127] - [task] [139835234313984] - INFO - {'uptime': '100', 'memoryTotal': 4201676800, 'memoryFree': 3578654720, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.123.44:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4514-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.123.44', 'ip': '172.23.123.44', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} [2021-02-22 17:54:39,130] - [rest_client] [139835234313984] - ERROR - GET http://172.23.123.44:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-02-22 17:54:39,131] - [rest_client] [139835234313984] - INFO - pools/default params : memoryQuota=2147 [2021-02-22 17:54:39,140] - [rest_client] [139835234313984] - INFO - --> in init_cluster...Administrator,password,8091 [2021-02-22 17:54:39,141] - [rest_client] [139835234313984] - INFO - settings/web params on 172.23.123.44:8091:port=8091&username=Administrator&password=password [2021-02-22 17:54:39,197] - [rest_client] [139835234313984] - INFO - --> status:True [2021-02-22 17:54:39,200] - [remote_util] [139835234313984] - INFO - SSH Connecting to 172.23.123.44 with username:root, attempt#1 of 5 [2021-02-22 17:54:39,315] - [remote_util] [139835234313984] - INFO - SSH Connected to 172.23.123.44 as root [2021-02-22 17:54:39,615] - [remote_util] [139835234313984] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 17:54:39,969] - [remote_util] [139835234313984] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 17:54:39,981] - [remote_util] [139835234313984] - INFO - running command.raw on 172.23.123.44: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-02-22 17:54:40,039] - [remote_util] [139835234313984] - INFO - command executed successfully with root [2021-02-22 17:54:40,039] - [remote_util] [139835234313984] - INFO - ['ok'] [2021-02-22 17:54:40,043] - [rest_client] [139835234313984] - INFO - /diag/eval status on 172.23.123.44:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-22 17:54:40,046] - [rest_client] [139835234313984] - INFO - /diag/eval status on 172.23.123.44:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-22 17:54:40,051] - [rest_client] [139835234313984] - INFO - settings/indexes params : storageMode=plasma [2021-02-22 17:54:40,074] - [task] [139835234313984] - INFO - server: ip:172.23.122.132 port:8091 ssh_username:root, nodes/self [2021-02-22 17:54:40,078] - [task] [139835234313984] - INFO - {'uptime': '85', 'memoryTotal': 4201684992, 'memoryFree': 3571904512, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.122.132:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4514-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.122.132', 'ip': '172.23.122.132', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} [2021-02-22 17:54:40,082] - [rest_client] [139835234313984] - ERROR - GET http://172.23.122.132:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-02-22 17:54:40,082] - [rest_client] [139835234313984] - INFO - pools/default params : memoryQuota=2147 [2021-02-22 17:54:40,089] - [rest_client] [139835234313984] - INFO - --> in init_cluster...Administrator,password,8091 [2021-02-22 17:54:40,090] - [rest_client] [139835234313984] - INFO - settings/web params on 172.23.122.132:8091:port=8091&username=Administrator&password=password [2021-02-22 17:54:40,146] - [rest_client] [139835234313984] - INFO - --> status:True [2021-02-22 17:54:40,150] - [remote_util] [139835234313984] - INFO - SSH Connecting to 172.23.122.132 with username:root, attempt#1 of 5 [2021-02-22 17:54:40,255] - [remote_util] [139835234313984] - INFO - SSH Connected to 172.23.122.132 as root [2021-02-22 17:54:40,521] - [remote_util] [139835234313984] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 17:54:40,821] - [remote_util] [139835234313984] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 17:54:40,833] - [remote_util] [139835234313984] - INFO - running command.raw on 172.23.122.132: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-02-22 17:54:40,886] - [remote_util] [139835234313984] - INFO - command executed successfully with root [2021-02-22 17:54:40,886] - [remote_util] [139835234313984] - INFO - ['ok'] [2021-02-22 17:54:40,891] - [rest_client] [139835234313984] - INFO - /diag/eval status on 172.23.122.132:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-22 17:54:40,894] - [rest_client] [139835234313984] - INFO - /diag/eval status on 172.23.122.132:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-22 17:54:40,899] - [rest_client] [139835234313984] - INFO - settings/indexes params : storageMode=plasma [2021-02-22 17:54:40,922] - [task] [139835234313984] - INFO - server: ip:172.23.122.133 port:8091 ssh_username:root, nodes/self [2021-02-22 17:54:40,926] - [task] [139835234313984] - INFO - {'uptime': '69', 'memoryTotal': 4201684992, 'memoryFree': 3584253952, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.122.133:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4514-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.122.133', 'ip': '172.23.122.133', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} [2021-02-22 17:54:40,929] - [rest_client] [139835234313984] - ERROR - GET http://172.23.122.133:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-02-22 17:54:40,930] - [rest_client] [139835234313984] - INFO - pools/default params : memoryQuota=2147 [2021-02-22 17:54:40,937] - [rest_client] [139835234313984] - INFO - --> in init_cluster...Administrator,password,8091 [2021-02-22 17:54:40,937] - [rest_client] [139835234313984] - INFO - settings/web params on 172.23.122.133:8091:port=8091&username=Administrator&password=password [2021-02-22 17:54:40,992] - [rest_client] [139835234313984] - INFO - --> status:True [2021-02-22 17:54:40,995] - [remote_util] [139835234313984] - INFO - SSH Connecting to 172.23.122.133 with username:root, attempt#1 of 5 [2021-02-22 17:54:41,098] - [remote_util] [139835234313984] - INFO - SSH Connected to 172.23.122.133 as root [2021-02-22 17:54:41,371] - [remote_util] [139835234313984] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 17:54:41,666] - [remote_util] [139835234313984] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 17:54:41,678] - [remote_util] [139835234313984] - INFO - running command.raw on 172.23.122.133: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-02-22 17:54:41,729] - [remote_util] [139835234313984] - INFO - command executed successfully with root [2021-02-22 17:54:41,730] - [remote_util] [139835234313984] - INFO - ['ok'] [2021-02-22 17:54:41,733] - [rest_client] [139835234313984] - INFO - /diag/eval status on 172.23.122.133:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-22 17:54:41,736] - [rest_client] [139835234313984] - INFO - /diag/eval status on 172.23.122.133:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-22 17:54:41,741] - [rest_client] [139835234313984] - INFO - settings/indexes params : storageMode=plasma [2021-02-22 17:54:41,763] - [task] [139835234313984] - INFO - server: ip:172.23.122.134 port:8091 ssh_username:root, nodes/self [2021-02-22 17:54:41,767] - [task] [139835234313984] - INFO - {'uptime': '60', 'memoryTotal': 4201684992, 'memoryFree': 3580637184, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.122.134:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4514-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.122.134', 'ip': '172.23.122.134', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} [2021-02-22 17:54:41,770] - [rest_client] [139835234313984] - ERROR - GET http://172.23.122.134:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-02-22 17:54:41,771] - [rest_client] [139835234313984] - INFO - pools/default params : memoryQuota=2147 [2021-02-22 17:54:41,779] - [rest_client] [139835234313984] - INFO - --> in init_cluster...Administrator,password,8091 [2021-02-22 17:54:41,779] - [rest_client] [139835234313984] - INFO - settings/web params on 172.23.122.134:8091:port=8091&username=Administrator&password=password [2021-02-22 17:54:41,835] - [rest_client] [139835234313984] - INFO - --> status:True [2021-02-22 17:54:41,838] - [remote_util] [139835234313984] - INFO - SSH Connecting to 172.23.122.134 with username:root, attempt#1 of 5 [2021-02-22 17:54:41,946] - [remote_util] [139835234313984] - INFO - SSH Connected to 172.23.122.134 as root [2021-02-22 17:54:42,223] - [remote_util] [139835234313984] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 17:54:42,523] - [remote_util] [139835234313984] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 17:54:42,533] - [remote_util] [139835234313984] - INFO - running command.raw on 172.23.122.134: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-02-22 17:54:42,588] - [remote_util] [139835234313984] - INFO - command executed successfully with root [2021-02-22 17:54:42,588] - [remote_util] [139835234313984] - INFO - ['ok'] [2021-02-22 17:54:42,591] - [rest_client] [139835234313984] - INFO - /diag/eval status on 172.23.122.134:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-22 17:54:42,594] - [rest_client] [139835234313984] - INFO - /diag/eval status on 172.23.122.134:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-22 17:54:42,599] - [rest_client] [139835234313984] - INFO - settings/indexes params : storageMode=plasma [2021-02-22 17:54:42,620] - [task] [139835234313984] - INFO - server: ip:172.23.122.131 port:8091 ssh_username:root, nodes/self [2021-02-22 17:54:42,624] - [task] [139835234313984] - INFO - {'uptime': '44', 'memoryTotal': 4201684992, 'memoryFree': 3578961920, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.122.131:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4514-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.122.131', 'ip': '172.23.122.131', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} [2021-02-22 17:54:42,627] - [rest_client] [139835234313984] - ERROR - GET http://172.23.122.131:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-02-22 17:54:42,628] - [rest_client] [139835234313984] - INFO - pools/default params : memoryQuota=2147 [2021-02-22 17:54:42,635] - [rest_client] [139835234313984] - INFO - --> in init_cluster...Administrator,password,8091 [2021-02-22 17:54:42,635] - [rest_client] [139835234313984] - INFO - settings/web params on 172.23.122.131:8091:port=8091&username=Administrator&password=password [2021-02-22 17:54:42,692] - [rest_client] [139835234313984] - INFO - --> status:True [2021-02-22 17:54:42,695] - [remote_util] [139835234313984] - INFO - SSH Connecting to 172.23.122.131 with username:root, attempt#1 of 5 [2021-02-22 17:54:42,800] - [remote_util] [139835234313984] - INFO - SSH Connected to 172.23.122.131 as root [2021-02-22 17:54:43,091] - [remote_util] [139835234313984] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 17:54:43,434] - [remote_util] [139835234313984] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 17:54:43,446] - [remote_util] [139835234313984] - INFO - running command.raw on 172.23.122.131: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-02-22 17:54:43,498] - [remote_util] [139835234313984] - INFO - command executed successfully with root [2021-02-22 17:54:43,498] - [remote_util] [139835234313984] - INFO - ['ok'] [2021-02-22 17:54:43,502] - [rest_client] [139835234313984] - INFO - /diag/eval status on 172.23.122.131:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-22 17:54:43,506] - [rest_client] [139835234313984] - INFO - /diag/eval status on 172.23.122.131:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-22 17:54:43,511] - [rest_client] [139835234313984] - INFO - settings/indexes params : storageMode=plasma [2021-02-22 17:54:43,534] - [task] [139835234313984] - INFO - server: ip:172.23.122.180 port:8091 ssh_username:root, nodes/self [2021-02-22 17:54:43,538] - [task] [139835234313984] - INFO - {'uptime': '29', 'memoryTotal': 4201684992, 'memoryFree': 3518197760, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.122.180:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4514-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.122.180', 'ip': '172.23.122.180', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} [2021-02-22 17:54:43,541] - [rest_client] [139835234313984] - ERROR - GET http://172.23.122.180:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-02-22 17:54:43,542] - [rest_client] [139835234313984] - INFO - pools/default params : memoryQuota=2147 [2021-02-22 17:54:43,550] - [rest_client] [139835234313984] - INFO - --> in init_cluster...Administrator,password,8091 [2021-02-22 17:54:43,550] - [rest_client] [139835234313984] - INFO - settings/web params on 172.23.122.180:8091:port=8091&username=Administrator&password=password [2021-02-22 17:54:43,609] - [rest_client] [139835234313984] - INFO - --> status:True [2021-02-22 17:54:43,612] - [remote_util] [139835234313984] - INFO - SSH Connecting to 172.23.122.180 with username:root, attempt#1 of 5 [2021-02-22 17:54:43,719] - [remote_util] [139835234313984] - INFO - SSH Connected to 172.23.122.180 as root [2021-02-22 17:54:43,996] - [remote_util] [139835234313984] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 17:54:44,301] - [remote_util] [139835234313984] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 17:54:44,313] - [remote_util] [139835234313984] - INFO - running command.raw on 172.23.122.180: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-02-22 17:54:44,367] - [remote_util] [139835234313984] - INFO - command executed successfully with root [2021-02-22 17:54:44,368] - [remote_util] [139835234313984] - INFO - ['ok'] [2021-02-22 17:54:44,372] - [rest_client] [139835234313984] - INFO - /diag/eval status on 172.23.122.180:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-22 17:54:44,375] - [rest_client] [139835234313984] - INFO - /diag/eval status on 172.23.122.180:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-22 17:54:44,380] - [rest_client] [139835234313984] - INFO - settings/indexes params : storageMode=plasma [2021-02-22 17:54:44,406] - [task] [139835234313984] - INFO - server: ip:172.23.122.130 port:8091 ssh_username:root, nodes/self [2021-02-22 17:54:44,411] - [task] [139835234313984] - INFO - {'uptime': '19', 'memoryTotal': 4201689088, 'memoryFree': 3540885504, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.122.130:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4514-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@cb.local', 'ip': '172.23.122.130', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} [2021-02-22 17:54:44,414] - [rest_client] [139835234313984] - ERROR - GET http://172.23.122.130:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-02-22 17:54:44,414] - [rest_client] [139835234313984] - INFO - pools/default params : memoryQuota=2147 [2021-02-22 17:54:44,422] - [rest_client] [139835234313984] - INFO - --> in init_cluster...Administrator,password,8091 [2021-02-22 17:54:44,422] - [rest_client] [139835234313984] - INFO - settings/web params on 172.23.122.130:8091:port=8091&username=Administrator&password=password [2021-02-22 17:54:44,477] - [rest_client] [139835234313984] - INFO - --> status:True [2021-02-22 17:54:44,481] - [remote_util] [139835234313984] - INFO - SSH Connecting to 172.23.122.130 with username:root, attempt#1 of 5 [2021-02-22 17:54:44,589] - [remote_util] [139835234313984] - INFO - SSH Connected to 172.23.122.130 as root [2021-02-22 17:54:44,852] - [remote_util] [139835234313984] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 17:54:45,199] - [remote_util] [139835234313984] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 17:54:45,211] - [remote_util] [139835234313984] - INFO - running command.raw on 172.23.122.130: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-02-22 17:54:45,263] - [remote_util] [139835234313984] - INFO - command executed successfully with root [2021-02-22 17:54:45,264] - [remote_util] [139835234313984] - INFO - ['ok'] [2021-02-22 17:54:45,269] - [rest_client] [139835234313984] - INFO - /diag/eval status on 172.23.122.130:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-22 17:54:45,272] - [rest_client] [139835234313984] - INFO - /diag/eval status on 172.23.122.130:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-22 17:54:45,277] - [rest_client] [139835234313984] - INFO - settings/indexes params : storageMode=plasma [2021-02-22 17:54:45,303] - [basetestcase] [139837458200384] - INFO - **** add built-in 'cbadminbucket' user to node 172.23.123.42 **** [2021-02-22 17:54:45,319] - [rest_client] [139837458200384] - ERROR - DELETE http://172.23.123.42:8091/settings/rbac/users/local/cbadminbucket body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:password [2021-02-22 17:54:45,320] - [internal_user] [139837458200384] - INFO - Exception while deleting user. Exception is -b'"User was not found."' [2021-02-22 17:54:45,386] - [basetestcase] [139837458200384] - INFO - sleep for 5 secs. ... [2021-02-22 17:54:50,391] - [basetestcase] [139837458200384] - INFO - **** add 'admin' role to 'cbadminbucket' user **** [2021-02-22 17:54:50,407] - [basetestcase] [139837458200384] - INFO - done initializing cluster [2021-02-22 17:54:51,297] - [task] [139835234313984] - INFO - adding node 172.23.123.44:8091 to cluster [2021-02-22 17:54:51,298] - [rest_client] [139835234313984] - INFO - adding remote node @172.23.123.44:8091 to this cluster @172.23.123.42:8091 [2021-02-22 17:55:01,309] - [rest_client] [139835234313984] - INFO - rebalance progress took 10.01 seconds [2021-02-22 17:55:01,309] - [rest_client] [139835234313984] - INFO - sleep for 10 seconds after rebalance... [2021-02-22 17:55:13,382] - [task] [139835234313984] - INFO - adding node 172.23.122.132:8091 to cluster [2021-02-22 17:55:13,383] - [rest_client] [139835234313984] - INFO - adding remote node @172.23.122.132:8091 to this cluster @172.23.123.42:8091 [2021-02-22 17:55:23,401] - [rest_client] [139835234313984] - INFO - rebalance progress took 10.02 seconds [2021-02-22 17:55:23,401] - [rest_client] [139835234313984] - INFO - sleep for 10 seconds after rebalance... [2021-02-22 17:55:35,766] - [task] [139835234313984] - INFO - adding node 172.23.122.133:8091 to cluster [2021-02-22 17:55:35,766] - [rest_client] [139835234313984] - INFO - adding remote node @172.23.122.133:8091 to this cluster @172.23.123.42:8091 [2021-02-22 17:55:45,783] - [rest_client] [139835234313984] - INFO - rebalance progress took 10.02 seconds [2021-02-22 17:55:45,784] - [rest_client] [139835234313984] - INFO - sleep for 10 seconds after rebalance... [2021-02-22 17:55:57,999] - [task] [139835234313984] - INFO - adding node 172.23.122.134:8091 to cluster [2021-02-22 17:55:57,999] - [rest_client] [139835234313984] - INFO - adding remote node @172.23.122.134:8091 to this cluster @172.23.123.42:8091 [2021-02-22 17:56:08,016] - [rest_client] [139835234313984] - INFO - rebalance progress took 10.02 seconds [2021-02-22 17:56:08,016] - [rest_client] [139835234313984] - INFO - sleep for 10 seconds after rebalance... [2021-02-22 17:56:20,318] - [rest_client] [139835234313984] - INFO - rebalance params : {'knownNodes': 'ns_1@172.23.122.132,ns_1@172.23.122.133,ns_1@172.23.122.134,ns_1@172.23.123.42,ns_1@172.23.123.44', 'ejectedNodes': '', 'user': 'Administrator', 'password': 'password'} [2021-02-22 17:56:20,347] - [rest_client] [139835234313984] - INFO - rebalance operation started [2021-02-22 17:56:20,351] - [rest_client] [139835234313984] - INFO - rebalance percentage : 0.00 % [2021-02-22 17:56:20,351] - [task] [139835234313984] - INFO - Rebalance - status: running, progress: 0.00% [2021-02-22 17:56:30,377] - [task] [139835234313984] - INFO - Rebalance - status: none, progress: 100.00% [2021-02-22 17:56:30,386] - [task] [139835234313984] - INFO - rebalancing was completed with progress: 100% in 10.038559913635254 sec [2021-02-22 17:56:30,390] - [remote_util] [139837458200384] - INFO - SSH Connecting to 172.23.123.42 with username:root, attempt#1 of 5 [2021-02-22 17:56:30,498] - [remote_util] [139837458200384] - INFO - SSH Connected to 172.23.123.42 as root [2021-02-22 17:56:30,775] - [remote_util] [139837458200384] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 17:56:31,090] - [remote_util] [139837458200384] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 17:56:31,103] - [remote_util] [139837458200384] - INFO - running command.raw on 172.23.123.42: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-02-22 17:56:31,158] - [remote_util] [139837458200384] - INFO - command executed successfully with root [2021-02-22 17:56:31,158] - [remote_util] [139837458200384] - INFO - ['ok'] [2021-02-22 17:56:31,159] - [basetestcase] [139837458200384] - INFO - Enabled diag/eval for non-local hosts from 172.23.123.42 [2021-02-22 17:56:31,159] - [basetestcase] [139837458200384] - INFO - ============== basetestcase setup was finished for test #8 suite_tearDown ============== [2021-02-22 17:56:31,172] - [remote_util] [139837458200384] - INFO - SSH Connecting to 172.23.123.42 with username:root, attempt#1 of 5 [2021-02-22 17:56:31,281] - [remote_util] [139837458200384] - INFO - SSH Connected to 172.23.123.42 as root [2021-02-22 17:56:31,546] - [remote_util] [139837458200384] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 17:56:31,856] - [remote_util] [139837458200384] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 17:56:31,862] - [remote_util] [139837458200384] - INFO - SSH Connecting to 172.23.123.44 with username:root, attempt#1 of 5 [2021-02-22 17:56:31,969] - [remote_util] [139837458200384] - INFO - SSH Connected to 172.23.123.44 as root [2021-02-22 17:56:32,260] - [remote_util] [139837458200384] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 17:56:32,573] - [remote_util] [139837458200384] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 17:56:32,577] - [remote_util] [139837458200384] - INFO - SSH Connecting to 172.23.122.132 with username:root, attempt#1 of 5 [2021-02-22 17:56:32,683] - [remote_util] [139837458200384] - INFO - SSH Connected to 172.23.122.132 as root [2021-02-22 17:56:32,978] - [remote_util] [139837458200384] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 17:56:33,299] - [remote_util] [139837458200384] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 17:56:33,308] - [remote_util] [139837458200384] - INFO - SSH Connecting to 172.23.122.133 with username:root, attempt#1 of 5 [2021-02-22 17:56:33,414] - [remote_util] [139837458200384] - INFO - SSH Connected to 172.23.122.133 as root [2021-02-22 17:56:33,706] - [remote_util] [139837458200384] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 17:56:34,063] - [remote_util] [139837458200384] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 17:56:34,069] - [remote_util] [139837458200384] - INFO - SSH Connecting to 172.23.122.134 with username:root, attempt#1 of 5 [2021-02-22 17:56:34,182] - [remote_util] [139837458200384] - INFO - SSH Connected to 172.23.122.134 as root [2021-02-22 17:56:34,498] - [remote_util] [139837458200384] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 17:56:34,817] - [remote_util] [139837458200384] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 17:56:34,822] - [remote_util] [139837458200384] - INFO - SSH Connecting to 172.23.122.131 with username:root, attempt#1 of 5 [2021-02-22 17:56:34,932] - [remote_util] [139837458200384] - INFO - SSH Connected to 172.23.122.131 as root [2021-02-22 17:56:35,232] - [remote_util] [139837458200384] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 17:56:35,542] - [remote_util] [139837458200384] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 17:56:35,547] - [remote_util] [139837458200384] - INFO - SSH Connecting to 172.23.122.180 with username:root, attempt#1 of 5 [2021-02-22 17:56:35,653] - [remote_util] [139837458200384] - INFO - SSH Connected to 172.23.122.180 as root [2021-02-22 17:56:35,940] - [remote_util] [139837458200384] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 17:56:36,294] - [remote_util] [139837458200384] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 17:56:36,302] - [remote_util] [139837458200384] - INFO - SSH Connecting to 172.23.122.130 with username:root, attempt#1 of 5 [2021-02-22 17:56:36,412] - [remote_util] [139837458200384] - INFO - SSH Connected to 172.23.122.130 as root [2021-02-22 17:56:36,715] - [remote_util] [139837458200384] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 17:56:37,070] - [remote_util] [139837458200384] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 17:57:53,130] - [basetestcase] [139837458200384] - INFO - ------- Cluster statistics ------- [2021-02-22 17:57:53,130] - [basetestcase] [139837458200384] - INFO - 172.23.122.132:8091 => {'services': ['kv'], 'cpu_utilization': 0.8270676691729323, 'mem_free': 3546443776, 'mem_total': 4201684992, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-02-22 17:57:53,130] - [basetestcase] [139837458200384] - INFO - 172.23.122.133:8091 => {'services': ['eventing'], 'cpu_utilization': 0.4764292878635908, 'mem_free': 3548200960, 'mem_total': 4201684992, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-02-22 17:57:53,131] - [basetestcase] [139837458200384] - INFO - 172.23.122.134:8091 => {'services': ['index', 'n1ql'], 'cpu_utilization': 0.4522613065326633, 'mem_free': 3505020928, 'mem_total': 4201684992, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-02-22 17:57:53,131] - [basetestcase] [139837458200384] - INFO - 172.23.123.42:8091 => {'services': ['kv'], 'cpu_utilization': 0.7788944723618091, 'mem_free': 3522936832, 'mem_total': 4201672704, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-02-22 17:57:53,131] - [basetestcase] [139837458200384] - INFO - 172.23.123.44:8091 => {'services': ['kv'], 'cpu_utilization': 0.6264094211976948, 'mem_free': 3556175872, 'mem_total': 4201676800, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-02-22 17:57:53,132] - [basetestcase] [139837458200384] - INFO - --- End of cluster statistics --- [2021-02-22 17:57:53,135] - [remote_util] [139837458200384] - INFO - SSH Connecting to 172.23.123.42 with username:root, attempt#1 of 5 [2021-02-22 17:57:53,246] - [remote_util] [139837458200384] - INFO - SSH Connected to 172.23.123.42 as root [2021-02-22 17:57:53,528] - [remote_util] [139837458200384] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 17:57:53,847] - [remote_util] [139837458200384] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 17:57:56,155] - [basetestcase] [139837458200384] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.134 port:8091 ssh_username:root] [2021-02-22 17:57:56,164] - [basetestcase] [139837458200384] - INFO - list of eventing nodes in cluster: [ip:172.23.122.133 port:8091 ssh_username:root] [2021-02-22 17:57:56,170] - [rest_client] [139837458200384] - INFO - settings/indexes params : storageMode=plasma [2021-02-22 17:57:56,175] - [eventing_base] [139837458200384] - INFO - Setting the min possible memory quota so that adding mode nodes to the cluster wouldn't be a problem. [2021-02-22 17:57:56,175] - [rest_client] [139837458200384] - INFO - pools/default params : memoryQuota=330 [2021-02-22 17:57:56,183] - [rest_client] [139837458200384] - INFO - pools/default params : indexMemoryQuota=256 [2021-02-22 17:57:56,191] - [rest_client] [139837458200384] - INFO - pools/default params : eventingMemoryQuota=512 [2021-02-22 17:57:56,209] - [basetestcase] [139837458200384] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.134 port:8091 ssh_username:root] [2021-02-22 17:57:56,215] - [rest_client] [139837458200384] - INFO - pools/default params : memoryQuota=700 [2021-02-22 17:57:56,220] - [eventing_rebalance] [139837458200384] - INFO - 100 [2021-02-22 17:57:57,162] - [rest_client] [139835234313984] - INFO - http://172.23.123.42:8091/pools/default/buckets with param: name=src_bucket&ramQuotaMB=100&authType=none&replicaNumber=0&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore [2021-02-22 17:57:57,193] - [rest_client] [139835234313984] - INFO - 0.03 seconds to create bucket src_bucket [2021-02-22 17:57:57,193] - [bucket_helper] [139835234313984] - INFO - waiting for memcached bucket : src_bucket in 172.23.123.42 to accept set ops [2021-02-22 17:57:58,041] - [data_helper] [139835234313984] - INFO - creating direct client 172.23.122.132:11210 src_bucket [2021-02-22 17:57:58,149] - [data_helper] [139835234313984] - INFO - creating direct client 172.23.123.42:11210 src_bucket [2021-02-22 17:57:58,254] - [data_helper] [139835234313984] - INFO - creating direct client 172.23.123.44:11210 src_bucket [2021-02-22 17:57:58,419] - [data_helper] [139835234313984] - INFO - creating direct client 172.23.122.132:11210 src_bucket [2021-02-22 17:57:58,724] - [data_helper] [139835234313984] - INFO - creating direct client 172.23.123.42:11210 src_bucket [2021-02-22 17:57:58,851] - [data_helper] [139835234313984] - INFO - creating direct client 172.23.123.44:11210 src_bucket [2021-02-22 17:57:58,945] - [task] [139835234313984] - WARNING - vbucket map not ready after try 0 [2021-02-22 17:57:58,945] - [bucket_helper] [139835234313984] - INFO - waiting for memcached bucket : src_bucket in 172.23.123.42 to accept set ops [2021-02-22 17:57:59,269] - [data_helper] [139835234313984] - INFO - creating direct client 172.23.122.132:11210 src_bucket [2021-02-22 17:57:59,375] - [data_helper] [139835234313984] - INFO - creating direct client 172.23.123.42:11210 src_bucket [2021-02-22 17:57:59,484] - [data_helper] [139835234313984] - INFO - creating direct client 172.23.123.44:11210 src_bucket [2021-02-22 17:57:59,649] - [data_helper] [139835234313984] - INFO - creating direct client 172.23.122.132:11210 src_bucket [2021-02-22 17:57:59,933] - [data_helper] [139835234313984] - INFO - creating direct client 172.23.123.42:11210 src_bucket [2021-02-22 17:58:00,034] - [data_helper] [139835234313984] - INFO - creating direct client 172.23.123.44:11210 src_bucket [2021-02-22 17:58:00,131] - [task] [139835234313984] - WARNING - vbucket map not ready after try 1 [2021-02-22 17:58:00,131] - [bucket_helper] [139835234313984] - INFO - waiting for memcached bucket : src_bucket in 172.23.123.42 to accept set ops [2021-02-22 17:58:00,455] - [data_helper] [139835234313984] - INFO - creating direct client 172.23.122.132:11210 src_bucket [2021-02-22 17:58:00,570] - [data_helper] [139835234313984] - INFO - creating direct client 172.23.123.42:11210 src_bucket [2021-02-22 17:58:00,676] - [data_helper] [139835234313984] - INFO - creating direct client 172.23.123.44:11210 src_bucket [2021-02-22 17:58:01,007] - [data_helper] [139835234313984] - INFO - creating direct client 172.23.122.132:11210 src_bucket [2021-02-22 17:58:01,116] - [data_helper] [139835234313984] - INFO - creating direct client 172.23.123.42:11210 src_bucket [2021-02-22 17:58:01,227] - [data_helper] [139835234313984] - INFO - creating direct client 172.23.123.44:11210 src_bucket [2021-02-22 17:58:01,323] - [task] [139835234313984] - WARNING - vbucket map not ready after try 2 [2021-02-22 17:58:01,323] - [bucket_helper] [139835234313984] - INFO - waiting for memcached bucket : src_bucket in 172.23.123.42 to accept set ops [2021-02-22 17:58:01,646] - [data_helper] [139835234313984] - INFO - creating direct client 172.23.122.132:11210 src_bucket [2021-02-22 17:58:01,756] - [data_helper] [139835234313984] - INFO - creating direct client 172.23.123.42:11210 src_bucket [2021-02-22 17:58:01,859] - [data_helper] [139835234313984] - INFO - creating direct client 172.23.123.44:11210 src_bucket [2021-02-22 17:58:02,187] - [data_helper] [139835234313984] - INFO - creating direct client 172.23.122.132:11210 src_bucket [2021-02-22 17:58:02,301] - [data_helper] [139835234313984] - INFO - creating direct client 172.23.123.42:11210 src_bucket [2021-02-22 17:58:02,405] - [data_helper] [139835234313984] - INFO - creating direct client 172.23.123.44:11210 src_bucket [2021-02-22 17:58:02,508] - [task] [139835234313984] - WARNING - vbucket map not ready after try 3 [2021-02-22 17:58:02,508] - [bucket_helper] [139835234313984] - INFO - waiting for memcached bucket : src_bucket in 172.23.123.42 to accept set ops [2021-02-22 17:58:02,829] - [data_helper] [139835234313984] - INFO - creating direct client 172.23.122.132:11210 src_bucket [2021-02-22 17:58:02,934] - [data_helper] [139835234313984] - INFO - creating direct client 172.23.123.42:11210 src_bucket [2021-02-22 17:58:03,056] - [data_helper] [139835234313984] - INFO - creating direct client 172.23.123.44:11210 src_bucket [2021-02-22 17:58:03,394] - [data_helper] [139835234313984] - INFO - creating direct client 172.23.122.132:11210 src_bucket [2021-02-22 17:58:03,505] - [data_helper] [139835234313984] - INFO - creating direct client 172.23.123.42:11210 src_bucket [2021-02-22 17:58:03,619] - [data_helper] [139835234313984] - INFO - creating direct client 172.23.123.44:11210 src_bucket [2021-02-22 17:58:03,717] - [task] [139835234313984] - WARNING - vbucket map not ready after try 4 [2021-02-22 17:58:03,718] - [bucket_helper] [139835234313984] - INFO - waiting for memcached bucket : src_bucket in 172.23.123.42 to accept set ops [2021-02-22 17:58:04,004] - [data_helper] [139835234313984] - INFO - creating direct client 172.23.122.132:11210 src_bucket [2021-02-22 17:58:04,078] - [data_helper] [139835234313984] - INFO - creating direct client 172.23.123.42:11210 src_bucket [2021-02-22 17:58:04,152] - [data_helper] [139835234313984] - INFO - creating direct client 172.23.123.44:11210 src_bucket [2021-02-22 17:58:04,457] - [data_helper] [139835234313984] - INFO - creating direct client 172.23.122.132:11210 src_bucket [2021-02-22 17:58:04,540] - [data_helper] [139835234313984] - INFO - creating direct client 172.23.123.42:11210 src_bucket [2021-02-22 17:58:04,615] - [data_helper] [139835234313984] - INFO - creating direct client 172.23.123.44:11210 src_bucket [2021-02-22 17:58:04,674] - [task] [139835234313984] - WARNING - vbucket map not ready after try 5 [2021-02-22 17:58:05,707] - [rest_client] [139835234313984] - INFO - http://172.23.123.42:8091/pools/default/buckets with param: name=dst_bucket&ramQuotaMB=100&authType=none&replicaNumber=0&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore [2021-02-22 17:58:05,737] - [rest_client] [139835234313984] - INFO - 0.03 seconds to create bucket dst_bucket [2021-02-22 17:58:05,738] - [bucket_helper] [139835234313984] - INFO - waiting for memcached bucket : dst_bucket in 172.23.123.42 to accept set ops [2021-02-22 17:58:06,409] - [data_helper] [139835234313984] - INFO - creating direct client 172.23.122.132:11210 dst_bucket [2021-02-22 17:58:06,514] - [data_helper] [139835234313984] - INFO - creating direct client 172.23.123.42:11210 dst_bucket [2021-02-22 17:58:06,640] - [data_helper] [139835234313984] - INFO - creating direct client 172.23.123.44:11210 dst_bucket [2021-02-22 17:58:06,968] - [data_helper] [139835234313984] - INFO - creating direct client 172.23.122.132:11210 dst_bucket [2021-02-22 17:58:07,070] - [data_helper] [139835234313984] - INFO - creating direct client 172.23.123.42:11210 dst_bucket [2021-02-22 17:58:07,179] - [data_helper] [139835234313984] - INFO - creating direct client 172.23.123.44:11210 dst_bucket [2021-02-22 17:58:07,441] - [task] [139835234313984] - WARNING - vbucket map not ready after try 0 [2021-02-22 17:58:07,441] - [bucket_helper] [139835234313984] - INFO - waiting for memcached bucket : dst_bucket in 172.23.123.42 to accept set ops [2021-02-22 17:58:07,610] - [data_helper] [139835234313984] - INFO - creating direct client 172.23.122.132:11210 dst_bucket [2021-02-22 17:58:07,734] - [data_helper] [139835234313984] - INFO - creating direct client 172.23.123.42:11210 dst_bucket [2021-02-22 17:58:08,002] - [data_helper] [139835234313984] - INFO - creating direct client 172.23.123.44:11210 dst_bucket [2021-02-22 17:58:08,158] - [data_helper] [139835234313984] - INFO - creating direct client 172.23.122.132:11210 dst_bucket [2021-02-22 17:58:08,258] - [data_helper] [139835234313984] - INFO - creating direct client 172.23.123.42:11210 dst_bucket [2021-02-22 17:58:08,357] - [data_helper] [139835234313984] - INFO - creating direct client 172.23.123.44:11210 dst_bucket [2021-02-22 17:58:08,615] - [task] [139835234313984] - WARNING - vbucket map not ready after try 1 [2021-02-22 17:58:08,616] - [bucket_helper] [139835234313984] - INFO - waiting for memcached bucket : dst_bucket in 172.23.123.42 to accept set ops [2021-02-22 17:58:08,767] - [data_helper] [139835234313984] - INFO - creating direct client 172.23.122.132:11210 dst_bucket [2021-02-22 17:58:08,874] - [data_helper] [139835234313984] - INFO - creating direct client 172.23.123.42:11210 dst_bucket [2021-02-22 17:58:09,162] - [data_helper] [139835234313984] - INFO - creating direct client 172.23.123.44:11210 dst_bucket [2021-02-22 17:58:09,318] - [data_helper] [139835234313984] - INFO - creating direct client 172.23.122.132:11210 dst_bucket [2021-02-22 17:58:09,427] - [data_helper] [139835234313984] - INFO - creating direct client 172.23.123.42:11210 dst_bucket [2021-02-22 17:58:09,535] - [data_helper] [139835234313984] - INFO - creating direct client 172.23.123.44:11210 dst_bucket [2021-02-22 17:58:09,791] - [task] [139835234313984] - WARNING - vbucket map not ready after try 2 [2021-02-22 17:58:09,791] - [bucket_helper] [139835234313984] - INFO - waiting for memcached bucket : dst_bucket in 172.23.123.42 to accept set ops [2021-02-22 17:58:09,945] - [data_helper] [139835234313984] - INFO - creating direct client 172.23.122.132:11210 dst_bucket [2021-02-22 17:58:10,054] - [data_helper] [139835234313984] - INFO - creating direct client 172.23.123.42:11210 dst_bucket [2021-02-22 17:58:10,335] - [data_helper] [139835234313984] - INFO - creating direct client 172.23.123.44:11210 dst_bucket [2021-02-22 17:58:10,494] - [data_helper] [139835234313984] - INFO - creating direct client 172.23.122.132:11210 dst_bucket [2021-02-22 17:58:10,600] - [data_helper] [139835234313984] - INFO - creating direct client 172.23.123.42:11210 dst_bucket [2021-02-22 17:58:10,888] - [data_helper] [139835234313984] - INFO - creating direct client 172.23.123.44:11210 dst_bucket [2021-02-22 17:58:10,974] - [task] [139835234313984] - WARNING - vbucket map not ready after try 3 [2021-02-22 17:58:10,975] - [bucket_helper] [139835234313984] - INFO - waiting for memcached bucket : dst_bucket in 172.23.123.42 to accept set ops [2021-02-22 17:58:11,139] - [data_helper] [139835234313984] - INFO - creating direct client 172.23.122.132:11210 dst_bucket [2021-02-22 17:58:11,247] - [data_helper] [139835234313984] - INFO - creating direct client 172.23.123.42:11210 dst_bucket [2021-02-22 17:58:11,517] - [data_helper] [139835234313984] - INFO - creating direct client 172.23.123.44:11210 dst_bucket [2021-02-22 17:58:11,643] - [data_helper] [139835234313984] - INFO - creating direct client 172.23.122.132:11210 dst_bucket [2021-02-22 17:58:11,713] - [data_helper] [139835234313984] - INFO - creating direct client 172.23.123.42:11210 dst_bucket [2021-02-22 17:58:11,962] - [data_helper] [139835234313984] - INFO - creating direct client 172.23.123.44:11210 dst_bucket [2021-02-22 17:58:12,017] - [task] [139835234313984] - WARNING - vbucket map not ready after try 4 [2021-02-22 17:58:12,018] - [bucket_helper] [139835234313984] - INFO - waiting for memcached bucket : dst_bucket in 172.23.123.42 to accept set ops [2021-02-22 17:58:12,115] - [data_helper] [139835234313984] - INFO - creating direct client 172.23.122.132:11210 dst_bucket [2021-02-22 17:58:12,190] - [data_helper] [139835234313984] - INFO - creating direct client 172.23.123.42:11210 dst_bucket [2021-02-22 17:58:12,438] - [data_helper] [139835234313984] - INFO - creating direct client 172.23.123.44:11210 dst_bucket [2021-02-22 17:58:12,543] - [data_helper] [139835234313984] - INFO - creating direct client 172.23.122.132:11210 dst_bucket [2021-02-22 17:58:12,620] - [data_helper] [139835234313984] - INFO - creating direct client 172.23.123.42:11210 dst_bucket [2021-02-22 17:58:12,857] - [data_helper] [139835234313984] - INFO - creating direct client 172.23.123.44:11210 dst_bucket [2021-02-22 17:58:12,919] - [task] [139835234313984] - WARNING - vbucket map not ready after try 5 [2021-02-22 17:58:13,952] - [rest_client] [139835234313984] - INFO - http://172.23.123.42:8091/pools/default/buckets with param: name=metadata&ramQuotaMB=400&authType=none&replicaNumber=0&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore [2021-02-22 17:58:13,983] - [rest_client] [139835234313984] - INFO - 0.03 seconds to create bucket metadata [2021-02-22 17:58:13,983] - [bucket_helper] [139835234313984] - INFO - waiting for memcached bucket : metadata in 172.23.123.42 to accept set ops [2021-02-22 17:58:14,695] - [data_helper] [139835234313984] - INFO - creating direct client 172.23.122.132:11210 metadata [2021-02-22 17:58:14,806] - [data_helper] [139835234313984] - INFO - creating direct client 172.23.123.42:11210 metadata [2021-02-22 17:58:15,093] - [data_helper] [139835234313984] - INFO - creating direct client 172.23.123.44:11210 metadata [2021-02-22 17:58:15,275] - [data_helper] [139835234313984] - INFO - creating direct client 172.23.122.132:11210 metadata [2021-02-22 17:58:15,385] - [data_helper] [139835234313984] - INFO - creating direct client 172.23.123.42:11210 metadata [2021-02-22 17:58:15,498] - [data_helper] [139835234313984] - INFO - creating direct client 172.23.123.44:11210 metadata [2021-02-22 17:58:15,780] - [task] [139835234313984] - WARNING - vbucket map not ready after try 0 [2021-02-22 17:58:15,781] - [bucket_helper] [139835234313984] - INFO - waiting for memcached bucket : metadata in 172.23.123.42 to accept set ops [2021-02-22 17:58:15,943] - [data_helper] [139835234313984] - INFO - creating direct client 172.23.122.132:11210 metadata [2021-02-22 17:58:16,055] - [data_helper] [139835234313984] - INFO - creating direct client 172.23.123.42:11210 metadata [2021-02-22 17:58:16,343] - [data_helper] [139835234313984] - INFO - creating direct client 172.23.123.44:11210 metadata [2021-02-22 17:58:16,511] - [data_helper] [139835234313984] - INFO - creating direct client 172.23.122.132:11210 metadata [2021-02-22 17:58:16,625] - [data_helper] [139835234313984] - INFO - creating direct client 172.23.123.42:11210 metadata [2021-02-22 17:58:16,734] - [data_helper] [139835234313984] - INFO - creating direct client 172.23.123.44:11210 metadata [2021-02-22 17:58:17,001] - [task] [139835234313984] - WARNING - vbucket map not ready after try 1 [2021-02-22 17:58:17,001] - [bucket_helper] [139835234313984] - INFO - waiting for memcached bucket : metadata in 172.23.123.42 to accept set ops [2021-02-22 17:58:17,168] - [data_helper] [139835234313984] - INFO - creating direct client 172.23.122.132:11210 metadata [2021-02-22 17:58:17,277] - [data_helper] [139835234313984] - INFO - creating direct client 172.23.123.42:11210 metadata [2021-02-22 17:58:17,557] - [data_helper] [139835234313984] - INFO - creating direct client 172.23.123.44:11210 metadata [2021-02-22 17:58:17,715] - [data_helper] [139835234313984] - INFO - creating direct client 172.23.122.132:11210 metadata [2021-02-22 17:58:17,831] - [data_helper] [139835234313984] - INFO - creating direct client 172.23.123.42:11210 metadata [2021-02-22 17:58:17,944] - [data_helper] [139835234313984] - INFO - creating direct client 172.23.123.44:11210 metadata [2021-02-22 17:58:18,219] - [task] [139835234313984] - WARNING - vbucket map not ready after try 2 [2021-02-22 17:58:18,220] - [bucket_helper] [139835234313984] - INFO - waiting for memcached bucket : metadata in 172.23.123.42 to accept set ops [2021-02-22 17:58:18,384] - [data_helper] [139835234313984] - INFO - creating direct client 172.23.122.132:11210 metadata [2021-02-22 17:58:18,493] - [data_helper] [139835234313984] - INFO - creating direct client 172.23.123.42:11210 metadata [2021-02-22 17:58:18,789] - [data_helper] [139835234313984] - INFO - creating direct client 172.23.123.44:11210 metadata [2021-02-22 17:58:18,950] - [data_helper] [139835234313984] - INFO - creating direct client 172.23.122.132:11210 metadata [2021-02-22 17:58:19,069] - [data_helper] [139835234313984] - INFO - creating direct client 172.23.123.42:11210 metadata [2021-02-22 17:58:19,184] - [data_helper] [139835234313984] - INFO - creating direct client 172.23.123.44:11210 metadata [2021-02-22 17:58:19,451] - [task] [139835234313984] - WARNING - vbucket map not ready after try 3 [2021-02-22 17:58:19,452] - [bucket_helper] [139835234313984] - INFO - waiting for memcached bucket : metadata in 172.23.123.42 to accept set ops [2021-02-22 17:58:19,621] - [data_helper] [139835234313984] - INFO - creating direct client 172.23.122.132:11210 metadata [2021-02-22 17:58:19,728] - [data_helper] [139835234313984] - INFO - creating direct client 172.23.123.42:11210 metadata [2021-02-22 17:58:20,016] - [data_helper] [139835234313984] - INFO - creating direct client 172.23.123.44:11210 metadata [2021-02-22 17:58:20,185] - [data_helper] [139835234313984] - INFO - creating direct client 172.23.122.132:11210 metadata [2021-02-22 17:58:20,303] - [data_helper] [139835234313984] - INFO - creating direct client 172.23.123.42:11210 metadata [2021-02-22 17:58:20,406] - [data_helper] [139835234313984] - INFO - creating direct client 172.23.123.44:11210 metadata [2021-02-22 17:58:20,687] - [task] [139835234313984] - WARNING - vbucket map not ready after try 4 [2021-02-22 17:58:20,688] - [bucket_helper] [139835234313984] - INFO - waiting for memcached bucket : metadata in 172.23.123.42 to accept set ops [2021-02-22 17:58:20,841] - [data_helper] [139835234313984] - INFO - creating direct client 172.23.122.132:11210 metadata [2021-02-22 17:58:20,943] - [data_helper] [139835234313984] - INFO - creating direct client 172.23.123.42:11210 metadata [2021-02-22 17:58:21,045] - [data_helper] [139835234313984] - INFO - creating direct client 172.23.123.44:11210 metadata [2021-02-22 17:58:21,376] - [data_helper] [139835234313984] - INFO - creating direct client 172.23.122.132:11210 metadata [2021-02-22 17:58:21,488] - [data_helper] [139835234313984] - INFO - creating direct client 172.23.123.42:11210 metadata [2021-02-22 17:58:21,603] - [data_helper] [139835234313984] - INFO - creating direct client 172.23.123.44:11210 metadata [2021-02-22 17:58:21,873] - [task] [139835234313984] - WARNING - vbucket map not ready after try 5 [2021-02-22 17:58:22,074] - [basetestcase] [139837458200384] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.134 port:8091 ssh_username:root] [2021-02-22 17:58:22,082] - [tuq_helper] [139837458200384] - INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary' [2021-02-22 17:58:22,087] - [rest_client] [139837458200384] - INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27 [2021-02-22 17:58:22,422] - [tuq_helper] [139837458200384] - INFO - TOTAL ELAPSED TIME: 331.264302ms [2021-02-22 17:58:22,428] - [tuq_helper] [139837458200384] - INFO - RUN QUERY CREATE PRIMARY INDEX ON dst_bucket USING GSI [2021-02-22 17:58:22,433] - [rest_client] [139837458200384] - INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+dst_bucket++USING+GSI [2021-02-22 17:58:24,228] - [tuq_helper] [139837458200384] - INFO - TOTAL ELAPSED TIME: 1.790951651s [2021-02-22 17:58:24,236] - [tuq_helper] [139837458200384] - INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary' [2021-02-22 17:58:24,242] - [rest_client] [139837458200384] - INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27 [2021-02-22 17:58:24,276] - [tuq_helper] [139837458200384] - INFO - TOTAL ELAPSED TIME: 31.612069ms [2021-02-22 17:58:24,282] - [tuq_helper] [139837458200384] - INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary' [2021-02-22 17:58:24,288] - [rest_client] [139837458200384] - INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27 [2021-02-22 17:58:24,292] - [tuq_helper] [139837458200384] - INFO - TOTAL ELAPSED TIME: 1.215561ms [2021-02-22 17:58:24,297] - [tuq_helper] [139837458200384] - INFO - RUN QUERY CREATE PRIMARY INDEX ON metadata USING GSI [2021-02-22 17:58:24,303] - [rest_client] [139837458200384] - INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+metadata++USING+GSI [2021-02-22 17:58:26,335] - [tuq_helper] [139837458200384] - INFO - TOTAL ELAPSED TIME: 2.029086742s [2021-02-22 17:58:26,348] - [tuq_helper] [139837458200384] - INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary' [2021-02-22 17:58:26,355] - [rest_client] [139837458200384] - INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27 [2021-02-22 17:58:26,381] - [tuq_helper] [139837458200384] - INFO - TOTAL ELAPSED TIME: 22.174719ms [2021-02-22 17:58:26,387] - [tuq_helper] [139837458200384] - INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary' [2021-02-22 17:58:26,393] - [rest_client] [139837458200384] - INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27 [2021-02-22 17:58:26,397] - [tuq_helper] [139837458200384] - INFO - TOTAL ELAPSED TIME: 1.046295ms [2021-02-22 17:58:26,402] - [tuq_helper] [139837458200384] - INFO - RUN QUERY CREATE PRIMARY INDEX ON src_bucket USING GSI [2021-02-22 17:58:26,408] - [rest_client] [139837458200384] - INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+src_bucket++USING+GSI [2021-02-22 17:58:28,214] - [tuq_helper] [139837458200384] - INFO - TOTAL ELAPSED TIME: 1.802261023s [2021-02-22 17:58:28,228] - [tuq_helper] [139837458200384] - INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary' [2021-02-22 17:58:28,234] - [rest_client] [139837458200384] - INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27 [2021-02-22 17:58:28,255] - [tuq_helper] [139837458200384] - INFO - TOTAL ELAPSED TIME: 18.91478ms [2021-02-22 17:58:28,274] - [basetestcase] [139837458200384] - INFO - list of eventing nodes in cluster: [ip:172.23.122.133 port:8091 ssh_username:root] [2021-02-22 17:58:28,286] - [eventing_base] [139837458200384] - INFO - Stats for Node 172.23.122.133 is [] [2021-02-22 17:58:28,305] - [basetestcase] [139837458200384] - INFO - list of eventing nodes in cluster: [ip:172.23.122.133 port:8091 ssh_username:root] [2021-02-22 17:58:28,308] - [remote_util] [139837458200384] - INFO - SSH Connecting to 172.23.122.133 with username:root, attempt#1 of 5 [2021-02-22 17:58:28,420] - [remote_util] [139837458200384] - INFO - SSH Connected to 172.23.122.133 as root [2021-02-22 17:58:28,724] - [remote_util] [139837458200384] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 17:58:29,039] - [remote_util] [139837458200384] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 17:58:29,049] - [rest_client] [139837458200384] - INFO - /diag/eval status on 172.23.122.133:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). [2021-02-22 17:58:29,050] - [remote_util] [139837458200384] - INFO - running command.raw on 172.23.122.133: zgrep "panic" "/opt/couchbase/var/lib/couchbase/logs"/eventing.log* | wc -l [2021-02-22 17:58:30,227] - [remote_util] [139837458200384] - INFO - command executed successfully with root [2021-02-22 17:58:30,227] - [remote_util] [139837458200384] - INFO - running command.raw on 172.23.122.133: ls "/opt/couchbase/var/lib/couchbase/logs"/../crash/| wc -l [2021-02-22 17:58:30,284] - [remote_util] [139837458200384] - INFO - command executed successfully with root [2021-02-22 17:58:30,539] - [basetestcase] [139837458200384] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.134 port:8091 ssh_username:root] [2021-02-22 17:58:30,545] - [rest_client] [139837458200384] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+metadata [2021-02-22 17:58:30,558] - [eventing_base] [139837458200384] - INFO - number of documents in metadata bucket 0 [2021-02-22 17:58:30,569] - [basetestcase] [139837458200384] - INFO - ------- Cluster statistics ------- [2021-02-22 17:58:30,569] - [basetestcase] [139837458200384] - INFO - 172.23.122.132:8091 => {'services': ['kv'], 'cpu_utilization': 8.863003291972651, 'mem_free': 3423850496, 'mem_total': 4201684992, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-02-22 17:58:30,569] - [basetestcase] [139837458200384] - INFO - 172.23.122.133:8091 => {'services': ['eventing'], 'cpu_utilization': 1.053423626787058, 'mem_free': 3506769920, 'mem_total': 4201684992, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-02-22 17:58:30,569] - [basetestcase] [139837458200384] - INFO - 172.23.122.134:8091 => {'services': ['index', 'n1ql'], 'cpu_utilization': 3.144654088050315, 'mem_free': 3368046592, 'mem_total': 4201684992, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-02-22 17:58:30,570] - [basetestcase] [139837458200384] - INFO - 172.23.123.42:8091 => {'services': ['kv'], 'cpu_utilization': 13.04569823844779, 'mem_free': 3384213504, 'mem_total': 4201672704, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-02-22 17:58:30,570] - [basetestcase] [139837458200384] - INFO - 172.23.123.44:8091 => {'services': ['kv'], 'cpu_utilization': 10.56105610561056, 'mem_free': 3424018432, 'mem_total': 4201676800, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-02-22 17:58:30,570] - [basetestcase] [139837458200384] - INFO - --- End of cluster statistics --- [2021-02-22 17:58:30,573] - [remote_util] [139837458200384] - INFO - SSH Connecting to 172.23.123.42 with username:root, attempt#1 of 5 [2021-02-22 17:58:30,686] - [remote_util] [139837458200384] - INFO - SSH Connected to 172.23.123.42 as root [2021-02-22 17:58:30,956] - [remote_util] [139837458200384] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 17:58:31,274] - [remote_util] [139837458200384] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 17:58:31,278] - [remote_util] [139837458200384] - INFO - SSH Connecting to 172.23.123.44 with username:root, attempt#1 of 5 [2021-02-22 17:58:31,383] - [remote_util] [139837458200384] - INFO - SSH Connected to 172.23.123.44 as root [2021-02-22 17:58:31,673] - [remote_util] [139837458200384] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 17:58:31,991] - [remote_util] [139837458200384] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 17:58:31,995] - [remote_util] [139837458200384] - INFO - SSH Connecting to 172.23.122.132 with username:root, attempt#1 of 5 [2021-02-22 17:58:32,097] - [remote_util] [139837458200384] - INFO - SSH Connected to 172.23.122.132 as root [2021-02-22 17:58:32,374] - [remote_util] [139837458200384] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 17:58:32,681] - [remote_util] [139837458200384] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 17:58:32,686] - [remote_util] [139837458200384] - INFO - SSH Connecting to 172.23.122.133 with username:root, attempt#1 of 5 [2021-02-22 17:58:32,788] - [remote_util] [139837458200384] - INFO - SSH Connected to 172.23.122.133 as root [2021-02-22 17:58:33,060] - [remote_util] [139837458200384] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 17:58:33,366] - [remote_util] [139837458200384] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 17:58:33,373] - [remote_util] [139837458200384] - INFO - SSH Connecting to 172.23.122.134 with username:root, attempt#1 of 5 [2021-02-22 17:58:33,479] - [remote_util] [139837458200384] - INFO - SSH Connected to 172.23.122.134 as root [2021-02-22 17:58:33,761] - [remote_util] [139837458200384] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 17:58:34,068] - [remote_util] [139837458200384] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 17:58:34,073] - [remote_util] [139837458200384] - INFO - SSH Connecting to 172.23.122.131 with username:root, attempt#1 of 5 [2021-02-22 17:58:34,178] - [remote_util] [139837458200384] - INFO - SSH Connected to 172.23.122.131 as root [2021-02-22 17:58:34,471] - [remote_util] [139837458200384] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 17:58:34,779] - [remote_util] [139837458200384] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 17:58:34,784] - [remote_util] [139837458200384] - INFO - SSH Connecting to 172.23.122.180 with username:root, attempt#1 of 5 [2021-02-22 17:58:34,886] - [remote_util] [139837458200384] - INFO - SSH Connected to 172.23.122.180 as root [2021-02-22 17:58:35,160] - [remote_util] [139837458200384] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 17:58:35,464] - [remote_util] [139837458200384] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 17:58:35,469] - [remote_util] [139837458200384] - INFO - SSH Connecting to 172.23.122.130 with username:root, attempt#1 of 5 [2021-02-22 17:58:35,573] - [remote_util] [139837458200384] - INFO - SSH Connected to 172.23.122.130 as root [2021-02-22 17:58:35,841] - [remote_util] [139837458200384] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 17:58:36,146] - [remote_util] [139837458200384] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 17:59:51,927] - [basetestcase] [139837458200384] - INFO - ============== basetestcase cleanup was started for test #8 suite_tearDown ============== [2021-02-22 17:59:52,010] - [bucket_helper] [139837458200384] - INFO - deleting existing buckets ['dst_bucket', 'metadata', 'src_bucket'] on 172.23.123.42 [2021-02-22 17:59:53,206] - [bucket_helper] [139837458200384] - INFO - waiting for bucket deletion to complete.... [2021-02-22 17:59:53,241] - [rest_client] [139837458200384] - INFO - node 172.23.123.42 existing buckets : ['metadata', 'src_bucket'] [2021-02-22 17:59:53,244] - [bucket_helper] [139837458200384] - INFO - deleted bucket : dst_bucket from 172.23.123.42 [2021-02-22 17:59:54,398] - [bucket_helper] [139837458200384] - INFO - waiting for bucket deletion to complete.... [2021-02-22 17:59:54,414] - [rest_client] [139837458200384] - INFO - node 172.23.123.42 existing buckets : ['src_bucket'] [2021-02-22 17:59:54,416] - [bucket_helper] [139837458200384] - INFO - deleted bucket : metadata from 172.23.123.42 [2021-02-22 17:59:55,719] - [bucket_helper] [139837458200384] - INFO - waiting for bucket deletion to complete.... [2021-02-22 17:59:55,724] - [rest_client] [139837458200384] - INFO - node 172.23.123.42 existing buckets : [] [2021-02-22 17:59:55,724] - [bucket_helper] [139837458200384] - INFO - deleted bucket : src_bucket from 172.23.123.42 [2021-02-22 17:59:55,735] - [bucket_helper] [139837458200384] - INFO - Could not find any buckets for node 172.23.123.44, nothing to delete [2021-02-22 17:59:55,744] - [bucket_helper] [139837458200384] - INFO - Could not find any buckets for node 172.23.122.132, nothing to delete [2021-02-22 17:59:55,752] - [bucket_helper] [139837458200384] - INFO - Could not find any buckets for node 172.23.122.133, nothing to delete [2021-02-22 17:59:55,761] - [bucket_helper] [139837458200384] - INFO - Could not find any buckets for node 172.23.122.134, nothing to delete [2021-02-22 17:59:55,769] - [bucket_helper] [139837458200384] - INFO - Could not find any buckets for node 172.23.122.131, nothing to delete [2021-02-22 17:59:55,778] - [bucket_helper] [139837458200384] - INFO - Could not find any buckets for node 172.23.122.180, nothing to delete [2021-02-22 17:59:55,784] - [bucket_helper] [139837458200384] - INFO - Could not find any buckets for node 172.23.122.130, nothing to delete [2021-02-22 17:59:55,791] - [rest_client] [139837458200384] - INFO - -->is_ns_server_running? [2021-02-22 17:59:55,805] - [cluster_helper] [139837458200384] - INFO - rebalancing all nodes in order to remove nodes [2021-02-22 17:59:55,808] - [rest_client] [139837458200384] - INFO - rebalance params : {'knownNodes': 'ns_1@172.23.122.132,ns_1@172.23.122.133,ns_1@172.23.122.134,ns_1@172.23.123.42,ns_1@172.23.123.44', 'ejectedNodes': 'ns_1@172.23.122.132,ns_1@172.23.122.133,ns_1@172.23.122.134,ns_1@172.23.123.44', 'user': 'Administrator', 'password': 'password'} [2021-02-22 17:59:55,838] - [rest_client] [139837458200384] - INFO - rebalance operation started [2021-02-22 17:59:55,842] - [rest_client] [139837458200384] - INFO - rebalance percentage : 0.00 % [2021-02-22 18:00:05,849] - [rest_client] [139837458200384] - INFO - rebalance percentage : 70.00 % [2021-02-22 18:00:15,854] - [rest_client] [139837458200384] - INFO - rebalance percentage : 70.00 % [2021-02-22 18:00:35,876] - [rest_client] [139837458200384] - INFO - rebalance progress took 40.04 seconds [2021-02-22 18:00:35,876] - [rest_client] [139837458200384] - INFO - sleep for 10 seconds after rebalance... [2021-02-22 18:00:45,913] - [cluster_helper] [139837458200384] - INFO - removed all the nodes from cluster associated with ip:172.23.123.42 port:8091 ssh_username:root ? [('ns_1@172.23.122.132', 8091), ('ns_1@172.23.122.133', 8091), ('ns_1@172.23.122.134', 8091), ('ns_1@172.23.123.44', 8091)] [2021-02-22 18:00:45,919] - [cluster_helper] [139837458200384] - INFO - waiting for ns_server @ 172.23.123.42:8091 [2021-02-22 18:00:45,920] - [rest_client] [139837458200384] - INFO - -->is_ns_server_running? [2021-02-22 18:00:45,924] - [cluster_helper] [139837458200384] - INFO - ns_server @ 172.23.123.42:8091 is running [2021-02-22 18:00:45,929] - [cluster_helper] [139837458200384] - INFO - waiting for ns_server @ 172.23.123.44:8091 [2021-02-22 18:00:45,929] - [rest_client] [139837458200384] - INFO - -->is_ns_server_running? [2021-02-22 18:00:45,934] - [cluster_helper] [139837458200384] - INFO - ns_server @ 172.23.123.44:8091 is running [2021-02-22 18:00:45,939] - [cluster_helper] [139837458200384] - INFO - waiting for ns_server @ 172.23.122.132:8091 [2021-02-22 18:00:45,939] - [rest_client] [139837458200384] - INFO - -->is_ns_server_running? [2021-02-22 18:00:45,944] - [cluster_helper] [139837458200384] - INFO - ns_server @ 172.23.122.132:8091 is running [2021-02-22 18:00:45,949] - [cluster_helper] [139837458200384] - INFO - waiting for ns_server @ 172.23.122.133:8091 [2021-02-22 18:00:45,949] - [rest_client] [139837458200384] - INFO - -->is_ns_server_running? [2021-02-22 18:00:45,953] - [cluster_helper] [139837458200384] - INFO - ns_server @ 172.23.122.133:8091 is running [2021-02-22 18:00:45,958] - [cluster_helper] [139837458200384] - INFO - waiting for ns_server @ 172.23.122.134:8091 [2021-02-22 18:00:45,958] - [rest_client] [139837458200384] - INFO - -->is_ns_server_running? [2021-02-22 18:00:45,963] - [cluster_helper] [139837458200384] - INFO - ns_server @ 172.23.122.134:8091 is running [2021-02-22 18:00:45,969] - [cluster_helper] [139837458200384] - INFO - waiting for ns_server @ 172.23.122.131:8091 [2021-02-22 18:00:45,969] - [rest_client] [139837458200384] - INFO - -->is_ns_server_running? [2021-02-22 18:00:45,974] - [cluster_helper] [139837458200384] - INFO - ns_server @ 172.23.122.131:8091 is running [2021-02-22 18:00:45,978] - [cluster_helper] [139837458200384] - INFO - waiting for ns_server @ 172.23.122.180:8091 [2021-02-22 18:00:45,979] - [rest_client] [139837458200384] - INFO - -->is_ns_server_running? [2021-02-22 18:00:45,983] - [cluster_helper] [139837458200384] - INFO - ns_server @ 172.23.122.180:8091 is running [2021-02-22 18:00:45,988] - [cluster_helper] [139837458200384] - INFO - waiting for ns_server @ 172.23.122.130:8091 [2021-02-22 18:00:45,988] - [rest_client] [139837458200384] - INFO - -->is_ns_server_running? [2021-02-22 18:00:45,994] - [cluster_helper] [139837458200384] - INFO - ns_server @ 172.23.122.130:8091 is running [2021-02-22 18:00:45,994] - [basetestcase] [139837458200384] - INFO - ============== basetestcase cleanup was finished for test #8 suite_tearDown ============== [2021-02-22 18:00:45,994] - [basetestcase] [139837458200384] - INFO - closing all ssh connections [2021-02-22 18:00:45,998] - [basetestcase] [139837458200384] - INFO - closing all memcached connections [2021-02-22 18:00:46,001] - [remote_util] [139837458200384] - INFO - SSH Connecting to 172.23.123.42 with username:root, attempt#1 of 5 [2021-02-22 18:00:46,107] - [remote_util] [139837458200384] - INFO - SSH Connected to 172.23.123.42 as root [2021-02-22 18:00:46,368] - [remote_util] [139837458200384] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 18:00:46,681] - [remote_util] [139837458200384] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 18:00:46,697] - [remote_util] [139837458200384] - INFO - SSH Connecting to 172.23.123.42 with username:root, attempt#1 of 5 [2021-02-22 18:00:46,804] - [remote_util] [139837458200384] - INFO - SSH Connected to 172.23.123.42 as root [2021-02-22 18:00:47,062] - [remote_util] [139837458200384] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 18:00:47,371] - [remote_util] [139837458200384] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 18:00:47,431] - [remote_util] [139837458200384] - INFO - 172.23.123.42 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-22 18:00:47,583] - [remote_util] [139837458200384] - INFO - Running systemd command on this server [2021-02-22 18:00:47,583] - [remote_util] [139837458200384] - INFO - running command.raw on 172.23.123.42: systemctl stop couchbase-server.service [2021-02-22 18:00:49,297] - [remote_util] [139837458200384] - INFO - command executed successfully with root [2021-02-22 18:00:49,297] - [basetestcase] [139837458200384] - INFO - Couchbase stopped [2021-02-22 18:00:49,298] - [basetestcase] [139837458200384] - INFO - sleep for 5 secs. ... [2021-02-22 18:00:54,304] - [remote_util] [139837458200384] - INFO - running command.raw on 172.23.123.42: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-22 18:00:54,322] - [remote_util] [139837458200384] - INFO - command executed successfully with root [2021-02-22 18:00:54,322] - [remote_util] [139837458200384] - INFO - running command.raw on 172.23.123.42: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-22 18:00:54,378] - [remote_util] [139837458200384] - INFO - command executed successfully with root [2021-02-22 18:00:54,382] - [remote_util] [139837458200384] - INFO - SSH Connecting to 172.23.123.42 with username:root, attempt#1 of 5 [2021-02-22 18:00:54,489] - [remote_util] [139837458200384] - INFO - SSH Connected to 172.23.123.42 as root [2021-02-22 18:00:54,746] - [remote_util] [139837458200384] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 18:00:55,055] - [remote_util] [139837458200384] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 18:00:55,120] - [remote_util] [139837458200384] - INFO - 172.23.123.42 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-22 18:00:55,120] - [remote_util] [139837458200384] - INFO - Checking for process beam.smp on linux [2021-02-22 18:00:55,196] - [remote_util] [139837458200384] - INFO - Starting couchbase server [2021-02-22 18:00:55,303] - [remote_util] [139837458200384] - INFO - Running systemd command on this server [2021-02-22 18:00:55,304] - [remote_util] [139837458200384] - INFO - running command.raw on 172.23.123.42: systemctl start couchbase-server.service [2021-02-22 18:00:55,326] - [remote_util] [139837458200384] - INFO - command executed successfully with root [2021-02-22 18:00:55,327] - [remote_util] [139837458200384] - INFO - 172.23.123.42:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-22 18:01:00,329] - [remote_util] [139837458200384] - INFO - running command.raw on 172.23.123.42: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-22 18:01:00,361] - [remote_util] [139837458200384] - INFO - command executed successfully with root [2021-02-22 18:01:00,361] - [remote_util] [139837458200384] - INFO - Couchbase server status: [' Process: 61540 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-22 18:01:00,362] - [remote_util] [139837458200384] - INFO - Checking for process beam.smp on linux [2021-02-22 18:01:00,437] - [remote_util] [139837458200384] - INFO - process beam.smp is running on 172.23.123.42: with pid 61682 [2021-02-22 18:01:00,437] - [basetestcase] [139837458200384] - INFO - Couchbase started [2021-02-22 18:01:00,441] - [remote_util] [139837458200384] - INFO - SSH Connecting to 172.23.123.44 with username:root, attempt#1 of 5 [2021-02-22 18:01:00,552] - [remote_util] [139837458200384] - INFO - SSH Connected to 172.23.123.44 as root [2021-02-22 18:01:00,843] - [remote_util] [139837458200384] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 18:01:01,154] - [remote_util] [139837458200384] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 18:01:01,170] - [remote_util] [139837458200384] - INFO - SSH Connecting to 172.23.123.44 with username:root, attempt#1 of 5 [2021-02-22 18:01:01,273] - [remote_util] [139837458200384] - INFO - SSH Connected to 172.23.123.44 as root [2021-02-22 18:01:01,564] - [remote_util] [139837458200384] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 18:01:01,878] - [remote_util] [139837458200384] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 18:01:01,941] - [remote_util] [139837458200384] - INFO - 172.23.123.44 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-22 18:01:02,127] - [remote_util] [139837458200384] - INFO - Running systemd command on this server [2021-02-22 18:01:02,127] - [remote_util] [139837458200384] - INFO - running command.raw on 172.23.123.44: systemctl stop couchbase-server.service [2021-02-22 18:01:03,705] - [remote_util] [139837458200384] - INFO - command executed successfully with root [2021-02-22 18:01:03,706] - [basetestcase] [139837458200384] - INFO - Couchbase stopped [2021-02-22 18:01:03,706] - [basetestcase] [139837458200384] - INFO - sleep for 5 secs. ... [2021-02-22 18:01:08,712] - [remote_util] [139837458200384] - INFO - running command.raw on 172.23.123.44: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-22 18:01:08,728] - [remote_util] [139837458200384] - INFO - command executed successfully with root [2021-02-22 18:01:08,729] - [remote_util] [139837458200384] - INFO - running command.raw on 172.23.123.44: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-22 18:01:08,784] - [remote_util] [139837458200384] - INFO - command executed successfully with root [2021-02-22 18:01:08,788] - [remote_util] [139837458200384] - INFO - SSH Connecting to 172.23.123.44 with username:root, attempt#1 of 5 [2021-02-22 18:01:08,893] - [remote_util] [139837458200384] - INFO - SSH Connected to 172.23.123.44 as root [2021-02-22 18:01:09,177] - [remote_util] [139837458200384] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 18:01:09,486] - [remote_util] [139837458200384] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 18:01:09,553] - [remote_util] [139837458200384] - INFO - 172.23.123.44 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-22 18:01:09,553] - [remote_util] [139837458200384] - INFO - Checking for process beam.smp on linux [2021-02-22 18:01:09,634] - [remote_util] [139837458200384] - INFO - Starting couchbase server [2021-02-22 18:01:09,745] - [remote_util] [139837458200384] - INFO - Running systemd command on this server [2021-02-22 18:01:09,746] - [remote_util] [139837458200384] - INFO - running command.raw on 172.23.123.44: systemctl start couchbase-server.service [2021-02-22 18:01:09,770] - [remote_util] [139837458200384] - INFO - command executed successfully with root [2021-02-22 18:01:09,771] - [remote_util] [139837458200384] - INFO - 172.23.123.44:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-22 18:01:14,776] - [remote_util] [139837458200384] - INFO - running command.raw on 172.23.123.44: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-22 18:01:14,805] - [remote_util] [139837458200384] - INFO - command executed successfully with root [2021-02-22 18:01:14,805] - [remote_util] [139837458200384] - INFO - Couchbase server status: [' Process: 23877 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-22 18:01:14,806] - [remote_util] [139837458200384] - INFO - Checking for process beam.smp on linux [2021-02-22 18:01:14,875] - [remote_util] [139837458200384] - INFO - process beam.smp is running on 172.23.123.44: with pid 24019 [2021-02-22 18:01:14,876] - [basetestcase] [139837458200384] - INFO - Couchbase started [2021-02-22 18:01:14,879] - [remote_util] [139837458200384] - INFO - SSH Connecting to 172.23.122.132 with username:root, attempt#1 of 5 [2021-02-22 18:01:14,986] - [remote_util] [139837458200384] - INFO - SSH Connected to 172.23.122.132 as root [2021-02-22 18:01:15,271] - [remote_util] [139837458200384] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 18:01:15,583] - [remote_util] [139837458200384] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 18:01:15,595] - [remote_util] [139837458200384] - INFO - SSH Connecting to 172.23.122.132 with username:root, attempt#1 of 5 [2021-02-22 18:01:15,697] - [remote_util] [139837458200384] - INFO - SSH Connected to 172.23.122.132 as root [2021-02-22 18:01:15,967] - [remote_util] [139837458200384] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 18:01:16,275] - [remote_util] [139837458200384] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 18:01:16,338] - [remote_util] [139837458200384] - INFO - 172.23.122.132 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-22 18:01:16,445] - [remote_util] [139837458200384] - INFO - Running systemd command on this server [2021-02-22 18:01:16,445] - [remote_util] [139837458200384] - INFO - running command.raw on 172.23.122.132: systemctl stop couchbase-server.service [2021-02-22 18:01:18,131] - [remote_util] [139837458200384] - INFO - command executed successfully with root [2021-02-22 18:01:18,132] - [basetestcase] [139837458200384] - INFO - Couchbase stopped [2021-02-22 18:01:18,133] - [basetestcase] [139837458200384] - INFO - sleep for 5 secs. ... [2021-02-22 18:01:23,138] - [remote_util] [139837458200384] - INFO - running command.raw on 172.23.122.132: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-22 18:01:23,155] - [remote_util] [139837458200384] - INFO - command executed successfully with root [2021-02-22 18:01:23,155] - [remote_util] [139837458200384] - INFO - running command.raw on 172.23.122.132: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-22 18:01:23,209] - [remote_util] [139837458200384] - INFO - command executed successfully with root [2021-02-22 18:01:23,214] - [remote_util] [139837458200384] - INFO - SSH Connecting to 172.23.122.132 with username:root, attempt#1 of 5 [2021-02-22 18:01:23,319] - [remote_util] [139837458200384] - INFO - SSH Connected to 172.23.122.132 as root [2021-02-22 18:01:23,592] - [remote_util] [139837458200384] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 18:01:23,911] - [remote_util] [139837458200384] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 18:01:23,974] - [remote_util] [139837458200384] - INFO - 172.23.122.132 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-22 18:01:23,974] - [remote_util] [139837458200384] - INFO - Checking for process beam.smp on linux [2021-02-22 18:01:24,052] - [remote_util] [139837458200384] - INFO - Starting couchbase server [2021-02-22 18:01:24,201] - [remote_util] [139837458200384] - INFO - Running systemd command on this server [2021-02-22 18:01:24,202] - [remote_util] [139837458200384] - INFO - running command.raw on 172.23.122.132: systemctl start couchbase-server.service [2021-02-22 18:01:24,226] - [remote_util] [139837458200384] - INFO - command executed successfully with root [2021-02-22 18:01:24,226] - [remote_util] [139837458200384] - INFO - 172.23.122.132:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-22 18:01:29,231] - [remote_util] [139837458200384] - INFO - running command.raw on 172.23.122.132: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-22 18:01:29,262] - [remote_util] [139837458200384] - INFO - command executed successfully with root [2021-02-22 18:01:29,263] - [remote_util] [139837458200384] - INFO - Couchbase server status: [' Process: 52971 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-22 18:01:29,264] - [remote_util] [139837458200384] - INFO - Checking for process beam.smp on linux [2021-02-22 18:01:29,338] - [remote_util] [139837458200384] - INFO - process beam.smp is running on 172.23.122.132: with pid 53115 [2021-02-22 18:01:29,339] - [basetestcase] [139837458200384] - INFO - Couchbase started [2021-02-22 18:01:29,342] - [remote_util] [139837458200384] - INFO - SSH Connecting to 172.23.122.133 with username:root, attempt#1 of 5 [2021-02-22 18:01:29,459] - [remote_util] [139837458200384] - INFO - SSH Connected to 172.23.122.133 as root [2021-02-22 18:01:29,777] - [remote_util] [139837458200384] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 18:01:30,107] - [remote_util] [139837458200384] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 18:01:30,121] - [remote_util] [139837458200384] - INFO - SSH Connecting to 172.23.122.133 with username:root, attempt#1 of 5 [2021-02-22 18:01:30,225] - [remote_util] [139837458200384] - INFO - SSH Connected to 172.23.122.133 as root [2021-02-22 18:01:30,536] - [remote_util] [139837458200384] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 18:01:30,864] - [remote_util] [139837458200384] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 18:01:30,927] - [remote_util] [139837458200384] - INFO - 172.23.122.133 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-22 18:01:31,040] - [remote_util] [139837458200384] - INFO - Running systemd command on this server [2021-02-22 18:01:31,040] - [remote_util] [139837458200384] - INFO - running command.raw on 172.23.122.133: systemctl stop couchbase-server.service [2021-02-22 18:01:32,879] - [remote_util] [139837458200384] - INFO - command executed successfully with root [2021-02-22 18:01:32,880] - [basetestcase] [139837458200384] - INFO - Couchbase stopped [2021-02-22 18:01:32,881] - [basetestcase] [139837458200384] - INFO - sleep for 5 secs. ... [2021-02-22 18:01:37,886] - [remote_util] [139837458200384] - INFO - running command.raw on 172.23.122.133: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-22 18:01:37,905] - [remote_util] [139837458200384] - INFO - command executed successfully with root [2021-02-22 18:01:37,906] - [remote_util] [139837458200384] - INFO - running command.raw on 172.23.122.133: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-22 18:01:37,962] - [remote_util] [139837458200384] - INFO - command executed successfully with root [2021-02-22 18:01:37,966] - [remote_util] [139837458200384] - INFO - SSH Connecting to 172.23.122.133 with username:root, attempt#1 of 5 [2021-02-22 18:01:38,074] - [remote_util] [139837458200384] - INFO - SSH Connected to 172.23.122.133 as root [2021-02-22 18:01:38,380] - [remote_util] [139837458200384] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 18:01:38,694] - [remote_util] [139837458200384] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 18:01:38,759] - [remote_util] [139837458200384] - INFO - 172.23.122.133 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-22 18:01:38,759] - [remote_util] [139837458200384] - INFO - Checking for process beam.smp on linux [2021-02-22 18:01:38,841] - [remote_util] [139837458200384] - INFO - Starting couchbase server [2021-02-22 18:01:38,952] - [remote_util] [139837458200384] - INFO - Running systemd command on this server [2021-02-22 18:01:38,952] - [remote_util] [139837458200384] - INFO - running command.raw on 172.23.122.133: systemctl start couchbase-server.service [2021-02-22 18:01:38,982] - [remote_util] [139837458200384] - INFO - command executed successfully with root [2021-02-22 18:01:38,983] - [remote_util] [139837458200384] - INFO - 172.23.122.133:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-22 18:01:43,988] - [remote_util] [139837458200384] - INFO - running command.raw on 172.23.122.133: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-22 18:01:44,018] - [remote_util] [139837458200384] - INFO - command executed successfully with root [2021-02-22 18:01:44,018] - [remote_util] [139837458200384] - INFO - Couchbase server status: [' Process: 58161 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-22 18:01:44,018] - [remote_util] [139837458200384] - INFO - Checking for process beam.smp on linux [2021-02-22 18:01:44,092] - [remote_util] [139837458200384] - INFO - process beam.smp is running on 172.23.122.133: with pid 58305 [2021-02-22 18:01:44,092] - [basetestcase] [139837458200384] - INFO - Couchbase started [2021-02-22 18:01:44,096] - [remote_util] [139837458200384] - INFO - SSH Connecting to 172.23.122.134 with username:root, attempt#1 of 5 [2021-02-22 18:01:44,211] - [remote_util] [139837458200384] - INFO - SSH Connected to 172.23.122.134 as root [2021-02-22 18:01:44,510] - [remote_util] [139837458200384] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 18:01:44,826] - [remote_util] [139837458200384] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 18:01:44,839] - [remote_util] [139837458200384] - INFO - SSH Connecting to 172.23.122.134 with username:root, attempt#1 of 5 [2021-02-22 18:01:44,944] - [remote_util] [139837458200384] - INFO - SSH Connected to 172.23.122.134 as root [2021-02-22 18:01:45,233] - [remote_util] [139837458200384] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 18:01:45,555] - [remote_util] [139837458200384] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 18:01:45,618] - [remote_util] [139837458200384] - INFO - 172.23.122.134 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-22 18:01:45,723] - [remote_util] [139837458200384] - INFO - Running systemd command on this server [2021-02-22 18:01:45,724] - [remote_util] [139837458200384] - INFO - running command.raw on 172.23.122.134: systemctl stop couchbase-server.service [2021-02-22 18:01:47,488] - [remote_util] [139837458200384] - INFO - command executed successfully with root [2021-02-22 18:01:47,489] - [basetestcase] [139837458200384] - INFO - Couchbase stopped [2021-02-22 18:01:47,489] - [basetestcase] [139837458200384] - INFO - sleep for 5 secs. ... [2021-02-22 18:01:52,495] - [remote_util] [139837458200384] - INFO - running command.raw on 172.23.122.134: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-22 18:01:52,513] - [remote_util] [139837458200384] - INFO - command executed successfully with root [2021-02-22 18:01:52,514] - [remote_util] [139837458200384] - INFO - running command.raw on 172.23.122.134: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-22 18:01:52,569] - [remote_util] [139837458200384] - INFO - command executed successfully with root [2021-02-22 18:01:52,573] - [remote_util] [139837458200384] - INFO - SSH Connecting to 172.23.122.134 with username:root, attempt#1 of 5 [2021-02-22 18:01:52,675] - [remote_util] [139837458200384] - INFO - SSH Connected to 172.23.122.134 as root [2021-02-22 18:01:52,957] - [remote_util] [139837458200384] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 18:01:53,267] - [remote_util] [139837458200384] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 18:01:53,331] - [remote_util] [139837458200384] - INFO - 172.23.122.134 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-22 18:01:53,332] - [remote_util] [139837458200384] - INFO - Checking for process beam.smp on linux [2021-02-22 18:01:53,411] - [remote_util] [139837458200384] - INFO - Starting couchbase server [2021-02-22 18:01:53,516] - [remote_util] [139837458200384] - INFO - Running systemd command on this server [2021-02-22 18:01:53,516] - [remote_util] [139837458200384] - INFO - running command.raw on 172.23.122.134: systemctl start couchbase-server.service [2021-02-22 18:01:53,541] - [remote_util] [139837458200384] - INFO - command executed successfully with root [2021-02-22 18:01:53,541] - [remote_util] [139837458200384] - INFO - 172.23.122.134:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-22 18:01:58,542] - [remote_util] [139837458200384] - INFO - running command.raw on 172.23.122.134: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-22 18:01:58,572] - [remote_util] [139837458200384] - INFO - command executed successfully with root [2021-02-22 18:01:58,573] - [remote_util] [139837458200384] - INFO - Couchbase server status: [' Process: 10510 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-22 18:01:58,573] - [remote_util] [139837458200384] - INFO - Checking for process beam.smp on linux [2021-02-22 18:01:58,644] - [remote_util] [139837458200384] - INFO - process beam.smp is running on 172.23.122.134: with pid 10655 [2021-02-22 18:01:58,645] - [basetestcase] [139837458200384] - INFO - Couchbase started [2021-02-22 18:01:58,648] - [remote_util] [139837458200384] - INFO - SSH Connecting to 172.23.122.131 with username:root, attempt#1 of 5 [2021-02-22 18:01:58,756] - [remote_util] [139837458200384] - INFO - SSH Connected to 172.23.122.131 as root [2021-02-22 18:01:59,067] - [remote_util] [139837458200384] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 18:01:59,378] - [remote_util] [139837458200384] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 18:01:59,391] - [remote_util] [139837458200384] - INFO - SSH Connecting to 172.23.122.131 with username:root, attempt#1 of 5 [2021-02-22 18:01:59,495] - [remote_util] [139837458200384] - INFO - SSH Connected to 172.23.122.131 as root [2021-02-22 18:01:59,779] - [remote_util] [139837458200384] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 18:02:00,092] - [remote_util] [139837458200384] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 18:02:00,154] - [remote_util] [139837458200384] - INFO - 172.23.122.131 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-22 18:02:00,337] - [remote_util] [139837458200384] - INFO - Running systemd command on this server [2021-02-22 18:02:00,337] - [remote_util] [139837458200384] - INFO - running command.raw on 172.23.122.131: systemctl stop couchbase-server.service [2021-02-22 18:02:02,445] - [remote_util] [139837458200384] - INFO - command executed successfully with root [2021-02-22 18:02:02,445] - [basetestcase] [139837458200384] - INFO - Couchbase stopped [2021-02-22 18:02:02,445] - [basetestcase] [139837458200384] - INFO - sleep for 5 secs. ... [2021-02-22 18:02:07,448] - [remote_util] [139837458200384] - INFO - running command.raw on 172.23.122.131: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-22 18:02:07,464] - [remote_util] [139837458200384] - INFO - command executed successfully with root [2021-02-22 18:02:07,465] - [remote_util] [139837458200384] - INFO - running command.raw on 172.23.122.131: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-22 18:02:07,521] - [remote_util] [139837458200384] - INFO - command executed successfully with root [2021-02-22 18:02:07,525] - [remote_util] [139837458200384] - INFO - SSH Connecting to 172.23.122.131 with username:root, attempt#1 of 5 [2021-02-22 18:02:07,633] - [remote_util] [139837458200384] - INFO - SSH Connected to 172.23.122.131 as root [2021-02-22 18:02:07,932] - [remote_util] [139837458200384] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 18:02:08,243] - [remote_util] [139837458200384] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 18:02:08,307] - [remote_util] [139837458200384] - INFO - 172.23.122.131 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-22 18:02:08,307] - [remote_util] [139837458200384] - INFO - Checking for process beam.smp on linux [2021-02-22 18:02:08,389] - [remote_util] [139837458200384] - INFO - Starting couchbase server [2021-02-22 18:02:08,536] - [remote_util] [139837458200384] - INFO - Running systemd command on this server [2021-02-22 18:02:08,536] - [remote_util] [139837458200384] - INFO - running command.raw on 172.23.122.131: systemctl start couchbase-server.service [2021-02-22 18:02:08,562] - [remote_util] [139837458200384] - INFO - command executed successfully with root [2021-02-22 18:02:08,563] - [remote_util] [139837458200384] - INFO - 172.23.122.131:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-22 18:02:13,568] - [remote_util] [139837458200384] - INFO - running command.raw on 172.23.122.131: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-22 18:02:13,594] - [remote_util] [139837458200384] - INFO - command executed successfully with root [2021-02-22 18:02:13,595] - [remote_util] [139837458200384] - INFO - Couchbase server status: [' Process: 107469 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-22 18:02:13,595] - [remote_util] [139837458200384] - INFO - Checking for process beam.smp on linux [2021-02-22 18:02:13,664] - [remote_util] [139837458200384] - INFO - process beam.smp is running on 172.23.122.131: with pid 107614 [2021-02-22 18:02:13,664] - [basetestcase] [139837458200384] - INFO - Couchbase started [2021-02-22 18:02:13,668] - [remote_util] [139837458200384] - INFO - SSH Connecting to 172.23.122.180 with username:root, attempt#1 of 5 [2021-02-22 18:02:13,771] - [remote_util] [139837458200384] - INFO - SSH Connected to 172.23.122.180 as root [2021-02-22 18:02:14,051] - [remote_util] [139837458200384] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 18:02:14,354] - [remote_util] [139837458200384] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 18:02:14,368] - [remote_util] [139837458200384] - INFO - SSH Connecting to 172.23.122.180 with username:root, attempt#1 of 5 [2021-02-22 18:02:14,467] - [remote_util] [139837458200384] - INFO - SSH Connected to 172.23.122.180 as root [2021-02-22 18:02:14,735] - [remote_util] [139837458200384] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 18:02:15,041] - [remote_util] [139837458200384] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 18:02:15,102] - [remote_util] [139837458200384] - INFO - 172.23.122.180 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-22 18:02:15,241] - [remote_util] [139837458200384] - INFO - Running systemd command on this server [2021-02-22 18:02:15,241] - [remote_util] [139837458200384] - INFO - running command.raw on 172.23.122.180: systemctl stop couchbase-server.service [2021-02-22 18:02:16,875] - [remote_util] [139837458200384] - INFO - command executed successfully with root [2021-02-22 18:02:16,876] - [basetestcase] [139837458200384] - INFO - Couchbase stopped [2021-02-22 18:02:16,876] - [basetestcase] [139837458200384] - INFO - sleep for 5 secs. ... [2021-02-22 18:02:21,881] - [remote_util] [139837458200384] - INFO - running command.raw on 172.23.122.180: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-22 18:02:21,898] - [remote_util] [139837458200384] - INFO - command executed successfully with root [2021-02-22 18:02:21,899] - [remote_util] [139837458200384] - INFO - running command.raw on 172.23.122.180: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-22 18:02:21,953] - [remote_util] [139837458200384] - INFO - command executed successfully with root [2021-02-22 18:02:21,957] - [remote_util] [139837458200384] - INFO - SSH Connecting to 172.23.122.180 with username:root, attempt#1 of 5 [2021-02-22 18:02:22,061] - [remote_util] [139837458200384] - INFO - SSH Connected to 172.23.122.180 as root [2021-02-22 18:02:22,326] - [remote_util] [139837458200384] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 18:02:22,639] - [remote_util] [139837458200384] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 18:02:22,701] - [remote_util] [139837458200384] - INFO - 172.23.122.180 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-22 18:02:22,702] - [remote_util] [139837458200384] - INFO - Checking for process beam.smp on linux [2021-02-22 18:02:22,780] - [remote_util] [139837458200384] - INFO - Starting couchbase server [2021-02-22 18:02:22,922] - [remote_util] [139837458200384] - INFO - Running systemd command on this server [2021-02-22 18:02:22,922] - [remote_util] [139837458200384] - INFO - running command.raw on 172.23.122.180: systemctl start couchbase-server.service [2021-02-22 18:02:22,945] - [remote_util] [139837458200384] - INFO - command executed successfully with root [2021-02-22 18:02:22,945] - [remote_util] [139837458200384] - INFO - 172.23.122.180:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-22 18:02:27,947] - [remote_util] [139837458200384] - INFO - running command.raw on 172.23.122.180: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-22 18:02:27,974] - [remote_util] [139837458200384] - INFO - command executed successfully with root [2021-02-22 18:02:27,974] - [remote_util] [139837458200384] - INFO - Couchbase server status: [' Process: 118863 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-22 18:02:27,975] - [remote_util] [139837458200384] - INFO - Checking for process beam.smp on linux [2021-02-22 18:02:28,045] - [remote_util] [139837458200384] - INFO - process beam.smp is running on 172.23.122.180: with pid 119007 [2021-02-22 18:02:28,046] - [basetestcase] [139837458200384] - INFO - Couchbase started [2021-02-22 18:02:28,049] - [remote_util] [139837458200384] - INFO - SSH Connecting to 172.23.122.130 with username:root, attempt#1 of 5 [2021-02-22 18:02:28,153] - [remote_util] [139837458200384] - INFO - SSH Connected to 172.23.122.130 as root [2021-02-22 18:02:28,416] - [remote_util] [139837458200384] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 18:02:28,753] - [remote_util] [139837458200384] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 18:02:28,767] - [remote_util] [139837458200384] - INFO - SSH Connecting to 172.23.122.130 with username:root, attempt#1 of 5 [2021-02-22 18:02:28,867] - [remote_util] [139837458200384] - INFO - SSH Connected to 172.23.122.130 as root [2021-02-22 18:02:29,126] - [remote_util] [139837458200384] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 18:02:29,428] - [remote_util] [139837458200384] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 18:02:29,494] - [remote_util] [139837458200384] - INFO - 172.23.122.130 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-22 18:02:29,681] - [remote_util] [139837458200384] - INFO - Running systemd command on this server [2021-02-22 18:02:29,681] - [remote_util] [139837458200384] - INFO - running command.raw on 172.23.122.130: systemctl stop couchbase-server.service [2021-02-22 18:02:31,536] - [remote_util] [139837458200384] - INFO - command executed successfully with root [2021-02-22 18:02:31,536] - [basetestcase] [139837458200384] - INFO - Couchbase stopped [2021-02-22 18:02:31,537] - [basetestcase] [139837458200384] - INFO - sleep for 5 secs. ... [2021-02-22 18:02:36,537] - [remote_util] [139837458200384] - INFO - running command.raw on 172.23.122.130: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-22 18:02:36,553] - [remote_util] [139837458200384] - INFO - command executed successfully with root [2021-02-22 18:02:36,554] - [remote_util] [139837458200384] - INFO - running command.raw on 172.23.122.130: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-22 18:02:36,608] - [remote_util] [139837458200384] - INFO - command executed successfully with root [2021-02-22 18:02:36,612] - [remote_util] [139837458200384] - INFO - SSH Connecting to 172.23.122.130 with username:root, attempt#1 of 5 [2021-02-22 18:02:36,711] - [remote_util] [139837458200384] - INFO - SSH Connected to 172.23.122.130 as root [2021-02-22 18:02:36,978] - [remote_util] [139837458200384] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 18:02:37,277] - [remote_util] [139837458200384] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 18:02:37,337] - [remote_util] [139837458200384] - INFO - 172.23.122.130 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-22 18:02:37,338] - [remote_util] [139837458200384] - INFO - Checking for process beam.smp on linux [2021-02-22 18:02:37,412] - [remote_util] [139837458200384] - INFO - Starting couchbase server [2021-02-22 18:02:37,517] - [remote_util] [139837458200384] - INFO - Running systemd command on this server [2021-02-22 18:02:37,517] - [remote_util] [139837458200384] - INFO - running command.raw on 172.23.122.130: systemctl start couchbase-server.service [2021-02-22 18:02:37,542] - [remote_util] [139837458200384] - INFO - command executed successfully with root [2021-02-22 18:02:37,542] - [remote_util] [139837458200384] - INFO - 172.23.122.130:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-22 18:02:42,548] - [remote_util] [139837458200384] - INFO - running command.raw on 172.23.122.130: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-22 18:02:42,587] - [remote_util] [139837458200384] - INFO - command executed successfully with root [2021-02-22 18:02:42,588] - [remote_util] [139837458200384] - INFO - Couchbase server status: [' Process: 9056 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-22 18:02:42,588] - [remote_util] [139837458200384] - INFO - Checking for process beam.smp on linux [2021-02-22 18:02:42,661] - [remote_util] [139837458200384] - INFO - process beam.smp is running on 172.23.122.130: with pid 9202 [2021-02-22 18:02:42,662] - [basetestcase] [139837458200384] - INFO - Couchbase started [2021-02-22 18:02:42,662] - [basetestcase] [139837458200384] - INFO - sleep for 10 secs. ...