[2021-02-06 19:50:39,912] - [eventing_base] [140487281329920] - INFO - Starting Test: test_kv_swap_rebalance_when_existing_eventing_node_is_processing_mutations [2021-02-06 19:50:39,923] - [remote_util] [140487281329920] - INFO - SSH Connecting to 172.23.122.170 with username:root, attempt#1 of 5 [2021-02-06 19:50:40,024] - [remote_util] [140487281329920] - INFO - SSH Connected to 172.23.122.170 as root [2021-02-06 19:50:40,283] - [remote_util] [140487281329920] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-06 19:50:40,573] - [remote_util] [140487281329920] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-06 19:50:41,488] - [rest_client] [140487281329920] - INFO - -->is_ns_server_running? [2021-02-06 19:50:41,509] - [rest_client] [140487281329920] - INFO - Node version in cluster 7.0.0-4390-enterprise [2021-02-06 19:50:41,525] - [rest_client] [140487281329920] - ERROR - GET http://172.23.122.170:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-02-06 19:50:41,566] - [rest_client] [140487281329920] - INFO - Node versions in cluster [] [2021-02-06 19:50:41,577] - [basetestcase] [140487281329920] - INFO - ============== basetestcase setup was started for test #6 test_kv_swap_rebalance_when_existing_eventing_node_is_processing_mutations============== [2021-02-06 19:50:41,611] - [rest_client] [140487281329920] - ERROR - GET http://172.23.122.170:8091/pools/default/ body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-02-06 19:50:41,676] - [bucket_helper] [140487281329920] - INFO - Could not find any buckets for node 172.23.122.170, nothing to delete [2021-02-06 19:50:41,686] - [bucket_helper] [140487281329920] - INFO - Could not find any buckets for node 172.23.122.172, nothing to delete [2021-02-06 19:50:41,697] - [bucket_helper] [140487281329920] - INFO - Could not find any buckets for node 172.23.122.177, nothing to delete [2021-02-06 19:50:41,707] - [bucket_helper] [140487281329920] - INFO - Could not find any buckets for node 172.23.122.178, nothing to delete [2021-02-06 19:50:41,716] - [bucket_helper] [140487281329920] - INFO - Could not find any buckets for node 172.23.122.175, nothing to delete [2021-02-06 19:50:41,732] - [bucket_helper] [140487281329920] - INFO - Could not find any buckets for node 172.23.122.173, nothing to delete [2021-02-06 19:50:41,743] - [bucket_helper] [140487281329920] - INFO - Could not find any buckets for node 172.23.122.174, nothing to delete [2021-02-06 19:50:41,754] - [bucket_helper] [140487281329920] - INFO - Could not find any buckets for node 172.23.122.176, nothing to delete [2021-02-06 19:50:41,795] - [rest_client] [140487281329920] - INFO - -->is_ns_server_running? [2021-02-06 19:50:41,809] - [cluster_helper] [140487281329920] - INFO - waiting for ns_server @ 172.23.122.170:8091 [2021-02-06 19:50:41,809] - [rest_client] [140487281329920] - INFO - -->is_ns_server_running? [2021-02-06 19:50:41,812] - [cluster_helper] [140487281329920] - INFO - ns_server @ 172.23.122.170:8091 is running [2021-02-06 19:50:41,815] - [cluster_helper] [140487281329920] - INFO - waiting for ns_server @ 172.23.122.172:8091 [2021-02-06 19:50:41,815] - [rest_client] [140487281329920] - INFO - -->is_ns_server_running? [2021-02-06 19:50:41,818] - [cluster_helper] [140487281329920] - INFO - ns_server @ 172.23.122.172:8091 is running [2021-02-06 19:50:41,821] - [cluster_helper] [140487281329920] - INFO - waiting for ns_server @ 172.23.122.177:8091 [2021-02-06 19:50:41,821] - [rest_client] [140487281329920] - INFO - -->is_ns_server_running? [2021-02-06 19:50:41,824] - [cluster_helper] [140487281329920] - INFO - ns_server @ 172.23.122.177:8091 is running [2021-02-06 19:50:41,827] - [cluster_helper] [140487281329920] - INFO - waiting for ns_server @ 172.23.122.178:8091 [2021-02-06 19:50:41,827] - [rest_client] [140487281329920] - INFO - -->is_ns_server_running? [2021-02-06 19:50:41,830] - [cluster_helper] [140487281329920] - INFO - ns_server @ 172.23.122.178:8091 is running [2021-02-06 19:50:41,833] - [cluster_helper] [140487281329920] - INFO - waiting for ns_server @ 172.23.122.175:8091 [2021-02-06 19:50:41,833] - [rest_client] [140487281329920] - INFO - -->is_ns_server_running? [2021-02-06 19:50:41,836] - [cluster_helper] [140487281329920] - INFO - ns_server @ 172.23.122.175:8091 is running [2021-02-06 19:50:41,839] - [cluster_helper] [140487281329920] - INFO - waiting for ns_server @ 172.23.122.173:8091 [2021-02-06 19:50:41,839] - [rest_client] [140487281329920] - INFO - -->is_ns_server_running? [2021-02-06 19:50:41,841] - [cluster_helper] [140487281329920] - INFO - ns_server @ 172.23.122.173:8091 is running [2021-02-06 19:50:41,844] - [cluster_helper] [140487281329920] - INFO - waiting for ns_server @ 172.23.122.174:8091 [2021-02-06 19:50:41,845] - [rest_client] [140487281329920] - INFO - -->is_ns_server_running? [2021-02-06 19:50:41,847] - [cluster_helper] [140487281329920] - INFO - ns_server @ 172.23.122.174:8091 is running [2021-02-06 19:50:41,850] - [cluster_helper] [140487281329920] - INFO - waiting for ns_server @ 172.23.122.176:8091 [2021-02-06 19:50:41,851] - [rest_client] [140487281329920] - INFO - -->is_ns_server_running? [2021-02-06 19:50:41,854] - [cluster_helper] [140487281329920] - INFO - ns_server @ 172.23.122.176:8091 is running [2021-02-06 19:50:41,860] - [basetestcase] [140487281329920] - INFO - initializing cluster [2021-02-06 19:50:41,861] - [remote_util] [140487281329920] - INFO - SSH Connecting to 172.23.122.170 with username:root, attempt#1 of 5 [2021-02-06 19:50:41,961] - [remote_util] [140487281329920] - INFO - SSH Connected to 172.23.122.170 as root [2021-02-06 19:50:42,213] - [remote_util] [140487281329920] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-06 19:50:42,507] - [remote_util] [140487281329920] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-06 19:50:42,514] - [remote_util] [140487281329920] - INFO - SSH Connecting to 172.23.122.170 with username:root, attempt#1 of 5 [2021-02-06 19:50:42,609] - [remote_util] [140487281329920] - INFO - SSH Connected to 172.23.122.170 as root [2021-02-06 19:50:42,855] - [remote_util] [140487281329920] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-06 19:50:43,147] - [remote_util] [140487281329920] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-06 19:50:43,206] - [remote_util] [140487281329920] - INFO - 172.23.122.170 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-06 19:50:43,260] - [remote_util] [140487281329920] - INFO - Running systemd command on this server [2021-02-06 19:50:43,260] - [remote_util] [140487281329920] - INFO - running command.raw on 172.23.122.170: systemctl stop couchbase-server.service [2021-02-06 19:50:45,124] - [remote_util] [140487281329920] - INFO - command executed successfully with root [2021-02-06 19:50:45,125] - [basetestcase] [140487281329920] - INFO - Couchbase stopped [2021-02-06 19:50:45,125] - [basetestcase] [140487281329920] - INFO - sleep for 5 secs. ... [2021-02-06 19:50:50,130] - [remote_util] [140487281329920] - INFO - running command.raw on 172.23.122.170: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-06 19:50:50,146] - [remote_util] [140487281329920] - INFO - command executed successfully with root [2021-02-06 19:50:50,146] - [remote_util] [140487281329920] - INFO - running command.raw on 172.23.122.170: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-06 19:50:50,198] - [remote_util] [140487281329920] - INFO - command executed successfully with root [2021-02-06 19:50:50,293] - [remote_util] [140487281329920] - INFO - SSH Connecting to 172.23.122.170 with username:root, attempt#1 of 5 [2021-02-06 19:50:50,596] - [remote_util] [140487281329920] - INFO - SSH Connected to 172.23.122.170 as root [2021-02-06 19:50:50,849] - [remote_util] [140487281329920] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-06 19:50:51,136] - [remote_util] [140487281329920] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-06 19:50:51,155] - [remote_util] [140487281329920] - INFO - 172.23.122.170 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-06 19:50:51,155] - [remote_util] [140487281329920] - INFO - Checking for process beam.smp on linux [2021-02-06 19:50:51,190] - [remote_util] [140487281329920] - INFO - Starting couchbase server [2021-02-06 19:50:51,283] - [remote_util] [140487281329920] - INFO - Running systemd command on this server [2021-02-06 19:50:51,284] - [remote_util] [140487281329920] - INFO - running command.raw on 172.23.122.170: systemctl start couchbase-server.service [2021-02-06 19:50:51,304] - [remote_util] [140487281329920] - INFO - command executed successfully with root [2021-02-06 19:50:51,305] - [remote_util] [140487281329920] - INFO - 172.23.122.170:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-06 19:50:56,310] - [remote_util] [140487281329920] - INFO - running command.raw on 172.23.122.170: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-06 19:50:56,333] - [remote_util] [140487281329920] - INFO - command executed successfully with root [2021-02-06 19:50:56,333] - [remote_util] [140487281329920] - INFO - Couchbase server status: [' Process: 62163 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-06 19:50:56,333] - [remote_util] [140487281329920] - INFO - Checking for process beam.smp on linux [2021-02-06 19:50:56,402] - [remote_util] [140487281329920] - INFO - process beam.smp is running on 172.23.122.170: with pid 62307 [2021-02-06 19:50:56,402] - [basetestcase] [140487281329920] - INFO - Couchbase started [2021-02-06 19:50:56,405] - [remote_util] [140487281329920] - INFO - SSH Connecting to 172.23.122.172 with username:root, attempt#1 of 5 [2021-02-06 19:50:56,508] - [remote_util] [140487281329920] - INFO - SSH Connected to 172.23.122.172 as root [2021-02-06 19:50:56,789] - [remote_util] [140487281329920] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-06 19:50:57,081] - [remote_util] [140487281329920] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-06 19:50:57,089] - [remote_util] [140487281329920] - INFO - SSH Connecting to 172.23.122.172 with username:root, attempt#1 of 5 [2021-02-06 19:50:57,189] - [remote_util] [140487281329920] - INFO - SSH Connected to 172.23.122.172 as root [2021-02-06 19:50:57,462] - [remote_util] [140487281329920] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-06 19:50:57,760] - [remote_util] [140487281329920] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-06 19:50:57,779] - [remote_util] [140487281329920] - INFO - 172.23.122.172 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-06 19:50:57,834] - [remote_util] [140487281329920] - INFO - Running systemd command on this server [2021-02-06 19:50:57,834] - [remote_util] [140487281329920] - INFO - running command.raw on 172.23.122.172: systemctl stop couchbase-server.service [2021-02-06 19:50:59,443] - [remote_util] [140487281329920] - INFO - command executed successfully with root [2021-02-06 19:50:59,455] - [basetestcase] [140487281329920] - INFO - Couchbase stopped [2021-02-06 19:50:59,456] - [basetestcase] [140487281329920] - INFO - sleep for 5 secs. ... [2021-02-06 19:51:04,461] - [remote_util] [140487281329920] - INFO - running command.raw on 172.23.122.172: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-06 19:51:04,475] - [remote_util] [140487281329920] - INFO - command executed successfully with root [2021-02-06 19:51:04,475] - [remote_util] [140487281329920] - INFO - running command.raw on 172.23.122.172: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-06 19:51:04,529] - [remote_util] [140487281329920] - INFO - command executed successfully with root [2021-02-06 19:51:04,531] - [remote_util] [140487281329920] - INFO - SSH Connecting to 172.23.122.172 with username:root, attempt#1 of 5 [2021-02-06 19:51:04,628] - [remote_util] [140487281329920] - INFO - SSH Connected to 172.23.122.172 as root [2021-02-06 19:51:04,898] - [remote_util] [140487281329920] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-06 19:51:05,181] - [remote_util] [140487281329920] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-06 19:51:05,240] - [remote_util] [140487281329920] - INFO - 172.23.122.172 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-06 19:51:05,240] - [remote_util] [140487281329920] - INFO - Checking for process beam.smp on linux [2021-02-06 19:51:05,276] - [remote_util] [140487281329920] - INFO - Starting couchbase server [2021-02-06 19:51:05,369] - [remote_util] [140487281329920] - INFO - Running systemd command on this server [2021-02-06 19:51:05,369] - [remote_util] [140487281329920] - INFO - running command.raw on 172.23.122.172: systemctl start couchbase-server.service [2021-02-06 19:51:05,389] - [remote_util] [140487281329920] - INFO - command executed successfully with root [2021-02-06 19:51:05,389] - [remote_util] [140487281329920] - INFO - 172.23.122.172:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-06 19:51:10,392] - [remote_util] [140487281329920] - INFO - running command.raw on 172.23.122.172: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-06 19:51:10,419] - [remote_util] [140487281329920] - INFO - command executed successfully with root [2021-02-06 19:51:10,419] - [remote_util] [140487281329920] - INFO - Couchbase server status: [' Process: 125534 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-06 19:51:10,419] - [remote_util] [140487281329920] - INFO - Checking for process beam.smp on linux [2021-02-06 19:51:10,485] - [remote_util] [140487281329920] - INFO - process beam.smp is running on 172.23.122.172: with pid 125722 [2021-02-06 19:51:10,485] - [basetestcase] [140487281329920] - INFO - Couchbase started [2021-02-06 19:51:10,487] - [remote_util] [140487281329920] - INFO - SSH Connecting to 172.23.122.177 with username:root, attempt#1 of 5 [2021-02-06 19:51:10,593] - [remote_util] [140487281329920] - INFO - SSH Connected to 172.23.122.177 as root [2021-02-06 19:51:10,858] - [remote_util] [140487281329920] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-06 19:51:11,154] - [remote_util] [140487281329920] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-06 19:51:11,164] - [remote_util] [140487281329920] - INFO - SSH Connecting to 172.23.122.177 with username:root, attempt#1 of 5 [2021-02-06 19:51:11,262] - [remote_util] [140487281329920] - INFO - SSH Connected to 172.23.122.177 as root [2021-02-06 19:51:11,526] - [remote_util] [140487281329920] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-06 19:51:11,823] - [remote_util] [140487281329920] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-06 19:51:11,880] - [remote_util] [140487281329920] - INFO - 172.23.122.177 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-06 19:51:11,932] - [remote_util] [140487281329920] - INFO - Running systemd command on this server [2021-02-06 19:51:11,932] - [remote_util] [140487281329920] - INFO - running command.raw on 172.23.122.177: systemctl stop couchbase-server.service [2021-02-06 19:51:13,879] - [remote_util] [140487281329920] - INFO - command executed successfully with root [2021-02-06 19:51:13,879] - [basetestcase] [140487281329920] - INFO - Couchbase stopped [2021-02-06 19:51:13,879] - [basetestcase] [140487281329920] - INFO - sleep for 5 secs. ... [2021-02-06 19:51:18,885] - [remote_util] [140487281329920] - INFO - running command.raw on 172.23.122.177: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-06 19:51:18,899] - [remote_util] [140487281329920] - INFO - command executed successfully with root [2021-02-06 19:51:18,900] - [remote_util] [140487281329920] - INFO - running command.raw on 172.23.122.177: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-06 19:51:18,951] - [remote_util] [140487281329920] - INFO - command executed successfully with root [2021-02-06 19:51:18,954] - [remote_util] [140487281329920] - INFO - SSH Connecting to 172.23.122.177 with username:root, attempt#1 of 5 [2021-02-06 19:51:19,050] - [remote_util] [140487281329920] - INFO - SSH Connected to 172.23.122.177 as root [2021-02-06 19:51:19,305] - [remote_util] [140487281329920] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-06 19:51:19,593] - [remote_util] [140487281329920] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-06 19:51:19,651] - [remote_util] [140487281329920] - INFO - 172.23.122.177 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-06 19:51:19,651] - [remote_util] [140487281329920] - INFO - Checking for process beam.smp on linux [2021-02-06 19:51:19,748] - [remote_util] [140487281329920] - INFO - Starting couchbase server [2021-02-06 19:51:19,841] - [remote_util] [140487281329920] - INFO - Running systemd command on this server [2021-02-06 19:51:19,841] - [remote_util] [140487281329920] - INFO - running command.raw on 172.23.122.177: systemctl start couchbase-server.service [2021-02-06 19:51:19,862] - [remote_util] [140487281329920] - INFO - command executed successfully with root [2021-02-06 19:51:19,862] - [remote_util] [140487281329920] - INFO - 172.23.122.177:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-06 19:51:24,867] - [remote_util] [140487281329920] - INFO - running command.raw on 172.23.122.177: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-06 19:51:24,894] - [remote_util] [140487281329920] - INFO - command executed successfully with root [2021-02-06 19:51:24,894] - [remote_util] [140487281329920] - INFO - Couchbase server status: [' Process: 59395 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-06 19:51:24,894] - [remote_util] [140487281329920] - INFO - Checking for process beam.smp on linux [2021-02-06 19:51:24,968] - [remote_util] [140487281329920] - INFO - process beam.smp is running on 172.23.122.177: with pid 59540 [2021-02-06 19:51:24,968] - [basetestcase] [140487281329920] - INFO - Couchbase started [2021-02-06 19:51:24,970] - [remote_util] [140487281329920] - INFO - SSH Connecting to 172.23.122.178 with username:root, attempt#1 of 5 [2021-02-06 19:51:25,074] - [remote_util] [140487281329920] - INFO - SSH Connected to 172.23.122.178 as root [2021-02-06 19:51:25,339] - [remote_util] [140487281329920] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-06 19:51:25,638] - [remote_util] [140487281329920] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-06 19:51:25,647] - [remote_util] [140487281329920] - INFO - SSH Connecting to 172.23.122.178 with username:root, attempt#1 of 5 [2021-02-06 19:51:25,745] - [remote_util] [140487281329920] - INFO - SSH Connected to 172.23.122.178 as root [2021-02-06 19:51:25,999] - [remote_util] [140487281329920] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-06 19:51:26,298] - [remote_util] [140487281329920] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-06 19:51:26,358] - [remote_util] [140487281329920] - INFO - 172.23.122.178 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-06 19:51:26,412] - [remote_util] [140487281329920] - INFO - Running systemd command on this server [2021-02-06 19:51:26,412] - [remote_util] [140487281329920] - INFO - running command.raw on 172.23.122.178: systemctl stop couchbase-server.service [2021-02-06 19:51:28,318] - [remote_util] [140487281329920] - INFO - command executed successfully with root [2021-02-06 19:51:28,318] - [basetestcase] [140487281329920] - INFO - Couchbase stopped [2021-02-06 19:51:28,318] - [basetestcase] [140487281329920] - INFO - sleep for 5 secs. ... [2021-02-06 19:51:33,324] - [remote_util] [140487281329920] - INFO - running command.raw on 172.23.122.178: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-06 19:51:33,339] - [remote_util] [140487281329920] - INFO - command executed successfully with root [2021-02-06 19:51:33,339] - [remote_util] [140487281329920] - INFO - running command.raw on 172.23.122.178: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-06 19:51:33,393] - [remote_util] [140487281329920] - INFO - command executed successfully with root [2021-02-06 19:51:33,395] - [remote_util] [140487281329920] - INFO - SSH Connecting to 172.23.122.178 with username:root, attempt#1 of 5 [2021-02-06 19:51:33,495] - [remote_util] [140487281329920] - INFO - SSH Connected to 172.23.122.178 as root [2021-02-06 19:51:33,753] - [remote_util] [140487281329920] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-06 19:51:34,049] - [remote_util] [140487281329920] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-06 19:51:34,107] - [remote_util] [140487281329920] - INFO - 172.23.122.178 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-06 19:51:34,108] - [remote_util] [140487281329920] - INFO - Checking for process beam.smp on linux [2021-02-06 19:51:34,143] - [remote_util] [140487281329920] - INFO - Starting couchbase server [2021-02-06 19:51:34,244] - [remote_util] [140487281329920] - INFO - Running systemd command on this server [2021-02-06 19:51:34,244] - [remote_util] [140487281329920] - INFO - running command.raw on 172.23.122.178: systemctl start couchbase-server.service [2021-02-06 19:51:34,269] - [remote_util] [140487281329920] - INFO - command executed successfully with root [2021-02-06 19:51:34,269] - [remote_util] [140487281329920] - INFO - 172.23.122.178:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-06 19:51:39,274] - [remote_util] [140487281329920] - INFO - running command.raw on 172.23.122.178: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-06 19:51:39,298] - [remote_util] [140487281329920] - INFO - command executed successfully with root [2021-02-06 19:51:39,298] - [remote_util] [140487281329920] - INFO - Couchbase server status: [' Process: 32257 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-06 19:51:39,298] - [remote_util] [140487281329920] - INFO - Checking for process beam.smp on linux [2021-02-06 19:51:39,365] - [remote_util] [140487281329920] - INFO - process beam.smp is running on 172.23.122.178: with pid 32402 [2021-02-06 19:51:39,366] - [basetestcase] [140487281329920] - INFO - Couchbase started [2021-02-06 19:51:39,367] - [remote_util] [140487281329920] - INFO - SSH Connecting to 172.23.122.175 with username:root, attempt#1 of 5 [2021-02-06 19:51:39,473] - [remote_util] [140487281329920] - INFO - SSH Connected to 172.23.122.175 as root [2021-02-06 19:51:39,739] - [remote_util] [140487281329920] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-06 19:51:40,035] - [remote_util] [140487281329920] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-06 19:51:40,043] - [remote_util] [140487281329920] - INFO - SSH Connecting to 172.23.122.175 with username:root, attempt#1 of 5 [2021-02-06 19:51:40,139] - [remote_util] [140487281329920] - INFO - SSH Connected to 172.23.122.175 as root [2021-02-06 19:51:40,394] - [remote_util] [140487281329920] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-06 19:51:40,689] - [remote_util] [140487281329920] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-06 19:51:40,709] - [remote_util] [140487281329920] - INFO - 172.23.122.175 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-06 19:51:40,762] - [remote_util] [140487281329920] - INFO - Running systemd command on this server [2021-02-06 19:51:40,762] - [remote_util] [140487281329920] - INFO - running command.raw on 172.23.122.175: systemctl stop couchbase-server.service [2021-02-06 19:51:42,554] - [remote_util] [140487281329920] - INFO - command executed successfully with root [2021-02-06 19:51:42,554] - [basetestcase] [140487281329920] - INFO - Couchbase stopped [2021-02-06 19:51:42,554] - [basetestcase] [140487281329920] - INFO - sleep for 5 secs. ... [2021-02-06 19:51:47,557] - [remote_util] [140487281329920] - INFO - running command.raw on 172.23.122.175: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-06 19:51:47,572] - [remote_util] [140487281329920] - INFO - command executed successfully with root [2021-02-06 19:51:47,572] - [remote_util] [140487281329920] - INFO - running command.raw on 172.23.122.175: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-06 19:51:47,626] - [remote_util] [140487281329920] - INFO - command executed successfully with root [2021-02-06 19:51:47,628] - [remote_util] [140487281329920] - INFO - SSH Connecting to 172.23.122.175 with username:root, attempt#1 of 5 [2021-02-06 19:51:47,726] - [remote_util] [140487281329920] - INFO - SSH Connected to 172.23.122.175 as root [2021-02-06 19:51:47,990] - [remote_util] [140487281329920] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-06 19:51:48,285] - [remote_util] [140487281329920] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-06 19:51:48,306] - [remote_util] [140487281329920] - INFO - 172.23.122.175 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-06 19:51:48,306] - [remote_util] [140487281329920] - INFO - Checking for process beam.smp on linux [2021-02-06 19:51:48,385] - [remote_util] [140487281329920] - INFO - Starting couchbase server [2021-02-06 19:51:48,483] - [remote_util] [140487281329920] - INFO - Running systemd command on this server [2021-02-06 19:51:48,483] - [remote_util] [140487281329920] - INFO - running command.raw on 172.23.122.175: systemctl start couchbase-server.service [2021-02-06 19:51:48,506] - [remote_util] [140487281329920] - INFO - command executed successfully with root [2021-02-06 19:51:48,506] - [remote_util] [140487281329920] - INFO - 172.23.122.175:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-06 19:51:53,511] - [remote_util] [140487281329920] - INFO - running command.raw on 172.23.122.175: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-06 19:51:53,538] - [remote_util] [140487281329920] - INFO - command executed successfully with root [2021-02-06 19:51:53,538] - [remote_util] [140487281329920] - INFO - Couchbase server status: [' Process: 46455 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-06 19:51:53,538] - [remote_util] [140487281329920] - INFO - Checking for process beam.smp on linux [2021-02-06 19:51:53,606] - [remote_util] [140487281329920] - INFO - process beam.smp is running on 172.23.122.175: with pid 46599 [2021-02-06 19:51:53,606] - [basetestcase] [140487281329920] - INFO - Couchbase started [2021-02-06 19:51:53,608] - [remote_util] [140487281329920] - INFO - SSH Connecting to 172.23.122.173 with username:root, attempt#1 of 5 [2021-02-06 19:51:53,713] - [remote_util] [140487281329920] - INFO - SSH Connected to 172.23.122.173 as root [2021-02-06 19:51:54,066] - [remote_util] [140487281329920] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-06 19:51:54,361] - [remote_util] [140487281329920] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-06 19:51:54,369] - [remote_util] [140487281329920] - INFO - SSH Connecting to 172.23.122.173 with username:root, attempt#1 of 5 [2021-02-06 19:51:54,617] - [remote_util] [140487281329920] - INFO - SSH Connected to 172.23.122.173 as root [2021-02-06 19:51:54,863] - [remote_util] [140487281329920] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-06 19:51:55,155] - [remote_util] [140487281329920] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-06 19:51:55,176] - [remote_util] [140487281329920] - INFO - 172.23.122.173 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-06 19:51:55,231] - [remote_util] [140487281329920] - INFO - Running systemd command on this server [2021-02-06 19:51:55,231] - [remote_util] [140487281329920] - INFO - running command.raw on 172.23.122.173: systemctl stop couchbase-server.service [2021-02-06 19:51:56,870] - [remote_util] [140487281329920] - INFO - command executed successfully with root [2021-02-06 19:51:56,871] - [basetestcase] [140487281329920] - INFO - Couchbase stopped [2021-02-06 19:51:56,871] - [basetestcase] [140487281329920] - INFO - sleep for 5 secs. ... [2021-02-06 19:52:01,876] - [remote_util] [140487281329920] - INFO - running command.raw on 172.23.122.173: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-06 19:52:01,890] - [remote_util] [140487281329920] - INFO - command executed successfully with root [2021-02-06 19:52:01,890] - [remote_util] [140487281329920] - INFO - running command.raw on 172.23.122.173: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-06 19:52:01,942] - [remote_util] [140487281329920] - INFO - command executed successfully with root [2021-02-06 19:52:01,944] - [remote_util] [140487281329920] - INFO - SSH Connecting to 172.23.122.173 with username:root, attempt#1 of 5 [2021-02-06 19:52:02,039] - [remote_util] [140487281329920] - INFO - SSH Connected to 172.23.122.173 as root [2021-02-06 19:52:02,288] - [remote_util] [140487281329920] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-06 19:52:02,578] - [remote_util] [140487281329920] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-06 19:52:02,638] - [remote_util] [140487281329920] - INFO - 172.23.122.173 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-06 19:52:02,639] - [remote_util] [140487281329920] - INFO - Checking for process beam.smp on linux [2021-02-06 19:52:02,674] - [remote_util] [140487281329920] - INFO - Starting couchbase server [2021-02-06 19:52:02,766] - [remote_util] [140487281329920] - INFO - Running systemd command on this server [2021-02-06 19:52:02,766] - [remote_util] [140487281329920] - INFO - running command.raw on 172.23.122.173: systemctl start couchbase-server.service [2021-02-06 19:52:02,787] - [remote_util] [140487281329920] - INFO - command executed successfully with root [2021-02-06 19:52:02,787] - [remote_util] [140487281329920] - INFO - 172.23.122.173:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-06 19:52:07,793] - [remote_util] [140487281329920] - INFO - running command.raw on 172.23.122.173: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-06 19:52:07,819] - [remote_util] [140487281329920] - INFO - command executed successfully with root [2021-02-06 19:52:07,819] - [remote_util] [140487281329920] - INFO - Couchbase server status: [' Process: 93185 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-06 19:52:07,819] - [remote_util] [140487281329920] - INFO - Checking for process beam.smp on linux [2021-02-06 19:52:07,888] - [remote_util] [140487281329920] - INFO - process beam.smp is running on 172.23.122.173: with pid 93330 [2021-02-06 19:52:07,888] - [basetestcase] [140487281329920] - INFO - Couchbase started [2021-02-06 19:52:07,890] - [remote_util] [140487281329920] - INFO - SSH Connecting to 172.23.122.174 with username:root, attempt#1 of 5 [2021-02-06 19:52:07,994] - [remote_util] [140487281329920] - INFO - SSH Connected to 172.23.122.174 as root [2021-02-06 19:52:08,249] - [remote_util] [140487281329920] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-06 19:52:08,545] - [remote_util] [140487281329920] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-06 19:52:08,555] - [remote_util] [140487281329920] - INFO - SSH Connecting to 172.23.122.174 with username:root, attempt#1 of 5 [2021-02-06 19:52:08,652] - [remote_util] [140487281329920] - INFO - SSH Connected to 172.23.122.174 as root [2021-02-06 19:52:08,903] - [remote_util] [140487281329920] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-06 19:52:09,194] - [remote_util] [140487281329920] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-06 19:52:09,253] - [remote_util] [140487281329920] - INFO - 172.23.122.174 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-06 19:52:09,306] - [remote_util] [140487281329920] - INFO - Running systemd command on this server [2021-02-06 19:52:09,306] - [remote_util] [140487281329920] - INFO - running command.raw on 172.23.122.174: systemctl stop couchbase-server.service [2021-02-06 19:52:10,954] - [remote_util] [140487281329920] - INFO - command executed successfully with root [2021-02-06 19:52:10,954] - [basetestcase] [140487281329920] - INFO - Couchbase stopped [2021-02-06 19:52:10,955] - [basetestcase] [140487281329920] - INFO - sleep for 5 secs. ... [2021-02-06 19:52:15,956] - [remote_util] [140487281329920] - INFO - running command.raw on 172.23.122.174: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-06 19:52:15,972] - [remote_util] [140487281329920] - INFO - command executed successfully with root [2021-02-06 19:52:15,972] - [remote_util] [140487281329920] - INFO - running command.raw on 172.23.122.174: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-06 19:52:16,025] - [remote_util] [140487281329920] - INFO - command executed successfully with root [2021-02-06 19:52:16,027] - [remote_util] [140487281329920] - INFO - SSH Connecting to 172.23.122.174 with username:root, attempt#1 of 5 [2021-02-06 19:52:16,124] - [remote_util] [140487281329920] - INFO - SSH Connected to 172.23.122.174 as root [2021-02-06 19:52:16,369] - [remote_util] [140487281329920] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-06 19:52:16,664] - [remote_util] [140487281329920] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-06 19:52:16,684] - [remote_util] [140487281329920] - INFO - 172.23.122.174 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-06 19:52:16,684] - [remote_util] [140487281329920] - INFO - Checking for process beam.smp on linux [2021-02-06 19:52:16,720] - [remote_util] [140487281329920] - INFO - Starting couchbase server [2021-02-06 19:52:16,814] - [remote_util] [140487281329920] - INFO - Running systemd command on this server [2021-02-06 19:52:16,814] - [remote_util] [140487281329920] - INFO - running command.raw on 172.23.122.174: systemctl start couchbase-server.service [2021-02-06 19:52:16,834] - [remote_util] [140487281329920] - INFO - command executed successfully with root [2021-02-06 19:52:16,834] - [remote_util] [140487281329920] - INFO - 172.23.122.174:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-06 19:52:21,839] - [remote_util] [140487281329920] - INFO - running command.raw on 172.23.122.174: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-06 19:52:21,868] - [remote_util] [140487281329920] - INFO - command executed successfully with root [2021-02-06 19:52:21,868] - [remote_util] [140487281329920] - INFO - Couchbase server status: [' Process: 37532 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-06 19:52:21,869] - [remote_util] [140487281329920] - INFO - Checking for process beam.smp on linux [2021-02-06 19:52:21,938] - [remote_util] [140487281329920] - INFO - process beam.smp is running on 172.23.122.174: with pid 37677 [2021-02-06 19:52:21,938] - [basetestcase] [140487281329920] - INFO - Couchbase started [2021-02-06 19:52:21,940] - [remote_util] [140487281329920] - INFO - SSH Connecting to 172.23.122.176 with username:root, attempt#1 of 5 [2021-02-06 19:52:22,043] - [remote_util] [140487281329920] - INFO - SSH Connected to 172.23.122.176 as root [2021-02-06 19:52:22,302] - [remote_util] [140487281329920] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-06 19:52:22,593] - [remote_util] [140487281329920] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-06 19:52:22,601] - [remote_util] [140487281329920] - INFO - SSH Connecting to 172.23.122.176 with username:root, attempt#1 of 5 [2021-02-06 19:52:22,697] - [remote_util] [140487281329920] - INFO - SSH Connected to 172.23.122.176 as root [2021-02-06 19:52:22,953] - [remote_util] [140487281329920] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-06 19:52:23,250] - [remote_util] [140487281329920] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-06 19:52:23,310] - [remote_util] [140487281329920] - INFO - 172.23.122.176 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-06 19:52:23,364] - [remote_util] [140487281329920] - INFO - Running systemd command on this server [2021-02-06 19:52:23,364] - [remote_util] [140487281329920] - INFO - running command.raw on 172.23.122.176: systemctl stop couchbase-server.service [2021-02-06 19:52:25,094] - [remote_util] [140487281329920] - INFO - command executed successfully with root [2021-02-06 19:52:25,094] - [basetestcase] [140487281329920] - INFO - Couchbase stopped [2021-02-06 19:52:25,095] - [basetestcase] [140487281329920] - INFO - sleep for 5 secs. ... [2021-02-06 19:52:30,100] - [remote_util] [140487281329920] - INFO - running command.raw on 172.23.122.176: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-06 19:52:30,117] - [remote_util] [140487281329920] - INFO - command executed successfully with root [2021-02-06 19:52:30,117] - [remote_util] [140487281329920] - INFO - running command.raw on 172.23.122.176: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-06 19:52:30,171] - [remote_util] [140487281329920] - INFO - command executed successfully with root [2021-02-06 19:52:30,173] - [remote_util] [140487281329920] - INFO - SSH Connecting to 172.23.122.176 with username:root, attempt#1 of 5 [2021-02-06 19:52:30,279] - [remote_util] [140487281329920] - INFO - SSH Connected to 172.23.122.176 as root [2021-02-06 19:52:30,548] - [remote_util] [140487281329920] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-06 19:52:30,846] - [remote_util] [140487281329920] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-06 19:52:30,866] - [remote_util] [140487281329920] - INFO - 172.23.122.176 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-06 19:52:30,866] - [remote_util] [140487281329920] - INFO - Checking for process beam.smp on linux [2021-02-06 19:52:30,901] - [remote_util] [140487281329920] - INFO - Starting couchbase server [2021-02-06 19:52:31,002] - [remote_util] [140487281329920] - INFO - Running systemd command on this server [2021-02-06 19:52:31,002] - [remote_util] [140487281329920] - INFO - running command.raw on 172.23.122.176: systemctl start couchbase-server.service [2021-02-06 19:52:31,025] - [remote_util] [140487281329920] - INFO - command executed successfully with root [2021-02-06 19:52:31,025] - [remote_util] [140487281329920] - INFO - 172.23.122.176:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-06 19:52:36,038] - [remote_util] [140487281329920] - INFO - running command.raw on 172.23.122.176: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-06 19:52:36,085] - [remote_util] [140487281329920] - INFO - command executed successfully with root [2021-02-06 19:52:36,086] - [remote_util] [140487281329920] - INFO - Couchbase server status: [' Process: 39345 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-06 19:52:36,086] - [remote_util] [140487281329920] - INFO - Checking for process beam.smp on linux [2021-02-06 19:52:36,154] - [remote_util] [140487281329920] - INFO - process beam.smp is running on 172.23.122.176: with pid 39489 [2021-02-06 19:52:36,154] - [basetestcase] [140487281329920] - INFO - Couchbase started [2021-02-06 19:52:36,167] - [basetestcase] [140487281329920] - INFO - sleep for 10 secs. ... [2021-02-06 19:52:49,856] - [task] [140488573183744] - INFO - server: ip:172.23.122.170 port:8091 ssh_username:root, nodes/self [2021-02-06 19:52:49,956] - [task] [140488573183744] - INFO - {'uptime': '115', 'memoryTotal': 4201684992, 'memoryFree': 3543597056, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.122.170:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4390-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.122.170', 'ip': '172.23.122.170', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} [2021-02-06 19:52:49,968] - [rest_client] [140488573183744] - ERROR - GET http://172.23.122.170:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-02-06 19:52:50,029] - [rest_client] [140488573183744] - INFO - pools/default params : memoryQuota=2147 [2021-02-06 19:52:50,047] - [rest_client] [140488573183744] - INFO - --> init_node_services(Administrator,password,172.23.122.170,8091,['kv']) [2021-02-06 19:52:50,047] - [rest_client] [140488573183744] - INFO - /node/controller/setupServices params on 172.23.122.170: 8091:hostname=172.23.122.170&user=Administrator&password=password&services=kv [2021-02-06 19:52:50,075] - [rest_client] [140488573183744] - INFO - --> in init_cluster...Administrator,password,8091 [2021-02-06 19:52:50,075] - [rest_client] [140488573183744] - INFO - settings/web params on 172.23.122.170:8091:port=8091&username=Administrator&password=password [2021-02-06 19:52:50,130] - [rest_client] [140488573183744] - INFO - --> status:True [2021-02-06 19:52:50,734] - [remote_util] [140488573183744] - INFO - SSH Connecting to 172.23.122.170 with username:root, attempt#1 of 5 [2021-02-06 19:52:52,518] - [remote_util] [140488573183744] - INFO - SSH Connected to 172.23.122.170 as root [2021-02-06 19:52:53,898] - [remote_util] [140488573183744] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-06 19:52:54,286] - [remote_util] [140488573183744] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-06 19:52:54,307] - [remote_util] [140488573183744] - INFO - running command.raw on 172.23.122.170: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-02-06 19:52:54,350] - [remote_util] [140488573183744] - INFO - command executed successfully with root [2021-02-06 19:52:54,351] - [remote_util] [140488573183744] - INFO - ['ok'] [2021-02-06 19:52:54,355] - [rest_client] [140488573183744] - INFO - /diag/eval status on 172.23.122.170:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-06 19:52:54,357] - [rest_client] [140488573183744] - INFO - /diag/eval status on 172.23.122.170:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-06 19:52:54,361] - [rest_client] [140488573183744] - INFO - settings/indexes params : storageMode=plasma [2021-02-06 19:52:54,379] - [task] [140488573183744] - INFO - server: ip:172.23.122.172 port:8091 ssh_username:root, nodes/self [2021-02-06 19:52:54,383] - [task] [140488573183744] - INFO - {'uptime': '104', 'memoryTotal': 4201684992, 'memoryFree': 3561005056, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.122.172:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4390-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.122.172', 'ip': '172.23.122.172', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} [2021-02-06 19:52:54,385] - [rest_client] [140488573183744] - ERROR - GET http://172.23.122.172:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-02-06 19:52:54,385] - [rest_client] [140488573183744] - INFO - pools/default params : memoryQuota=2147 [2021-02-06 19:52:54,391] - [rest_client] [140488573183744] - INFO - --> in init_cluster...Administrator,password,8091 [2021-02-06 19:52:54,391] - [rest_client] [140488573183744] - INFO - settings/web params on 172.23.122.172:8091:port=8091&username=Administrator&password=password [2021-02-06 19:52:54,448] - [rest_client] [140488573183744] - INFO - --> status:True [2021-02-06 19:52:54,450] - [remote_util] [140488573183744] - INFO - SSH Connecting to 172.23.122.172 with username:root, attempt#1 of 5 [2021-02-06 19:52:54,552] - [remote_util] [140488573183744] - INFO - SSH Connected to 172.23.122.172 as root [2021-02-06 19:52:54,837] - [remote_util] [140488573183744] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-06 19:52:55,132] - [remote_util] [140488573183744] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-06 19:52:55,140] - [remote_util] [140488573183744] - INFO - running command.raw on 172.23.122.172: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-02-06 19:52:55,198] - [remote_util] [140488573183744] - INFO - command executed successfully with root [2021-02-06 19:52:55,198] - [remote_util] [140488573183744] - INFO - ['ok'] [2021-02-06 19:52:55,200] - [rest_client] [140488573183744] - INFO - /diag/eval status on 172.23.122.172:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-06 19:52:55,202] - [rest_client] [140488573183744] - INFO - /diag/eval status on 172.23.122.172:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-06 19:52:55,208] - [rest_client] [140488573183744] - INFO - settings/indexes params : storageMode=plasma [2021-02-06 19:52:55,242] - [task] [140488573183744] - INFO - server: ip:172.23.122.177 port:8091 ssh_username:root, nodes/self [2021-02-06 19:52:55,245] - [task] [140488573183744] - INFO - {'uptime': '90', 'memoryTotal': 4201684992, 'memoryFree': 3582226432, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.122.177:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4390-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.122.177', 'ip': '172.23.122.177', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} [2021-02-06 19:52:55,247] - [rest_client] [140488573183744] - ERROR - GET http://172.23.122.177:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-02-06 19:52:55,248] - [rest_client] [140488573183744] - INFO - pools/default params : memoryQuota=2147 [2021-02-06 19:52:55,254] - [rest_client] [140488573183744] - INFO - --> in init_cluster...Administrator,password,8091 [2021-02-06 19:52:55,254] - [rest_client] [140488573183744] - INFO - settings/web params on 172.23.122.177:8091:port=8091&username=Administrator&password=password [2021-02-06 19:52:55,309] - [rest_client] [140488573183744] - INFO - --> status:True [2021-02-06 19:52:55,311] - [remote_util] [140488573183744] - INFO - SSH Connecting to 172.23.122.177 with username:root, attempt#1 of 5 [2021-02-06 19:52:55,415] - [remote_util] [140488573183744] - INFO - SSH Connected to 172.23.122.177 as root [2021-02-06 19:52:55,688] - [remote_util] [140488573183744] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-06 19:52:55,993] - [remote_util] [140488573183744] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-06 19:52:56,002] - [remote_util] [140488573183744] - INFO - running command.raw on 172.23.122.177: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-02-06 19:52:56,058] - [remote_util] [140488573183744] - INFO - command executed successfully with root [2021-02-06 19:52:56,058] - [remote_util] [140488573183744] - INFO - ['ok'] [2021-02-06 19:52:56,060] - [rest_client] [140488573183744] - INFO - /diag/eval status on 172.23.122.177:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-06 19:52:56,062] - [rest_client] [140488573183744] - INFO - /diag/eval status on 172.23.122.177:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-06 19:52:56,066] - [rest_client] [140488573183744] - INFO - settings/indexes params : storageMode=plasma [2021-02-06 19:52:56,084] - [task] [140488573183744] - INFO - server: ip:172.23.122.178 port:8091 ssh_username:root, nodes/self [2021-02-06 19:52:56,087] - [task] [140488573183744] - INFO - {'uptime': '79', 'memoryTotal': 4201684992, 'memoryFree': 3589365760, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.122.178:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4390-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.122.178', 'ip': '172.23.122.178', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} [2021-02-06 19:52:56,089] - [rest_client] [140488573183744] - ERROR - GET http://172.23.122.178:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-02-06 19:52:56,089] - [rest_client] [140488573183744] - INFO - pools/default params : memoryQuota=2147 [2021-02-06 19:52:56,095] - [rest_client] [140488573183744] - INFO - --> in init_cluster...Administrator,password,8091 [2021-02-06 19:52:56,096] - [rest_client] [140488573183744] - INFO - settings/web params on 172.23.122.178:8091:port=8091&username=Administrator&password=password [2021-02-06 19:52:56,151] - [rest_client] [140488573183744] - INFO - --> status:True [2021-02-06 19:52:56,153] - [remote_util] [140488573183744] - INFO - SSH Connecting to 172.23.122.178 with username:root, attempt#1 of 5 [2021-02-06 19:52:56,278] - [remote_util] [140488573183744] - INFO - SSH Connected to 172.23.122.178 as root [2021-02-06 19:52:56,548] - [remote_util] [140488573183744] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-06 19:52:56,855] - [remote_util] [140488573183744] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-06 19:52:56,864] - [remote_util] [140488573183744] - INFO - running command.raw on 172.23.122.178: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-02-06 19:52:56,920] - [remote_util] [140488573183744] - INFO - command executed successfully with root [2021-02-06 19:52:56,920] - [remote_util] [140488573183744] - INFO - ['ok'] [2021-02-06 19:52:56,922] - [rest_client] [140488573183744] - INFO - /diag/eval status on 172.23.122.178:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-06 19:52:56,924] - [rest_client] [140488573183744] - INFO - /diag/eval status on 172.23.122.178:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-06 19:52:56,928] - [rest_client] [140488573183744] - INFO - settings/indexes params : storageMode=plasma [2021-02-06 19:52:56,948] - [task] [140488573183744] - INFO - server: ip:172.23.122.175 port:8091 ssh_username:root, nodes/self [2021-02-06 19:52:56,951] - [task] [140488573183744] - INFO - {'uptime': '64', 'memoryTotal': 4201684992, 'memoryFree': 3603189760, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.122.175:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4390-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.122.175', 'ip': '172.23.122.175', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} [2021-02-06 19:52:56,953] - [rest_client] [140488573183744] - ERROR - GET http://172.23.122.175:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-02-06 19:52:56,953] - [rest_client] [140488573183744] - INFO - pools/default params : memoryQuota=2147 [2021-02-06 19:52:56,959] - [rest_client] [140488573183744] - INFO - --> in init_cluster...Administrator,password,8091 [2021-02-06 19:52:56,959] - [rest_client] [140488573183744] - INFO - settings/web params on 172.23.122.175:8091:port=8091&username=Administrator&password=password [2021-02-06 19:52:57,015] - [rest_client] [140488573183744] - INFO - --> status:True [2021-02-06 19:52:57,018] - [remote_util] [140488573183744] - INFO - SSH Connecting to 172.23.122.175 with username:root, attempt#1 of 5 [2021-02-06 19:52:57,121] - [remote_util] [140488573183744] - INFO - SSH Connected to 172.23.122.175 as root [2021-02-06 19:52:57,395] - [remote_util] [140488573183744] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-06 19:52:57,690] - [remote_util] [140488573183744] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-06 19:52:57,699] - [remote_util] [140488573183744] - INFO - running command.raw on 172.23.122.175: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-02-06 19:52:57,755] - [remote_util] [140488573183744] - INFO - command executed successfully with root [2021-02-06 19:52:57,755] - [remote_util] [140488573183744] - INFO - ['ok'] [2021-02-06 19:52:57,757] - [rest_client] [140488573183744] - INFO - /diag/eval status on 172.23.122.175:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-06 19:52:57,759] - [rest_client] [140488573183744] - INFO - /diag/eval status on 172.23.122.175:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-06 19:52:57,763] - [rest_client] [140488573183744] - INFO - settings/indexes params : storageMode=plasma [2021-02-06 19:52:57,782] - [task] [140488573183744] - INFO - server: ip:172.23.122.173 port:8091 ssh_username:root, nodes/self [2021-02-06 19:52:57,785] - [task] [140488573183744] - INFO - {'uptime': '49', 'memoryTotal': 4201684992, 'memoryFree': 3586306048, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.122.173:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4390-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.122.173', 'ip': '172.23.122.173', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} [2021-02-06 19:52:57,787] - [rest_client] [140488573183744] - ERROR - GET http://172.23.122.173:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-02-06 19:52:57,787] - [rest_client] [140488573183744] - INFO - pools/default params : memoryQuota=2147 [2021-02-06 19:52:57,793] - [rest_client] [140488573183744] - INFO - --> in init_cluster...Administrator,password,8091 [2021-02-06 19:52:57,793] - [rest_client] [140488573183744] - INFO - settings/web params on 172.23.122.173:8091:port=8091&username=Administrator&password=password [2021-02-06 19:52:57,849] - [rest_client] [140488573183744] - INFO - --> status:True [2021-02-06 19:52:57,851] - [remote_util] [140488573183744] - INFO - SSH Connecting to 172.23.122.173 with username:root, attempt#1 of 5 [2021-02-06 19:52:57,954] - [remote_util] [140488573183744] - INFO - SSH Connected to 172.23.122.173 as root [2021-02-06 19:52:58,210] - [remote_util] [140488573183744] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-06 19:52:58,507] - [remote_util] [140488573183744] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-06 19:52:58,515] - [remote_util] [140488573183744] - INFO - running command.raw on 172.23.122.173: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-02-06 19:52:58,571] - [remote_util] [140488573183744] - INFO - command executed successfully with root [2021-02-06 19:52:58,571] - [remote_util] [140488573183744] - INFO - ['ok'] [2021-02-06 19:52:58,574] - [rest_client] [140488573183744] - INFO - /diag/eval status on 172.23.122.173:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-06 19:52:58,575] - [rest_client] [140488573183744] - INFO - /diag/eval status on 172.23.122.173:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-06 19:52:58,579] - [rest_client] [140488573183744] - INFO - settings/indexes params : storageMode=plasma [2021-02-06 19:52:58,598] - [task] [140488573183744] - INFO - server: ip:172.23.122.174 port:8091 ssh_username:root, nodes/self [2021-02-06 19:52:58,601] - [task] [140488573183744] - INFO - {'uptime': '39', 'memoryTotal': 4201684992, 'memoryFree': 3640717312, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.122.174:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4390-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.122.174', 'ip': '172.23.122.174', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} [2021-02-06 19:52:58,603] - [rest_client] [140488573183744] - ERROR - GET http://172.23.122.174:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-02-06 19:52:58,603] - [rest_client] [140488573183744] - INFO - pools/default params : memoryQuota=2147 [2021-02-06 19:52:58,609] - [rest_client] [140488573183744] - INFO - --> in init_cluster...Administrator,password,8091 [2021-02-06 19:52:58,609] - [rest_client] [140488573183744] - INFO - settings/web params on 172.23.122.174:8091:port=8091&username=Administrator&password=password [2021-02-06 19:52:58,665] - [rest_client] [140488573183744] - INFO - --> status:True [2021-02-06 19:52:58,666] - [remote_util] [140488573183744] - INFO - SSH Connecting to 172.23.122.174 with username:root, attempt#1 of 5 [2021-02-06 19:52:58,769] - [remote_util] [140488573183744] - INFO - SSH Connected to 172.23.122.174 as root [2021-02-06 19:52:59,032] - [remote_util] [140488573183744] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-06 19:52:59,334] - [remote_util] [140488573183744] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-06 19:52:59,342] - [remote_util] [140488573183744] - INFO - running command.raw on 172.23.122.174: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-02-06 19:52:59,398] - [remote_util] [140488573183744] - INFO - command executed successfully with root [2021-02-06 19:52:59,398] - [remote_util] [140488573183744] - INFO - ['ok'] [2021-02-06 19:52:59,400] - [rest_client] [140488573183744] - INFO - /diag/eval status on 172.23.122.174:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-06 19:52:59,402] - [rest_client] [140488573183744] - INFO - /diag/eval status on 172.23.122.174:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-06 19:52:59,406] - [rest_client] [140488573183744] - INFO - settings/indexes params : storageMode=plasma [2021-02-06 19:52:59,423] - [task] [140488573183744] - INFO - server: ip:172.23.122.176 port:8091 ssh_username:root, nodes/self [2021-02-06 19:52:59,427] - [task] [140488573183744] - INFO - {'uptime': '24', 'memoryTotal': 4201689088, 'memoryFree': 3571474432, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.122.176:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4390-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@cb.local', 'ip': '172.23.122.176', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} [2021-02-06 19:52:59,429] - [rest_client] [140488573183744] - ERROR - GET http://172.23.122.176:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-02-06 19:52:59,429] - [rest_client] [140488573183744] - INFO - pools/default params : memoryQuota=2147 [2021-02-06 19:52:59,435] - [rest_client] [140488573183744] - INFO - --> in init_cluster...Administrator,password,8091 [2021-02-06 19:52:59,435] - [rest_client] [140488573183744] - INFO - settings/web params on 172.23.122.176:8091:port=8091&username=Administrator&password=password [2021-02-06 19:52:59,492] - [rest_client] [140488573183744] - INFO - --> status:True [2021-02-06 19:52:59,494] - [remote_util] [140488573183744] - INFO - SSH Connecting to 172.23.122.176 with username:root, attempt#1 of 5 [2021-02-06 19:52:59,595] - [remote_util] [140488573183744] - INFO - SSH Connected to 172.23.122.176 as root [2021-02-06 19:52:59,860] - [remote_util] [140488573183744] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-06 19:53:00,153] - [remote_util] [140488573183744] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-06 19:53:00,163] - [remote_util] [140488573183744] - INFO - running command.raw on 172.23.122.176: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-02-06 19:53:00,219] - [remote_util] [140488573183744] - INFO - command executed successfully with root [2021-02-06 19:53:00,219] - [remote_util] [140488573183744] - INFO - ['ok'] [2021-02-06 19:53:00,222] - [rest_client] [140488573183744] - INFO - /diag/eval status on 172.23.122.176:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-06 19:53:00,224] - [rest_client] [140488573183744] - INFO - /diag/eval status on 172.23.122.176:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-06 19:53:00,228] - [rest_client] [140488573183744] - INFO - settings/indexes params : storageMode=plasma [2021-02-06 19:53:00,538] - [basetestcase] [140487281329920] - INFO - **** add built-in 'cbadminbucket' user to node 172.23.122.170 **** [2021-02-06 19:53:00,715] - [rest_client] [140487281329920] - ERROR - DELETE http://172.23.122.170:8091/settings/rbac/users/local/cbadminbucket body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:password [2021-02-06 19:53:00,740] - [internal_user] [140487281329920] - INFO - Exception while deleting user. Exception is -b'"User was not found."' [2021-02-06 19:53:00,818] - [basetestcase] [140487281329920] - INFO - sleep for 5 secs. ... [2021-02-06 19:53:05,823] - [basetestcase] [140487281329920] - INFO - **** add 'admin' role to 'cbadminbucket' user **** [2021-02-06 19:53:05,875] - [basetestcase] [140487281329920] - INFO - done initializing cluster [2021-02-06 19:53:07,603] - [task] [140488573183744] - INFO - adding node 172.23.122.172:8091 to cluster [2021-02-06 19:53:07,711] - [rest_client] [140488573183744] - INFO - adding remote node @172.23.122.172:8091 to this cluster @172.23.122.170:8091 [2021-02-06 19:53:10,247] - [task] [140488573183744] - INFO - adding node 172.23.122.177:8091 to cluster [2021-02-06 19:53:10,247] - [rest_client] [140488573183744] - INFO - adding remote node @172.23.122.177:8091 to this cluster @172.23.122.170:8091 [2021-02-06 19:53:12,427] - [task] [140488573183744] - INFO - adding node 172.23.122.178:8091 to cluster [2021-02-06 19:53:12,428] - [rest_client] [140488573183744] - INFO - adding remote node @172.23.122.178:8091 to this cluster @172.23.122.170:8091 [2021-02-06 19:53:14,853] - [task] [140488573183744] - INFO - adding node 172.23.122.175:8091 to cluster [2021-02-06 19:53:14,853] - [rest_client] [140488573183744] - INFO - adding remote node @172.23.122.175:8091 to this cluster @172.23.122.170:8091 [2021-02-06 19:53:17,153] - [task] [140488573183744] - INFO - adding node 172.23.122.173:8091 to cluster [2021-02-06 19:53:17,154] - [rest_client] [140488573183744] - INFO - adding remote node @172.23.122.173:8091 to this cluster @172.23.122.170:8091 [2021-02-06 19:53:19,480] - [rest_client] [140488573183744] - INFO - rebalance params : {'knownNodes': 'ns_1@172.23.122.170,ns_1@172.23.122.172,ns_1@172.23.122.173,ns_1@172.23.122.175,ns_1@172.23.122.177,ns_1@172.23.122.178', 'ejectedNodes': '', 'user': 'Administrator', 'password': 'password'} [2021-02-06 19:53:19,508] - [rest_client] [140488573183744] - INFO - rebalance operation started [2021-02-06 19:53:19,511] - [rest_client] [140488573183744] - INFO - rebalance percentage : 0.00 % [2021-02-06 19:53:19,511] - [task] [140488573183744] - INFO - Rebalance - status: running, progress: 0.00% [2021-02-06 19:53:29,531] - [task] [140488573183744] - INFO - Rebalance - status: none, progress: 100.00% [2021-02-06 19:53:29,538] - [task] [140488573183744] - INFO - rebalancing was completed with progress: 100% in 10.02992582321167 sec [2021-02-06 19:53:29,569] - [remote_util] [140487281329920] - INFO - SSH Connecting to 172.23.122.170 with username:root, attempt#1 of 5 [2021-02-06 19:53:29,668] - [remote_util] [140487281329920] - INFO - SSH Connected to 172.23.122.170 as root [2021-02-06 19:53:29,918] - [remote_util] [140487281329920] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-06 19:53:30,209] - [remote_util] [140487281329920] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-06 19:53:30,217] - [remote_util] [140487281329920] - INFO - running command.raw on 172.23.122.170: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-02-06 19:53:30,273] - [remote_util] [140487281329920] - INFO - command executed successfully with root [2021-02-06 19:53:30,273] - [remote_util] [140487281329920] - INFO - ['ok'] [2021-02-06 19:53:30,273] - [basetestcase] [140487281329920] - INFO - Enabled diag/eval for non-local hosts from 172.23.122.170 [2021-02-06 19:53:30,421] - [basetestcase] [140487281329920] - INFO - ============== basetestcase setup was finished for test #6 test_kv_swap_rebalance_when_existing_eventing_node_is_processing_mutations ============== [2021-02-06 19:53:31,286] - [remote_util] [140487281329920] - INFO - SSH Connecting to 172.23.122.170 with username:root, attempt#1 of 5 [2021-02-06 19:53:31,477] - [remote_util] [140487281329920] - INFO - SSH Connected to 172.23.122.170 as root [2021-02-06 19:53:31,948] - [remote_util] [140487281329920] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-06 19:53:32,241] - [remote_util] [140487281329920] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-06 19:53:32,245] - [remote_util] [140487281329920] - INFO - SSH Connecting to 172.23.122.172 with username:root, attempt#1 of 5 [2021-02-06 19:53:32,346] - [remote_util] [140487281329920] - INFO - SSH Connected to 172.23.122.172 as root [2021-02-06 19:53:33,385] - [remote_util] [140487281329920] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-06 19:53:33,753] - [remote_util] [140487281329920] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-06 19:53:33,765] - [remote_util] [140487281329920] - INFO - SSH Connecting to 172.23.122.177 with username:root, attempt#1 of 5 [2021-02-06 19:53:33,907] - [remote_util] [140487281329920] - INFO - SSH Connected to 172.23.122.177 as root [2021-02-06 19:53:34,172] - [remote_util] [140487281329920] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-06 19:53:34,474] - [remote_util] [140487281329920] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-06 19:53:34,477] - [remote_util] [140487281329920] - INFO - SSH Connecting to 172.23.122.178 with username:root, attempt#1 of 5 [2021-02-06 19:53:34,717] - [remote_util] [140487281329920] - INFO - SSH Connected to 172.23.122.178 as root [2021-02-06 19:53:35,487] - [remote_util] [140487281329920] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-06 19:53:35,780] - [remote_util] [140487281329920] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-06 19:53:35,782] - [remote_util] [140487281329920] - INFO - SSH Connecting to 172.23.122.175 with username:root, attempt#1 of 5 [2021-02-06 19:53:36,106] - [remote_util] [140487281329920] - INFO - SSH Connected to 172.23.122.175 as root [2021-02-06 19:53:37,137] - [remote_util] [140487281329920] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-06 19:53:37,761] - [remote_util] [140487281329920] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-06 19:53:37,764] - [remote_util] [140487281329920] - INFO - SSH Connecting to 172.23.122.173 with username:root, attempt#1 of 5 [2021-02-06 19:53:37,864] - [remote_util] [140487281329920] - INFO - SSH Connected to 172.23.122.173 as root [2021-02-06 19:53:38,689] - [remote_util] [140487281329920] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-06 19:53:38,978] - [remote_util] [140487281329920] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-06 19:53:38,982] - [remote_util] [140487281329920] - INFO - SSH Connecting to 172.23.122.174 with username:root, attempt#1 of 5 [2021-02-06 19:53:39,125] - [remote_util] [140487281329920] - INFO - SSH Connected to 172.23.122.174 as root [2021-02-06 19:53:39,562] - [remote_util] [140487281329920] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-06 19:53:39,860] - [remote_util] [140487281329920] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-06 19:53:39,862] - [remote_util] [140487281329920] - INFO - SSH Connecting to 172.23.122.176 with username:root, attempt#1 of 5 [2021-02-06 19:53:39,963] - [remote_util] [140487281329920] - INFO - SSH Connected to 172.23.122.176 as root [2021-02-06 19:53:40,336] - [remote_util] [140487281329920] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-06 19:53:40,633] - [remote_util] [140487281329920] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-06 19:53:53,517] - [basetestcase] [140487281329920] - INFO - ------- Cluster statistics ------- [2021-02-06 19:53:53,517] - [basetestcase] [140487281329920] - INFO - 172.23.122.170:8091 => {'services': ['kv'], 'cpu_utilization': 5.293672800604991, 'mem_free': 3507830784, 'mem_total': 4201684992, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-02-06 19:53:53,517] - [basetestcase] [140487281329920] - INFO - 172.23.122.172:8091 => {'services': ['kv'], 'cpu_utilization': 7.067404426559357, 'mem_free': 3534319616, 'mem_total': 4201684992, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-02-06 19:53:53,517] - [basetestcase] [140487281329920] - INFO - 172.23.122.173:8091 => {'services': ['index', 'n1ql'], 'cpu_utilization': 1.828657314629259, 'mem_free': 3507523584, 'mem_total': 4201684992, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-02-06 19:53:53,517] - [basetestcase] [140487281329920] - INFO - 172.23.122.175:8091 => {'services': ['eventing'], 'cpu_utilization': 1.703406813627254, 'mem_free': 3562770432, 'mem_total': 4201684992, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-02-06 19:53:53,517] - [basetestcase] [140487281329920] - INFO - 172.23.122.177:8091 => {'services': ['eventing'], 'cpu_utilization': 7.02416918429003, 'mem_free': 3550212096, 'mem_total': 4201684992, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-02-06 19:53:53,517] - [basetestcase] [140487281329920] - INFO - 172.23.122.178:8091 => {'services': ['eventing'], 'cpu_utilization': 2.855711422845691, 'mem_free': 3551174656, 'mem_total': 4201684992, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-02-06 19:53:53,517] - [basetestcase] [140487281329920] - INFO - --- End of cluster statistics --- [2021-02-06 19:53:53,524] - [remote_util] [140487281329920] - INFO - SSH Connecting to 172.23.122.170 with username:root, attempt#1 of 5 [2021-02-06 19:53:53,621] - [remote_util] [140487281329920] - INFO - SSH Connected to 172.23.122.170 as root [2021-02-06 19:53:53,877] - [remote_util] [140487281329920] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-06 19:53:54,166] - [remote_util] [140487281329920] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-06 19:54:17,258] - [basetestcase] [140487281329920] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.173 port:8091 ssh_username:root] [2021-02-06 19:54:17,265] - [basetestcase] [140487281329920] - INFO - list of eventing nodes in cluster: [ip:172.23.122.175 port:8091 ssh_username:root, ip:172.23.122.177 port:8091 ssh_username:root, ip:172.23.122.178 port:8091 ssh_username:root] [2021-02-06 19:54:17,273] - [rest_client] [140487281329920] - INFO - settings/indexes params : storageMode=plasma [2021-02-06 19:54:17,277] - [eventing_base] [140487281329920] - INFO - Setting the min possible memory quota so that adding mode nodes to the cluster wouldn't be a problem. [2021-02-06 19:54:17,278] - [rest_client] [140487281329920] - INFO - pools/default params : memoryQuota=330 [2021-02-06 19:54:17,286] - [rest_client] [140487281329920] - INFO - pools/default params : indexMemoryQuota=256 [2021-02-06 19:54:17,292] - [rest_client] [140487281329920] - INFO - pools/default params : eventingMemoryQuota=512 [2021-02-06 19:54:17,317] - [basetestcase] [140487281329920] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.173 port:8091 ssh_username:root] [2021-02-06 19:54:17,321] - [rest_client] [140487281329920] - INFO - pools/default params : memoryQuota=700 [2021-02-06 19:54:17,326] - [eventing_rebalance] [140487281329920] - INFO - 100 [2021-02-06 19:54:18,159] - [rest_client] [140488573183744] - INFO - http://172.23.122.170:8091/pools/default/buckets with param: name=src_bucket&ramQuotaMB=100&authType=none&replicaNumber=0&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore [2021-02-06 19:54:18,204] - [rest_client] [140488573183744] - INFO - 0.05 seconds to create bucket src_bucket [2021-02-06 19:54:18,205] - [bucket_helper] [140488573183744] - INFO - waiting for memcached bucket : src_bucket in 172.23.122.170 to accept set ops [2021-02-06 19:54:19,128] - [data_helper] [140488573183744] - INFO - creating direct client 172.23.122.170:11210 src_bucket [2021-02-06 19:54:19,212] - [data_helper] [140488573183744] - INFO - creating direct client 172.23.122.172:11210 src_bucket [2021-02-06 19:54:19,343] - [data_helper] [140488573183744] - INFO - creating direct client 172.23.122.170:11210 src_bucket [2021-02-06 19:54:19,432] - [data_helper] [140488573183744] - INFO - creating direct client 172.23.122.172:11210 src_bucket [2021-02-06 19:54:19,512] - [task] [140488573183744] - WARNING - vbucket map not ready after try 0 [2021-02-06 19:54:19,512] - [bucket_helper] [140488573183744] - INFO - waiting for memcached bucket : src_bucket in 172.23.122.170 to accept set ops [2021-02-06 19:54:19,851] - [data_helper] [140488573183744] - INFO - creating direct client 172.23.122.170:11210 src_bucket [2021-02-06 19:54:19,921] - [data_helper] [140488573183744] - INFO - creating direct client 172.23.122.172:11210 src_bucket [2021-02-06 19:54:20,026] - [data_helper] [140488573183744] - INFO - creating direct client 172.23.122.170:11210 src_bucket [2021-02-06 19:54:20,097] - [data_helper] [140488573183744] - INFO - creating direct client 172.23.122.172:11210 src_bucket [2021-02-06 19:54:20,158] - [task] [140488573183744] - WARNING - vbucket map not ready after try 1 [2021-02-06 19:54:20,158] - [bucket_helper] [140488573183744] - INFO - waiting for memcached bucket : src_bucket in 172.23.122.170 to accept set ops [2021-02-06 19:54:20,480] - [data_helper] [140488573183744] - INFO - creating direct client 172.23.122.170:11210 src_bucket [2021-02-06 19:54:20,554] - [data_helper] [140488573183744] - INFO - creating direct client 172.23.122.172:11210 src_bucket [2021-02-06 19:54:20,673] - [data_helper] [140488573183744] - INFO - creating direct client 172.23.122.170:11210 src_bucket [2021-02-06 19:54:20,746] - [data_helper] [140488573183744] - INFO - creating direct client 172.23.122.172:11210 src_bucket [2021-02-06 19:54:20,812] - [task] [140488573183744] - WARNING - vbucket map not ready after try 2 [2021-02-06 19:54:20,813] - [bucket_helper] [140488573183744] - INFO - waiting for memcached bucket : src_bucket in 172.23.122.170 to accept set ops [2021-02-06 19:54:21,062] - [data_helper] [140488573183744] - INFO - creating direct client 172.23.122.170:11210 src_bucket [2021-02-06 19:54:21,133] - [data_helper] [140488573183744] - INFO - creating direct client 172.23.122.172:11210 src_bucket [2021-02-06 19:54:21,239] - [data_helper] [140488573183744] - INFO - creating direct client 172.23.122.170:11210 src_bucket [2021-02-06 19:54:21,308] - [data_helper] [140488573183744] - INFO - creating direct client 172.23.122.172:11210 src_bucket [2021-02-06 19:54:21,375] - [task] [140488573183744] - WARNING - vbucket map not ready after try 3 [2021-02-06 19:54:21,375] - [bucket_helper] [140488573183744] - INFO - waiting for memcached bucket : src_bucket in 172.23.122.170 to accept set ops [2021-02-06 19:54:21,634] - [data_helper] [140488573183744] - INFO - creating direct client 172.23.122.170:11210 src_bucket [2021-02-06 19:54:21,706] - [data_helper] [140488573183744] - INFO - creating direct client 172.23.122.172:11210 src_bucket [2021-02-06 19:54:21,817] - [data_helper] [140488573183744] - INFO - creating direct client 172.23.122.170:11210 src_bucket [2021-02-06 19:54:21,886] - [data_helper] [140488573183744] - INFO - creating direct client 172.23.122.172:11210 src_bucket [2021-02-06 19:54:21,947] - [task] [140488573183744] - WARNING - vbucket map not ready after try 4 [2021-02-06 19:54:21,947] - [bucket_helper] [140488573183744] - INFO - waiting for memcached bucket : src_bucket in 172.23.122.170 to accept set ops [2021-02-06 19:54:22,199] - [data_helper] [140488573183744] - INFO - creating direct client 172.23.122.170:11210 src_bucket [2021-02-06 19:54:22,269] - [data_helper] [140488573183744] - INFO - creating direct client 172.23.122.172:11210 src_bucket [2021-02-06 19:54:22,375] - [data_helper] [140488573183744] - INFO - creating direct client 172.23.122.170:11210 src_bucket [2021-02-06 19:54:22,445] - [data_helper] [140488573183744] - INFO - creating direct client 172.23.122.172:11210 src_bucket [2021-02-06 19:54:22,508] - [task] [140488573183744] - WARNING - vbucket map not ready after try 5 [2021-02-06 19:54:23,530] - [rest_client] [140488573183744] - INFO - http://172.23.122.170:8091/pools/default/buckets with param: name=dst_bucket&ramQuotaMB=100&authType=none&replicaNumber=0&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore [2021-02-06 19:54:23,568] - [rest_client] [140488573183744] - INFO - 0.04 seconds to create bucket dst_bucket [2021-02-06 19:54:23,569] - [bucket_helper] [140488573183744] - INFO - waiting for memcached bucket : dst_bucket in 172.23.122.170 to accept set ops [2021-02-06 19:54:23,965] - [data_helper] [140488573183744] - INFO - creating direct client 172.23.122.170:11210 dst_bucket [2021-02-06 19:54:24,041] - [data_helper] [140488573183744] - INFO - creating direct client 172.23.122.172:11210 dst_bucket [2021-02-06 19:54:24,155] - [data_helper] [140488573183744] - INFO - creating direct client 172.23.122.170:11210 dst_bucket [2021-02-06 19:54:24,227] - [data_helper] [140488573183744] - INFO - creating direct client 172.23.122.172:11210 dst_bucket [2021-02-06 19:54:24,290] - [task] [140488573183744] - WARNING - vbucket map not ready after try 0 [2021-02-06 19:54:24,290] - [bucket_helper] [140488573183744] - INFO - waiting for memcached bucket : dst_bucket in 172.23.122.170 to accept set ops [2021-02-06 19:54:24,562] - [data_helper] [140488573183744] - INFO - creating direct client 172.23.122.170:11210 dst_bucket [2021-02-06 19:54:24,633] - [data_helper] [140488573183744] - INFO - creating direct client 172.23.122.172:11210 dst_bucket [2021-02-06 19:54:24,747] - [data_helper] [140488573183744] - INFO - creating direct client 172.23.122.170:11210 dst_bucket [2021-02-06 19:54:24,833] - [data_helper] [140488573183744] - INFO - creating direct client 172.23.122.172:11210 dst_bucket [2021-02-06 19:54:24,935] - [task] [140488573183744] - WARNING - vbucket map not ready after try 1 [2021-02-06 19:54:24,935] - [bucket_helper] [140488573183744] - INFO - waiting for memcached bucket : dst_bucket in 172.23.122.170 to accept set ops [2021-02-06 19:54:25,275] - [data_helper] [140488573183744] - INFO - creating direct client 172.23.122.170:11210 dst_bucket [2021-02-06 19:54:25,352] - [data_helper] [140488573183744] - INFO - creating direct client 172.23.122.172:11210 dst_bucket [2021-02-06 19:54:25,469] - [data_helper] [140488573183744] - INFO - creating direct client 172.23.122.170:11210 dst_bucket [2021-02-06 19:54:25,552] - [data_helper] [140488573183744] - INFO - creating direct client 172.23.122.172:11210 dst_bucket [2021-02-06 19:54:25,618] - [task] [140488573183744] - WARNING - vbucket map not ready after try 2 [2021-02-06 19:54:25,618] - [bucket_helper] [140488573183744] - INFO - waiting for memcached bucket : dst_bucket in 172.23.122.170 to accept set ops [2021-02-06 19:54:25,964] - [data_helper] [140488573183744] - INFO - creating direct client 172.23.122.170:11210 dst_bucket [2021-02-06 19:54:26,037] - [data_helper] [140488573183744] - INFO - creating direct client 172.23.122.172:11210 dst_bucket [2021-02-06 19:54:26,156] - [data_helper] [140488573183744] - INFO - creating direct client 172.23.122.170:11210 dst_bucket [2021-02-06 19:54:26,231] - [data_helper] [140488573183744] - INFO - creating direct client 172.23.122.172:11210 dst_bucket [2021-02-06 19:54:26,296] - [task] [140488573183744] - WARNING - vbucket map not ready after try 3 [2021-02-06 19:54:26,296] - [bucket_helper] [140488573183744] - INFO - waiting for memcached bucket : dst_bucket in 172.23.122.170 to accept set ops [2021-02-06 19:54:26,622] - [data_helper] [140488573183744] - INFO - creating direct client 172.23.122.170:11210 dst_bucket [2021-02-06 19:54:26,698] - [data_helper] [140488573183744] - INFO - creating direct client 172.23.122.172:11210 dst_bucket [2021-02-06 19:54:26,810] - [data_helper] [140488573183744] - INFO - creating direct client 172.23.122.170:11210 dst_bucket [2021-02-06 19:54:26,884] - [data_helper] [140488573183744] - INFO - creating direct client 172.23.122.172:11210 dst_bucket [2021-02-06 19:54:26,952] - [task] [140488573183744] - WARNING - vbucket map not ready after try 4 [2021-02-06 19:54:26,952] - [bucket_helper] [140488573183744] - INFO - waiting for memcached bucket : dst_bucket in 172.23.122.170 to accept set ops [2021-02-06 19:54:27,072] - [data_helper] [140488573183744] - INFO - creating direct client 172.23.122.170:11210 dst_bucket [2021-02-06 19:54:27,376] - [data_helper] [140488573183744] - INFO - creating direct client 172.23.122.172:11210 dst_bucket [2021-02-06 19:54:27,503] - [data_helper] [140488573183744] - INFO - creating direct client 172.23.122.170:11210 dst_bucket [2021-02-06 19:54:27,581] - [data_helper] [140488573183744] - INFO - creating direct client 172.23.122.172:11210 dst_bucket [2021-02-06 19:54:27,648] - [task] [140488573183744] - WARNING - vbucket map not ready after try 5 [2021-02-06 19:54:28,674] - [rest_client] [140488573183744] - INFO - http://172.23.122.170:8091/pools/default/buckets with param: name=metadata&ramQuotaMB=400&authType=none&replicaNumber=0&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore [2021-02-06 19:54:28,709] - [rest_client] [140488573183744] - INFO - 0.03 seconds to create bucket metadata [2021-02-06 19:54:28,709] - [bucket_helper] [140488573183744] - INFO - waiting for memcached bucket : metadata in 172.23.122.170 to accept set ops [2021-02-06 19:54:29,431] - [data_helper] [140488573183744] - INFO - creating direct client 172.23.122.170:11210 metadata [2021-02-06 19:54:29,737] - [data_helper] [140488573183744] - INFO - creating direct client 172.23.122.172:11210 metadata [2021-02-06 19:54:29,868] - [data_helper] [140488573183744] - INFO - creating direct client 172.23.122.170:11210 metadata [2021-02-06 19:54:29,989] - [data_helper] [140488573183744] - INFO - creating direct client 172.23.122.172:11210 metadata [2021-02-06 19:54:30,074] - [task] [140488573183744] - WARNING - vbucket map not ready after try 0 [2021-02-06 19:54:30,074] - [bucket_helper] [140488573183744] - INFO - waiting for memcached bucket : metadata in 172.23.122.170 to accept set ops [2021-02-06 19:54:30,202] - [data_helper] [140488573183744] - INFO - creating direct client 172.23.122.170:11210 metadata [2021-02-06 19:54:30,502] - [data_helper] [140488573183744] - INFO - creating direct client 172.23.122.172:11210 metadata [2021-02-06 19:54:30,618] - [data_helper] [140488573183744] - INFO - creating direct client 172.23.122.170:11210 metadata [2021-02-06 19:54:30,695] - [data_helper] [140488573183744] - INFO - creating direct client 172.23.122.172:11210 metadata [2021-02-06 19:54:30,763] - [task] [140488573183744] - WARNING - vbucket map not ready after try 1 [2021-02-06 19:54:30,763] - [bucket_helper] [140488573183744] - INFO - waiting for memcached bucket : metadata in 172.23.122.170 to accept set ops [2021-02-06 19:54:30,878] - [data_helper] [140488573183744] - INFO - creating direct client 172.23.122.170:11210 metadata [2021-02-06 19:54:30,956] - [data_helper] [140488573183744] - INFO - creating direct client 172.23.122.172:11210 metadata [2021-02-06 19:54:31,236] - [data_helper] [140488573183744] - INFO - creating direct client 172.23.122.170:11210 metadata [2021-02-06 19:54:31,311] - [data_helper] [140488573183744] - INFO - creating direct client 172.23.122.172:11210 metadata [2021-02-06 19:54:31,384] - [task] [140488573183744] - WARNING - vbucket map not ready after try 2 [2021-02-06 19:54:31,384] - [bucket_helper] [140488573183744] - INFO - waiting for memcached bucket : metadata in 172.23.122.170 to accept set ops [2021-02-06 19:54:31,501] - [data_helper] [140488573183744] - INFO - creating direct client 172.23.122.170:11210 metadata [2021-02-06 19:54:31,582] - [data_helper] [140488573183744] - INFO - creating direct client 172.23.122.172:11210 metadata [2021-02-06 19:54:31,912] - [data_helper] [140488573183744] - INFO - creating direct client 172.23.122.170:11210 metadata [2021-02-06 19:54:31,993] - [data_helper] [140488573183744] - INFO - creating direct client 172.23.122.172:11210 metadata [2021-02-06 19:54:32,060] - [task] [140488573183744] - WARNING - vbucket map not ready after try 3 [2021-02-06 19:54:32,060] - [bucket_helper] [140488573183744] - INFO - waiting for memcached bucket : metadata in 172.23.122.170 to accept set ops [2021-02-06 19:54:32,172] - [data_helper] [140488573183744] - INFO - creating direct client 172.23.122.170:11210 metadata [2021-02-06 19:54:32,256] - [data_helper] [140488573183744] - INFO - creating direct client 172.23.122.172:11210 metadata [2021-02-06 19:54:32,528] - [data_helper] [140488573183744] - INFO - creating direct client 172.23.122.170:11210 metadata [2021-02-06 19:54:32,613] - [data_helper] [140488573183744] - INFO - creating direct client 172.23.122.172:11210 metadata [2021-02-06 19:54:32,686] - [task] [140488573183744] - WARNING - vbucket map not ready after try 4 [2021-02-06 19:54:32,686] - [bucket_helper] [140488573183744] - INFO - waiting for memcached bucket : metadata in 172.23.122.170 to accept set ops [2021-02-06 19:54:32,802] - [data_helper] [140488573183744] - INFO - creating direct client 172.23.122.170:11210 metadata [2021-02-06 19:54:32,880] - [data_helper] [140488573183744] - INFO - creating direct client 172.23.122.172:11210 metadata [2021-02-06 19:54:33,219] - [data_helper] [140488573183744] - INFO - creating direct client 172.23.122.170:11210 metadata [2021-02-06 19:54:33,304] - [data_helper] [140488573183744] - INFO - creating direct client 172.23.122.172:11210 metadata [2021-02-06 19:54:33,389] - [task] [140488573183744] - WARNING - vbucket map not ready after try 5 [2021-02-06 19:54:33,486] - [basetestcase] [140487281329920] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.173 port:8091 ssh_username:root] [2021-02-06 19:54:33,493] - [tuq_helper] [140487281329920] - INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary' [2021-02-06 19:54:33,497] - [rest_client] [140487281329920] - INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27 [2021-02-06 19:54:33,827] - [tuq_helper] [140487281329920] - INFO - TOTAL ELAPSED TIME: 327.539713ms [2021-02-06 19:54:33,834] - [tuq_helper] [140487281329920] - INFO - RUN QUERY CREATE PRIMARY INDEX ON dst_bucket USING GSI [2021-02-06 19:54:33,838] - [rest_client] [140487281329920] - INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+dst_bucket++USING+GSI [2021-02-06 19:54:35,629] - [tuq_helper] [140487281329920] - INFO - TOTAL ELAPSED TIME: 1.789275532s [2021-02-06 19:54:35,634] - [tuq_helper] [140487281329920] - INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary' [2021-02-06 19:54:35,639] - [rest_client] [140487281329920] - INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27 [2021-02-06 19:54:35,667] - [tuq_helper] [140487281329920] - INFO - TOTAL ELAPSED TIME: 26.520955ms [2021-02-06 19:54:35,674] - [tuq_helper] [140487281329920] - INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary' [2021-02-06 19:54:35,679] - [rest_client] [140487281329920] - INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27 [2021-02-06 19:54:35,682] - [tuq_helper] [140487281329920] - INFO - TOTAL ELAPSED TIME: 1.104382ms [2021-02-06 19:54:35,686] - [tuq_helper] [140487281329920] - INFO - RUN QUERY CREATE PRIMARY INDEX ON metadata USING GSI [2021-02-06 19:54:35,690] - [rest_client] [140487281329920] - INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+metadata++USING+GSI [2021-02-06 19:54:37,499] - [tuq_helper] [140487281329920] - INFO - TOTAL ELAPSED TIME: 1.807146841s [2021-02-06 19:54:37,503] - [tuq_helper] [140487281329920] - INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary' [2021-02-06 19:54:37,509] - [rest_client] [140487281329920] - INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27 [2021-02-06 19:54:37,529] - [tuq_helper] [140487281329920] - INFO - TOTAL ELAPSED TIME: 17.963697ms [2021-02-06 19:54:37,533] - [tuq_helper] [140487281329920] - INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary' [2021-02-06 19:54:37,537] - [rest_client] [140487281329920] - INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27 [2021-02-06 19:54:37,539] - [tuq_helper] [140487281329920] - INFO - TOTAL ELAPSED TIME: 1.052373ms [2021-02-06 19:54:37,543] - [tuq_helper] [140487281329920] - INFO - RUN QUERY CREATE PRIMARY INDEX ON src_bucket USING GSI [2021-02-06 19:54:37,551] - [rest_client] [140487281329920] - INFO - query params : statement=CREATE+PRIMARY+INDEX+ON+src_bucket++USING+GSI [2021-02-06 19:54:39,346] - [tuq_helper] [140487281329920] - INFO - TOTAL ELAPSED TIME: 1.793833089s [2021-02-06 19:54:39,353] - [tuq_helper] [140487281329920] - INFO - RUN QUERY SELECT * FROM system:indexes where name = '#primary' [2021-02-06 19:54:39,358] - [rest_client] [140487281329920] - INFO - query params : statement=SELECT+%2A+FROM+system%3Aindexes+where+name+%3D+%27%23primary%27 [2021-02-06 19:54:39,379] - [tuq_helper] [140487281329920] - INFO - TOTAL ELAPSED TIME: 18.888433ms [2021-02-06 19:54:40,620] - [eventing_base] [140487281329920] - INFO - saving function b'{\n "code": 0,\n "info": {\n "status": "Stored function: \'Function_161864479_test_kv_swap_rebalance_when_existing_eventing_node_is_processing_mutati\' in metakv",\n "warnings": null\n }\n}' [2021-02-06 19:54:40,620] - [eventing_base] [140487281329920] - INFO - Deploying the following handler code : Function_161864479_test_kv_swap_rebalance_when_existing_eventing_node_is_processing_mutati with bindings: {'buckets': [{'alias': 'dst_bucket', 'bucket_name': 'dst_bucket', 'access': 'rw'}], 'metadata_bucket': 'metadata', 'source_bucket': 'src_bucket', 'curl': []} and settings: {'checkpoint_interval': 20000, 'cleanup_timers': False, 'dcp_stream_boundary': 'everything', 'deployment_status': False, 'description': 'Sample Description', 'log_level': 'INFO', 'skip_timer_threshold': 86400, 'sock_batch_size': 1, 'tick_duration': 5000, 'timer_processing_tick_interval': 500, 'timer_worker_pool_size': 3, 'worker_count': 3, 'processing_status': False, 'cpp_worker_thread_count': 1, 'execution_timeout': 20, 'data_chan_size': 10000, 'worker_queue_cap': 100000, 'use_memory_manager': True, 'deadline_timeout': 21, 'timer_storage_chan_size': 10000, 'dcp_gen_chan_size': 10000, 'language_compatibility': '6.5.0'} [2021-02-06 19:54:40,621] - [eventing_base] [140487281329920] - INFO - function OnUpdate(doc, meta) { log("Doc created/updated", meta.id); var result1= couchbase.insert(dst_bucket,meta,doc); log(result1); } function OnDelete(meta, options) { log("Doc deleted/expired", meta.id); var doc={"id":meta.id} var result = couchbase.delete(dst_bucket,doc); log(result); } [2021-02-06 19:54:41,843] - [eventing_base] [140487281329920] - INFO - deploy Application : b'' [2021-02-06 19:54:41,843] - [basetestcase] [140487281329920] - INFO - sleep for 5 secs. Waiting for Function_161864479_test_kv_swap_rebalance_when_existing_eventing_node_is_processing_mutati to deployed... ... [2021-02-06 19:54:46,864] - [basetestcase] [140487281329920] - INFO - sleep for 5 secs. Waiting for Function_161864479_test_kv_swap_rebalance_when_existing_eventing_node_is_processing_mutati to deployed... ... [2021-02-06 19:54:51,886] - [basetestcase] [140487281329920] - INFO - sleep for 5 secs. Waiting for Function_161864479_test_kv_swap_rebalance_when_existing_eventing_node_is_processing_mutati to deployed... ... [2021-02-06 19:54:56,904] - [basetestcase] [140487281329920] - INFO - sleep for 5 secs. Waiting for Function_161864479_test_kv_swap_rebalance_when_existing_eventing_node_is_processing_mutati to deployed... ... [2021-02-06 19:55:02,380] - [basetestcase] [140487281329920] - INFO - create 40320.0 to src_bucket documents... [2021-02-06 19:55:02,424] - [data_helper] [140487281329920] - INFO - creating direct client 172.23.122.170:11210 src_bucket [2021-02-06 19:55:02,475] - [data_helper] [140487281329920] - INFO - creating direct client 172.23.122.172:11210 src_bucket [2021-02-06 19:55:32,456] - [basetestcase] [140487281329920] - INFO - LOAD IS FINISHED [2021-02-06 19:55:33,501] - [rest_client] [140488573183744] - INFO - Node versions in cluster ['7.0.0-4390-enterprise', '7.0.0-4390-enterprise', '7.0.0-4390-enterprise', '7.0.0-4390-enterprise', '7.0.0-4390-enterprise', '7.0.0-4390-enterprise'] [2021-02-06 19:55:33,852] - [task] [140488573183744] - INFO - This is swap rebalance and we will monitor vbuckets shuffling [2021-02-06 19:55:33,852] - [task] [140488573183744] - INFO - adding node 172.23.122.174:8091 to cluster [2021-02-06 19:55:33,852] - [rest_client] [140488573183744] - INFO - adding remote node @172.23.122.174:8091 to this cluster @172.23.122.170:8091 [2021-02-06 19:55:35,479] - [rest_client] [140487281329920] - INFO - rebalance reached >100% in 3.0064971446990967 seconds [2021-02-06 19:55:36,384] - [rest_client] [140488573183744] - INFO - rebalance params : {'knownNodes': 'ns_1@172.23.122.170,ns_1@172.23.122.172,ns_1@172.23.122.173,ns_1@172.23.122.174,ns_1@172.23.122.175,ns_1@172.23.122.177,ns_1@172.23.122.178', 'ejectedNodes': 'ns_1@172.23.122.172', 'user': 'Administrator', 'password': 'password'} [2021-02-06 19:55:36,418] - [rest_client] [140488573183744] - INFO - rebalance operation started [2021-02-06 19:55:36,418] - [task] [140488573183744] - INFO - This is swap rebalance and we will monitor vbuckets shuffling [2021-02-06 19:55:36,890] - [rest_client] [140488573183744] - INFO - rebalance percentage : 0.00 % [2021-02-06 19:55:36,890] - [task] [140488573183744] - INFO - Rebalance - status: running, progress: 0.00% [2021-02-06 19:55:46,909] - [task] [140488573183744] - INFO - This is swap rebalance and we will monitor vbuckets shuffling [2021-02-06 19:55:47,759] - [rest_client] [140488573183744] - INFO - rebalance percentage : 7.00 % [2021-02-06 19:55:47,760] - [task] [140488573183744] - INFO - Rebalance - status: running, progress: 7.00% [2021-02-06 19:55:57,792] - [task] [140488573183744] - INFO - This is swap rebalance and we will monitor vbuckets shuffling [2021-02-06 19:55:58,248] - [rest_client] [140488573183744] - INFO - rebalance percentage : 9.00 % [2021-02-06 19:55:58,248] - [task] [140488573183744] - INFO - Rebalance - status: running, progress: 9.00% [2021-02-06 19:56:08,270] - [task] [140488573183744] - INFO - This is swap rebalance and we will monitor vbuckets shuffling [2021-02-06 19:56:08,746] - [rest_client] [140488573183744] - INFO - rebalance percentage : 19.00 % [2021-02-06 19:56:08,746] - [task] [140488573183744] - INFO - Rebalance - status: running, progress: 19.00% [2021-02-06 19:56:18,773] - [task] [140488573183744] - INFO - This is swap rebalance and we will monitor vbuckets shuffling [2021-02-06 19:56:19,204] - [rest_client] [140488573183744] - INFO - rebalance percentage : 23.00 % [2021-02-06 19:56:19,204] - [task] [140488573183744] - INFO - Rebalance - status: running, progress: 23.00% [2021-02-06 19:56:29,223] - [task] [140488573183744] - INFO - This is swap rebalance and we will monitor vbuckets shuffling [2021-02-06 19:56:29,793] - [rest_client] [140488573183744] - INFO - rebalance percentage : 32.00 % [2021-02-06 19:56:29,793] - [task] [140488573183744] - INFO - Rebalance - status: running, progress: 32.00% [2021-02-06 19:56:39,814] - [task] [140488573183744] - INFO - This is swap rebalance and we will monitor vbuckets shuffling [2021-02-06 19:56:40,284] - [rest_client] [140488573183744] - INFO - rebalance percentage : 38.00 % [2021-02-06 19:56:40,284] - [task] [140488573183744] - INFO - Rebalance - status: running, progress: 38.00% [2021-02-06 19:56:50,307] - [task] [140488573183744] - INFO - This is swap rebalance and we will monitor vbuckets shuffling [2021-02-06 19:56:50,764] - [rest_client] [140488573183744] - INFO - rebalance percentage : 50.00 % [2021-02-06 19:56:50,764] - [task] [140488573183744] - INFO - Rebalance - status: running, progress: 50.00% [2021-02-06 19:57:00,784] - [task] [140488573183744] - INFO - This is swap rebalance and we will monitor vbuckets shuffling [2021-02-06 19:57:01,203] - [rest_client] [140488573183744] - INFO - rebalance percentage : 50.00 % [2021-02-06 19:57:01,203] - [task] [140488573183744] - INFO - Rebalance - status: running, progress: 50.00% [2021-02-06 19:57:11,224] - [task] [140488573183744] - INFO - This is swap rebalance and we will monitor vbuckets shuffling [2021-02-06 19:57:11,635] - [task] [140488573183744] - INFO - Rebalance - status: none, progress: 100.00% [2021-02-06 19:57:11,654] - [task] [140488573183744] - INFO - rebalancing was completed with progress: 100% in 95.23617768287659 sec [2021-02-06 19:57:11,666] - [basetestcase] [140487281329920] - INFO - list of eventing nodes in cluster: [ip:172.23.122.175 port:8091 ssh_username:root, ip:172.23.122.177 port:8091 ssh_username:root, ip:172.23.122.178 port:8091 ssh_username:root] [2021-02-06 19:57:11,677] - [basetestcase] [140487281329920] - INFO - list of eventing nodes in cluster: [ip:172.23.122.175 port:8091 ssh_username:root, ip:172.23.122.177 port:8091 ssh_username:root, ip:172.23.122.178 port:8091 ssh_username:root] [2021-02-06 19:57:12,923] - [eventing_base] [140487281329920] - INFO - Documents in source bucket : 40320 [2021-02-06 19:57:12,924] - [eventing_base] [140487281329920] - INFO - Final docs count... Current : 40320 Expected : 40320 [2021-02-06 19:57:13,782] - [eventing_base] [140487281329920] - INFO - Stats for Node 172.23.122.175 is [ { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 6, "agg_messages_sent_to_worker": 16037, "agg_queue_memory": 0, "agg_queue_memory_cap": 106954752, "agg_queue_size": 0, "agg_queue_size_cap": 300000, "agg_timer_feedback_queue_cap": 1500, "dcp_mutation": 13571, "dcp_mutation_sent_to_worker": 13571, "dcp_snapshot": 13569, "dcp_stream_req_counter": 342, "dcp_streamreq": 342, "execution_stats": 420, "failure_stats": 420, "latency_stats": 420, "lcb_exception_stats": 420, "log_level": 3, "num_processed_events": 13571, "processed_events_size": 9791368, "thr_count": 3, "thr_map": 3, "v8_init": 3, "v8_load": 3 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13571, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13571, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 16025, "num_processed_events": 13571, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13571, "processed_events_size": 9791368, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "49362": "2021-02-07T03:57:12Z", "49367": "2021-02-07T03:57:12Z", "49371": "2021-02-07T03:57:12Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "49362": "2021-02-07T03:57:12Z", "49367": "2021-02-07T03:57:12Z", "49371": "2021-02-07T03:57:12Z" }, "v8worker_events_lost": 0 }, "function_id": 2830811561, "function_name": "Function_161864479_test_kv_swap_rebalance_when_existing_eventing_node_is_processing_mutati", "gocb_creds_request_counter": 32, "internal_vb_distribution_stats": { "worker_Function_161864479_test_kv_swap_rebalance_when_existing_eventing_node_is_processing_mutati_0": "[0-113]", "worker_Function_161864479_test_kv_swap_rebalance_when_existing_eventing_node_is_processing_mutati_1": "[114-227]", "worker_Function_161864479_test_kv_swap_rebalance_when_existing_eventing_node_is_processing_mutati_2": "[228-341]" }, "latency_percentile_stats": { "100": 12200, "50": 400, "80": 500, "90": 500, "95": 600, "99": 1300 }, "lcb_creds_request_counter": 12, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.175:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.177:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.178:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.175:8096": { "worker_Function_161864479_test_kv_swap_rebalance_when_existing_eventing_node_is_processing_mutati_0": "[0-113]", "worker_Function_161864479_test_kv_swap_rebalance_when_existing_eventing_node_is_processing_mutati_1": "[114-227]", "worker_Function_161864479_test_kv_swap_rebalance_when_existing_eventing_node_is_processing_mutati_2": "[228-341]" }, "172.23.122.177:8096": { "worker_Function_161864479_test_kv_swap_rebalance_when_existing_eventing_node_is_processing_mutati_0": "[342-455]", "worker_Function_161864479_test_kv_swap_rebalance_when_existing_eventing_node_is_processing_mutati_1": "[456-569]", "worker_Function_161864479_test_kv_swap_rebalance_when_existing_eventing_node_is_processing_mutati_2": "[570-641, 647-682]" }, "172.23.122.178:8096": { "worker_Function_161864479_test_kv_swap_rebalance_when_existing_eventing_node_is_processing_mutati_0": "[683-796]", "worker_Function_161864479_test_kv_swap_rebalance_when_existing_eventing_node_is_processing_mutati_1": "[797-874, 896-910]", "worker_Function_161864479_test_kv_swap_rebalance_when_existing_eventing_node_is_processing_mutati_2": "[911-1023]" } }, "worker_pids": { "worker_Function_161864479_test_kv_swap_rebalance_when_existing_eventing_node_is_processing_mutati_0": 49362, "worker_Function_161864479_test_kv_swap_rebalance_when_existing_eventing_node_is_processing_mutati_1": 49371, "worker_Function_161864479_test_kv_swap_rebalance_when_existing_eventing_node_is_processing_mutati_2": 49367 } } ] [2021-02-06 19:57:14,775] - [eventing_base] [140487281329920] - INFO - Stats for Node 172.23.122.177 is [ { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 6, "agg_messages_sent_to_worker": 16262, "agg_queue_memory": 0, "agg_queue_memory_cap": 106954752, "agg_queue_size": 0, "agg_queue_size_cap": 300000, "agg_timer_feedback_queue_cap": 1500, "dcp_mutation": 13328, "dcp_mutation_sent_to_worker": 13328, "dcp_snapshot": 13328, "dcp_stream_req_counter": 15437, "dcp_stream_req_err_counter": 14719, "dcp_streamend": 227, "dcp_streamreq": 718, "execution_stats": 423, "failure_stats": 423, "latency_stats": 423, "lcb_exception_stats": 423, "log_level": 3, "num_processed_events": 13328, "processed_events_size": 9621412, "thr_count": 3, "thr_map": 3, "v8_init": 3, "v8_load": 3 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13328, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13328, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 16250, "num_processed_events": 13328, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13328, "processed_events_size": 9621412, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "62289": "2021-02-07T03:57:12Z", "62299": "2021-02-07T03:57:13Z", "62310": "2021-02-07T03:57:13Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "62289": "2021-02-07T03:57:12Z", "62299": "2021-02-07T03:57:13Z", "62310": "2021-02-07T03:57:13Z" }, "v8worker_events_lost": 0 }, "function_id": 2830811561, "function_name": "Function_161864479_test_kv_swap_rebalance_when_existing_eventing_node_is_processing_mutati", "gocb_creds_request_counter": 34, "internal_vb_distribution_stats": { "worker_Function_161864479_test_kv_swap_rebalance_when_existing_eventing_node_is_processing_mutati_0": "[342-455]", "worker_Function_161864479_test_kv_swap_rebalance_when_existing_eventing_node_is_processing_mutati_1": "[456-569]", "worker_Function_161864479_test_kv_swap_rebalance_when_existing_eventing_node_is_processing_mutati_2": "[570-682]" }, "latency_percentile_stats": { "100": 14100, "50": 400, "80": 500, "90": 500, "95": 600, "99": 1200 }, "lcb_creds_request_counter": 16, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.175:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.177:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.178:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.175:8096": { "worker_Function_161864479_test_kv_swap_rebalance_when_existing_eventing_node_is_processing_mutati_0": "[0-113]", "worker_Function_161864479_test_kv_swap_rebalance_when_existing_eventing_node_is_processing_mutati_1": "[114-227]", "worker_Function_161864479_test_kv_swap_rebalance_when_existing_eventing_node_is_processing_mutati_2": "[228-341]" }, "172.23.122.177:8096": { "worker_Function_161864479_test_kv_swap_rebalance_when_existing_eventing_node_is_processing_mutati_0": "[342-455]", "worker_Function_161864479_test_kv_swap_rebalance_when_existing_eventing_node_is_processing_mutati_1": "[456-569]", "worker_Function_161864479_test_kv_swap_rebalance_when_existing_eventing_node_is_processing_mutati_2": "[570-641, 647-682]" }, "172.23.122.178:8096": { "worker_Function_161864479_test_kv_swap_rebalance_when_existing_eventing_node_is_processing_mutati_0": "[683-796]", "worker_Function_161864479_test_kv_swap_rebalance_when_existing_eventing_node_is_processing_mutati_1": "[797-879, 896-910]", "worker_Function_161864479_test_kv_swap_rebalance_when_existing_eventing_node_is_processing_mutati_2": "[911-1023]" } }, "worker_pids": { "worker_Function_161864479_test_kv_swap_rebalance_when_existing_eventing_node_is_processing_mutati_0": 62289, "worker_Function_161864479_test_kv_swap_rebalance_when_existing_eventing_node_is_processing_mutati_1": 62299, "worker_Function_161864479_test_kv_swap_rebalance_when_existing_eventing_node_is_processing_mutati_2": 62310 } } ] [2021-02-06 19:57:15,777] - [eventing_base] [140487281329920] - INFO - Stats for Node 172.23.122.178 is [ { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 6, "agg_messages_sent_to_worker": 16609, "agg_queue_memory": 0, "agg_queue_memory_cap": 106954752, "agg_queue_size": 0, "agg_queue_size_cap": 300000, "agg_timer_feedback_queue_cap": 1500, "dcp_mutation": 13421, "dcp_mutation_sent_to_worker": 13421, "dcp_snapshot": 13421, "dcp_stream_req_counter": 39428, "dcp_stream_req_err_counter": 38468, "dcp_streamend": 352, "dcp_streamreq": 960, "execution_stats": 426, "failure_stats": 426, "latency_stats": 426, "lcb_exception_stats": 426, "log_level": 3, "num_processed_events": 13421, "processed_events_size": 9690752, "thr_count": 3, "thr_map": 3, "v8_init": 3, "v8_load": 3 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13421, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13421, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 16597, "num_processed_events": 13421, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13421, "processed_events_size": 9690752, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "34999": "2021-02-07T03:57:14Z", "35004": "2021-02-07T03:57:14Z", "35010": "2021-02-07T03:57:14Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "34999": "2021-02-07T03:57:14Z", "35004": "2021-02-07T03:57:14Z", "35010": "2021-02-07T03:57:14Z" }, "v8worker_events_lost": 0 }, "function_id": 2830811561, "function_name": "Function_161864479_test_kv_swap_rebalance_when_existing_eventing_node_is_processing_mutati", "gocb_creds_request_counter": 33, "internal_vb_distribution_stats": { "worker_Function_161864479_test_kv_swap_rebalance_when_existing_eventing_node_is_processing_mutati_0": "[683-796]", "worker_Function_161864479_test_kv_swap_rebalance_when_existing_eventing_node_is_processing_mutati_1": "[797-910]", "worker_Function_161864479_test_kv_swap_rebalance_when_existing_eventing_node_is_processing_mutati_2": "[911-1023]" }, "latency_percentile_stats": { "100": 16600, "50": 400, "80": 500, "90": 500, "95": 600, "99": 1200 }, "lcb_creds_request_counter": 18, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.175:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.177:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.178:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.175:8096": { "worker_Function_161864479_test_kv_swap_rebalance_when_existing_eventing_node_is_processing_mutati_0": "[0-113]", "worker_Function_161864479_test_kv_swap_rebalance_when_existing_eventing_node_is_processing_mutati_1": "[114-227]", "worker_Function_161864479_test_kv_swap_rebalance_when_existing_eventing_node_is_processing_mutati_2": "[228-341]" }, "172.23.122.177:8096": { "worker_Function_161864479_test_kv_swap_rebalance_when_existing_eventing_node_is_processing_mutati_0": "[342-455]", "worker_Function_161864479_test_kv_swap_rebalance_when_existing_eventing_node_is_processing_mutati_1": "[456-569]", "worker_Function_161864479_test_kv_swap_rebalance_when_existing_eventing_node_is_processing_mutati_2": "[570-641, 647-682]" }, "172.23.122.178:8096": { "worker_Function_161864479_test_kv_swap_rebalance_when_existing_eventing_node_is_processing_mutati_0": "[683-796]", "worker_Function_161864479_test_kv_swap_rebalance_when_existing_eventing_node_is_processing_mutati_1": "[797-874, 896-910]", "worker_Function_161864479_test_kv_swap_rebalance_when_existing_eventing_node_is_processing_mutati_2": "[911-1023]" } }, "worker_pids": { "worker_Function_161864479_test_kv_swap_rebalance_when_existing_eventing_node_is_processing_mutati_0": 34999, "worker_Function_161864479_test_kv_swap_rebalance_when_existing_eventing_node_is_processing_mutati_1": 35010, "worker_Function_161864479_test_kv_swap_rebalance_when_existing_eventing_node_is_processing_mutati_2": 35004 } } ] [2021-02-06 19:57:16,423] - [basetestcase] [140487281329920] - INFO - delete 40320.0 to src_bucket documents... [2021-02-06 19:57:16,471] - [data_helper] [140487281329920] - INFO - creating direct client 172.23.122.170:11210 src_bucket [2021-02-06 19:57:16,525] - [data_helper] [140487281329920] - INFO - creating direct client 172.23.122.174:11210 src_bucket [2021-02-06 19:57:35,641] - [data_helper] [140487264544512] - INFO - creating direct client 172.23.122.170:11210 src_bucket [2021-02-06 19:57:35,682] - [data_helper] [140487264544512] - WARNING - unable to establish connection to 172.23.122.170. cleanup open connections [2021-02-06 19:57:35,683] - [task] [140487264544512] - ERROR - Memcached error #1 'Not found': b'', key: query-testemployee62603.52967833974-6 delete operation. [2021-02-06 19:57:35,692] - [basetestcase] [140487281329920] - INFO - list of eventing nodes in cluster: [ip:172.23.122.175 port:8091 ssh_username:root, ip:172.23.122.177 port:8091 ssh_username:root, ip:172.23.122.178 port:8091 ssh_username:root] [2021-02-06 19:57:35,713] - [eventing_base] [140487281329920] - INFO - Stats for Node 172.23.122.175 is [ { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 9, "agg_messages_sent_to_worker": 30012, "agg_queue_memory": 0, "agg_queue_memory_cap": 106954752, "agg_queue_size": 0, "agg_queue_size_cap": 300000, "agg_timer_feedback_queue_cap": 1500, "dcp_deletion": 13288, "dcp_deletion_sent_to_worker": 13288, "dcp_mutation": 13571, "dcp_mutation_sent_to_worker": 13571, "dcp_snapshot": 26856, "dcp_stream_req_counter": 342, "dcp_streamend": 342, "dcp_streamreq": 342, "execution_stats": 489, "failure_stats": 489, "latency_stats": 489, "lcb_exception_stats": 489, "log_level": 3, "num_processed_events": 26859, "processed_events_size": 14148028, "reb_vb_remaining_to_own": 2, "reb_vb_remaining_to_stream_req": 2, "thr_count": 3, "thr_map": 3, "v8_init": 3, "v8_load": 3 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 13092, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13571, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 13092, "enqueued_dcp_mutation_msg_counter": 13571, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 29462, "num_processed_events": 26662, "on_delete_failure": 0, "on_delete_success": 13091, "on_update_failure": 0, "on_update_success": 13571, "processed_events_size": 14083484, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "49362": "2021-02-07T03:57:35Z", "49367": "2021-02-07T03:57:35Z", "49371": "2021-02-07T03:57:35Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "49362": "2021-02-07T03:57:35Z", "49367": "2021-02-07T03:57:35Z", "49371": "2021-02-07T03:57:35Z" }, "v8worker_events_lost": 0 }, "function_id": 2830811561, "function_name": "Function_161864479_test_kv_swap_rebalance_when_existing_eventing_node_is_processing_mutati", "gocb_creds_request_counter": 32, "internal_vb_distribution_stats": { "worker_Function_161864479_test_kv_swap_rebalance_when_existing_eventing_node_is_processing_mutati_0": "[0-113]", "worker_Function_161864479_test_kv_swap_rebalance_when_existing_eventing_node_is_processing_mutati_1": "[114-146, 148-227]", "worker_Function_161864479_test_kv_swap_rebalance_when_existing_eventing_node_is_processing_mutati_2": "[228-244, 246-341]" }, "latency_percentile_stats": { "100": 12200, "50": 400, "80": 500, "90": 500, "95": 700, "99": 1300 }, "lcb_creds_request_counter": 18, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.175:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.177:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.178:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.175:8096": { "worker_Function_161864479_test_kv_swap_rebalance_when_existing_eventing_node_is_processing_mutati_0": "[0-113]", "worker_Function_161864479_test_kv_swap_rebalance_when_existing_eventing_node_is_processing_mutati_1": "[114-227]", "worker_Function_161864479_test_kv_swap_rebalance_when_existing_eventing_node_is_processing_mutati_2": "[228-341]" }, "172.23.122.177:8096": { "worker_Function_161864479_test_kv_swap_rebalance_when_existing_eventing_node_is_processing_mutati_0": "[342-455]", "worker_Function_161864479_test_kv_swap_rebalance_when_existing_eventing_node_is_processing_mutati_1": "[456-569]", "worker_Function_161864479_test_kv_swap_rebalance_when_existing_eventing_node_is_processing_mutati_2": "[570-682]" }, "172.23.122.178:8096": { "worker_Function_161864479_test_kv_swap_rebalance_when_existing_eventing_node_is_processing_mutati_0": "[683-796]", "worker_Function_161864479_test_kv_swap_rebalance_when_existing_eventing_node_is_processing_mutati_1": "[797-910]", "worker_Function_161864479_test_kv_swap_rebalance_when_existing_eventing_node_is_processing_mutati_2": "[911-1023]" } }, "worker_pids": { "worker_Function_161864479_test_kv_swap_rebalance_when_existing_eventing_node_is_processing_mutati_0": 49362, "worker_Function_161864479_test_kv_swap_rebalance_when_existing_eventing_node_is_processing_mutati_1": 49371, "worker_Function_161864479_test_kv_swap_rebalance_when_existing_eventing_node_is_processing_mutati_2": 49367 } } ] [2021-02-06 19:57:35,733] - [eventing_base] [140487281329920] - INFO - Stats for Node 172.23.122.177 is [ { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 9, "agg_messages_sent_to_worker": 29812, "agg_queue_memory": 469, "agg_queue_memory_cap": 106954752, "agg_queue_size": 1, "agg_queue_size_cap": 300000, "agg_timer_feedback_queue_cap": 1500, "dcp_deletion": 13050, "dcp_deletion_sent_to_worker": 13050, "dcp_mutation": 13328, "dcp_mutation_sent_to_worker": 13328, "dcp_snapshot": 26377, "dcp_stream_req_counter": 15437, "dcp_stream_req_err_counter": 14719, "dcp_streamend": 397, "dcp_streamreq": 718, "execution_stats": 489, "failure_stats": 489, "latency_stats": 489, "lcb_exception_stats": 489, "log_level": 3, "num_processed_events": 26377, "processed_events_size": 13900892, "thr_count": 3, "thr_map": 3, "v8_init": 3, "v8_load": 3 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 12838, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13328, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 12838, "enqueued_dcp_mutation_msg_counter": 13328, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 29418, "num_processed_events": 26166, "on_delete_failure": 0, "on_delete_success": 12838, "on_update_failure": 0, "on_update_success": 13328, "processed_events_size": 13831732, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "62289": "2021-02-07T03:57:34Z", "62299": "2021-02-07T03:57:35Z", "62310": "2021-02-07T03:57:35Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "62289": "2021-02-07T03:57:34Z", "62299": "2021-02-07T03:57:35Z", "62310": "2021-02-07T03:57:35Z" }, "v8worker_events_lost": 0 }, "function_id": 2830811561, "function_name": "Function_161864479_test_kv_swap_rebalance_when_existing_eventing_node_is_processing_mutati", "gocb_creds_request_counter": 34, "internal_vb_distribution_stats": { "worker_Function_161864479_test_kv_swap_rebalance_when_existing_eventing_node_is_processing_mutati_0": "[342-455]", "worker_Function_161864479_test_kv_swap_rebalance_when_existing_eventing_node_is_processing_mutati_1": "[456-569]", "worker_Function_161864479_test_kv_swap_rebalance_when_existing_eventing_node_is_processing_mutati_2": "[570-682]" }, "latency_percentile_stats": { "100": 19500, "50": 400, "80": 400, "90": 500, "95": 600, "99": 1300 }, "lcb_creds_request_counter": 24, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.175:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.177:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.178:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.175:8096": { "worker_Function_161864479_test_kv_swap_rebalance_when_existing_eventing_node_is_processing_mutati_0": "[0-113]", "worker_Function_161864479_test_kv_swap_rebalance_when_existing_eventing_node_is_processing_mutati_1": "[114-227]", "worker_Function_161864479_test_kv_swap_rebalance_when_existing_eventing_node_is_processing_mutati_2": "[228-341]" }, "172.23.122.177:8096": { "worker_Function_161864479_test_kv_swap_rebalance_when_existing_eventing_node_is_processing_mutati_0": "[342-455]", "worker_Function_161864479_test_kv_swap_rebalance_when_existing_eventing_node_is_processing_mutati_1": "[456-569]", "worker_Function_161864479_test_kv_swap_rebalance_when_existing_eventing_node_is_processing_mutati_2": "[570-682]" }, "172.23.122.178:8096": { "worker_Function_161864479_test_kv_swap_rebalance_when_existing_eventing_node_is_processing_mutati_0": "[683-796]", "worker_Function_161864479_test_kv_swap_rebalance_when_existing_eventing_node_is_processing_mutati_1": "[797-910]", "worker_Function_161864479_test_kv_swap_rebalance_when_existing_eventing_node_is_processing_mutati_2": "[911-1023]" } }, "worker_pids": { "worker_Function_161864479_test_kv_swap_rebalance_when_existing_eventing_node_is_processing_mutati_0": 62289, "worker_Function_161864479_test_kv_swap_rebalance_when_existing_eventing_node_is_processing_mutati_1": 62299, "worker_Function_161864479_test_kv_swap_rebalance_when_existing_eventing_node_is_processing_mutati_2": 62310 } } ] [2021-02-06 19:57:35,752] - [eventing_base] [140487281329920] - INFO - Stats for Node 172.23.122.178 is [ { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 9, "agg_messages_sent_to_worker": 30070, "agg_queue_memory": 1407, "agg_queue_memory_cap": 106954752, "agg_queue_size": 3, "agg_queue_size_cap": 300000, "agg_timer_feedback_queue_cap": 1500, "dcp_deletion": 13146, "dcp_deletion_sent_to_worker": 13146, "dcp_mutation": 13421, "dcp_mutation_sent_to_worker": 13421, "dcp_snapshot": 26567, "dcp_stream_req_counter": 39428, "dcp_stream_req_err_counter": 38468, "dcp_streamend": 352, "dcp_streamreq": 960, "execution_stats": 489, "failure_stats": 489, "latency_stats": 489, "lcb_exception_stats": 489, "log_level": 3, "num_processed_events": 26564, "processed_events_size": 14001740, "thr_count": 3, "thr_map": 3, "v8_init": 3, "v8_load": 3 }, "events_remaining": { "dcp_backlog": 1 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 12990, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13421, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 12990, "enqueued_dcp_mutation_msg_counter": 13421, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 29902, "num_processed_events": 26410, "on_delete_failure": 0, "on_delete_success": 12989, "on_update_failure": 0, "on_update_success": 13421, "processed_events_size": 13951248, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "34999": "2021-02-07T03:57:35Z", "35004": "2021-02-07T03:57:35Z", "35010": "2021-02-07T03:57:35Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "34999": "2021-02-07T03:57:35Z", "35004": "2021-02-07T03:57:35Z", "35010": "2021-02-07T03:57:35Z" }, "v8worker_events_lost": 0 }, "function_id": 2830811561, "function_name": "Function_161864479_test_kv_swap_rebalance_when_existing_eventing_node_is_processing_mutati", "gocb_creds_request_counter": 33, "internal_vb_distribution_stats": { "worker_Function_161864479_test_kv_swap_rebalance_when_existing_eventing_node_is_processing_mutati_0": "[683-796]", "worker_Function_161864479_test_kv_swap_rebalance_when_existing_eventing_node_is_processing_mutati_1": "[797-910]", "worker_Function_161864479_test_kv_swap_rebalance_when_existing_eventing_node_is_processing_mutati_2": "[911-1023]" }, "latency_percentile_stats": { "100": 18600, "50": 400, "80": 500, "90": 500, "95": 600, "99": 1100 }, "lcb_creds_request_counter": 30, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.175:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.177:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.178:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.175:8096": { "worker_Function_161864479_test_kv_swap_rebalance_when_existing_eventing_node_is_processing_mutati_0": "[0-113]", "worker_Function_161864479_test_kv_swap_rebalance_when_existing_eventing_node_is_processing_mutati_1": "[114-227]", "worker_Function_161864479_test_kv_swap_rebalance_when_existing_eventing_node_is_processing_mutati_2": "[228-341]" }, "172.23.122.177:8096": { "worker_Function_161864479_test_kv_swap_rebalance_when_existing_eventing_node_is_processing_mutati_0": "[342-455]", "worker_Function_161864479_test_kv_swap_rebalance_when_existing_eventing_node_is_processing_mutati_1": "[456-569]", "worker_Function_161864479_test_kv_swap_rebalance_when_existing_eventing_node_is_processing_mutati_2": "[570-682]" }, "172.23.122.178:8096": { "worker_Function_161864479_test_kv_swap_rebalance_when_existing_eventing_node_is_processing_mutati_0": "[683-796]", "worker_Function_161864479_test_kv_swap_rebalance_when_existing_eventing_node_is_processing_mutati_1": "[797-910]", "worker_Function_161864479_test_kv_swap_rebalance_when_existing_eventing_node_is_processing_mutati_2": "[911-1023]" } }, "worker_pids": { "worker_Function_161864479_test_kv_swap_rebalance_when_existing_eventing_node_is_processing_mutati_0": 34999, "worker_Function_161864479_test_kv_swap_rebalance_when_existing_eventing_node_is_processing_mutati_1": 35010, "worker_Function_161864479_test_kv_swap_rebalance_when_existing_eventing_node_is_processing_mutati_2": 35004 } } ] [2021-02-06 19:57:35,776] - [basetestcase] [140487281329920] - INFO - list of eventing nodes in cluster: [ip:172.23.122.175 port:8091 ssh_username:root, ip:172.23.122.177 port:8091 ssh_username:root, ip:172.23.122.178 port:8091 ssh_username:root] [2021-02-06 19:57:35,779] - [remote_util] [140487281329920] - INFO - SSH Connecting to 172.23.122.175 with username:root, attempt#1 of 5 [2021-02-06 19:57:35,883] - [remote_util] [140487281329920] - INFO - SSH Connected to 172.23.122.175 as root [2021-02-06 19:57:36,162] - [remote_util] [140487281329920] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-06 19:57:36,469] - [remote_util] [140487281329920] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-06 19:57:36,477] - [rest_client] [140487281329920] - INFO - /diag/eval status on 172.23.122.175:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). [2021-02-06 19:57:36,477] - [remote_util] [140487281329920] - INFO - running command.raw on 172.23.122.175: zgrep "panic" "/opt/couchbase/var/lib/couchbase/logs"/eventing.log* | wc -l [2021-02-06 19:57:36,728] - [remote_util] [140487281329920] - INFO - command executed successfully with root [2021-02-06 19:57:36,728] - [remote_util] [140487281329920] - INFO - running command.raw on 172.23.122.175: ls "/opt/couchbase/var/lib/couchbase/logs"/../crash/| wc -l [2021-02-06 19:57:36,742] - [remote_util] [140487281329920] - INFO - command executed successfully with root [2021-02-06 19:57:36,744] - [remote_util] [140487281329920] - INFO - SSH Connecting to 172.23.122.177 with username:root, attempt#1 of 5 [2021-02-06 19:57:36,846] - [remote_util] [140487281329920] - INFO - SSH Connected to 172.23.122.177 as root [2021-02-06 19:57:37,125] - [remote_util] [140487281329920] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-06 19:57:37,425] - [remote_util] [140487281329920] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-06 19:57:37,433] - [rest_client] [140487281329920] - INFO - /diag/eval status on 172.23.122.177:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). [2021-02-06 19:57:37,433] - [remote_util] [140487281329920] - INFO - running command.raw on 172.23.122.177: zgrep "panic" "/opt/couchbase/var/lib/couchbase/logs"/eventing.log* | wc -l [2021-02-06 19:57:37,870] - [remote_util] [140487281329920] - INFO - command executed successfully with root [2021-02-06 19:57:37,870] - [remote_util] [140487281329920] - INFO - running command.raw on 172.23.122.177: ls "/opt/couchbase/var/lib/couchbase/logs"/../crash/| wc -l [2021-02-06 19:57:37,884] - [remote_util] [140487281329920] - INFO - command executed successfully with root [2021-02-06 19:57:37,886] - [remote_util] [140487281329920] - INFO - SSH Connecting to 172.23.122.178 with username:root, attempt#1 of 5 [2021-02-06 19:57:37,991] - [remote_util] [140487281329920] - INFO - SSH Connected to 172.23.122.178 as root [2021-02-06 19:57:38,259] - [remote_util] [140487281329920] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-06 19:57:38,555] - [remote_util] [140487281329920] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-06 19:57:38,562] - [rest_client] [140487281329920] - INFO - /diag/eval status on 172.23.122.178:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). [2021-02-06 19:57:38,562] - [remote_util] [140487281329920] - INFO - running command.raw on 172.23.122.178: zgrep "panic" "/opt/couchbase/var/lib/couchbase/logs"/eventing.log* | wc -l [2021-02-06 19:57:39,183] - [remote_util] [140487281329920] - INFO - command executed successfully with root [2021-02-06 19:57:39,184] - [remote_util] [140487281329920] - INFO - running command.raw on 172.23.122.178: ls "/opt/couchbase/var/lib/couchbase/logs"/../crash/| wc -l [2021-02-06 19:57:39,199] - [remote_util] [140487281329920] - INFO - command executed successfully with root [2021-02-06 19:57:39,251] - [basetestcase] [140487281329920] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.173 port:8091 ssh_username:root] [2021-02-06 19:57:39,257] - [rest_client] [140487281329920] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+metadata [2021-02-06 19:57:39,270] - [eventing_base] [140487281329920] - INFO - number of documents in metadata bucket 0 [2021-02-06 19:57:39,278] - [basetestcase] [140487281329920] - INFO - ------- Cluster statistics ------- [2021-02-06 19:57:39,278] - [basetestcase] [140487281329920] - INFO - 172.23.122.170:8091 => {'services': ['kv'], 'cpu_utilization': 67.08595387840671, 'mem_free': 2767298560, 'mem_total': 4201684992, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-02-06 19:57:39,278] - [basetestcase] [140487281329920] - INFO - 172.23.122.173:8091 => {'services': ['index', 'n1ql'], 'cpu_utilization': 15.36216503051207, 'mem_free': 3090636800, 'mem_total': 4201684992, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-02-06 19:57:39,278] - [basetestcase] [140487281329920] - INFO - 172.23.122.174:8091 => {'services': ['kv'], 'cpu_utilization': 64.80192719486081, 'mem_free': 3078864896, 'mem_total': 4201684992, 'swap_mem_used': 98660352, 'swap_mem_total': 3758092288} [2021-02-06 19:57:39,278] - [basetestcase] [140487281329920] - INFO - 172.23.122.175:8091 => {'services': ['eventing'], 'cpu_utilization': 10.42440318302387, 'mem_free': 3322966016, 'mem_total': 4201684992, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-02-06 19:57:39,278] - [basetestcase] [140487281329920] - INFO - 172.23.122.177:8091 => {'services': ['eventing'], 'cpu_utilization': 9.27473712590995, 'mem_free': 3300438016, 'mem_total': 4201684992, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-02-06 19:57:39,278] - [basetestcase] [140487281329920] - INFO - 172.23.122.178:8091 => {'services': ['eventing'], 'cpu_utilization': 9.909429941395844, 'mem_free': 3272323072, 'mem_total': 4201684992, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-02-06 19:57:39,278] - [basetestcase] [140487281329920] - INFO - --- End of cluster statistics --- [2021-02-06 19:57:39,280] - [remote_util] [140487281329920] - INFO - SSH Connecting to 172.23.122.170 with username:root, attempt#1 of 5 [2021-02-06 19:57:39,380] - [remote_util] [140487281329920] - INFO - SSH Connected to 172.23.122.170 as root [2021-02-06 19:57:39,643] - [remote_util] [140487281329920] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-06 19:57:39,939] - [remote_util] [140487281329920] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-06 19:57:39,941] - [remote_util] [140487281329920] - INFO - SSH Connecting to 172.23.122.172 with username:root, attempt#1 of 5 [2021-02-06 19:57:40,044] - [remote_util] [140487281329920] - INFO - SSH Connected to 172.23.122.172 as root [2021-02-06 19:57:40,329] - [remote_util] [140487281329920] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-06 19:57:40,630] - [remote_util] [140487281329920] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-06 19:57:40,632] - [remote_util] [140487281329920] - INFO - SSH Connecting to 172.23.122.177 with username:root, attempt#1 of 5 [2021-02-06 19:57:40,738] - [remote_util] [140487281329920] - INFO - SSH Connected to 172.23.122.177 as root [2021-02-06 19:57:41,012] - [remote_util] [140487281329920] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-06 19:57:41,312] - [remote_util] [140487281329920] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-06 19:57:41,313] - [remote_util] [140487281329920] - INFO - SSH Connecting to 172.23.122.178 with username:root, attempt#1 of 5 [2021-02-06 19:57:41,419] - [remote_util] [140487281329920] - INFO - SSH Connected to 172.23.122.178 as root [2021-02-06 19:57:41,682] - [remote_util] [140487281329920] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-06 19:57:41,983] - [remote_util] [140487281329920] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-06 19:57:41,985] - [remote_util] [140487281329920] - INFO - SSH Connecting to 172.23.122.175 with username:root, attempt#1 of 5 [2021-02-06 19:57:42,085] - [remote_util] [140487281329920] - INFO - SSH Connected to 172.23.122.175 as root [2021-02-06 19:57:42,342] - [remote_util] [140487281329920] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-06 19:57:42,640] - [remote_util] [140487281329920] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-06 19:57:42,642] - [remote_util] [140487281329920] - INFO - SSH Connecting to 172.23.122.173 with username:root, attempt#1 of 5 [2021-02-06 19:57:42,742] - [remote_util] [140487281329920] - INFO - SSH Connected to 172.23.122.173 as root [2021-02-06 19:57:43,001] - [remote_util] [140487281329920] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-06 19:57:43,305] - [remote_util] [140487281329920] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-06 19:57:43,307] - [remote_util] [140487281329920] - INFO - SSH Connecting to 172.23.122.174 with username:root, attempt#1 of 5 [2021-02-06 19:57:43,411] - [remote_util] [140487281329920] - INFO - SSH Connected to 172.23.122.174 as root [2021-02-06 19:57:43,666] - [remote_util] [140487281329920] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-06 19:57:43,961] - [remote_util] [140487281329920] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-06 19:57:43,963] - [remote_util] [140487281329920] - INFO - SSH Connecting to 172.23.122.176 with username:root, attempt#1 of 5 [2021-02-06 19:57:44,073] - [remote_util] [140487281329920] - INFO - SSH Connected to 172.23.122.176 as root [2021-02-06 19:57:44,364] - [remote_util] [140487281329920] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-06 19:57:44,668] - [remote_util] [140487281329920] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-06 19:58:07,948] - [basetestcase] [140487281329920] - INFO - closing all ssh connections [2021-02-06 19:58:07,955] - [basetestcase] [140487281329920] - INFO - closing all memcached connections [2021-02-06 19:58:07,957] - [remote_util] [140487281329920] - INFO - SSH Connecting to 172.23.122.170 with username:root, attempt#1 of 5 [2021-02-06 19:58:08,062] - [remote_util] [140487281329920] - INFO - SSH Connected to 172.23.122.170 as root [2021-02-06 19:58:08,335] - [remote_util] [140487281329920] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-06 19:58:08,631] - [remote_util] [140487281329920] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-06 19:58:08,645] - [remote_util] [140487281329920] - INFO - SSH Connecting to 172.23.122.170 with username:root, attempt#1 of 5 [2021-02-06 19:58:08,743] - [remote_util] [140487281329920] - INFO - SSH Connected to 172.23.122.170 as root [2021-02-06 19:58:09,004] - [remote_util] [140487281329920] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-06 19:58:09,305] - [remote_util] [140487281329920] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-06 19:58:09,363] - [remote_util] [140487281329920] - INFO - 172.23.122.170 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-06 19:58:09,457] - [remote_util] [140487281329920] - INFO - Running systemd command on this server [2021-02-06 19:58:09,457] - [remote_util] [140487281329920] - INFO - running command.raw on 172.23.122.170: systemctl stop couchbase-server.service [2021-02-06 19:58:13,117] - [remote_util] [140487281329920] - INFO - command executed successfully with root [2021-02-06 19:58:13,118] - [basetestcase] [140487281329920] - INFO - Couchbase stopped [2021-02-06 19:58:13,118] - [basetestcase] [140487281329920] - INFO - sleep for 5 secs. ... [2021-02-06 19:58:18,123] - [remote_util] [140487281329920] - INFO - running command.raw on 172.23.122.170: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-06 19:58:18,315] - [remote_util] [140487281329920] - INFO - command executed successfully with root [2021-02-06 19:58:18,315] - [remote_util] [140487281329920] - INFO - running command.raw on 172.23.122.170: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-06 19:58:18,329] - [remote_util] [140487281329920] - INFO - command executed successfully with root [2021-02-06 19:58:18,332] - [remote_util] [140487281329920] - INFO - SSH Connecting to 172.23.122.170 with username:root, attempt#1 of 5 [2021-02-06 19:58:18,548] - [remote_util] [140487281329920] - INFO - SSH Connected to 172.23.122.170 as root [2021-02-06 19:58:18,803] - [remote_util] [140487281329920] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-06 19:58:19,098] - [remote_util] [140487281329920] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-06 19:58:19,157] - [remote_util] [140487281329920] - INFO - 172.23.122.170 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-06 19:58:19,157] - [remote_util] [140487281329920] - INFO - Checking for process beam.smp on linux [2021-02-06 19:58:19,195] - [remote_util] [140487281329920] - INFO - Starting couchbase server [2021-02-06 19:58:19,291] - [remote_util] [140487281329920] - INFO - Running systemd command on this server [2021-02-06 19:58:19,291] - [remote_util] [140487281329920] - INFO - running command.raw on 172.23.122.170: systemctl start couchbase-server.service [2021-02-06 19:58:19,314] - [remote_util] [140487281329920] - INFO - command executed successfully with root [2021-02-06 19:58:19,314] - [remote_util] [140487281329920] - INFO - 172.23.122.170:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-06 19:58:24,319] - [remote_util] [140487281329920] - INFO - running command.raw on 172.23.122.170: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-06 19:58:24,343] - [remote_util] [140487281329920] - INFO - command executed successfully with root [2021-02-06 19:58:24,343] - [remote_util] [140487281329920] - INFO - Couchbase server status: [' Process: 68713 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-06 19:58:24,343] - [remote_util] [140487281329920] - INFO - Checking for process beam.smp on linux [2021-02-06 19:58:24,411] - [remote_util] [140487281329920] - INFO - process beam.smp is running on 172.23.122.170: with pid 68869 [2021-02-06 19:58:24,411] - [basetestcase] [140487281329920] - INFO - Couchbase started [2021-02-06 19:58:24,412] - [remote_util] [140487281329920] - INFO - SSH Connecting to 172.23.122.172 with username:root, attempt#1 of 5 [2021-02-06 19:58:24,511] - [remote_util] [140487281329920] - INFO - SSH Connected to 172.23.122.172 as root [2021-02-06 19:58:26,812] - [remote_util] [140487281329920] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-06 19:58:27,112] - [remote_util] [140487281329920] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-06 19:58:27,120] - [remote_util] [140487281329920] - INFO - SSH Connecting to 172.23.122.172 with username:root, attempt#1 of 5 [2021-02-06 19:58:27,218] - [remote_util] [140487281329920] - INFO - SSH Connected to 172.23.122.172 as root [2021-02-06 19:58:27,492] - [remote_util] [140487281329920] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-06 19:58:27,785] - [remote_util] [140487281329920] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-06 19:58:27,805] - [remote_util] [140487281329920] - INFO - 172.23.122.172 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-06 19:58:27,858] - [remote_util] [140487281329920] - INFO - Running systemd command on this server [2021-02-06 19:58:27,858] - [remote_util] [140487281329920] - INFO - running command.raw on 172.23.122.172: systemctl stop couchbase-server.service [2021-02-06 19:58:29,653] - [remote_util] [140487281329920] - INFO - command executed successfully with root [2021-02-06 19:58:29,653] - [basetestcase] [140487281329920] - INFO - Couchbase stopped [2021-02-06 19:58:29,654] - [basetestcase] [140487281329920] - INFO - sleep for 5 secs. ... [2021-02-06 19:58:34,659] - [remote_util] [140487281329920] - INFO - running command.raw on 172.23.122.172: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-06 19:58:34,680] - [remote_util] [140487281329920] - INFO - command executed successfully with root [2021-02-06 19:58:34,680] - [remote_util] [140487281329920] - INFO - running command.raw on 172.23.122.172: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-06 19:58:34,733] - [remote_util] [140487281329920] - INFO - command executed successfully with root [2021-02-06 19:58:34,734] - [remote_util] [140487281329920] - INFO - SSH Connecting to 172.23.122.172 with username:root, attempt#1 of 5 [2021-02-06 19:58:34,832] - [remote_util] [140487281329920] - INFO - SSH Connected to 172.23.122.172 as root [2021-02-06 19:58:35,133] - [remote_util] [140487281329920] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-06 19:58:35,433] - [remote_util] [140487281329920] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-06 19:58:35,452] - [remote_util] [140487281329920] - INFO - 172.23.122.172 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-06 19:58:35,452] - [remote_util] [140487281329920] - INFO - Checking for process beam.smp on linux [2021-02-06 19:58:35,488] - [remote_util] [140487281329920] - INFO - Starting couchbase server [2021-02-06 19:58:35,581] - [remote_util] [140487281329920] - INFO - Running systemd command on this server [2021-02-06 19:58:35,581] - [remote_util] [140487281329920] - INFO - running command.raw on 172.23.122.172: systemctl start couchbase-server.service [2021-02-06 19:58:35,602] - [remote_util] [140487281329920] - INFO - command executed successfully with root [2021-02-06 19:58:35,603] - [remote_util] [140487281329920] - INFO - 172.23.122.172:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-06 19:58:40,607] - [remote_util] [140487281329920] - INFO - running command.raw on 172.23.122.172: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-06 19:58:40,630] - [remote_util] [140487281329920] - INFO - command executed successfully with root [2021-02-06 19:58:40,630] - [remote_util] [140487281329920] - INFO - Couchbase server status: [' Process: 506 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-06 19:58:40,630] - [remote_util] [140487281329920] - INFO - Checking for process beam.smp on linux [2021-02-06 19:58:40,701] - [remote_util] [140487281329920] - INFO - process beam.smp is running on 172.23.122.172: with pid 653 [2021-02-06 19:58:40,701] - [basetestcase] [140487281329920] - INFO - Couchbase started [2021-02-06 19:58:40,703] - [remote_util] [140487281329920] - INFO - SSH Connecting to 172.23.122.177 with username:root, attempt#1 of 5 [2021-02-06 19:58:40,803] - [remote_util] [140487281329920] - INFO - SSH Connected to 172.23.122.177 as root [2021-02-06 19:58:41,068] - [remote_util] [140487281329920] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-06 19:58:41,364] - [remote_util] [140487281329920] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-06 19:58:41,377] - [remote_util] [140487281329920] - INFO - SSH Connecting to 172.23.122.177 with username:root, attempt#1 of 5 [2021-02-06 19:58:41,476] - [remote_util] [140487281329920] - INFO - SSH Connected to 172.23.122.177 as root [2021-02-06 19:58:41,737] - [remote_util] [140487281329920] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-06 19:58:42,039] - [remote_util] [140487281329920] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-06 19:58:42,104] - [remote_util] [140487281329920] - INFO - 172.23.122.177 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-06 19:58:42,163] - [remote_util] [140487281329920] - INFO - Running systemd command on this server [2021-02-06 19:58:42,163] - [remote_util] [140487281329920] - INFO - running command.raw on 172.23.122.177: systemctl stop couchbase-server.service [2021-02-06 19:58:43,855] - [remote_util] [140487281329920] - INFO - command executed successfully with root [2021-02-06 19:58:43,856] - [basetestcase] [140487281329920] - INFO - Couchbase stopped [2021-02-06 19:58:43,856] - [basetestcase] [140487281329920] - INFO - sleep for 5 secs. ... [2021-02-06 19:58:48,861] - [remote_util] [140487281329920] - INFO - running command.raw on 172.23.122.177: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-06 19:58:48,878] - [remote_util] [140487281329920] - INFO - command executed successfully with root [2021-02-06 19:58:48,878] - [remote_util] [140487281329920] - INFO - running command.raw on 172.23.122.177: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-06 19:58:48,931] - [remote_util] [140487281329920] - INFO - command executed successfully with root [2021-02-06 19:58:48,933] - [remote_util] [140487281329920] - INFO - SSH Connecting to 172.23.122.177 with username:root, attempt#1 of 5 [2021-02-06 19:58:49,032] - [remote_util] [140487281329920] - INFO - SSH Connected to 172.23.122.177 as root [2021-02-06 19:58:49,294] - [remote_util] [140487281329920] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-06 19:58:49,587] - [remote_util] [140487281329920] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-06 19:58:49,605] - [remote_util] [140487281329920] - INFO - 172.23.122.177 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-06 19:58:49,605] - [remote_util] [140487281329920] - INFO - Checking for process beam.smp on linux [2021-02-06 19:58:49,641] - [remote_util] [140487281329920] - INFO - Starting couchbase server [2021-02-06 19:58:49,735] - [remote_util] [140487281329920] - INFO - Running systemd command on this server [2021-02-06 19:58:49,735] - [remote_util] [140487281329920] - INFO - running command.raw on 172.23.122.177: systemctl start couchbase-server.service [2021-02-06 19:58:49,758] - [remote_util] [140487281329920] - INFO - command executed successfully with root [2021-02-06 19:58:49,758] - [remote_util] [140487281329920] - INFO - 172.23.122.177:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-06 19:58:54,763] - [remote_util] [140487281329920] - INFO - running command.raw on 172.23.122.177: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-06 19:58:54,786] - [remote_util] [140487281329920] - INFO - command executed successfully with root [2021-02-06 19:58:54,786] - [remote_util] [140487281329920] - INFO - Couchbase server status: [' Process: 64853 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-06 19:58:54,786] - [remote_util] [140487281329920] - INFO - Checking for process beam.smp on linux [2021-02-06 19:58:54,854] - [remote_util] [140487281329920] - INFO - process beam.smp is running on 172.23.122.177: with pid 64997 [2021-02-06 19:58:54,854] - [basetestcase] [140487281329920] - INFO - Couchbase started [2021-02-06 19:58:54,856] - [remote_util] [140487281329920] - INFO - SSH Connecting to 172.23.122.178 with username:root, attempt#1 of 5 [2021-02-06 19:58:54,959] - [remote_util] [140487281329920] - INFO - SSH Connected to 172.23.122.178 as root [2021-02-06 19:58:55,225] - [remote_util] [140487281329920] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-06 19:58:55,528] - [remote_util] [140487281329920] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-06 19:58:55,540] - [remote_util] [140487281329920] - INFO - SSH Connecting to 172.23.122.178 with username:root, attempt#1 of 5 [2021-02-06 19:58:55,638] - [remote_util] [140487281329920] - INFO - SSH Connected to 172.23.122.178 as root [2021-02-06 19:58:55,896] - [remote_util] [140487281329920] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-06 19:58:56,196] - [remote_util] [140487281329920] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-06 19:58:56,217] - [remote_util] [140487281329920] - INFO - 172.23.122.178 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-06 19:58:56,274] - [remote_util] [140487281329920] - INFO - Running systemd command on this server [2021-02-06 19:58:56,274] - [remote_util] [140487281329920] - INFO - running command.raw on 172.23.122.178: systemctl stop couchbase-server.service [2021-02-06 19:58:58,037] - [remote_util] [140487281329920] - INFO - command executed successfully with root [2021-02-06 19:58:58,037] - [basetestcase] [140487281329920] - INFO - Couchbase stopped [2021-02-06 19:58:58,038] - [basetestcase] [140487281329920] - INFO - sleep for 5 secs. ... [2021-02-06 19:59:03,043] - [remote_util] [140487281329920] - INFO - running command.raw on 172.23.122.178: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-06 19:59:03,060] - [remote_util] [140487281329920] - INFO - command executed successfully with root [2021-02-06 19:59:03,060] - [remote_util] [140487281329920] - INFO - running command.raw on 172.23.122.178: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-06 19:59:03,113] - [remote_util] [140487281329920] - INFO - command executed successfully with root [2021-02-06 19:59:03,115] - [remote_util] [140487281329920] - INFO - SSH Connecting to 172.23.122.178 with username:root, attempt#1 of 5 [2021-02-06 19:59:03,213] - [remote_util] [140487281329920] - INFO - SSH Connected to 172.23.122.178 as root [2021-02-06 19:59:03,470] - [remote_util] [140487281329920] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-06 19:59:03,767] - [remote_util] [140487281329920] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-06 19:59:03,786] - [remote_util] [140487281329920] - INFO - 172.23.122.178 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-06 19:59:03,786] - [remote_util] [140487281329920] - INFO - Checking for process beam.smp on linux [2021-02-06 19:59:03,821] - [remote_util] [140487281329920] - INFO - Starting couchbase server [2021-02-06 19:59:03,917] - [remote_util] [140487281329920] - INFO - Running systemd command on this server [2021-02-06 19:59:03,917] - [remote_util] [140487281329920] - INFO - running command.raw on 172.23.122.178: systemctl start couchbase-server.service [2021-02-06 19:59:03,940] - [remote_util] [140487281329920] - INFO - command executed successfully with root [2021-02-06 19:59:03,940] - [remote_util] [140487281329920] - INFO - 172.23.122.178:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-06 19:59:08,945] - [remote_util] [140487281329920] - INFO - running command.raw on 172.23.122.178: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-06 19:59:08,970] - [remote_util] [140487281329920] - INFO - command executed successfully with root [2021-02-06 19:59:08,970] - [remote_util] [140487281329920] - INFO - Couchbase server status: [' Process: 37484 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-06 19:59:08,971] - [remote_util] [140487281329920] - INFO - Checking for process beam.smp on linux [2021-02-06 19:59:09,039] - [remote_util] [140487281329920] - INFO - process beam.smp is running on 172.23.122.178: with pid 37628 [2021-02-06 19:59:09,039] - [basetestcase] [140487281329920] - INFO - Couchbase started [2021-02-06 19:59:09,040] - [remote_util] [140487281329920] - INFO - SSH Connecting to 172.23.122.175 with username:root, attempt#1 of 5 [2021-02-06 19:59:09,144] - [remote_util] [140487281329920] - INFO - SSH Connected to 172.23.122.175 as root [2021-02-06 19:59:09,405] - [remote_util] [140487281329920] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-06 19:59:09,704] - [remote_util] [140487281329920] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-06 19:59:09,716] - [remote_util] [140487281329920] - INFO - SSH Connecting to 172.23.122.175 with username:root, attempt#1 of 5 [2021-02-06 19:59:09,815] - [remote_util] [140487281329920] - INFO - SSH Connected to 172.23.122.175 as root [2021-02-06 19:59:10,078] - [remote_util] [140487281329920] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-06 19:59:10,371] - [remote_util] [140487281329920] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-06 19:59:10,390] - [remote_util] [140487281329920] - INFO - 172.23.122.175 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-06 19:59:10,442] - [remote_util] [140487281329920] - INFO - Running systemd command on this server [2021-02-06 19:59:10,442] - [remote_util] [140487281329920] - INFO - running command.raw on 172.23.122.175: systemctl stop couchbase-server.service [2021-02-06 19:59:12,151] - [remote_util] [140487281329920] - INFO - command executed successfully with root [2021-02-06 19:59:12,152] - [basetestcase] [140487281329920] - INFO - Couchbase stopped [2021-02-06 19:59:12,152] - [basetestcase] [140487281329920] - INFO - sleep for 5 secs. ... [2021-02-06 19:59:17,157] - [remote_util] [140487281329920] - INFO - running command.raw on 172.23.122.175: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-06 19:59:17,175] - [remote_util] [140487281329920] - INFO - command executed successfully with root [2021-02-06 19:59:17,175] - [remote_util] [140487281329920] - INFO - running command.raw on 172.23.122.175: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-06 19:59:17,228] - [remote_util] [140487281329920] - INFO - command executed successfully with root [2021-02-06 19:59:17,230] - [remote_util] [140487281329920] - INFO - SSH Connecting to 172.23.122.175 with username:root, attempt#1 of 5 [2021-02-06 19:59:17,327] - [remote_util] [140487281329920] - INFO - SSH Connected to 172.23.122.175 as root [2021-02-06 19:59:17,585] - [remote_util] [140487281329920] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-06 19:59:17,878] - [remote_util] [140487281329920] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-06 19:59:17,938] - [remote_util] [140487281329920] - INFO - 172.23.122.175 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-06 19:59:17,938] - [remote_util] [140487281329920] - INFO - Checking for process beam.smp on linux [2021-02-06 19:59:17,975] - [remote_util] [140487281329920] - INFO - Starting couchbase server [2021-02-06 19:59:18,070] - [remote_util] [140487281329920] - INFO - Running systemd command on this server [2021-02-06 19:59:18,070] - [remote_util] [140487281329920] - INFO - running command.raw on 172.23.122.175: systemctl start couchbase-server.service [2021-02-06 19:59:18,092] - [remote_util] [140487281329920] - INFO - command executed successfully with root [2021-02-06 19:59:18,092] - [remote_util] [140487281329920] - INFO - 172.23.122.175:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-06 19:59:23,098] - [remote_util] [140487281329920] - INFO - running command.raw on 172.23.122.175: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-06 19:59:23,128] - [remote_util] [140487281329920] - INFO - command executed successfully with root [2021-02-06 19:59:23,128] - [remote_util] [140487281329920] - INFO - Couchbase server status: [' Process: 51892 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-06 19:59:23,128] - [remote_util] [140487281329920] - INFO - Checking for process beam.smp on linux [2021-02-06 19:59:23,198] - [remote_util] [140487281329920] - INFO - process beam.smp is running on 172.23.122.175: with pid 52037 [2021-02-06 19:59:23,198] - [basetestcase] [140487281329920] - INFO - Couchbase started [2021-02-06 19:59:23,200] - [remote_util] [140487281329920] - INFO - SSH Connecting to 172.23.122.173 with username:root, attempt#1 of 5 [2021-02-06 19:59:23,301] - [remote_util] [140487281329920] - INFO - SSH Connected to 172.23.122.173 as root [2021-02-06 19:59:23,558] - [remote_util] [140487281329920] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-06 19:59:23,856] - [remote_util] [140487281329920] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-06 19:59:23,869] - [remote_util] [140487281329920] - INFO - SSH Connecting to 172.23.122.173 with username:root, attempt#1 of 5 [2021-02-06 19:59:23,971] - [remote_util] [140487281329920] - INFO - SSH Connected to 172.23.122.173 as root [2021-02-06 19:59:24,223] - [remote_util] [140487281329920] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-06 19:59:24,518] - [remote_util] [140487281329920] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-06 19:59:24,579] - [remote_util] [140487281329920] - INFO - 172.23.122.173 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-06 19:59:24,632] - [remote_util] [140487281329920] - INFO - Running systemd command on this server [2021-02-06 19:59:24,632] - [remote_util] [140487281329920] - INFO - running command.raw on 172.23.122.173: systemctl stop couchbase-server.service [2021-02-06 19:59:26,504] - [remote_util] [140487281329920] - INFO - command executed successfully with root [2021-02-06 19:59:26,504] - [basetestcase] [140487281329920] - INFO - Couchbase stopped [2021-02-06 19:59:26,504] - [basetestcase] [140487281329920] - INFO - sleep for 5 secs. ... [2021-02-06 19:59:31,509] - [remote_util] [140487281329920] - INFO - running command.raw on 172.23.122.173: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-06 19:59:31,526] - [remote_util] [140487281329920] - INFO - command executed successfully with root [2021-02-06 19:59:31,527] - [remote_util] [140487281329920] - INFO - running command.raw on 172.23.122.173: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-06 19:59:31,580] - [remote_util] [140487281329920] - INFO - command executed successfully with root [2021-02-06 19:59:31,582] - [remote_util] [140487281329920] - INFO - SSH Connecting to 172.23.122.173 with username:root, attempt#1 of 5 [2021-02-06 19:59:31,679] - [remote_util] [140487281329920] - INFO - SSH Connected to 172.23.122.173 as root [2021-02-06 19:59:31,929] - [remote_util] [140487281329920] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-06 19:59:32,228] - [remote_util] [140487281329920] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-06 19:59:32,289] - [remote_util] [140487281329920] - INFO - 172.23.122.173 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-06 19:59:32,289] - [remote_util] [140487281329920] - INFO - Checking for process beam.smp on linux [2021-02-06 19:59:32,326] - [remote_util] [140487281329920] - INFO - Starting couchbase server [2021-02-06 19:59:32,421] - [remote_util] [140487281329920] - INFO - Running systemd command on this server [2021-02-06 19:59:32,421] - [remote_util] [140487281329920] - INFO - running command.raw on 172.23.122.173: systemctl start couchbase-server.service [2021-02-06 19:59:32,444] - [remote_util] [140487281329920] - INFO - command executed successfully with root [2021-02-06 19:59:32,444] - [remote_util] [140487281329920] - INFO - 172.23.122.173:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-06 19:59:37,446] - [remote_util] [140487281329920] - INFO - running command.raw on 172.23.122.173: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-06 19:59:37,469] - [remote_util] [140487281329920] - INFO - command executed successfully with root [2021-02-06 19:59:37,470] - [remote_util] [140487281329920] - INFO - Couchbase server status: [' Process: 98180 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-06 19:59:37,470] - [remote_util] [140487281329920] - INFO - Checking for process beam.smp on linux [2021-02-06 19:59:37,539] - [remote_util] [140487281329920] - INFO - process beam.smp is running on 172.23.122.173: with pid 98324 [2021-02-06 19:59:37,539] - [basetestcase] [140487281329920] - INFO - Couchbase started [2021-02-06 19:59:37,541] - [remote_util] [140487281329920] - INFO - SSH Connecting to 172.23.122.174 with username:root, attempt#1 of 5 [2021-02-06 19:59:37,643] - [remote_util] [140487281329920] - INFO - SSH Connected to 172.23.122.174 as root [2021-02-06 19:59:37,899] - [remote_util] [140487281329920] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-06 19:59:38,201] - [remote_util] [140487281329920] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-06 19:59:38,214] - [remote_util] [140487281329920] - INFO - SSH Connecting to 172.23.122.174 with username:root, attempt#1 of 5 [2021-02-06 19:59:38,314] - [remote_util] [140487281329920] - INFO - SSH Connected to 172.23.122.174 as root [2021-02-06 19:59:38,565] - [remote_util] [140487281329920] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-06 19:59:38,863] - [remote_util] [140487281329920] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-06 19:59:38,883] - [remote_util] [140487281329920] - INFO - 172.23.122.174 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-06 19:59:38,940] - [remote_util] [140487281329920] - INFO - Running systemd command on this server [2021-02-06 19:59:38,940] - [remote_util] [140487281329920] - INFO - running command.raw on 172.23.122.174: systemctl stop couchbase-server.service [2021-02-06 19:59:50,941] - [remote_util] [140487281329920] - INFO - command executed successfully with root [2021-02-06 19:59:50,941] - [basetestcase] [140487281329920] - INFO - Couchbase stopped [2021-02-06 19:59:50,941] - [basetestcase] [140487281329920] - INFO - sleep for 5 secs. ... [2021-02-06 19:59:55,946] - [remote_util] [140487281329920] - INFO - running command.raw on 172.23.122.174: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-06 19:59:56,057] - [remote_util] [140487281329920] - INFO - command executed successfully with root [2021-02-06 19:59:56,057] - [remote_util] [140487281329920] - INFO - running command.raw on 172.23.122.174: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-06 19:59:56,072] - [remote_util] [140487281329920] - INFO - command executed successfully with root [2021-02-06 19:59:56,073] - [remote_util] [140487281329920] - INFO - SSH Connecting to 172.23.122.174 with username:root, attempt#1 of 5 [2021-02-06 19:59:56,173] - [remote_util] [140487281329920] - INFO - SSH Connected to 172.23.122.174 as root [2021-02-06 19:59:56,427] - [remote_util] [140487281329920] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-06 19:59:56,723] - [remote_util] [140487281329920] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-06 19:59:56,782] - [remote_util] [140487281329920] - INFO - 172.23.122.174 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-06 19:59:56,783] - [remote_util] [140487281329920] - INFO - Checking for process beam.smp on linux [2021-02-06 19:59:56,817] - [remote_util] [140487281329920] - INFO - Starting couchbase server [2021-02-06 19:59:56,912] - [remote_util] [140487281329920] - INFO - Running systemd command on this server [2021-02-06 19:59:56,913] - [remote_util] [140487281329920] - INFO - running command.raw on 172.23.122.174: systemctl start couchbase-server.service [2021-02-06 19:59:56,935] - [remote_util] [140487281329920] - INFO - command executed successfully with root [2021-02-06 19:59:56,935] - [remote_util] [140487281329920] - INFO - 172.23.122.174:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-06 20:00:01,938] - [remote_util] [140487281329920] - INFO - running command.raw on 172.23.122.174: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-06 20:00:01,963] - [remote_util] [140487281329920] - INFO - command executed successfully with root [2021-02-06 20:00:01,964] - [remote_util] [140487281329920] - INFO - Couchbase server status: [' Process: 42732 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-06 20:00:01,964] - [remote_util] [140487281329920] - INFO - Checking for process beam.smp on linux [2021-02-06 20:00:02,033] - [remote_util] [140487281329920] - INFO - process beam.smp is running on 172.23.122.174: with pid 42901 [2021-02-06 20:00:02,033] - [basetestcase] [140487281329920] - INFO - Couchbase started [2021-02-06 20:00:02,035] - [remote_util] [140487281329920] - INFO - SSH Connecting to 172.23.122.176 with username:root, attempt#1 of 5 [2021-02-06 20:00:02,138] - [remote_util] [140487281329920] - INFO - SSH Connected to 172.23.122.176 as root [2021-02-06 20:00:02,401] - [remote_util] [140487281329920] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-06 20:00:02,698] - [remote_util] [140487281329920] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-06 20:00:02,707] - [remote_util] [140487281329920] - INFO - SSH Connecting to 172.23.122.176 with username:root, attempt#1 of 5 [2021-02-06 20:00:02,812] - [remote_util] [140487281329920] - INFO - SSH Connected to 172.23.122.176 as root [2021-02-06 20:00:03,070] - [remote_util] [140487281329920] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-06 20:00:03,363] - [remote_util] [140487281329920] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-06 20:00:03,381] - [remote_util] [140487281329920] - INFO - 172.23.122.176 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-06 20:00:03,433] - [remote_util] [140487281329920] - INFO - Running systemd command on this server [2021-02-06 20:00:03,433] - [remote_util] [140487281329920] - INFO - running command.raw on 172.23.122.176: systemctl stop couchbase-server.service [2021-02-06 20:00:05,359] - [remote_util] [140487281329920] - INFO - command executed successfully with root [2021-02-06 20:00:05,359] - [basetestcase] [140487281329920] - INFO - Couchbase stopped [2021-02-06 20:00:05,359] - [basetestcase] [140487281329920] - INFO - sleep for 5 secs. ... [2021-02-06 20:00:10,364] - [remote_util] [140487281329920] - INFO - running command.raw on 172.23.122.176: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-06 20:00:10,380] - [remote_util] [140487281329920] - INFO - command executed successfully with root [2021-02-06 20:00:10,380] - [remote_util] [140487281329920] - INFO - running command.raw on 172.23.122.176: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-06 20:00:10,435] - [remote_util] [140487281329920] - INFO - command executed successfully with root [2021-02-06 20:00:10,436] - [remote_util] [140487281329920] - INFO - SSH Connecting to 172.23.122.176 with username:root, attempt#1 of 5 [2021-02-06 20:00:10,534] - [remote_util] [140487281329920] - INFO - SSH Connected to 172.23.122.176 as root [2021-02-06 20:00:10,787] - [remote_util] [140487281329920] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-06 20:00:11,083] - [remote_util] [140487281329920] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-06 20:00:11,143] - [remote_util] [140487281329920] - INFO - 172.23.122.176 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-06 20:00:11,143] - [remote_util] [140487281329920] - INFO - Checking for process beam.smp on linux [2021-02-06 20:00:11,178] - [remote_util] [140487281329920] - INFO - Starting couchbase server [2021-02-06 20:00:11,272] - [remote_util] [140487281329920] - INFO - Running systemd command on this server [2021-02-06 20:00:11,272] - [remote_util] [140487281329920] - INFO - running command.raw on 172.23.122.176: systemctl start couchbase-server.service [2021-02-06 20:00:11,294] - [remote_util] [140487281329920] - INFO - command executed successfully with root [2021-02-06 20:00:11,295] - [remote_util] [140487281329920] - INFO - 172.23.122.176:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-06 20:00:16,299] - [remote_util] [140487281329920] - INFO - running command.raw on 172.23.122.176: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-06 20:00:16,323] - [remote_util] [140487281329920] - INFO - command executed successfully with root [2021-02-06 20:00:16,324] - [remote_util] [140487281329920] - INFO - Couchbase server status: [' Process: 44001 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-06 20:00:16,324] - [remote_util] [140487281329920] - INFO - Checking for process beam.smp on linux [2021-02-06 20:00:16,390] - [remote_util] [140487281329920] - INFO - process beam.smp is running on 172.23.122.176: with pid 44145 [2021-02-06 20:00:16,390] - [basetestcase] [140487281329920] - INFO - Couchbase started [2021-02-06 20:00:16,390] - [basetestcase] [140487281329920] - INFO - sleep for 10 secs. ... [2021-02-06 20:00:26,401] - [remote_util] [140487281329920] - INFO - SSH Connecting to 172.23.122.170 with username:root, attempt#1 of 5 [2021-02-06 20:00:26,402] - [remote_util] [140489396868864] - INFO - SSH Connecting to 172.23.122.172 with username:root, attempt#1 of 5 [2021-02-06 20:00:26,405] - [remote_util] [140489185556224] - INFO - SSH Connecting to 172.23.122.177 with username:root, attempt#1 of 5 [2021-02-06 20:00:26,407] - [remote_util] [140489160382208] - INFO - SSH Connecting to 172.23.122.178 with username:root, attempt#1 of 5 [2021-02-06 20:00:26,409] - [remote_util] [140489151989504] - INFO - SSH Connecting to 172.23.122.175 with username:root, attempt#1 of 5 [2021-02-06 20:00:26,412] - [remote_util] [140489143596800] - INFO - SSH Connecting to 172.23.122.173 with username:root, attempt#1 of 5 [2021-02-06 20:00:26,415] - [remote_util] [140488581576448] - INFO - SSH Connecting to 172.23.122.174 with username:root, attempt#1 of 5 [2021-02-06 20:00:26,417] - [remote_util] [140488556398336] - INFO - SSH Connecting to 172.23.122.176 with username:root, attempt#1 of 5 [2021-02-06 20:00:26,506] - [remote_util] [140487281329920] - INFO - SSH Connected to 172.23.122.170 as root [2021-02-06 20:00:26,513] - [remote_util] [140489396868864] - INFO - SSH Connected to 172.23.122.172 as root [2021-02-06 20:00:26,520] - [remote_util] [140489143596800] - INFO - SSH Connected to 172.23.122.173 as root [2021-02-06 20:00:26,520] - [remote_util] [140489185556224] - INFO - SSH Connected to 172.23.122.177 as root [2021-02-06 20:00:26,522] - [remote_util] [140488556398336] - INFO - SSH Connected to 172.23.122.176 as root [2021-02-06 20:00:26,522] - [remote_util] [140489160382208] - INFO - SSH Connected to 172.23.122.178 as root [2021-02-06 20:00:26,529] - [remote_util] [140489151989504] - INFO - SSH Connected to 172.23.122.175 as root [2021-02-06 20:00:26,530] - [remote_util] [140488581576448] - INFO - SSH Connected to 172.23.122.174 as root [2021-02-06 20:00:26,833] - [remote_util] [140487281329920] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-06 20:00:26,833] - [remote_util] [140489143596800] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-06 20:00:26,840] - [remote_util] [140488556398336] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-06 20:00:26,847] - [remote_util] [140489160382208] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-06 20:00:26,853] - [remote_util] [140489185556224] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-06 20:00:26,854] - [remote_util] [140488581576448] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-06 20:00:26,857] - [remote_util] [140489396868864] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-06 20:00:26,859] - [remote_util] [140489151989504] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-06 20:00:27,141] - [remote_util] [140487281329920] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-06 20:00:27,142] - [remote_util] [140487281329920] - INFO - running command.raw on 172.23.122.170: /opt/couchbase/bin/cbcollect_info 172.23.122.170-20210206-2000-diag.zip [2021-02-06 20:00:27,153] - [remote_util] [140489160382208] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-06 20:00:27,154] - [remote_util] [140489160382208] - INFO - running command.raw on 172.23.122.178: /opt/couchbase/bin/cbcollect_info 172.23.122.178-20210206-2000-diag.zip [2021-02-06 20:00:27,157] - [remote_util] [140488581576448] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-06 20:00:27,157] - [remote_util] [140488581576448] - INFO - running command.raw on 172.23.122.174: /opt/couchbase/bin/cbcollect_info 172.23.122.174-20210206-2000-diag.zip [2021-02-06 20:00:27,165] - [remote_util] [140489151989504] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-06 20:00:27,166] - [remote_util] [140489185556224] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-06 20:00:27,166] - [remote_util] [140489185556224] - INFO - running command.raw on 172.23.122.177: /opt/couchbase/bin/cbcollect_info 172.23.122.177-20210206-2000-diag.zip [2021-02-06 20:00:27,167] - [remote_util] [140489151989504] - INFO - running command.raw on 172.23.122.175: /opt/couchbase/bin/cbcollect_info 172.23.122.175-20210206-2000-diag.zip [2021-02-06 20:00:27,186] - [remote_util] [140489143596800] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-06 20:00:27,187] - [remote_util] [140489143596800] - INFO - running command.raw on 172.23.122.173: /opt/couchbase/bin/cbcollect_info 172.23.122.173-20210206-2000-diag.zip [2021-02-06 20:00:27,188] - [remote_util] [140488556398336] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-06 20:00:27,189] - [remote_util] [140488556398336] - INFO - running command.raw on 172.23.122.176: /opt/couchbase/bin/cbcollect_info 172.23.122.176-20210206-2000-diag.zip [2021-02-06 20:00:27,203] - [remote_util] [140489396868864] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-06 20:00:27,203] - [remote_util] [140489396868864] - INFO - running command.raw on 172.23.122.172: /opt/couchbase/bin/cbcollect_info 172.23.122.172-20210206-2000-diag.zip [2021-02-06 20:01:58,181] - [remote_util] [140489396868864] - INFO - command executed successfully with root [2021-02-06 20:01:58,626] - [remote_util] [140489396868864] - INFO - running command.raw on 172.23.122.172: rm -f /root/172.23.122.172-20210206-2000-diag.zip [2021-02-06 20:01:58,683] - [remote_util] [140489396868864] - INFO - command executed successfully with root [2021-02-06 20:01:58,983] - [remote_util] [140488556398336] - INFO - command executed successfully with root [2021-02-06 20:01:59,417] - [remote_util] [140488556398336] - INFO - running command.raw on 172.23.122.176: rm -f /root/172.23.122.176-20210206-2000-diag.zip [2021-02-06 20:01:59,435] - [remote_util] [140488556398336] - INFO - command executed successfully with root [2021-02-06 20:02:05,665] - [remote_util] [140489160382208] - INFO - command executed successfully with root [2021-02-06 20:02:06,178] - [remote_util] [140489160382208] - INFO - running command.raw on 172.23.122.178: rm -f /root/172.23.122.178-20210206-2000-diag.zip [2021-02-06 20:02:06,198] - [remote_util] [140489160382208] - INFO - command executed successfully with root [2021-02-06 20:02:06,339] - [remote_util] [140489185556224] - INFO - command executed successfully with root [2021-02-06 20:02:06,814] - [remote_util] [140489185556224] - INFO - running command.raw on 172.23.122.177: rm -f /root/172.23.122.177-20210206-2000-diag.zip [2021-02-06 20:02:06,834] - [remote_util] [140489185556224] - INFO - command executed successfully with root [2021-02-06 20:02:07,359] - [remote_util] [140489151989504] - INFO - command executed successfully with root [2021-02-06 20:02:07,928] - [remote_util] [140489151989504] - INFO - running command.raw on 172.23.122.175: rm -f /root/172.23.122.175-20210206-2000-diag.zip [2021-02-06 20:02:07,946] - [remote_util] [140489151989504] - INFO - command executed successfully with root [2021-02-06 20:02:08,950] - [remote_util] [140487281329920] - INFO - command executed successfully with root [2021-02-06 20:02:09,505] - [remote_util] [140488581576448] - INFO - command executed successfully with root [2021-02-06 20:02:09,540] - [remote_util] [140487281329920] - INFO - running command.raw on 172.23.122.170: rm -f /root/172.23.122.170-20210206-2000-diag.zip [2021-02-06 20:02:09,560] - [remote_util] [140487281329920] - INFO - command executed successfully with root [2021-02-06 20:02:10,112] - [remote_util] [140488581576448] - INFO - running command.raw on 172.23.122.174: rm -f /root/172.23.122.174-20210206-2000-diag.zip [2021-02-06 20:02:10,131] - [remote_util] [140488581576448] - INFO - command executed successfully with root [2021-02-06 20:02:12,863] - [remote_util] [140489143596800] - INFO - command executed successfully with root [2021-02-06 20:02:13,408] - [remote_util] [140489143596800] - INFO - running command.raw on 172.23.122.173: rm -f /root/172.23.122.173-20210206-2000-diag.zip [2021-02-06 20:02:13,426] - [remote_util] [140489143596800] - INFO - command executed successfully with root