[2021-02-12 22:33:24,028] - [eventing_base] [140097149204224] - INFO - Starting Test: test_eventing_rebalance_with_multiple_kv_nodes [2021-02-12 22:33:24,030] - [remote_util] [140097149204224] - INFO - SSH Connecting to 172.23.123.61 with username:root, attempt#1 of 5 [2021-02-12 22:33:24,133] - [remote_util] [140097149204224] - INFO - SSH Connected to 172.23.123.61 as root [2021-02-12 22:33:24,401] - [remote_util] [140097149204224] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 22:33:24,698] - [remote_util] [140097149204224] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 22:33:24,704] - [rest_client] [140097149204224] - INFO - -->is_ns_server_running? [2021-02-12 22:33:24,718] - [rest_client] [140097149204224] - INFO - Node version in cluster 7.0.0-4454-enterprise [2021-02-12 22:33:24,732] - [rest_client] [140097149204224] - INFO - Node versions in cluster ['7.0.0-4454-enterprise', '7.0.0-4454-enterprise', '7.0.0-4454-enterprise', '7.0.0-4454-enterprise', '7.0.0-4454-enterprise'] [2021-02-12 22:33:24,732] - [basetestcase] [140097149204224] - INFO - ============== basetestcase setup was started for test #8 test_eventing_rebalance_with_multiple_kv_nodes============== [2021-02-12 22:33:24,770] - [bucket_helper] [140097149204224] - INFO - deleting existing buckets ['dst_bucket', 'metadata', 'src_bucket'] on 172.23.123.61 [2021-02-12 22:33:25,347] - [bucket_helper] [140097149204224] - INFO - waiting for bucket deletion to complete.... [2021-02-12 22:33:25,507] - [rest_client] [140097149204224] - INFO - node 172.23.123.61 existing buckets : ['metadata', 'src_bucket'] [2021-02-12 22:33:25,508] - [bucket_helper] [140097149204224] - INFO - deleted bucket : dst_bucket from 172.23.123.61 [2021-02-12 22:33:26,014] - [bucket_helper] [140097149204224] - INFO - waiting for bucket deletion to complete.... [2021-02-12 22:33:26,024] - [rest_client] [140097149204224] - INFO - node 172.23.123.61 existing buckets : ['src_bucket'] [2021-02-12 22:33:26,025] - [bucket_helper] [140097149204224] - INFO - deleted bucket : metadata from 172.23.123.61 [2021-02-12 22:33:26,755] - [bucket_helper] [140097149204224] - INFO - waiting for bucket deletion to complete.... [2021-02-12 22:33:26,757] - [rest_client] [140097149204224] - INFO - node 172.23.123.61 existing buckets : [] [2021-02-12 22:33:26,757] - [bucket_helper] [140097149204224] - INFO - deleted bucket : src_bucket from 172.23.123.61 [2021-02-12 22:33:26,766] - [bucket_helper] [140097149204224] - INFO - Could not find any buckets for node 172.23.123.62, nothing to delete [2021-02-12 22:33:26,773] - [bucket_helper] [140097149204224] - INFO - Could not find any buckets for node 172.23.122.90, nothing to delete [2021-02-12 22:33:26,778] - [bucket_helper] [140097149204224] - INFO - Could not find any buckets for node 172.23.122.117, nothing to delete [2021-02-12 22:33:26,785] - [bucket_helper] [140097149204224] - INFO - Could not find any buckets for node 172.23.122.97, nothing to delete [2021-02-12 22:33:26,791] - [bucket_helper] [140097149204224] - INFO - Could not find any buckets for node 172.23.122.82, nothing to delete [2021-02-12 22:33:26,797] - [bucket_helper] [140097149204224] - INFO - Could not find any buckets for node 172.23.122.118, nothing to delete [2021-02-12 22:33:26,802] - [bucket_helper] [140097149204224] - INFO - Could not find any buckets for node 172.23.122.116, nothing to delete [2021-02-12 22:33:26,808] - [rest_client] [140097149204224] - INFO - -->is_ns_server_running? [2021-02-12 22:33:26,819] - [cluster_helper] [140097149204224] - INFO - rebalancing all nodes in order to remove nodes [2021-02-12 22:33:26,821] - [rest_client] [140097149204224] - INFO - rebalance params : {'knownNodes': 'ns_1@172.23.122.82,ns_1@172.23.122.90,ns_1@172.23.122.97,ns_1@172.23.123.61,ns_1@172.23.123.62', 'ejectedNodes': 'ns_1@172.23.122.82,ns_1@172.23.122.90,ns_1@172.23.122.97,ns_1@172.23.123.62', 'user': 'Administrator', 'password': 'password'} [2021-02-12 22:33:26,849] - [rest_client] [140097149204224] - INFO - rebalance operation started [2021-02-12 22:33:26,852] - [rest_client] [140097149204224] - INFO - rebalance percentage : 0.00 % [2021-02-12 22:33:36,865] - [rest_client] [140097149204224] - INFO - rebalance percentage : 30.00 % [2021-02-12 22:33:46,877] - [rest_client] [140097149204224] - INFO - rebalance percentage : 90.00 % [2021-02-12 22:34:06,903] - [rest_client] [140097149204224] - INFO - rebalance progress took 40.05 seconds [2021-02-12 22:34:06,903] - [rest_client] [140097149204224] - INFO - sleep for 10 seconds after rebalance... [2021-02-12 22:34:16,937] - [cluster_helper] [140097149204224] - INFO - removed all the nodes from cluster associated with ip:172.23.123.61 port:8091 ssh_username:root ? [('ns_1@172.23.122.82', 8091), ('ns_1@172.23.122.90', 8091), ('ns_1@172.23.122.97', 8091), ('ns_1@172.23.123.62', 8091)] [2021-02-12 22:34:16,942] - [cluster_helper] [140097149204224] - INFO - waiting for ns_server @ 172.23.123.61:8091 [2021-02-12 22:34:16,942] - [rest_client] [140097149204224] - INFO - -->is_ns_server_running? [2021-02-12 22:34:16,945] - [cluster_helper] [140097149204224] - INFO - ns_server @ 172.23.123.61:8091 is running [2021-02-12 22:34:16,948] - [cluster_helper] [140097149204224] - INFO - waiting for ns_server @ 172.23.123.62:8091 [2021-02-12 22:34:16,948] - [rest_client] [140097149204224] - INFO - -->is_ns_server_running? [2021-02-12 22:34:16,951] - [cluster_helper] [140097149204224] - INFO - ns_server @ 172.23.123.62:8091 is running [2021-02-12 22:34:16,954] - [cluster_helper] [140097149204224] - INFO - waiting for ns_server @ 172.23.122.90:8091 [2021-02-12 22:34:16,954] - [rest_client] [140097149204224] - INFO - -->is_ns_server_running? [2021-02-12 22:34:16,956] - [cluster_helper] [140097149204224] - INFO - ns_server @ 172.23.122.90:8091 is running [2021-02-12 22:34:16,960] - [cluster_helper] [140097149204224] - INFO - waiting for ns_server @ 172.23.122.117:8091 [2021-02-12 22:34:16,960] - [rest_client] [140097149204224] - INFO - -->is_ns_server_running? [2021-02-12 22:34:16,963] - [cluster_helper] [140097149204224] - INFO - ns_server @ 172.23.122.117:8091 is running [2021-02-12 22:34:16,967] - [cluster_helper] [140097149204224] - INFO - waiting for ns_server @ 172.23.122.97:8091 [2021-02-12 22:34:16,967] - [rest_client] [140097149204224] - INFO - -->is_ns_server_running? [2021-02-12 22:34:16,969] - [cluster_helper] [140097149204224] - INFO - ns_server @ 172.23.122.97:8091 is running [2021-02-12 22:34:16,973] - [cluster_helper] [140097149204224] - INFO - waiting for ns_server @ 172.23.122.82:8091 [2021-02-12 22:34:16,973] - [rest_client] [140097149204224] - INFO - -->is_ns_server_running? [2021-02-12 22:34:16,976] - [cluster_helper] [140097149204224] - INFO - ns_server @ 172.23.122.82:8091 is running [2021-02-12 22:34:16,980] - [cluster_helper] [140097149204224] - INFO - waiting for ns_server @ 172.23.122.118:8091 [2021-02-12 22:34:16,980] - [rest_client] [140097149204224] - INFO - -->is_ns_server_running? [2021-02-12 22:34:16,984] - [cluster_helper] [140097149204224] - INFO - ns_server @ 172.23.122.118:8091 is running [2021-02-12 22:34:16,988] - [cluster_helper] [140097149204224] - INFO - waiting for ns_server @ 172.23.122.116:8091 [2021-02-12 22:34:16,988] - [rest_client] [140097149204224] - INFO - -->is_ns_server_running? [2021-02-12 22:34:16,991] - [cluster_helper] [140097149204224] - INFO - ns_server @ 172.23.122.116:8091 is running [2021-02-12 22:34:16,991] - [basetestcase] [140097149204224] - INFO - Building docker image with java sdk client [2021-02-12 22:34:20,596] - [basetestcase] [140097149204224] - INFO - initializing cluster [2021-02-12 22:34:20,598] - [remote_util] [140097149204224] - INFO - SSH Connecting to 172.23.123.61 with username:root, attempt#1 of 5 [2021-02-12 22:34:20,696] - [remote_util] [140097149204224] - INFO - SSH Connected to 172.23.123.61 as root [2021-02-12 22:34:20,951] - [remote_util] [140097149204224] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 22:34:21,247] - [remote_util] [140097149204224] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 22:34:21,257] - [remote_util] [140097149204224] - INFO - SSH Connecting to 172.23.123.61 with username:root, attempt#1 of 5 [2021-02-12 22:34:21,353] - [remote_util] [140097149204224] - INFO - SSH Connected to 172.23.123.61 as root [2021-02-12 22:34:21,612] - [remote_util] [140097149204224] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 22:34:21,908] - [remote_util] [140097149204224] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 22:34:21,966] - [remote_util] [140097149204224] - INFO - 172.23.123.61 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 22:34:22,018] - [remote_util] [140097149204224] - INFO - Running systemd command on this server [2021-02-12 22:34:22,018] - [remote_util] [140097149204224] - INFO - running command.raw on 172.23.123.61: systemctl stop couchbase-server.service [2021-02-12 22:34:23,840] - [remote_util] [140097149204224] - INFO - command executed successfully with root [2021-02-12 22:34:23,840] - [basetestcase] [140097149204224] - INFO - Couchbase stopped [2021-02-12 22:34:23,841] - [basetestcase] [140097149204224] - INFO - sleep for 5 secs. ... [2021-02-12 22:34:28,846] - [remote_util] [140097149204224] - INFO - running command.raw on 172.23.123.61: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-12 22:34:28,862] - [remote_util] [140097149204224] - INFO - command executed successfully with root [2021-02-12 22:34:28,862] - [remote_util] [140097149204224] - INFO - running command.raw on 172.23.123.61: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-12 22:34:28,921] - [remote_util] [140097149204224] - INFO - command executed successfully with root [2021-02-12 22:34:28,923] - [remote_util] [140097149204224] - INFO - SSH Connecting to 172.23.123.61 with username:root, attempt#1 of 5 [2021-02-12 22:34:29,026] - [remote_util] [140097149204224] - INFO - SSH Connected to 172.23.123.61 as root [2021-02-12 22:34:29,295] - [remote_util] [140097149204224] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 22:34:29,598] - [remote_util] [140097149204224] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 22:34:29,657] - [remote_util] [140097149204224] - INFO - 172.23.123.61 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 22:34:29,657] - [remote_util] [140097149204224] - INFO - Checking for process beam.smp on linux [2021-02-12 22:34:29,696] - [remote_util] [140097149204224] - INFO - Starting couchbase server [2021-02-12 22:34:29,789] - [remote_util] [140097149204224] - INFO - Running systemd command on this server [2021-02-12 22:34:29,789] - [remote_util] [140097149204224] - INFO - running command.raw on 172.23.123.61: systemctl start couchbase-server.service [2021-02-12 22:34:29,812] - [remote_util] [140097149204224] - INFO - command executed successfully with root [2021-02-12 22:34:29,813] - [remote_util] [140097149204224] - INFO - 172.23.123.61:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-12 22:34:34,818] - [remote_util] [140097149204224] - INFO - running command.raw on 172.23.123.61: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-12 22:34:34,843] - [remote_util] [140097149204224] - INFO - command executed successfully with root [2021-02-12 22:34:34,843] - [remote_util] [140097149204224] - INFO - Couchbase server status: [' Process: 120221 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-12 22:34:34,844] - [remote_util] [140097149204224] - INFO - Checking for process beam.smp on linux [2021-02-12 22:34:34,911] - [remote_util] [140097149204224] - INFO - process beam.smp is running on 172.23.123.61: with pid 120367 [2021-02-12 22:34:34,911] - [basetestcase] [140097149204224] - INFO - Couchbase started [2021-02-12 22:34:34,913] - [remote_util] [140097149204224] - INFO - SSH Connecting to 172.23.123.62 with username:root, attempt#1 of 5 [2021-02-12 22:34:35,030] - [remote_util] [140097149204224] - INFO - SSH Connected to 172.23.123.62 as root [2021-02-12 22:34:35,308] - [remote_util] [140097149204224] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 22:34:35,612] - [remote_util] [140097149204224] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 22:34:35,621] - [remote_util] [140097149204224] - INFO - SSH Connecting to 172.23.123.62 with username:root, attempt#1 of 5 [2021-02-12 22:34:35,721] - [remote_util] [140097149204224] - INFO - SSH Connected to 172.23.123.62 as root [2021-02-12 22:34:35,986] - [remote_util] [140097149204224] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 22:34:36,287] - [remote_util] [140097149204224] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 22:34:36,347] - [remote_util] [140097149204224] - INFO - 172.23.123.62 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 22:34:36,406] - [remote_util] [140097149204224] - INFO - Running systemd command on this server [2021-02-12 22:34:36,406] - [remote_util] [140097149204224] - INFO - running command.raw on 172.23.123.62: systemctl stop couchbase-server.service [2021-02-12 22:34:38,092] - [remote_util] [140097149204224] - INFO - command executed successfully with root [2021-02-12 22:34:38,092] - [basetestcase] [140097149204224] - INFO - Couchbase stopped [2021-02-12 22:34:38,092] - [basetestcase] [140097149204224] - INFO - sleep for 5 secs. ... [2021-02-12 22:34:43,097] - [remote_util] [140097149204224] - INFO - running command.raw on 172.23.123.62: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-12 22:34:43,119] - [remote_util] [140097149204224] - INFO - command executed successfully with root [2021-02-12 22:34:43,119] - [remote_util] [140097149204224] - INFO - running command.raw on 172.23.123.62: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-12 22:34:43,172] - [remote_util] [140097149204224] - INFO - command executed successfully with root [2021-02-12 22:34:43,174] - [remote_util] [140097149204224] - INFO - SSH Connecting to 172.23.123.62 with username:root, attempt#1 of 5 [2021-02-12 22:34:43,277] - [remote_util] [140097149204224] - INFO - SSH Connected to 172.23.123.62 as root [2021-02-12 22:34:43,550] - [remote_util] [140097149204224] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 22:34:43,851] - [remote_util] [140097149204224] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 22:34:43,870] - [remote_util] [140097149204224] - INFO - 172.23.123.62 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 22:34:43,870] - [remote_util] [140097149204224] - INFO - Checking for process beam.smp on linux [2021-02-12 22:34:43,907] - [remote_util] [140097149204224] - INFO - Starting couchbase server [2021-02-12 22:34:44,000] - [remote_util] [140097149204224] - INFO - Running systemd command on this server [2021-02-12 22:34:44,000] - [remote_util] [140097149204224] - INFO - running command.raw on 172.23.123.62: systemctl start couchbase-server.service [2021-02-12 22:34:44,022] - [remote_util] [140097149204224] - INFO - command executed successfully with root [2021-02-12 22:34:44,022] - [remote_util] [140097149204224] - INFO - 172.23.123.62:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-12 22:34:49,028] - [remote_util] [140097149204224] - INFO - running command.raw on 172.23.123.62: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-12 22:34:49,058] - [remote_util] [140097149204224] - INFO - command executed successfully with root [2021-02-12 22:34:49,058] - [remote_util] [140097149204224] - INFO - Couchbase server status: [' Process: 11366 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-12 22:34:49,058] - [remote_util] [140097149204224] - INFO - Checking for process beam.smp on linux [2021-02-12 22:34:49,127] - [remote_util] [140097149204224] - INFO - process beam.smp is running on 172.23.123.62: with pid 11510 [2021-02-12 22:34:49,128] - [basetestcase] [140097149204224] - INFO - Couchbase started [2021-02-12 22:34:49,129] - [remote_util] [140097149204224] - INFO - SSH Connecting to 172.23.122.90 with username:root, attempt#1 of 5 [2021-02-12 22:34:49,256] - [remote_util] [140097149204224] - INFO - SSH Connected to 172.23.122.90 as root [2021-02-12 22:34:49,558] - [remote_util] [140097149204224] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 22:34:49,861] - [remote_util] [140097149204224] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 22:34:49,868] - [remote_util] [140097149204224] - INFO - SSH Connecting to 172.23.122.90 with username:root, attempt#1 of 5 [2021-02-12 22:34:49,974] - [remote_util] [140097149204224] - INFO - SSH Connected to 172.23.122.90 as root [2021-02-12 22:34:50,269] - [remote_util] [140097149204224] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 22:34:50,572] - [remote_util] [140097149204224] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 22:34:50,591] - [remote_util] [140097149204224] - INFO - 172.23.122.90 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 22:34:50,649] - [remote_util] [140097149204224] - INFO - Running systemd command on this server [2021-02-12 22:34:50,649] - [remote_util] [140097149204224] - INFO - running command.raw on 172.23.122.90: systemctl stop couchbase-server.service [2021-02-12 22:34:52,482] - [remote_util] [140097149204224] - INFO - command executed successfully with root [2021-02-12 22:34:52,482] - [basetestcase] [140097149204224] - INFO - Couchbase stopped [2021-02-12 22:34:52,482] - [basetestcase] [140097149204224] - INFO - sleep for 5 secs. ... [2021-02-12 22:34:57,488] - [remote_util] [140097149204224] - INFO - running command.raw on 172.23.122.90: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-12 22:34:57,511] - [remote_util] [140097149204224] - INFO - command executed successfully with root [2021-02-12 22:34:57,511] - [remote_util] [140097149204224] - INFO - running command.raw on 172.23.122.90: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-12 22:34:57,566] - [remote_util] [140097149204224] - INFO - command executed successfully with root [2021-02-12 22:34:57,567] - [remote_util] [140097149204224] - INFO - SSH Connecting to 172.23.122.90 with username:root, attempt#1 of 5 [2021-02-12 22:34:57,674] - [remote_util] [140097149204224] - INFO - SSH Connected to 172.23.122.90 as root [2021-02-12 22:34:57,983] - [remote_util] [140097149204224] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 22:34:58,292] - [remote_util] [140097149204224] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 22:34:58,312] - [remote_util] [140097149204224] - INFO - 172.23.122.90 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 22:34:58,312] - [remote_util] [140097149204224] - INFO - Checking for process beam.smp on linux [2021-02-12 22:34:58,354] - [remote_util] [140097149204224] - INFO - Starting couchbase server [2021-02-12 22:34:58,453] - [remote_util] [140097149204224] - INFO - Running systemd command on this server [2021-02-12 22:34:58,454] - [remote_util] [140097149204224] - INFO - running command.raw on 172.23.122.90: systemctl start couchbase-server.service [2021-02-12 22:34:58,481] - [remote_util] [140097149204224] - INFO - command executed successfully with root [2021-02-12 22:34:58,481] - [remote_util] [140097149204224] - INFO - 172.23.122.90:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-12 22:35:03,486] - [remote_util] [140097149204224] - INFO - running command.raw on 172.23.122.90: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-12 22:35:03,515] - [remote_util] [140097149204224] - INFO - command executed successfully with root [2021-02-12 22:35:03,515] - [remote_util] [140097149204224] - INFO - Couchbase server status: [' Process: 120679 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-12 22:35:03,515] - [remote_util] [140097149204224] - INFO - Checking for process beam.smp on linux [2021-02-12 22:35:03,585] - [remote_util] [140097149204224] - INFO - process beam.smp is running on 172.23.122.90: with pid 120867 [2021-02-12 22:35:03,585] - [basetestcase] [140097149204224] - INFO - Couchbase started [2021-02-12 22:35:03,586] - [remote_util] [140097149204224] - INFO - SSH Connecting to 172.23.122.117 with username:root, attempt#1 of 5 [2021-02-12 22:35:03,689] - [remote_util] [140097149204224] - INFO - SSH Connected to 172.23.122.117 as root [2021-02-12 22:35:03,947] - [remote_util] [140097149204224] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 22:35:04,235] - [remote_util] [140097149204224] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 22:35:04,243] - [remote_util] [140097149204224] - INFO - SSH Connecting to 172.23.122.117 with username:root, attempt#1 of 5 [2021-02-12 22:35:04,339] - [remote_util] [140097149204224] - INFO - SSH Connected to 172.23.122.117 as root [2021-02-12 22:35:04,586] - [remote_util] [140097149204224] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 22:35:04,879] - [remote_util] [140097149204224] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 22:35:04,899] - [remote_util] [140097149204224] - INFO - 172.23.122.117 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 22:35:04,951] - [remote_util] [140097149204224] - INFO - Running systemd command on this server [2021-02-12 22:35:04,951] - [remote_util] [140097149204224] - INFO - running command.raw on 172.23.122.117: systemctl stop couchbase-server.service [2021-02-12 22:35:06,643] - [remote_util] [140097149204224] - INFO - command executed successfully with root [2021-02-12 22:35:06,643] - [basetestcase] [140097149204224] - INFO - Couchbase stopped [2021-02-12 22:35:06,644] - [basetestcase] [140097149204224] - INFO - sleep for 5 secs. ... [2021-02-12 22:35:11,647] - [remote_util] [140097149204224] - INFO - running command.raw on 172.23.122.117: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-12 22:35:11,668] - [remote_util] [140097149204224] - INFO - command executed successfully with root [2021-02-12 22:35:11,668] - [remote_util] [140097149204224] - INFO - running command.raw on 172.23.122.117: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-12 22:35:11,721] - [remote_util] [140097149204224] - INFO - command executed successfully with root [2021-02-12 22:35:11,722] - [remote_util] [140097149204224] - INFO - SSH Connecting to 172.23.122.117 with username:root, attempt#1 of 5 [2021-02-12 22:35:11,821] - [remote_util] [140097149204224] - INFO - SSH Connected to 172.23.122.117 as root [2021-02-12 22:35:12,080] - [remote_util] [140097149204224] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 22:35:12,375] - [remote_util] [140097149204224] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 22:35:12,434] - [remote_util] [140097149204224] - INFO - 172.23.122.117 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 22:35:12,434] - [remote_util] [140097149204224] - INFO - Checking for process beam.smp on linux [2021-02-12 22:35:12,470] - [remote_util] [140097149204224] - INFO - Starting couchbase server [2021-02-12 22:35:12,563] - [remote_util] [140097149204224] - INFO - Running systemd command on this server [2021-02-12 22:35:12,564] - [remote_util] [140097149204224] - INFO - running command.raw on 172.23.122.117: systemctl start couchbase-server.service [2021-02-12 22:35:12,586] - [remote_util] [140097149204224] - INFO - command executed successfully with root [2021-02-12 22:35:12,586] - [remote_util] [140097149204224] - INFO - 172.23.122.117:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-12 22:35:17,591] - [remote_util] [140097149204224] - INFO - running command.raw on 172.23.122.117: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-12 22:35:17,616] - [remote_util] [140097149204224] - INFO - command executed successfully with root [2021-02-12 22:35:17,617] - [remote_util] [140097149204224] - INFO - Couchbase server status: [' Process: 81024 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-12 22:35:17,617] - [remote_util] [140097149204224] - INFO - Checking for process beam.smp on linux [2021-02-12 22:35:17,681] - [remote_util] [140097149204224] - INFO - process beam.smp is running on 172.23.122.117: with pid 81169 [2021-02-12 22:35:17,681] - [basetestcase] [140097149204224] - INFO - Couchbase started [2021-02-12 22:35:17,683] - [remote_util] [140097149204224] - INFO - SSH Connecting to 172.23.122.97 with username:root, attempt#1 of 5 [2021-02-12 22:35:17,883] - [remote_util] [140097149204224] - INFO - SSH Connected to 172.23.122.97 as root [2021-02-12 22:35:18,179] - [remote_util] [140097149204224] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 22:35:18,490] - [remote_util] [140097149204224] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 22:35:18,498] - [remote_util] [140097149204224] - INFO - SSH Connecting to 172.23.122.97 with username:root, attempt#1 of 5 [2021-02-12 22:35:18,602] - [remote_util] [140097149204224] - INFO - SSH Connected to 172.23.122.97 as root [2021-02-12 22:35:18,878] - [remote_util] [140097149204224] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 22:35:19,186] - [remote_util] [140097149204224] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 22:35:19,207] - [remote_util] [140097149204224] - INFO - 172.23.122.97 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 22:35:19,268] - [remote_util] [140097149204224] - INFO - Running systemd command on this server [2021-02-12 22:35:19,268] - [remote_util] [140097149204224] - INFO - running command.raw on 172.23.122.97: systemctl stop couchbase-server.service [2021-02-12 22:35:21,104] - [remote_util] [140097149204224] - INFO - command executed successfully with root [2021-02-12 22:35:21,104] - [basetestcase] [140097149204224] - INFO - Couchbase stopped [2021-02-12 22:35:21,105] - [basetestcase] [140097149204224] - INFO - sleep for 5 secs. ... [2021-02-12 22:35:26,105] - [remote_util] [140097149204224] - INFO - running command.raw on 172.23.122.97: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-12 22:35:26,127] - [remote_util] [140097149204224] - INFO - command executed successfully with root [2021-02-12 22:35:26,127] - [remote_util] [140097149204224] - INFO - running command.raw on 172.23.122.97: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-12 22:35:26,183] - [remote_util] [140097149204224] - INFO - command executed successfully with root [2021-02-12 22:35:26,184] - [remote_util] [140097149204224] - INFO - SSH Connecting to 172.23.122.97 with username:root, attempt#1 of 5 [2021-02-12 22:35:26,293] - [remote_util] [140097149204224] - INFO - SSH Connected to 172.23.122.97 as root [2021-02-12 22:35:26,576] - [remote_util] [140097149204224] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 22:35:26,889] - [remote_util] [140097149204224] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 22:35:26,912] - [remote_util] [140097149204224] - INFO - 172.23.122.97 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 22:35:26,912] - [remote_util] [140097149204224] - INFO - Checking for process beam.smp on linux [2021-02-12 22:35:26,951] - [remote_util] [140097149204224] - INFO - Starting couchbase server [2021-02-12 22:35:27,051] - [remote_util] [140097149204224] - INFO - Running systemd command on this server [2021-02-12 22:35:27,052] - [remote_util] [140097149204224] - INFO - running command.raw on 172.23.122.97: systemctl start couchbase-server.service [2021-02-12 22:35:27,077] - [remote_util] [140097149204224] - INFO - command executed successfully with root [2021-02-12 22:35:27,077] - [remote_util] [140097149204224] - INFO - 172.23.122.97:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-12 22:35:32,082] - [remote_util] [140097149204224] - INFO - running command.raw on 172.23.122.97: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-12 22:35:32,114] - [remote_util] [140097149204224] - INFO - command executed successfully with root [2021-02-12 22:35:32,114] - [remote_util] [140097149204224] - INFO - Couchbase server status: [' Process: 101610 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-12 22:35:32,114] - [remote_util] [140097149204224] - INFO - Checking for process beam.smp on linux [2021-02-12 22:35:32,184] - [remote_util] [140097149204224] - INFO - process beam.smp is running on 172.23.122.97: with pid 101756 [2021-02-12 22:35:32,184] - [basetestcase] [140097149204224] - INFO - Couchbase started [2021-02-12 22:35:32,186] - [remote_util] [140097149204224] - INFO - SSH Connecting to 172.23.122.82 with username:root, attempt#1 of 5 [2021-02-12 22:35:32,294] - [remote_util] [140097149204224] - INFO - SSH Connected to 172.23.122.82 as root [2021-02-12 22:35:32,574] - [remote_util] [140097149204224] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 22:35:32,876] - [remote_util] [140097149204224] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 22:35:32,886] - [remote_util] [140097149204224] - INFO - SSH Connecting to 172.23.122.82 with username:root, attempt#1 of 5 [2021-02-12 22:35:32,987] - [remote_util] [140097149204224] - INFO - SSH Connected to 172.23.122.82 as root [2021-02-12 22:35:33,260] - [remote_util] [140097149204224] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 22:35:33,562] - [remote_util] [140097149204224] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 22:35:33,582] - [remote_util] [140097149204224] - INFO - 172.23.122.82 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 22:35:33,639] - [remote_util] [140097149204224] - INFO - Running systemd command on this server [2021-02-12 22:35:33,639] - [remote_util] [140097149204224] - INFO - running command.raw on 172.23.122.82: systemctl stop couchbase-server.service [2021-02-12 22:35:35,484] - [remote_util] [140097149204224] - INFO - command executed successfully with root [2021-02-12 22:35:35,484] - [basetestcase] [140097149204224] - INFO - Couchbase stopped [2021-02-12 22:35:35,484] - [basetestcase] [140097149204224] - INFO - sleep for 5 secs. ... [2021-02-12 22:35:40,490] - [remote_util] [140097149204224] - INFO - running command.raw on 172.23.122.82: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-12 22:35:40,507] - [remote_util] [140097149204224] - INFO - command executed successfully with root [2021-02-12 22:35:40,507] - [remote_util] [140097149204224] - INFO - running command.raw on 172.23.122.82: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-12 22:35:40,561] - [remote_util] [140097149204224] - INFO - command executed successfully with root [2021-02-12 22:35:40,563] - [remote_util] [140097149204224] - INFO - SSH Connecting to 172.23.122.82 with username:root, attempt#1 of 5 [2021-02-12 22:35:40,667] - [remote_util] [140097149204224] - INFO - SSH Connected to 172.23.122.82 as root [2021-02-12 22:35:40,945] - [remote_util] [140097149204224] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 22:35:41,248] - [remote_util] [140097149204224] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 22:35:41,269] - [remote_util] [140097149204224] - INFO - 172.23.122.82 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 22:35:41,269] - [remote_util] [140097149204224] - INFO - Checking for process beam.smp on linux [2021-02-12 22:35:41,309] - [remote_util] [140097149204224] - INFO - Starting couchbase server [2021-02-12 22:35:41,406] - [remote_util] [140097149204224] - INFO - Running systemd command on this server [2021-02-12 22:35:41,406] - [remote_util] [140097149204224] - INFO - running command.raw on 172.23.122.82: systemctl start couchbase-server.service [2021-02-12 22:35:41,430] - [remote_util] [140097149204224] - INFO - command executed successfully with root [2021-02-12 22:35:41,430] - [remote_util] [140097149204224] - INFO - 172.23.122.82:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-12 22:35:46,435] - [remote_util] [140097149204224] - INFO - running command.raw on 172.23.122.82: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-12 22:35:46,461] - [remote_util] [140097149204224] - INFO - command executed successfully with root [2021-02-12 22:35:46,462] - [remote_util] [140097149204224] - INFO - Couchbase server status: [' Process: 79312 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-12 22:35:46,462] - [remote_util] [140097149204224] - INFO - Checking for process beam.smp on linux [2021-02-12 22:35:46,533] - [remote_util] [140097149204224] - INFO - process beam.smp is running on 172.23.122.82: with pid 79458 [2021-02-12 22:35:46,533] - [basetestcase] [140097149204224] - INFO - Couchbase started [2021-02-12 22:35:46,535] - [remote_util] [140097149204224] - INFO - SSH Connecting to 172.23.122.118 with username:root, attempt#1 of 5 [2021-02-12 22:35:46,640] - [remote_util] [140097149204224] - INFO - SSH Connected to 172.23.122.118 as root [2021-02-12 22:35:46,899] - [remote_util] [140097149204224] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 22:35:47,199] - [remote_util] [140097149204224] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 22:35:47,207] - [remote_util] [140097149204224] - INFO - SSH Connecting to 172.23.122.118 with username:root, attempt#1 of 5 [2021-02-12 22:35:47,305] - [remote_util] [140097149204224] - INFO - SSH Connected to 172.23.122.118 as root [2021-02-12 22:35:47,556] - [remote_util] [140097149204224] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 22:35:47,859] - [remote_util] [140097149204224] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 22:35:47,920] - [remote_util] [140097149204224] - INFO - 172.23.122.118 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 22:35:47,976] - [remote_util] [140097149204224] - INFO - Running systemd command on this server [2021-02-12 22:35:47,976] - [remote_util] [140097149204224] - INFO - running command.raw on 172.23.122.118: systemctl stop couchbase-server.service [2021-02-12 22:35:49,659] - [remote_util] [140097149204224] - INFO - command executed successfully with root [2021-02-12 22:35:49,659] - [basetestcase] [140097149204224] - INFO - Couchbase stopped [2021-02-12 22:35:49,660] - [basetestcase] [140097149204224] - INFO - sleep for 5 secs. ... [2021-02-12 22:35:54,665] - [remote_util] [140097149204224] - INFO - running command.raw on 172.23.122.118: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-12 22:35:54,681] - [remote_util] [140097149204224] - INFO - command executed successfully with root [2021-02-12 22:35:54,681] - [remote_util] [140097149204224] - INFO - running command.raw on 172.23.122.118: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-12 22:35:54,736] - [remote_util] [140097149204224] - INFO - command executed successfully with root [2021-02-12 22:35:54,738] - [remote_util] [140097149204224] - INFO - SSH Connecting to 172.23.122.118 with username:root, attempt#1 of 5 [2021-02-12 22:35:54,841] - [remote_util] [140097149204224] - INFO - SSH Connected to 172.23.122.118 as root [2021-02-12 22:35:55,102] - [remote_util] [140097149204224] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 22:35:55,403] - [remote_util] [140097149204224] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 22:35:55,422] - [remote_util] [140097149204224] - INFO - 172.23.122.118 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 22:35:55,422] - [remote_util] [140097149204224] - INFO - Checking for process beam.smp on linux [2021-02-12 22:35:55,460] - [remote_util] [140097149204224] - INFO - Starting couchbase server [2021-02-12 22:35:55,557] - [remote_util] [140097149204224] - INFO - Running systemd command on this server [2021-02-12 22:35:55,557] - [remote_util] [140097149204224] - INFO - running command.raw on 172.23.122.118: systemctl start couchbase-server.service [2021-02-12 22:35:55,579] - [remote_util] [140097149204224] - INFO - command executed successfully with root [2021-02-12 22:35:55,579] - [remote_util] [140097149204224] - INFO - 172.23.122.118:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-12 22:36:00,584] - [remote_util] [140097149204224] - INFO - running command.raw on 172.23.122.118: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-12 22:36:00,611] - [remote_util] [140097149204224] - INFO - command executed successfully with root [2021-02-12 22:36:00,611] - [remote_util] [140097149204224] - INFO - Couchbase server status: [' Process: 105837 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-12 22:36:00,611] - [remote_util] [140097149204224] - INFO - Checking for process beam.smp on linux [2021-02-12 22:36:00,679] - [remote_util] [140097149204224] - INFO - process beam.smp is running on 172.23.122.118: with pid 105980 [2021-02-12 22:36:00,680] - [basetestcase] [140097149204224] - INFO - Couchbase started [2021-02-12 22:36:00,681] - [remote_util] [140097149204224] - INFO - SSH Connecting to 172.23.122.116 with username:root, attempt#1 of 5 [2021-02-12 22:36:00,791] - [remote_util] [140097149204224] - INFO - SSH Connected to 172.23.122.116 as root [2021-02-12 22:36:01,075] - [remote_util] [140097149204224] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 22:36:01,377] - [remote_util] [140097149204224] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 22:36:01,386] - [remote_util] [140097149204224] - INFO - SSH Connecting to 172.23.122.116 with username:root, attempt#1 of 5 [2021-02-12 22:36:01,489] - [remote_util] [140097149204224] - INFO - SSH Connected to 172.23.122.116 as root [2021-02-12 22:36:01,772] - [remote_util] [140097149204224] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 22:36:02,078] - [remote_util] [140097149204224] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 22:36:02,098] - [remote_util] [140097149204224] - INFO - 172.23.122.116 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 22:36:02,154] - [remote_util] [140097149204224] - INFO - Running systemd command on this server [2021-02-12 22:36:02,154] - [remote_util] [140097149204224] - INFO - running command.raw on 172.23.122.116: systemctl stop couchbase-server.service [2021-02-12 22:36:04,103] - [remote_util] [140097149204224] - INFO - command executed successfully with root [2021-02-12 22:36:04,103] - [basetestcase] [140097149204224] - INFO - Couchbase stopped [2021-02-12 22:36:04,103] - [basetestcase] [140097149204224] - INFO - sleep for 5 secs. ... [2021-02-12 22:36:09,104] - [remote_util] [140097149204224] - INFO - running command.raw on 172.23.122.116: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-12 22:36:09,121] - [remote_util] [140097149204224] - INFO - command executed successfully with root [2021-02-12 22:36:09,121] - [remote_util] [140097149204224] - INFO - running command.raw on 172.23.122.116: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-12 22:36:09,176] - [remote_util] [140097149204224] - INFO - command executed successfully with root [2021-02-12 22:36:09,177] - [remote_util] [140097149204224] - INFO - SSH Connecting to 172.23.122.116 with username:root, attempt#1 of 5 [2021-02-12 22:36:09,281] - [remote_util] [140097149204224] - INFO - SSH Connected to 172.23.122.116 as root [2021-02-12 22:36:09,561] - [remote_util] [140097149204224] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 22:36:09,860] - [remote_util] [140097149204224] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 22:36:09,920] - [remote_util] [140097149204224] - INFO - 172.23.122.116 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 22:36:09,920] - [remote_util] [140097149204224] - INFO - Checking for process beam.smp on linux [2021-02-12 22:36:09,958] - [remote_util] [140097149204224] - INFO - Starting couchbase server [2021-02-12 22:36:10,053] - [remote_util] [140097149204224] - INFO - Running systemd command on this server [2021-02-12 22:36:10,054] - [remote_util] [140097149204224] - INFO - running command.raw on 172.23.122.116: systemctl start couchbase-server.service [2021-02-12 22:36:10,075] - [remote_util] [140097149204224] - INFO - command executed successfully with root [2021-02-12 22:36:10,075] - [remote_util] [140097149204224] - INFO - 172.23.122.116:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-12 22:36:15,079] - [remote_util] [140097149204224] - INFO - running command.raw on 172.23.122.116: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-12 22:36:15,104] - [remote_util] [140097149204224] - INFO - command executed successfully with root [2021-02-12 22:36:15,104] - [remote_util] [140097149204224] - INFO - Couchbase server status: [' Process: 67358 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-12 22:36:15,104] - [remote_util] [140097149204224] - INFO - Checking for process beam.smp on linux [2021-02-12 22:36:15,174] - [remote_util] [140097149204224] - INFO - process beam.smp is running on 172.23.122.116: with pid 67504 [2021-02-12 22:36:15,174] - [basetestcase] [140097149204224] - INFO - Couchbase started [2021-02-12 22:36:15,175] - [basetestcase] [140097149204224] - INFO - sleep for 10 secs. ... [2021-02-12 22:36:25,229] - [task] [140096791635712] - INFO - server: ip:172.23.123.61 port:8091 ssh_username:root, nodes/self [2021-02-12 22:36:25,232] - [task] [140096791635712] - INFO - {'uptime': '110', 'memoryTotal': 4201672704, 'memoryFree': 3517489152, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.123.61:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4454-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.123.61', 'ip': '172.23.123.61', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} [2021-02-12 22:36:25,234] - [rest_client] [140096791635712] - ERROR - GET http://172.23.123.61:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-02-12 22:36:25,234] - [rest_client] [140096791635712] - INFO - pools/default params : memoryQuota=2147 [2021-02-12 22:36:25,240] - [rest_client] [140096791635712] - INFO - --> init_node_services(Administrator,password,172.23.123.61,8091,['kv']) [2021-02-12 22:36:25,240] - [rest_client] [140096791635712] - INFO - /node/controller/setupServices params on 172.23.123.61: 8091:hostname=172.23.123.61&user=Administrator&password=password&services=kv [2021-02-12 22:36:25,268] - [rest_client] [140096791635712] - INFO - --> in init_cluster...Administrator,password,8091 [2021-02-12 22:36:25,268] - [rest_client] [140096791635712] - INFO - settings/web params on 172.23.123.61:8091:port=8091&username=Administrator&password=password [2021-02-12 22:36:25,323] - [rest_client] [140096791635712] - INFO - --> status:True [2021-02-12 22:36:25,324] - [remote_util] [140096791635712] - INFO - SSH Connecting to 172.23.123.61 with username:root, attempt#1 of 5 [2021-02-12 22:36:25,435] - [remote_util] [140096791635712] - INFO - SSH Connected to 172.23.123.61 as root [2021-02-12 22:36:25,703] - [remote_util] [140096791635712] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 22:36:26,003] - [remote_util] [140096791635712] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 22:36:26,012] - [remote_util] [140096791635712] - INFO - running command.raw on 172.23.123.61: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-02-12 22:36:26,075] - [remote_util] [140096791635712] - INFO - command executed successfully with root [2021-02-12 22:36:26,075] - [remote_util] [140096791635712] - INFO - ['ok'] [2021-02-12 22:36:26,078] - [rest_client] [140096791635712] - INFO - /diag/eval status on 172.23.123.61:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-12 22:36:26,080] - [rest_client] [140096791635712] - INFO - /diag/eval status on 172.23.123.61:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-12 22:36:26,083] - [rest_client] [140096791635712] - INFO - settings/indexes params : storageMode=plasma [2021-02-12 22:36:26,105] - [task] [140096791635712] - INFO - server: ip:172.23.123.62 port:8091 ssh_username:root, nodes/self [2021-02-12 22:36:26,108] - [task] [140096791635712] - INFO - {'uptime': '100', 'memoryTotal': 4201676800, 'memoryFree': 3487694848, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.123.62:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4454-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.123.62', 'ip': '172.23.123.62', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} [2021-02-12 22:36:26,109] - [rest_client] [140096791635712] - ERROR - GET http://172.23.123.62:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-02-12 22:36:26,110] - [rest_client] [140096791635712] - INFO - pools/default params : memoryQuota=2147 [2021-02-12 22:36:26,116] - [rest_client] [140096791635712] - INFO - --> in init_cluster...Administrator,password,8091 [2021-02-12 22:36:26,116] - [rest_client] [140096791635712] - INFO - settings/web params on 172.23.123.62:8091:port=8091&username=Administrator&password=password [2021-02-12 22:36:26,173] - [rest_client] [140096791635712] - INFO - --> status:True [2021-02-12 22:36:26,174] - [remote_util] [140096791635712] - INFO - SSH Connecting to 172.23.123.62 with username:root, attempt#1 of 5 [2021-02-12 22:36:26,279] - [remote_util] [140096791635712] - INFO - SSH Connected to 172.23.123.62 as root [2021-02-12 22:36:26,560] - [remote_util] [140096791635712] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 22:36:26,857] - [remote_util] [140096791635712] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 22:36:26,867] - [remote_util] [140096791635712] - INFO - running command.raw on 172.23.123.62: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-02-12 22:36:26,923] - [remote_util] [140096791635712] - INFO - command executed successfully with root [2021-02-12 22:36:26,923] - [remote_util] [140096791635712] - INFO - ['ok'] [2021-02-12 22:36:26,926] - [rest_client] [140096791635712] - INFO - /diag/eval status on 172.23.123.62:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-12 22:36:26,928] - [rest_client] [140096791635712] - INFO - /diag/eval status on 172.23.123.62:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-12 22:36:26,932] - [rest_client] [140096791635712] - INFO - settings/indexes params : storageMode=plasma [2021-02-12 22:36:26,952] - [task] [140096791635712] - INFO - server: ip:172.23.122.90 port:8091 ssh_username:root, nodes/self [2021-02-12 22:36:26,954] - [task] [140096791635712] - INFO - {'uptime': '85', 'memoryTotal': 4201684992, 'memoryFree': 3571372032, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.122.90:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4454-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.122.90', 'ip': '172.23.122.90', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} [2021-02-12 22:36:26,956] - [rest_client] [140096791635712] - ERROR - GET http://172.23.122.90:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-02-12 22:36:26,957] - [rest_client] [140096791635712] - INFO - pools/default params : memoryQuota=2147 [2021-02-12 22:36:26,963] - [rest_client] [140096791635712] - INFO - --> in init_cluster...Administrator,password,8091 [2021-02-12 22:36:26,963] - [rest_client] [140096791635712] - INFO - settings/web params on 172.23.122.90:8091:port=8091&username=Administrator&password=password [2021-02-12 22:36:27,020] - [rest_client] [140096791635712] - INFO - --> status:True [2021-02-12 22:36:27,021] - [remote_util] [140096791635712] - INFO - SSH Connecting to 172.23.122.90 with username:root, attempt#1 of 5 [2021-02-12 22:36:27,135] - [remote_util] [140096791635712] - INFO - SSH Connected to 172.23.122.90 as root [2021-02-12 22:36:27,445] - [remote_util] [140096791635712] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 22:36:27,749] - [remote_util] [140096791635712] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 22:36:27,757] - [remote_util] [140096791635712] - INFO - running command.raw on 172.23.122.90: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-02-12 22:36:27,819] - [remote_util] [140096791635712] - INFO - command executed successfully with root [2021-02-12 22:36:27,819] - [remote_util] [140096791635712] - INFO - ['ok'] [2021-02-12 22:36:27,822] - [rest_client] [140096791635712] - INFO - /diag/eval status on 172.23.122.90:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-12 22:36:27,824] - [rest_client] [140096791635712] - INFO - /diag/eval status on 172.23.122.90:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-12 22:36:27,828] - [rest_client] [140096791635712] - INFO - settings/indexes params : storageMode=plasma [2021-02-12 22:36:27,847] - [task] [140096791635712] - INFO - server: ip:172.23.122.117 port:8091 ssh_username:root, nodes/self [2021-02-12 22:36:27,850] - [task] [140096791635712] - INFO - {'uptime': '70', 'memoryTotal': 4201684992, 'memoryFree': 3530575872, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.122.117:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4454-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.122.117', 'ip': '172.23.122.117', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} [2021-02-12 22:36:27,852] - [rest_client] [140096791635712] - ERROR - GET http://172.23.122.117:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-02-12 22:36:27,852] - [rest_client] [140096791635712] - INFO - pools/default params : memoryQuota=2147 [2021-02-12 22:36:27,858] - [rest_client] [140096791635712] - INFO - --> in init_cluster...Administrator,password,8091 [2021-02-12 22:36:27,858] - [rest_client] [140096791635712] - INFO - settings/web params on 172.23.122.117:8091:port=8091&username=Administrator&password=password [2021-02-12 22:36:27,912] - [rest_client] [140096791635712] - INFO - --> status:True [2021-02-12 22:36:27,914] - [remote_util] [140096791635712] - INFO - SSH Connecting to 172.23.122.117 with username:root, attempt#1 of 5 [2021-02-12 22:36:28,018] - [remote_util] [140096791635712] - INFO - SSH Connected to 172.23.122.117 as root [2021-02-12 22:36:28,280] - [remote_util] [140096791635712] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 22:36:28,578] - [remote_util] [140096791635712] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 22:36:28,586] - [remote_util] [140096791635712] - INFO - running command.raw on 172.23.122.117: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-02-12 22:36:28,641] - [remote_util] [140096791635712] - INFO - command executed successfully with root [2021-02-12 22:36:28,641] - [remote_util] [140096791635712] - INFO - ['ok'] [2021-02-12 22:36:28,643] - [rest_client] [140096791635712] - INFO - /diag/eval status on 172.23.122.117:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-12 22:36:28,644] - [rest_client] [140096791635712] - INFO - /diag/eval status on 172.23.122.117:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-12 22:36:28,648] - [rest_client] [140096791635712] - INFO - settings/indexes params : storageMode=plasma [2021-02-12 22:36:28,670] - [task] [140096791635712] - INFO - server: ip:172.23.122.97 port:8091 ssh_username:root, nodes/self [2021-02-12 22:36:28,674] - [task] [140096791635712] - INFO - {'uptime': '55', 'memoryTotal': 4201684992, 'memoryFree': 3556184064, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.122.97:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4454-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.122.97', 'ip': '172.23.122.97', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} [2021-02-12 22:36:28,676] - [rest_client] [140096791635712] - ERROR - GET http://172.23.122.97:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-02-12 22:36:28,676] - [rest_client] [140096791635712] - INFO - pools/default params : memoryQuota=2147 [2021-02-12 22:36:28,682] - [rest_client] [140096791635712] - INFO - --> in init_cluster...Administrator,password,8091 [2021-02-12 22:36:28,682] - [rest_client] [140096791635712] - INFO - settings/web params on 172.23.122.97:8091:port=8091&username=Administrator&password=password [2021-02-12 22:36:28,738] - [rest_client] [140096791635712] - INFO - --> status:True [2021-02-12 22:36:28,740] - [remote_util] [140096791635712] - INFO - SSH Connecting to 172.23.122.97 with username:root, attempt#1 of 5 [2021-02-12 22:36:28,853] - [remote_util] [140096791635712] - INFO - SSH Connected to 172.23.122.97 as root [2021-02-12 22:36:29,141] - [remote_util] [140096791635712] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 22:36:29,451] - [remote_util] [140096791635712] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 22:36:29,460] - [remote_util] [140096791635712] - INFO - running command.raw on 172.23.122.97: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-02-12 22:36:29,523] - [remote_util] [140096791635712] - INFO - command executed successfully with root [2021-02-12 22:36:29,523] - [remote_util] [140096791635712] - INFO - ['ok'] [2021-02-12 22:36:29,526] - [rest_client] [140096791635712] - INFO - /diag/eval status on 172.23.122.97:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-12 22:36:29,528] - [rest_client] [140096791635712] - INFO - /diag/eval status on 172.23.122.97:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-12 22:36:29,532] - [rest_client] [140096791635712] - INFO - settings/indexes params : storageMode=plasma [2021-02-12 22:36:29,553] - [task] [140096791635712] - INFO - server: ip:172.23.122.82 port:8091 ssh_username:root, nodes/self [2021-02-12 22:36:29,556] - [task] [140096791635712] - INFO - {'uptime': '44', 'memoryTotal': 4201684992, 'memoryFree': 3585740800, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.122.82:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4454-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.122.82', 'ip': '172.23.122.82', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} [2021-02-12 22:36:29,558] - [rest_client] [140096791635712] - ERROR - GET http://172.23.122.82:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-02-12 22:36:29,558] - [rest_client] [140096791635712] - INFO - pools/default params : memoryQuota=2147 [2021-02-12 22:36:29,564] - [rest_client] [140096791635712] - INFO - --> in init_cluster...Administrator,password,8091 [2021-02-12 22:36:29,564] - [rest_client] [140096791635712] - INFO - settings/web params on 172.23.122.82:8091:port=8091&username=Administrator&password=password [2021-02-12 22:36:29,620] - [rest_client] [140096791635712] - INFO - --> status:True [2021-02-12 22:36:29,622] - [remote_util] [140096791635712] - INFO - SSH Connecting to 172.23.122.82 with username:root, attempt#1 of 5 [2021-02-12 22:36:29,731] - [remote_util] [140096791635712] - INFO - SSH Connected to 172.23.122.82 as root [2021-02-12 22:36:30,014] - [remote_util] [140096791635712] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 22:36:30,318] - [remote_util] [140096791635712] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 22:36:30,327] - [remote_util] [140096791635712] - INFO - running command.raw on 172.23.122.82: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-02-12 22:36:30,387] - [remote_util] [140096791635712] - INFO - command executed successfully with root [2021-02-12 22:36:30,387] - [remote_util] [140096791635712] - INFO - ['ok'] [2021-02-12 22:36:30,389] - [rest_client] [140096791635712] - INFO - /diag/eval status on 172.23.122.82:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-12 22:36:30,391] - [rest_client] [140096791635712] - INFO - /diag/eval status on 172.23.122.82:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-12 22:36:30,396] - [rest_client] [140096791635712] - INFO - settings/indexes params : storageMode=plasma [2021-02-12 22:36:30,413] - [task] [140096791635712] - INFO - server: ip:172.23.122.118 port:8091 ssh_username:root, nodes/self [2021-02-12 22:36:30,415] - [task] [140096791635712] - INFO - {'uptime': '29', 'memoryTotal': 4201689088, 'memoryFree': 3548561408, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.122.118:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4454-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.122.118', 'ip': '172.23.122.118', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} [2021-02-12 22:36:30,417] - [rest_client] [140096791635712] - ERROR - GET http://172.23.122.118:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-02-12 22:36:30,418] - [rest_client] [140096791635712] - INFO - pools/default params : memoryQuota=2147 [2021-02-12 22:36:30,423] - [rest_client] [140096791635712] - INFO - --> in init_cluster...Administrator,password,8091 [2021-02-12 22:36:30,423] - [rest_client] [140096791635712] - INFO - settings/web params on 172.23.122.118:8091:port=8091&username=Administrator&password=password [2021-02-12 22:36:30,481] - [rest_client] [140096791635712] - INFO - --> status:True [2021-02-12 22:36:30,482] - [remote_util] [140096791635712] - INFO - SSH Connecting to 172.23.122.118 with username:root, attempt#1 of 5 [2021-02-12 22:36:30,586] - [remote_util] [140096791635712] - INFO - SSH Connected to 172.23.122.118 as root [2021-02-12 22:36:30,846] - [remote_util] [140096791635712] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 22:36:31,147] - [remote_util] [140096791635712] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 22:36:31,155] - [remote_util] [140096791635712] - INFO - running command.raw on 172.23.122.118: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-02-12 22:36:31,213] - [remote_util] [140096791635712] - INFO - command executed successfully with root [2021-02-12 22:36:31,213] - [remote_util] [140096791635712] - INFO - ['ok'] [2021-02-12 22:36:31,214] - [rest_client] [140096791635712] - INFO - /diag/eval status on 172.23.122.118:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-12 22:36:31,216] - [rest_client] [140096791635712] - INFO - /diag/eval status on 172.23.122.118:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-12 22:36:31,220] - [rest_client] [140096791635712] - INFO - settings/indexes params : storageMode=plasma [2021-02-12 22:36:31,242] - [task] [140096791635712] - INFO - server: ip:172.23.122.116 port:8091 ssh_username:root, nodes/self [2021-02-12 22:36:31,245] - [task] [140096791635712] - INFO - {'uptime': '14', 'memoryTotal': 4201689088, 'memoryFree': 3535822848, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.122.116:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4454-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@cb.local', 'ip': '172.23.122.116', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} [2021-02-12 22:36:31,246] - [rest_client] [140096791635712] - ERROR - GET http://172.23.122.116:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-02-12 22:36:31,247] - [rest_client] [140096791635712] - INFO - pools/default params : memoryQuota=2147 [2021-02-12 22:36:31,253] - [rest_client] [140096791635712] - INFO - --> in init_cluster...Administrator,password,8091 [2021-02-12 22:36:31,253] - [rest_client] [140096791635712] - INFO - settings/web params on 172.23.122.116:8091:port=8091&username=Administrator&password=password [2021-02-12 22:36:31,312] - [rest_client] [140096791635712] - INFO - --> status:True [2021-02-12 22:36:31,313] - [remote_util] [140096791635712] - INFO - SSH Connecting to 172.23.122.116 with username:root, attempt#1 of 5 [2021-02-12 22:36:31,418] - [remote_util] [140096791635712] - INFO - SSH Connected to 172.23.122.116 as root [2021-02-12 22:36:31,704] - [remote_util] [140096791635712] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 22:36:31,998] - [remote_util] [140096791635712] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 22:36:32,006] - [remote_util] [140096791635712] - INFO - running command.raw on 172.23.122.116: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-02-12 22:36:32,063] - [remote_util] [140096791635712] - INFO - command executed successfully with root [2021-02-12 22:36:32,063] - [remote_util] [140096791635712] - INFO - ['ok'] [2021-02-12 22:36:32,066] - [rest_client] [140096791635712] - INFO - /diag/eval status on 172.23.122.116:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-12 22:36:32,068] - [rest_client] [140096791635712] - INFO - /diag/eval status on 172.23.122.116:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-12 22:36:32,073] - [rest_client] [140096791635712] - INFO - settings/indexes params : storageMode=plasma [2021-02-12 22:36:32,095] - [basetestcase] [140097149204224] - INFO - **** add built-in 'cbadminbucket' user to node 172.23.123.61 **** [2021-02-12 22:36:32,104] - [rest_client] [140097149204224] - ERROR - DELETE http://172.23.123.61:8091/settings/rbac/users/local/cbadminbucket body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:password [2021-02-12 22:36:32,104] - [internal_user] [140097149204224] - INFO - Exception while deleting user. Exception is -b'"User was not found."' [2021-02-12 22:36:32,167] - [basetestcase] [140097149204224] - INFO - sleep for 5 secs. ... [2021-02-12 22:36:37,172] - [basetestcase] [140097149204224] - INFO - **** add 'admin' role to 'cbadminbucket' user **** [2021-02-12 22:36:37,183] - [basetestcase] [140097149204224] - INFO - done initializing cluster [2021-02-12 22:36:38,094] - [task] [140096791635712] - INFO - adding node 172.23.123.62:8091 to cluster [2021-02-12 22:36:38,094] - [rest_client] [140096791635712] - INFO - adding remote node @172.23.123.62:8091 to this cluster @172.23.123.61:8091 [2021-02-12 22:36:40,286] - [task] [140096791635712] - INFO - adding node 172.23.122.90:8091 to cluster [2021-02-12 22:36:40,287] - [rest_client] [140096791635712] - INFO - adding remote node @172.23.122.90:8091 to this cluster @172.23.123.61:8091 [2021-02-12 22:36:42,522] - [task] [140096791635712] - INFO - adding node 172.23.122.117:8091 to cluster [2021-02-12 22:36:42,522] - [rest_client] [140096791635712] - INFO - adding remote node @172.23.122.117:8091 to this cluster @172.23.123.61:8091 [2021-02-12 22:36:45,138] - [task] [140096791635712] - INFO - adding node 172.23.122.97:8091 to cluster [2021-02-12 22:36:45,138] - [rest_client] [140096791635712] - INFO - adding remote node @172.23.122.97:8091 to this cluster @172.23.123.61:8091 [2021-02-12 22:36:47,514] - [rest_client] [140096791635712] - INFO - rebalance params : {'knownNodes': 'ns_1@172.23.122.117,ns_1@172.23.122.90,ns_1@172.23.122.97,ns_1@172.23.123.61,ns_1@172.23.123.62', 'ejectedNodes': '', 'user': 'Administrator', 'password': 'password'} [2021-02-12 22:36:47,544] - [rest_client] [140096791635712] - INFO - rebalance operation started [2021-02-12 22:36:47,546] - [rest_client] [140096791635712] - INFO - rebalance percentage : 0.00 % [2021-02-12 22:36:47,546] - [task] [140096791635712] - INFO - Rebalance - status: running, progress: 0.00% [2021-02-12 22:36:57,566] - [task] [140096791635712] - INFO - Rebalance - status: none, progress: 100.00% [2021-02-12 22:36:57,573] - [task] [140096791635712] - INFO - rebalancing was completed with progress: 100% in 10.029252052307129 sec [2021-02-12 22:36:57,575] - [remote_util] [140097149204224] - INFO - SSH Connecting to 172.23.123.61 with username:root, attempt#1 of 5 [2021-02-12 22:36:57,679] - [remote_util] [140097149204224] - INFO - SSH Connected to 172.23.123.61 as root [2021-02-12 22:36:57,953] - [remote_util] [140097149204224] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 22:36:58,254] - [remote_util] [140097149204224] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 22:36:58,263] - [remote_util] [140097149204224] - INFO - running command.raw on 172.23.123.61: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-02-12 22:36:58,318] - [remote_util] [140097149204224] - INFO - command executed successfully with root [2021-02-12 22:36:58,319] - [remote_util] [140097149204224] - INFO - ['ok'] [2021-02-12 22:36:58,319] - [basetestcase] [140097149204224] - INFO - Enabled diag/eval for non-local hosts from 172.23.123.61 [2021-02-12 22:36:58,319] - [basetestcase] [140097149204224] - INFO - ============== basetestcase setup was finished for test #8 test_eventing_rebalance_with_multiple_kv_nodes ============== [2021-02-12 22:36:58,327] - [remote_util] [140097149204224] - INFO - SSH Connecting to 172.23.123.61 with username:root, attempt#1 of 5 [2021-02-12 22:36:58,429] - [remote_util] [140097149204224] - INFO - SSH Connected to 172.23.123.61 as root [2021-02-12 22:36:58,699] - [remote_util] [140097149204224] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 22:36:59,001] - [remote_util] [140097149204224] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 22:36:59,002] - [remote_util] [140097149204224] - INFO - SSH Connecting to 172.23.123.62 with username:root, attempt#1 of 5 [2021-02-12 22:36:59,107] - [remote_util] [140097149204224] - INFO - SSH Connected to 172.23.123.62 as root [2021-02-12 22:36:59,392] - [remote_util] [140097149204224] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 22:36:59,696] - [remote_util] [140097149204224] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 22:36:59,698] - [remote_util] [140097149204224] - INFO - SSH Connecting to 172.23.122.90 with username:root, attempt#1 of 5 [2021-02-12 22:36:59,801] - [remote_util] [140097149204224] - INFO - SSH Connected to 172.23.122.90 as root [2021-02-12 22:37:00,100] - [remote_util] [140097149204224] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 22:37:00,407] - [remote_util] [140097149204224] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 22:37:00,409] - [remote_util] [140097149204224] - INFO - SSH Connecting to 172.23.122.117 with username:root, attempt#1 of 5 [2021-02-12 22:37:00,515] - [remote_util] [140097149204224] - INFO - SSH Connected to 172.23.122.117 as root [2021-02-12 22:37:00,783] - [remote_util] [140097149204224] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 22:37:01,088] - [remote_util] [140097149204224] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 22:37:01,091] - [remote_util] [140097149204224] - INFO - SSH Connecting to 172.23.122.97 with username:root, attempt#1 of 5 [2021-02-12 22:37:01,197] - [remote_util] [140097149204224] - INFO - SSH Connected to 172.23.122.97 as root [2021-02-12 22:37:01,479] - [remote_util] [140097149204224] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 22:37:01,789] - [remote_util] [140097149204224] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 22:37:01,791] - [remote_util] [140097149204224] - INFO - SSH Connecting to 172.23.122.82 with username:root, attempt#1 of 5 [2021-02-12 22:37:01,898] - [remote_util] [140097149204224] - INFO - SSH Connected to 172.23.122.82 as root [2021-02-12 22:37:02,190] - [remote_util] [140097149204224] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 22:37:02,501] - [remote_util] [140097149204224] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 22:37:02,503] - [remote_util] [140097149204224] - INFO - SSH Connecting to 172.23.122.118 with username:root, attempt#1 of 5 [2021-02-12 22:37:02,610] - [remote_util] [140097149204224] - INFO - SSH Connected to 172.23.122.118 as root [2021-02-12 22:37:02,867] - [remote_util] [140097149204224] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 22:37:03,167] - [remote_util] [140097149204224] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 22:37:03,169] - [remote_util] [140097149204224] - INFO - SSH Connecting to 172.23.122.116 with username:root, attempt#1 of 5 [2021-02-12 22:37:03,271] - [remote_util] [140097149204224] - INFO - SSH Connected to 172.23.122.116 as root [2021-02-12 22:37:03,559] - [remote_util] [140097149204224] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 22:37:03,862] - [remote_util] [140097149204224] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 22:38:08,627] - [basetestcase] [140097149204224] - INFO - ------- Cluster statistics ------- [2021-02-12 22:38:08,627] - [basetestcase] [140097149204224] - INFO - 172.23.122.117:8091 => {'services': ['eventing'], 'cpu_utilization': 0.6778809942254582, 'mem_free': 3493158912, 'mem_total': 4201684992, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-02-12 22:38:08,627] - [basetestcase] [140097149204224] - INFO - 172.23.122.90:8091 => {'services': ['kv'], 'cpu_utilization': 0.777722027094832, 'mem_free': 3542016000, 'mem_total': 4201684992, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-02-12 22:38:08,627] - [basetestcase] [140097149204224] - INFO - 172.23.122.97:8091 => {'services': ['index', 'n1ql'], 'cpu_utilization': 1.128385155466399, 'mem_free': 3460538368, 'mem_total': 4201684992, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-02-12 22:38:08,627] - [basetestcase] [140097149204224] - INFO - 172.23.123.61:8091 => {'services': ['kv'], 'cpu_utilization': 0.5537377296753083, 'mem_free': 3485298688, 'mem_total': 4201672704, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-02-12 22:38:08,627] - [basetestcase] [140097149204224] - INFO - 172.23.123.62:8091 => {'services': ['kv'], 'cpu_utilization': 0.7526342197691922, 'mem_free': 3463757824, 'mem_total': 4201676800, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-02-12 22:38:08,627] - [basetestcase] [140097149204224] - INFO - --- End of cluster statistics --- [2021-02-12 22:38:08,629] - [remote_util] [140097149204224] - INFO - SSH Connecting to 172.23.123.61 with username:root, attempt#1 of 5 [2021-02-12 22:38:08,736] - [remote_util] [140097149204224] - INFO - SSH Connected to 172.23.123.61 as root [2021-02-12 22:38:09,009] - [remote_util] [140097149204224] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 22:38:09,306] - [remote_util] [140097149204224] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 22:38:12,132] - [basetestcase] [140097149204224] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.97 port:8091 ssh_username:root] [2021-02-12 22:38:12,139] - [basetestcase] [140097149204224] - INFO - list of eventing nodes in cluster: [ip:172.23.122.117 port:8091 ssh_username:root] [2021-02-12 22:38:12,144] - [rest_client] [140097149204224] - INFO - settings/indexes params : storageMode=plasma [2021-02-12 22:38:12,149] - [eventing_base] [140097149204224] - INFO - Setting the min possible memory quota so that adding mode nodes to the cluster wouldn't be a problem. [2021-02-12 22:38:12,149] - [rest_client] [140097149204224] - INFO - pools/default params : memoryQuota=330 [2021-02-12 22:38:12,156] - [rest_client] [140097149204224] - INFO - pools/default params : indexMemoryQuota=256 [2021-02-12 22:38:12,162] - [rest_client] [140097149204224] - INFO - pools/default params : eventingMemoryQuota=512 [2021-02-12 22:38:12,180] - [basetestcase] [140097149204224] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.97 port:8091 ssh_username:root] [2021-02-12 22:38:12,183] - [rest_client] [140097149204224] - INFO - pools/default params : memoryQuota=700 [2021-02-12 22:38:12,188] - [eventing_rebalance_collection] [140097149204224] - INFO - 200 [2021-02-12 22:38:12,711] - [rest_client] [140096791635712] - INFO - http://172.23.123.61:8091/pools/default/buckets with param: name=src_bucket&ramQuotaMB=200&authType=none&replicaNumber=0&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore [2021-02-12 22:38:12,739] - [rest_client] [140096791635712] - INFO - 0.03 seconds to create bucket src_bucket [2021-02-12 22:38:12,740] - [bucket_helper] [140096791635712] - INFO - waiting for memcached bucket : src_bucket in 172.23.123.61 to accept set ops [2021-02-12 22:38:13,410] - [data_helper] [140096791635712] - INFO - creating direct client 172.23.122.90:11210 src_bucket [2021-02-12 22:38:13,495] - [data_helper] [140096791635712] - INFO - creating direct client 172.23.123.61:11210 src_bucket [2021-02-12 22:38:13,574] - [data_helper] [140096791635712] - INFO - creating direct client 172.23.123.62:11210 src_bucket [2021-02-12 22:38:13,886] - [data_helper] [140096791635712] - INFO - creating direct client 172.23.122.90:11210 src_bucket [2021-02-12 22:38:13,961] - [data_helper] [140096791635712] - INFO - creating direct client 172.23.123.61:11210 src_bucket [2021-02-12 22:38:14,036] - [data_helper] [140096791635712] - INFO - creating direct client 172.23.123.62:11210 src_bucket [2021-02-12 22:38:14,104] - [task] [140096791635712] - WARNING - vbucket map not ready after try 0 [2021-02-12 22:38:14,104] - [bucket_helper] [140096791635712] - INFO - waiting for memcached bucket : src_bucket in 172.23.123.61 to accept set ops [2021-02-12 22:38:14,213] - [data_helper] [140096791635712] - INFO - creating direct client 172.23.122.90:11210 src_bucket [2021-02-12 22:38:14,289] - [data_helper] [140096791635712] - INFO - creating direct client 172.23.123.61:11210 src_bucket [2021-02-12 22:38:14,556] - [data_helper] [140096791635712] - INFO - creating direct client 172.23.123.62:11210 src_bucket [2021-02-12 22:38:14,675] - [data_helper] [140096791635712] - INFO - creating direct client 172.23.122.90:11210 src_bucket [2021-02-12 22:38:14,751] - [data_helper] [140096791635712] - INFO - creating direct client 172.23.123.61:11210 src_bucket [2021-02-12 22:38:14,830] - [data_helper] [140096791635712] - INFO - creating direct client 172.23.123.62:11210 src_bucket [2021-02-12 22:38:14,901] - [task] [140096791635712] - WARNING - vbucket map not ready after try 1 [2021-02-12 22:38:14,901] - [bucket_helper] [140096791635712] - INFO - waiting for memcached bucket : src_bucket in 172.23.123.61 to accept set ops [2021-02-12 22:38:15,189] - [data_helper] [140096791635712] - INFO - creating direct client 172.23.122.90:11210 src_bucket [2021-02-12 22:38:15,264] - [data_helper] [140096791635712] - INFO - creating direct client 172.23.123.61:11210 src_bucket [2021-02-12 22:38:15,342] - [data_helper] [140096791635712] - INFO - creating direct client 172.23.123.62:11210 src_bucket [2021-02-12 22:38:15,463] - [data_helper] [140096791635712] - INFO - creating direct client 172.23.122.90:11210 src_bucket [2021-02-12 22:38:15,535] - [data_helper] [140096791635712] - INFO - creating direct client 172.23.123.61:11210 src_bucket [2021-02-12 22:38:15,617] - [data_helper] [140096791635712] - INFO - creating direct client 172.23.123.62:11210 src_bucket [2021-02-12 22:38:15,863] - [task] [140096791635712] - WARNING - vbucket map not ready after try 2 [2021-02-12 22:38:15,863] - [bucket_helper] [140096791635712] - INFO - waiting for memcached bucket : src_bucket in 172.23.123.61 to accept set ops [2021-02-12 22:38:15,974] - [data_helper] [140096791635712] - INFO - creating direct client 172.23.122.90:11210 src_bucket [2021-02-12 22:38:16,057] - [data_helper] [140096791635712] - INFO - creating direct client 172.23.123.61:11210 src_bucket [2021-02-12 22:38:16,138] - [data_helper] [140096791635712] - INFO - creating direct client 172.23.123.62:11210 src_bucket [2021-02-12 22:38:16,273] - [data_helper] [140096791635712] - INFO - creating direct client 172.23.122.90:11210 src_bucket [2021-02-12 22:38:16,350] - [data_helper] [140096791635712] - INFO - creating direct client 172.23.123.61:11210 src_bucket [2021-02-12 22:38:16,606] - [data_helper] [140096791635712] - INFO - creating direct client 172.23.123.62:11210 src_bucket [2021-02-12 22:38:16,676] - [task] [140096791635712] - WARNING - vbucket map not ready after try 3 [2021-02-12 22:38:16,676] - [bucket_helper] [140096791635712] - INFO - waiting for memcached bucket : src_bucket in 172.23.123.61 to accept set ops [2021-02-12 22:38:16,794] - [data_helper] [140096791635712] - INFO - creating direct client 172.23.122.90:11210 src_bucket [2021-02-12 22:38:16,869] - [data_helper] [140096791635712] - INFO - creating direct client 172.23.123.61:11210 src_bucket [2021-02-12 22:38:16,947] - [data_helper] [140096791635712] - INFO - creating direct client 172.23.123.62:11210 src_bucket [2021-02-12 22:38:17,239] - [data_helper] [140096791635712] - INFO - creating direct client 172.23.122.90:11210 src_bucket [2021-02-12 22:38:17,316] - [data_helper] [140096791635712] - INFO - creating direct client 172.23.123.61:11210 src_bucket [2021-02-12 22:38:17,392] - [data_helper] [140096791635712] - INFO - creating direct client 172.23.123.62:11210 src_bucket [2021-02-12 22:38:17,466] - [task] [140096791635712] - WARNING - vbucket map not ready after try 4 [2021-02-12 22:38:17,466] - [bucket_helper] [140096791635712] - INFO - waiting for memcached bucket : src_bucket in 172.23.123.61 to accept set ops [2021-02-12 22:38:17,581] - [data_helper] [140096791635712] - INFO - creating direct client 172.23.122.90:11210 src_bucket [2021-02-12 22:38:17,658] - [data_helper] [140096791635712] - INFO - creating direct client 172.23.123.61:11210 src_bucket [2021-02-12 22:38:17,918] - [data_helper] [140096791635712] - INFO - creating direct client 172.23.123.62:11210 src_bucket [2021-02-12 22:38:18,034] - [data_helper] [140096791635712] - INFO - creating direct client 172.23.122.90:11210 src_bucket [2021-02-12 22:38:18,108] - [data_helper] [140096791635712] - INFO - creating direct client 172.23.123.61:11210 src_bucket [2021-02-12 22:38:18,186] - [data_helper] [140096791635712] - INFO - creating direct client 172.23.123.62:11210 src_bucket [2021-02-12 22:38:18,253] - [task] [140096791635712] - WARNING - vbucket map not ready after try 5 [2021-02-12 22:38:19,279] - [rest_client] [140096791635712] - INFO - http://172.23.123.61:8091/pools/default/buckets with param: name=dst_bucket&ramQuotaMB=200&authType=none&replicaNumber=0&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore [2021-02-12 22:38:19,308] - [rest_client] [140096791635712] - INFO - 0.03 seconds to create bucket dst_bucket [2021-02-12 22:38:19,308] - [bucket_helper] [140096791635712] - INFO - waiting for memcached bucket : dst_bucket in 172.23.123.61 to accept set ops [2021-02-12 22:38:20,160] - [data_helper] [140096791635712] - INFO - creating direct client 172.23.122.90:11210 dst_bucket [2021-02-12 22:38:20,269] - [data_helper] [140096791635712] - INFO - creating direct client 172.23.123.61:11210 dst_bucket [2021-02-12 22:38:20,353] - [data_helper] [140096791635712] - INFO - creating direct client 172.23.123.62:11210 dst_bucket [2021-02-12 22:38:20,486] - [data_helper] [140096791635712] - INFO - creating direct client 172.23.122.90:11210 dst_bucket [2021-02-12 22:38:20,568] - [data_helper] [140096791635712] - INFO - creating direct client 172.23.123.61:11210 dst_bucket [2021-02-12 22:38:20,666] - [data_helper] [140096791635712] - INFO - creating direct client 172.23.123.62:11210 dst_bucket [2021-02-12 22:38:20,947] - [task] [140096791635712] - WARNING - vbucket map not ready after try 0 [2021-02-12 22:38:20,948] - [bucket_helper] [140096791635712] - INFO - waiting for memcached bucket : dst_bucket in 172.23.123.61 to accept set ops [2021-02-12 22:38:21,071] - [data_helper] [140096791635712] - INFO - creating direct client 172.23.122.90:11210 dst_bucket [2021-02-12 22:38:21,155] - [data_helper] [140096791635712] - INFO - creating direct client 172.23.123.61:11210 dst_bucket [2021-02-12 22:38:21,234] - [data_helper] [140096791635712] - INFO - creating direct client 172.23.123.62:11210 dst_bucket [2021-02-12 22:38:21,350] - [data_helper] [140096791635712] - INFO - creating direct client 172.23.122.90:11210 dst_bucket [2021-02-12 22:38:21,436] - [data_helper] [140096791635712] - INFO - creating direct client 172.23.123.61:11210 dst_bucket [2021-02-12 22:38:21,695] - [data_helper] [140096791635712] - INFO - creating direct client 172.23.123.62:11210 dst_bucket [2021-02-12 22:38:21,763] - [task] [140096791635712] - WARNING - vbucket map not ready after try 1 [2021-02-12 22:38:21,764] - [bucket_helper] [140096791635712] - INFO - waiting for memcached bucket : dst_bucket in 172.23.123.61 to accept set ops [2021-02-12 22:38:21,882] - [data_helper] [140096791635712] - INFO - creating direct client 172.23.122.90:11210 dst_bucket [2021-02-12 22:38:21,960] - [data_helper] [140096791635712] - INFO - creating direct client 172.23.123.61:11210 dst_bucket [2021-02-12 22:38:22,037] - [data_helper] [140096791635712] - INFO - creating direct client 172.23.123.62:11210 dst_bucket [2021-02-12 22:38:22,334] - [data_helper] [140096791635712] - INFO - creating direct client 172.23.122.90:11210 dst_bucket [2021-02-12 22:38:22,425] - [data_helper] [140096791635712] - INFO - creating direct client 172.23.123.61:11210 dst_bucket [2021-02-12 22:38:22,502] - [data_helper] [140096791635712] - INFO - creating direct client 172.23.123.62:11210 dst_bucket [2021-02-12 22:38:22,572] - [task] [140096791635712] - WARNING - vbucket map not ready after try 2 [2021-02-12 22:38:22,573] - [bucket_helper] [140096791635712] - INFO - waiting for memcached bucket : dst_bucket in 172.23.123.61 to accept set ops [2021-02-12 22:38:22,702] - [data_helper] [140096791635712] - INFO - creating direct client 172.23.122.90:11210 dst_bucket [2021-02-12 22:38:22,783] - [data_helper] [140096791635712] - INFO - creating direct client 172.23.123.61:11210 dst_bucket [2021-02-12 22:38:23,052] - [data_helper] [140096791635712] - INFO - creating direct client 172.23.123.62:11210 dst_bucket [2021-02-12 22:38:23,183] - [data_helper] [140096791635712] - INFO - creating direct client 172.23.122.90:11210 dst_bucket [2021-02-12 22:38:23,258] - [data_helper] [140096791635712] - INFO - creating direct client 172.23.123.61:11210 dst_bucket [2021-02-12 22:38:23,336] - [data_helper] [140096791635712] - INFO - creating direct client 172.23.123.62:11210 dst_bucket [2021-02-12 22:38:23,406] - [task] [140096791635712] - WARNING - vbucket map not ready after try 3 [2021-02-12 22:38:23,406] - [bucket_helper] [140096791635712] - INFO - waiting for memcached bucket : dst_bucket in 172.23.123.61 to accept set ops [2021-02-12 22:38:23,518] - [data_helper] [140096791635712] - INFO - creating direct client 172.23.122.90:11210 dst_bucket [2021-02-12 22:38:23,787] - [data_helper] [140096791635712] - INFO - creating direct client 172.23.123.61:11210 dst_bucket [2021-02-12 22:38:23,864] - [data_helper] [140096791635712] - INFO - creating direct client 172.23.123.62:11210 dst_bucket [2021-02-12 22:38:23,976] - [data_helper] [140096791635712] - INFO - creating direct client 172.23.122.90:11210 dst_bucket [2021-02-12 22:38:24,047] - [data_helper] [140096791635712] - INFO - creating direct client 172.23.123.61:11210 dst_bucket [2021-02-12 22:38:24,126] - [data_helper] [140096791635712] - INFO - creating direct client 172.23.123.62:11210 dst_bucket [2021-02-12 22:38:24,196] - [task] [140096791635712] - WARNING - vbucket map not ready after try 4 [2021-02-12 22:38:24,196] - [bucket_helper] [140096791635712] - INFO - waiting for memcached bucket : dst_bucket in 172.23.123.61 to accept set ops [2021-02-12 22:38:24,499] - [data_helper] [140096791635712] - INFO - creating direct client 172.23.122.90:11210 dst_bucket [2021-02-12 22:38:24,574] - [data_helper] [140096791635712] - INFO - creating direct client 172.23.123.61:11210 dst_bucket [2021-02-12 22:38:24,653] - [data_helper] [140096791635712] - INFO - creating direct client 172.23.123.62:11210 dst_bucket [2021-02-12 22:38:24,772] - [data_helper] [140096791635712] - INFO - creating direct client 172.23.122.90:11210 dst_bucket [2021-02-12 22:38:24,858] - [data_helper] [140096791635712] - INFO - creating direct client 172.23.123.61:11210 dst_bucket [2021-02-12 22:38:25,118] - [data_helper] [140096791635712] - INFO - creating direct client 172.23.123.62:11210 dst_bucket [2021-02-12 22:38:25,186] - [task] [140096791635712] - WARNING - vbucket map not ready after try 5 [2021-02-12 22:38:26,214] - [rest_client] [140096791635712] - INFO - http://172.23.123.61:8091/pools/default/buckets with param: name=metadata&ramQuotaMB=200&authType=none&replicaNumber=0&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore [2021-02-12 22:38:26,244] - [rest_client] [140096791635712] - INFO - 0.03 seconds to create bucket metadata [2021-02-12 22:38:26,244] - [bucket_helper] [140096791635712] - INFO - waiting for memcached bucket : metadata in 172.23.123.61 to accept set ops [2021-02-12 22:38:26,901] - [data_helper] [140096791635712] - INFO - creating direct client 172.23.122.90:11210 metadata [2021-02-12 22:38:26,992] - [data_helper] [140096791635712] - INFO - creating direct client 172.23.123.61:11210 metadata [2021-02-12 22:38:27,075] - [data_helper] [140096791635712] - INFO - creating direct client 172.23.123.62:11210 metadata [2021-02-12 22:38:27,229] - [data_helper] [140096791635712] - INFO - creating direct client 172.23.122.90:11210 metadata [2021-02-12 22:38:27,494] - [data_helper] [140096791635712] - INFO - creating direct client 172.23.123.61:11210 metadata [2021-02-12 22:38:27,594] - [data_helper] [140096791635712] - INFO - creating direct client 172.23.123.62:11210 metadata [2021-02-12 22:38:27,676] - [task] [140096791635712] - WARNING - vbucket map not ready after try 0 [2021-02-12 22:38:27,676] - [bucket_helper] [140096791635712] - INFO - waiting for memcached bucket : metadata in 172.23.123.61 to accept set ops [2021-02-12 22:38:27,809] - [data_helper] [140096791635712] - INFO - creating direct client 172.23.122.90:11210 metadata [2021-02-12 22:38:27,923] - [data_helper] [140096791635712] - INFO - creating direct client 172.23.123.61:11210 metadata [2021-02-12 22:38:28,004] - [data_helper] [140096791635712] - INFO - creating direct client 172.23.123.62:11210 metadata [2021-02-12 22:38:28,303] - [data_helper] [140096791635712] - INFO - creating direct client 172.23.122.90:11210 metadata [2021-02-12 22:38:28,386] - [data_helper] [140096791635712] - INFO - creating direct client 172.23.123.61:11210 metadata [2021-02-12 22:38:28,468] - [data_helper] [140096791635712] - INFO - creating direct client 172.23.123.62:11210 metadata [2021-02-12 22:38:28,541] - [task] [140096791635712] - WARNING - vbucket map not ready after try 1 [2021-02-12 22:38:28,541] - [bucket_helper] [140096791635712] - INFO - waiting for memcached bucket : metadata in 172.23.123.61 to accept set ops [2021-02-12 22:38:28,675] - [data_helper] [140096791635712] - INFO - creating direct client 172.23.122.90:11210 metadata [2021-02-12 22:38:28,947] - [data_helper] [140096791635712] - INFO - creating direct client 172.23.123.61:11210 metadata [2021-02-12 22:38:29,031] - [data_helper] [140096791635712] - INFO - creating direct client 172.23.123.62:11210 metadata [2021-02-12 22:38:29,165] - [data_helper] [140096791635712] - INFO - creating direct client 172.23.122.90:11210 metadata [2021-02-12 22:38:29,252] - [data_helper] [140096791635712] - INFO - creating direct client 172.23.123.61:11210 metadata [2021-02-12 22:38:29,343] - [data_helper] [140096791635712] - INFO - creating direct client 172.23.123.62:11210 metadata [2021-02-12 22:38:29,418] - [task] [140096791635712] - WARNING - vbucket map not ready after try 2 [2021-02-12 22:38:29,418] - [bucket_helper] [140096791635712] - INFO - waiting for memcached bucket : metadata in 172.23.123.61 to accept set ops [2021-02-12 22:38:29,738] - [data_helper] [140096791635712] - INFO - creating direct client 172.23.122.90:11210 metadata [2021-02-12 22:38:29,823] - [data_helper] [140096791635712] - INFO - creating direct client 172.23.123.61:11210 metadata [2021-02-12 22:38:29,908] - [data_helper] [140096791635712] - INFO - creating direct client 172.23.123.62:11210 metadata [2021-02-12 22:38:30,039] - [data_helper] [140096791635712] - INFO - creating direct client 172.23.122.90:11210 metadata [2021-02-12 22:38:30,134] - [data_helper] [140096791635712] - INFO - creating direct client 172.23.123.61:11210 metadata [2021-02-12 22:38:30,223] - [data_helper] [140096791635712] - INFO - creating direct client 172.23.123.62:11210 metadata [2021-02-12 22:38:30,478] - [task] [140096791635712] - WARNING - vbucket map not ready after try 3 [2021-02-12 22:38:30,478] - [bucket_helper] [140096791635712] - INFO - waiting for memcached bucket : metadata in 172.23.123.61 to accept set ops [2021-02-12 22:38:30,618] - [data_helper] [140096791635712] - INFO - creating direct client 172.23.122.90:11210 metadata [2021-02-12 22:38:30,704] - [data_helper] [140096791635712] - INFO - creating direct client 172.23.123.61:11210 metadata [2021-02-12 22:38:30,799] - [data_helper] [140096791635712] - INFO - creating direct client 172.23.123.62:11210 metadata [2021-02-12 22:38:30,932] - [data_helper] [140096791635712] - INFO - creating direct client 172.23.122.90:11210 metadata [2021-02-12 22:38:31,216] - [data_helper] [140096791635712] - INFO - creating direct client 172.23.123.61:11210 metadata [2021-02-12 22:38:31,304] - [data_helper] [140096791635712] - INFO - creating direct client 172.23.123.62:11210 metadata [2021-02-12 22:38:31,386] - [task] [140096791635712] - WARNING - vbucket map not ready after try 4 [2021-02-12 22:38:31,387] - [bucket_helper] [140096791635712] - INFO - waiting for memcached bucket : metadata in 172.23.123.61 to accept set ops [2021-02-12 22:38:31,511] - [data_helper] [140096791635712] - INFO - creating direct client 172.23.122.90:11210 metadata [2021-02-12 22:38:31,598] - [data_helper] [140096791635712] - INFO - creating direct client 172.23.123.61:11210 metadata [2021-02-12 22:38:31,681] - [data_helper] [140096791635712] - INFO - creating direct client 172.23.123.62:11210 metadata [2021-02-12 22:38:32,005] - [data_helper] [140096791635712] - INFO - creating direct client 172.23.122.90:11210 metadata [2021-02-12 22:38:32,088] - [data_helper] [140096791635712] - INFO - creating direct client 172.23.123.61:11210 metadata [2021-02-12 22:38:32,174] - [data_helper] [140096791635712] - INFO - creating direct client 172.23.123.62:11210 metadata [2021-02-12 22:38:32,248] - [task] [140096791635712] - WARNING - vbucket map not ready after try 5 [2021-02-12 22:38:32,323] - [rest_client] [140097149204224] - INFO - http://172.23.122.117:8091/pools/default/buckets/dst_bucket/scopes with params: name=scope_0 [2021-02-12 22:38:32,324] - [rest_client] [140097149204224] - INFO - Scope created dst_bucket->scope_0 {'uid': '1'} [2021-02-12 22:38:32,352] - [rest_client] [140097149204224] - INFO - http://172.23.122.117:8091/pools/default/buckets/dst_bucket/scopes with params: name=scope_1 [2021-02-12 22:38:32,353] - [rest_client] [140097149204224] - INFO - Scope created dst_bucket->scope_1 {'uid': '2'} [2021-02-12 22:38:32,381] - [rest_client] [140097149204224] - INFO - http://172.23.122.117:8091/pools/default/buckets/dst_bucket/scopes with params: name=scope_2 [2021-02-12 22:38:32,381] - [rest_client] [140097149204224] - INFO - Scope created dst_bucket->scope_2 {'uid': '3'} [2021-02-12 22:38:32,410] - [rest_client] [140097149204224] - INFO - http://172.23.122.117:8091/pools/default/buckets/dst_bucket/scopes with params: name=scope_3 [2021-02-12 22:38:32,410] - [rest_client] [140097149204224] - INFO - Scope created dst_bucket->scope_3 {'uid': '4'} [2021-02-12 22:38:32,441] - [rest_client] [140097149204224] - INFO - http://172.23.122.117:8091/pools/default/buckets/dst_bucket/scopes with params: name=scope_4 [2021-02-12 22:38:32,441] - [rest_client] [140097149204224] - INFO - Scope created dst_bucket->scope_4 {'uid': '5'} [2021-02-12 22:38:32,470] - [rest_client] [140097149204224] - INFO - http://172.23.122.117:8091/pools/default/buckets/src_bucket/scopes with params: name=scope_0 [2021-02-12 22:38:32,470] - [rest_client] [140097149204224] - INFO - Scope created src_bucket->scope_0 {'uid': '1'} [2021-02-12 22:38:32,501] - [rest_client] [140097149204224] - INFO - http://172.23.122.117:8091/pools/default/buckets/src_bucket/scopes with params: name=scope_1 [2021-02-12 22:38:32,501] - [rest_client] [140097149204224] - INFO - Scope created src_bucket->scope_1 {'uid': '2'} [2021-02-12 22:38:32,528] - [rest_client] [140097149204224] - INFO - http://172.23.122.117:8091/pools/default/buckets/src_bucket/scopes with params: name=scope_2 [2021-02-12 22:38:32,528] - [rest_client] [140097149204224] - INFO - Scope created src_bucket->scope_2 {'uid': '3'} [2021-02-12 22:38:32,561] - [rest_client] [140097149204224] - INFO - http://172.23.122.117:8091/pools/default/buckets/src_bucket/scopes with params: name=scope_3 [2021-02-12 22:38:32,561] - [rest_client] [140097149204224] - INFO - Scope created src_bucket->scope_3 {'uid': '4'} [2021-02-12 22:38:32,588] - [rest_client] [140097149204224] - INFO - http://172.23.122.117:8091/pools/default/buckets/src_bucket/scopes with params: name=scope_4 [2021-02-12 22:38:32,588] - [rest_client] [140097149204224] - INFO - Scope created src_bucket->scope_4 {'uid': '5'} [2021-02-12 22:38:32,625] - [rest_client] [140097149204224] - INFO - http://172.23.122.117:8091/pools/default/buckets/dst_bucket/scopes/scope_1/collections with params: name=coll_0 [2021-02-12 22:38:32,626] - [rest_client] [140097149204224] - INFO - Collection created dst_bucket->scope_1->coll_0 manifest:{'uid': '6'} [2021-02-12 22:38:32,656] - [rest_client] [140097149204224] - INFO - http://172.23.122.117:8091/pools/default/buckets/dst_bucket/scopes/scope_1/collections with params: name=coll_1 [2021-02-12 22:38:32,656] - [rest_client] [140097149204224] - INFO - Collection created dst_bucket->scope_1->coll_1 manifest:{'uid': '7'} [2021-02-12 22:38:32,687] - [rest_client] [140097149204224] - INFO - http://172.23.122.117:8091/pools/default/buckets/dst_bucket/scopes/scope_1/collections with params: name=coll_2 [2021-02-12 22:38:32,687] - [rest_client] [140097149204224] - INFO - Collection created dst_bucket->scope_1->coll_2 manifest:{'uid': '8'} [2021-02-12 22:38:32,717] - [rest_client] [140097149204224] - INFO - http://172.23.122.117:8091/pools/default/buckets/dst_bucket/scopes/scope_1/collections with params: name=coll_3 [2021-02-12 22:38:32,717] - [rest_client] [140097149204224] - INFO - Collection created dst_bucket->scope_1->coll_3 manifest:{'uid': '9'} [2021-02-12 22:38:32,748] - [rest_client] [140097149204224] - INFO - http://172.23.122.117:8091/pools/default/buckets/dst_bucket/scopes/scope_1/collections with params: name=coll_4 [2021-02-12 22:38:32,748] - [rest_client] [140097149204224] - INFO - Collection created dst_bucket->scope_1->coll_4 manifest:{'uid': 'a'} [2021-02-12 22:38:32,776] - [rest_client] [140097149204224] - INFO - http://172.23.122.117:8091/pools/default/buckets/src_bucket/scopes/scope_1/collections with params: name=coll_0 [2021-02-12 22:38:32,776] - [rest_client] [140097149204224] - INFO - Collection created src_bucket->scope_1->coll_0 manifest:{'uid': '6'} [2021-02-12 22:38:32,812] - [rest_client] [140097149204224] - INFO - http://172.23.122.117:8091/pools/default/buckets/src_bucket/scopes/scope_1/collections with params: name=coll_1 [2021-02-12 22:38:32,812] - [rest_client] [140097149204224] - INFO - Collection created src_bucket->scope_1->coll_1 manifest:{'uid': '7'} [2021-02-12 22:38:32,839] - [rest_client] [140097149204224] - INFO - http://172.23.122.117:8091/pools/default/buckets/src_bucket/scopes/scope_1/collections with params: name=coll_2 [2021-02-12 22:38:32,839] - [rest_client] [140097149204224] - INFO - Collection created src_bucket->scope_1->coll_2 manifest:{'uid': '8'} [2021-02-12 22:38:32,872] - [rest_client] [140097149204224] - INFO - http://172.23.122.117:8091/pools/default/buckets/src_bucket/scopes/scope_1/collections with params: name=coll_3 [2021-02-12 22:38:32,872] - [rest_client] [140097149204224] - INFO - Collection created src_bucket->scope_1->coll_3 manifest:{'uid': '9'} [2021-02-12 22:38:32,901] - [rest_client] [140097149204224] - INFO - http://172.23.122.117:8091/pools/default/buckets/src_bucket/scopes/scope_1/collections with params: name=coll_4 [2021-02-12 22:38:32,901] - [rest_client] [140097149204224] - INFO - Collection created src_bucket->scope_1->coll_4 manifest:{'uid': 'a'} [2021-02-12 22:38:34,069] - [eventing_base] [140097149204224] - INFO - saving function bucket_op [2021-02-12 22:38:35,236] - [eventing_base] [140097149204224] - INFO - saving function timers [2021-02-12 22:38:36,474] - [eventing_base] [140097149204224] - INFO - saving function sbm [2021-02-12 22:38:37,672] - [eventing_base] [140097149204224] - INFO - saving function curl [2021-02-12 22:38:38,892] - [eventing_base] [140097149204224] - INFO - saving function n1ql [2021-02-12 22:38:40,114] - [basetestcase] [140097149204224] - INFO - sleep for 5 secs. Waiting for bucket_op to deployed... ... [2021-02-12 22:38:45,168] - [basetestcase] [140097149204224] - INFO - sleep for 5 secs. Waiting for bucket_op to deployed... ... [2021-02-12 22:38:50,197] - [basetestcase] [140097149204224] - INFO - sleep for 5 secs. Waiting for bucket_op to deployed... ... [2021-02-12 22:38:55,233] - [basetestcase] [140097149204224] - INFO - sleep for 5 secs. Waiting for bucket_op to deployed... ... [2021-02-12 22:39:01,522] - [basetestcase] [140097149204224] - INFO - sleep for 5 secs. Waiting for timers to deployed... ... [2021-02-12 22:39:06,550] - [basetestcase] [140097149204224] - INFO - sleep for 5 secs. Waiting for timers to deployed... ... [2021-02-12 22:39:11,597] - [basetestcase] [140097149204224] - INFO - sleep for 5 secs. Waiting for timers to deployed... ... [2021-02-12 22:39:16,619] - [basetestcase] [140097149204224] - INFO - sleep for 5 secs. Waiting for timers to deployed... ... [2021-02-12 22:39:22,949] - [basetestcase] [140097149204224] - INFO - sleep for 5 secs. Waiting for sbm to deployed... ... [2021-02-12 22:39:27,971] - [basetestcase] [140097149204224] - INFO - sleep for 5 secs. Waiting for sbm to deployed... ... [2021-02-12 22:39:33,005] - [basetestcase] [140097149204224] - INFO - sleep for 5 secs. Waiting for sbm to deployed... ... [2021-02-12 22:39:38,031] - [basetestcase] [140097149204224] - INFO - sleep for 5 secs. Waiting for sbm to deployed... ... [2021-02-12 22:39:44,286] - [basetestcase] [140097149204224] - INFO - sleep for 5 secs. Waiting for curl to deployed... ... [2021-02-12 22:39:49,312] - [basetestcase] [140097149204224] - INFO - sleep for 5 secs. Waiting for curl to deployed... ... [2021-02-12 22:39:54,352] - [basetestcase] [140097149204224] - INFO - sleep for 5 secs. Waiting for curl to deployed... ... [2021-02-12 22:39:59,381] - [basetestcase] [140097149204224] - INFO - sleep for 5 secs. Waiting for curl to deployed... ... [2021-02-12 22:40:05,618] - [basetestcase] [140097149204224] - INFO - sleep for 5 secs. Waiting for n1ql to deployed... ... [2021-02-12 22:40:10,641] - [basetestcase] [140097149204224] - INFO - sleep for 5 secs. Waiting for n1ql to deployed... ... [2021-02-12 22:40:15,700] - [basetestcase] [140097149204224] - INFO - sleep for 5 secs. Waiting for n1ql to deployed... ... [2021-02-12 22:40:20,728] - [basetestcase] [140097149204224] - INFO - sleep for 5 secs. Waiting for n1ql to deployed... ... [2021-02-12 22:40:26,343] - [task] [140096791635712] - INFO - java -jar java_sdk_client/collections/target/javaclient/javaclient.jar -i 172.23.123.61 -u Administrator -p password -b src_bucket -s _default -c _default -n 40320 -pc 100 -pu 0 -pd 0 -l uniform -dsn 1 -dpx doc_ -dt Person -de 0 -ds 500 -ac False -st 0 -en 40319 -o False -sd False [2021-02-12 22:40:28,773] - [rest_client] [140097149204224] - INFO - rebalance reached >100% in 3.0053842067718506 seconds [2021-02-12 22:40:37,362] - [task] [140096791635712] - INFO - java -jar java_sdk_client/collections/target/javaclient/javaclient.jar -i 172.23.123.61 -u Administrator -p password -b src_bucket -s scope_1 -c coll_1 -n 40320 -pc 100 -pu 0 -pd 0 -l uniform -dsn 1 -dpx doc_ -dt Person -de 0 -ds 500 -ac False -st 0 -en 40319 -o False -sd False [2021-02-12 22:40:48,510] - [task] [140096791635712] - INFO - adding node 172.23.122.82:8091 to cluster [2021-02-12 22:40:48,510] - [rest_client] [140096791635712] - INFO - adding remote node @172.23.122.82:8091 to this cluster @172.23.123.61:8091 [2021-02-12 22:40:50,986] - [task] [140096791635712] - INFO - adding node 172.23.122.118:8091 to cluster [2021-02-12 22:40:50,986] - [rest_client] [140096791635712] - INFO - adding remote node @172.23.122.118:8091 to this cluster @172.23.123.61:8091 [2021-02-12 22:40:53,466] - [rest_client] [140096791635712] - INFO - rebalance params : {'knownNodes': 'ns_1@172.23.122.117,ns_1@172.23.122.118,ns_1@172.23.122.82,ns_1@172.23.122.90,ns_1@172.23.122.97,ns_1@172.23.123.61,ns_1@172.23.123.62', 'ejectedNodes': '', 'user': 'Administrator', 'password': 'password'} [2021-02-12 22:40:53,500] - [rest_client] [140096791635712] - INFO - rebalance operation started [2021-02-12 22:40:53,504] - [rest_client] [140096791635712] - INFO - rebalance percentage : 0.00 % [2021-02-12 22:40:53,504] - [task] [140096791635712] - INFO - Rebalance - status: running, progress: 0.00% [2021-02-12 22:41:03,529] - [rest_client] [140096791635712] - INFO - rebalance percentage : 42.00 % [2021-02-12 22:41:03,529] - [task] [140096791635712] - INFO - Rebalance - status: running, progress: 42.00% [2021-02-12 22:41:13,558] - [rest_client] [140096791635712] - INFO - rebalance percentage : 78.00 % [2021-02-12 22:41:13,558] - [task] [140096791635712] - INFO - Rebalance - status: running, progress: 78.00% [2021-02-12 22:41:23,585] - [rest_client] [140096791635712] - INFO - rebalance percentage : 78.00 % [2021-02-12 22:41:23,585] - [task] [140096791635712] - INFO - Rebalance - status: running, progress: 78.00% [2021-02-12 22:41:33,613] - [rest_client] [140096791635712] - INFO - rebalance percentage : 79.00 % [2021-02-12 22:41:33,613] - [task] [140096791635712] - INFO - Rebalance - status: running, progress: 79.00% [2021-02-12 22:41:43,639] - [rest_client] [140096791635712] - INFO - rebalance percentage : 84.00 % [2021-02-12 22:41:43,639] - [task] [140096791635712] - INFO - Rebalance - status: running, progress: 84.00% [2021-02-12 22:41:53,667] - [rest_client] [140096791635712] - INFO - rebalance percentage : 87.00 % [2021-02-12 22:41:53,668] - [task] [140096791635712] - INFO - Rebalance - status: running, progress: 87.00% [2021-02-12 22:42:03,694] - [rest_client] [140096791635712] - INFO - rebalance percentage : 87.00 % [2021-02-12 22:42:03,695] - [task] [140096791635712] - INFO - Rebalance - status: running, progress: 87.00% [2021-02-12 22:42:13,725] - [rest_client] [140096791635712] - INFO - rebalance percentage : 88.00 % [2021-02-12 22:42:13,725] - [task] [140096791635712] - INFO - Rebalance - status: running, progress: 88.00% [2021-02-12 22:42:23,755] - [rest_client] [140096791635712] - INFO - rebalance percentage : 89.00 % [2021-02-12 22:42:23,755] - [task] [140096791635712] - INFO - Rebalance - status: running, progress: 89.00% [2021-02-12 22:42:33,785] - [rest_client] [140096791635712] - INFO - rebalance percentage : 89.00 % [2021-02-12 22:42:33,785] - [task] [140096791635712] - INFO - Rebalance - status: running, progress: 89.00% [2021-02-12 22:42:43,813] - [rest_client] [140096791635712] - INFO - rebalance percentage : 90.00 % [2021-02-12 22:42:43,813] - [task] [140096791635712] - INFO - Rebalance - status: running, progress: 90.00% [2021-02-12 22:42:53,842] - [rest_client] [140096791635712] - INFO - rebalance percentage : 91.00 % [2021-02-12 22:42:53,842] - [task] [140096791635712] - INFO - Rebalance - status: running, progress: 91.00% [2021-02-12 22:43:03,871] - [task] [140096791635712] - INFO - Rebalance - status: none, progress: 100.00% [2021-02-12 22:43:03,886] - [task] [140096791635712] - INFO - rebalancing was completed with progress: 100% in 130.38536024093628 sec [2021-02-12 22:43:03,895] - [basetestcase] [140097149204224] - INFO - list of eventing nodes in cluster: [ip:172.23.122.117 port:8091 ssh_username:root] [2021-02-12 22:43:03,901] - [tuq_helper] [140097149204224] - INFO - RUN QUERY create primary index on dst_bucket.scope_1.coll_0 [2021-02-12 22:43:03,907] - [rest_client] [140097149204224] - INFO - query params : statement=create+primary+index+on+dst_bucket.scope_1.coll_0 [2021-02-12 22:43:06,756] - [tuq_helper] [140097149204224] - INFO - TOTAL ELAPSED TIME: 2.847784209s [2021-02-12 22:43:06,775] - [basetestcase] [140097149204224] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.97 port:8091 ssh_username:root] [2021-02-12 22:43:06,781] - [rest_client] [140097149204224] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_0 [2021-02-12 22:43:06,793] - [eventing_base] [140097149204224] - INFO - Final docs count... Current : 40320 Expected : 40320 [2021-02-12 22:43:06,801] - [basetestcase] [140097149204224] - INFO - list of eventing nodes in cluster: [ip:172.23.122.117 port:8091 ssh_username:root] [2021-02-12 22:43:06,807] - [tuq_helper] [140097149204224] - INFO - RUN QUERY create primary index on dst_bucket.scope_1.coll_1 [2021-02-12 22:43:06,812] - [rest_client] [140097149204224] - INFO - query params : statement=create+primary+index+on+dst_bucket.scope_1.coll_1 [2021-02-12 22:43:10,118] - [tuq_helper] [140097149204224] - INFO - TOTAL ELAPSED TIME: 3.304931024s [2021-02-12 22:43:10,139] - [basetestcase] [140097149204224] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.97 port:8091 ssh_username:root] [2021-02-12 22:43:10,145] - [rest_client] [140097149204224] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_1 [2021-02-12 22:43:10,161] - [basetestcase] [140097149204224] - INFO - sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_1 to complete bucket operations... Current : 31592 Expected : 40320 ... [2021-02-12 22:43:40,215] - [basetestcase] [140097149204224] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.97 port:8091 ssh_username:root] [2021-02-12 22:43:40,221] - [rest_client] [140097149204224] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_1 [2021-02-12 22:43:40,246] - [basetestcase] [140097149204224] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.97 port:8091 ssh_username:root] [2021-02-12 22:43:40,251] - [rest_client] [140097149204224] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_1 [2021-02-12 22:43:40,254] - [eventing_base] [140097149204224] - INFO - Final docs count... Current : 40320 Expected : 40320 [2021-02-12 22:43:40,262] - [basetestcase] [140097149204224] - INFO - list of eventing nodes in cluster: [ip:172.23.122.117 port:8091 ssh_username:root] [2021-02-12 22:43:40,267] - [tuq_helper] [140097149204224] - INFO - RUN QUERY create primary index on dst_bucket.scope_1.coll_3 [2021-02-12 22:43:40,272] - [rest_client] [140097149204224] - INFO - query params : statement=create+primary+index+on+dst_bucket.scope_1.coll_3 [2021-02-12 22:43:43,289] - [tuq_helper] [140097149204224] - INFO - TOTAL ELAPSED TIME: 3.014744932s [2021-02-12 22:43:43,308] - [basetestcase] [140097149204224] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.97 port:8091 ssh_username:root] [2021-02-12 22:43:43,314] - [rest_client] [140097149204224] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_3 [2021-02-12 22:43:43,326] - [eventing_base] [140097149204224] - INFO - Final docs count... Current : 40320 Expected : 40320 [2021-02-12 22:43:43,335] - [basetestcase] [140097149204224] - INFO - list of eventing nodes in cluster: [ip:172.23.122.117 port:8091 ssh_username:root] [2021-02-12 22:43:43,341] - [tuq_helper] [140097149204224] - INFO - RUN QUERY create primary index on dst_bucket.scope_1.coll_4 [2021-02-12 22:43:43,347] - [rest_client] [140097149204224] - INFO - query params : statement=create+primary+index+on+dst_bucket.scope_1.coll_4 [2021-02-12 22:43:46,348] - [tuq_helper] [140097149204224] - INFO - TOTAL ELAPSED TIME: 2.999725411s [2021-02-12 22:43:46,372] - [basetestcase] [140097149204224] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.97 port:8091 ssh_username:root] [2021-02-12 22:43:46,379] - [rest_client] [140097149204224] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-12 22:43:46,396] - [eventing_base] [140097149204224] - INFO - Final docs count... Current : 40320 Expected : 40320 [2021-02-12 22:43:46,405] - [basetestcase] [140097149204224] - INFO - list of eventing nodes in cluster: [ip:172.23.122.117 port:8091 ssh_username:root] [2021-02-12 22:43:46,413] - [tuq_helper] [140097149204224] - INFO - RUN QUERY create primary index on src_bucket.scope_1.coll_1 [2021-02-12 22:43:46,418] - [rest_client] [140097149204224] - INFO - query params : statement=create+primary+index+on+src_bucket.scope_1.coll_1 [2021-02-12 22:43:49,617] - [tuq_helper] [140097149204224] - INFO - TOTAL ELAPSED TIME: 3.19736779s [2021-02-12 22:43:49,635] - [basetestcase] [140097149204224] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.97 port:8091 ssh_username:root] [2021-02-12 22:43:49,640] - [rest_client] [140097149204224] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.scope_1.coll_1 [2021-02-12 22:43:49,658] - [eventing_base] [140097149204224] - INFO - Final docs count... Current : 80640 Expected : 80640 [2021-02-12 22:43:49,668] - [basetestcase] [140097149204224] - INFO - list of kv nodes in cluster: [ip:172.23.122.118 port:8091 ssh_username:root, ip:172.23.122.82 port:8091 ssh_username:root, ip:172.23.122.90 port:8091 ssh_username:root, ip:172.23.123.61 port:8091 ssh_username:root, ip:172.23.123.62 port:8091 ssh_username:root] [2021-02-12 22:43:49,938] - [task] [140096791635712] - INFO - java -jar java_sdk_client/collections/target/javaclient/javaclient.jar -i 172.23.123.61 -u Administrator -p password -b src_bucket -s _default -c _default -n 40320 -pc 0 -pu 0 -pd 100 -l uniform -dsn 1 -dpx doc_ -dt Person -de 0 -ds 500 -ac False -st 0 -en 40319 -o False -sd False [2021-02-12 22:43:52,542] - [task] [140096791635712] - INFO - java -jar java_sdk_client/collections/target/javaclient/javaclient.jar -i 172.23.123.61 -u Administrator -p password -b src_bucket -s scope_1 -c coll_1 -n 40320 -pc 0 -pu 0 -pd 100 -l uniform -dsn 1 -dpx doc_ -dt Person -de 0 -ds 500 -ac False -st 0 -en 40319 -o False -sd False [2021-02-12 22:43:52,676] - [rest_client] [140097149204224] - INFO - rebalance reached >100% in 3.002703905105591 seconds [2021-02-12 22:43:55,153] - [rest_client] [140096791635712] - INFO - rebalance params : {'knownNodes': 'ns_1@172.23.122.117,ns_1@172.23.122.118,ns_1@172.23.122.82,ns_1@172.23.122.90,ns_1@172.23.122.97,ns_1@172.23.123.61,ns_1@172.23.123.62', 'ejectedNodes': 'ns_1@172.23.122.118,ns_1@172.23.122.82', 'user': 'Administrator', 'password': 'password'} [2021-02-12 22:43:55,183] - [rest_client] [140096791635712] - INFO - rebalance operation started [2021-02-12 22:43:55,186] - [rest_client] [140096791635712] - INFO - rebalance percentage : 0.00 % [2021-02-12 22:43:55,186] - [task] [140096791635712] - INFO - Rebalance - status: running, progress: 0.00% [2021-02-12 22:44:05,221] - [rest_client] [140096791635712] - INFO - rebalance percentage : 23.00 % [2021-02-12 22:44:05,221] - [task] [140096791635712] - INFO - Rebalance - status: running, progress: 23.00% [2021-02-12 22:44:15,283] - [rest_client] [140096791635712] - INFO - rebalance percentage : 47.00 % [2021-02-12 22:44:15,284] - [task] [140096791635712] - INFO - Rebalance - status: running, progress: 47.00% [2021-02-12 22:44:25,331] - [rest_client] [140096791635712] - INFO - rebalance percentage : 50.00 % [2021-02-12 22:44:25,331] - [task] [140096791635712] - INFO - Rebalance - status: running, progress: 50.00% [2021-02-12 22:44:35,378] - [rest_client] [140096791635712] - INFO - rebalance percentage : 71.00 % [2021-02-12 22:44:35,378] - [task] [140096791635712] - INFO - Rebalance - status: running, progress: 71.00% [2021-02-12 22:44:45,405] - [rest_client] [140096791635712] - INFO - rebalance percentage : 78.00 % [2021-02-12 22:44:45,405] - [task] [140096791635712] - INFO - Rebalance - status: running, progress: 78.00% [2021-02-12 22:44:55,433] - [task] [140096791635712] - INFO - Rebalance - status: none, progress: 100.00% [2021-02-12 22:44:55,461] - [task] [140096791635712] - INFO - rebalancing was completed with progress: 100% in 60.277854681015015 sec [2021-02-12 22:44:55,469] - [basetestcase] [140097149204224] - INFO - list of eventing nodes in cluster: [ip:172.23.122.117 port:8091 ssh_username:root] [2021-02-12 22:44:55,476] - [tuq_helper] [140097149204224] - INFO - RUN QUERY create primary index on dst_bucket.scope_1.coll_0 [2021-02-12 22:44:55,483] - [rest_client] [140097149204224] - INFO - query params : statement=create+primary+index+on+dst_bucket.scope_1.coll_0 [2021-02-12 22:44:55,486] - [rest_client] [140097149204224] - ERROR - POST http://172.23.122.97:8093/query?statement=create+primary+index+on+dst_bucket.scope_1.coll_0 body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 500 reason: unknown b'{\n"requestID": "7c505c58-b898-4b12-9338-f6e1799d57e6",\n"signature": null,\n"results": [\n],\n"errors": [{"code":5000,"msg":"GSI CreatePrimaryIndex() - cause: Index #primary already exists."}],\n"status": "errors",\n"metrics": {"elapsedTime": "714.431\xc2\xb5s","executionTime": "649.763\xc2\xb5s","resultCount": 0,"resultSize": 0,"serviceLoad": 6,"errorCount": 1}\n}\n' auth: Administrator:password [2021-02-12 22:44:55,494] - [basetestcase] [140097149204224] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.97 port:8091 ssh_username:root] [2021-02-12 22:44:55,499] - [rest_client] [140097149204224] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_0 [2021-02-12 22:44:55,509] - [eventing_base] [140097149204224] - INFO - Final docs count... Current : 0 Expected : 0 [2021-02-12 22:44:55,516] - [basetestcase] [140097149204224] - INFO - list of eventing nodes in cluster: [ip:172.23.122.117 port:8091 ssh_username:root] [2021-02-12 22:44:55,521] - [tuq_helper] [140097149204224] - INFO - RUN QUERY create primary index on dst_bucket.scope_1.coll_1 [2021-02-12 22:44:55,526] - [rest_client] [140097149204224] - INFO - query params : statement=create+primary+index+on+dst_bucket.scope_1.coll_1 [2021-02-12 22:44:55,528] - [rest_client] [140097149204224] - ERROR - POST http://172.23.122.97:8093/query?statement=create+primary+index+on+dst_bucket.scope_1.coll_1 body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 500 reason: unknown b'{\n"requestID": "08ac0814-671c-4fa2-bf69-e1d9258ac7a4",\n"signature": null,\n"results": [\n],\n"errors": [{"code":5000,"msg":"GSI CreatePrimaryIndex() - cause: Index #primary already exists."}],\n"status": "errors",\n"metrics": {"elapsedTime": "775.22\xc2\xb5s","executionTime": "717.186\xc2\xb5s","resultCount": 0,"resultSize": 0,"serviceLoad": 6,"errorCount": 1}\n}\n' auth: Administrator:password [2021-02-12 22:44:55,536] - [basetestcase] [140097149204224] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.97 port:8091 ssh_username:root] [2021-02-12 22:44:55,541] - [rest_client] [140097149204224] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_1 [2021-02-12 22:44:55,544] - [basetestcase] [140097149204224] - INFO - sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_1 to complete bucket operations... Current : 27603 Expected : 0 ... [2021-02-12 22:45:25,577] - [basetestcase] [140097149204224] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.97 port:8091 ssh_username:root] [2021-02-12 22:45:25,583] - [rest_client] [140097149204224] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_1 [2021-02-12 22:45:25,600] - [basetestcase] [140097149204224] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.97 port:8091 ssh_username:root] [2021-02-12 22:45:25,605] - [rest_client] [140097149204224] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_1 [2021-02-12 22:45:25,608] - [basetestcase] [140097149204224] - INFO - sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_1 to complete bucket operations... Current : 7700 Expected : 0 ... [2021-02-12 22:45:55,658] - [basetestcase] [140097149204224] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.97 port:8091 ssh_username:root] [2021-02-12 22:45:55,664] - [rest_client] [140097149204224] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_1 [2021-02-12 22:45:55,681] - [basetestcase] [140097149204224] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.97 port:8091 ssh_username:root] [2021-02-12 22:45:55,686] - [rest_client] [140097149204224] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_1 [2021-02-12 22:45:55,690] - [basetestcase] [140097149204224] - INFO - sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_1 to complete bucket operations... Current : 3758 Expected : 0 ... [2021-02-12 22:46:25,737] - [basetestcase] [140097149204224] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.97 port:8091 ssh_username:root] [2021-02-12 22:46:25,743] - [rest_client] [140097149204224] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_1 [2021-02-12 22:46:25,760] - [basetestcase] [140097149204224] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.97 port:8091 ssh_username:root] [2021-02-12 22:46:25,764] - [rest_client] [140097149204224] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_1 [2021-02-12 22:46:25,767] - [basetestcase] [140097149204224] - INFO - sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_1 to complete bucket operations... Current : 416 Expected : 0 ... [2021-02-12 22:46:55,815] - [basetestcase] [140097149204224] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.97 port:8091 ssh_username:root] [2021-02-12 22:46:55,822] - [rest_client] [140097149204224] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_1 [2021-02-12 22:46:55,840] - [basetestcase] [140097149204224] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.97 port:8091 ssh_username:root] [2021-02-12 22:46:55,845] - [rest_client] [140097149204224] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_1 [2021-02-12 22:46:55,848] - [basetestcase] [140097149204224] - INFO - sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_1 to complete bucket operations... Current : 104 Expected : 0 ... [2021-02-12 22:47:25,897] - [basetestcase] [140097149204224] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.97 port:8091 ssh_username:root] [2021-02-12 22:47:25,904] - [rest_client] [140097149204224] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_1 [2021-02-12 22:47:25,921] - [basetestcase] [140097149204224] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.97 port:8091 ssh_username:root] [2021-02-12 22:47:25,927] - [rest_client] [140097149204224] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_1 [2021-02-12 22:47:25,930] - [basetestcase] [140097149204224] - INFO - sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_1 to complete bucket operations... Current : 1 Expected : 0 ... [2021-02-12 22:47:55,979] - [basetestcase] [140097149204224] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.97 port:8091 ssh_username:root] [2021-02-12 22:47:55,985] - [rest_client] [140097149204224] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_1 [2021-02-12 22:47:56,003] - [basetestcase] [140097149204224] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.97 port:8091 ssh_username:root] [2021-02-12 22:47:56,008] - [rest_client] [140097149204224] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_1 [2021-02-12 22:47:56,012] - [basetestcase] [140097149204224] - INFO - sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_1 to complete bucket operations... Current : 1 Expected : 0 ... [2021-02-12 22:48:26,063] - [basetestcase] [140097149204224] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.97 port:8091 ssh_username:root] [2021-02-12 22:48:26,069] - [rest_client] [140097149204224] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_1 [2021-02-12 22:48:26,089] - [basetestcase] [140097149204224] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.97 port:8091 ssh_username:root] [2021-02-12 22:48:26,094] - [rest_client] [140097149204224] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_1 [2021-02-12 22:48:26,097] - [basetestcase] [140097149204224] - INFO - sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_1 to complete bucket operations... Current : 1 Expected : 0 ... [2021-02-12 22:48:56,146] - [basetestcase] [140097149204224] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.97 port:8091 ssh_username:root] [2021-02-12 22:48:56,153] - [rest_client] [140097149204224] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_1 [2021-02-12 22:48:56,172] - [basetestcase] [140097149204224] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.97 port:8091 ssh_username:root] [2021-02-12 22:48:56,177] - [rest_client] [140097149204224] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_1 [2021-02-12 22:48:56,180] - [basetestcase] [140097149204224] - INFO - sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_1 to complete bucket operations... Current : 1 Expected : 0 ... [2021-02-12 22:49:26,230] - [basetestcase] [140097149204224] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.97 port:8091 ssh_username:root] [2021-02-12 22:49:26,235] - [rest_client] [140097149204224] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_1 [2021-02-12 22:49:26,253] - [basetestcase] [140097149204224] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.97 port:8091 ssh_username:root] [2021-02-12 22:49:26,258] - [rest_client] [140097149204224] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_1 [2021-02-12 22:49:26,260] - [basetestcase] [140097149204224] - INFO - sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_1 to complete bucket operations... Current : 1 Expected : 0 ... [2021-02-12 22:49:56,312] - [basetestcase] [140097149204224] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.97 port:8091 ssh_username:root] [2021-02-12 22:49:56,332] - [rest_client] [140097149204224] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_1 [2021-02-12 22:49:56,370] - [basetestcase] [140097149204224] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.97 port:8091 ssh_username:root] [2021-02-12 22:49:56,382] - [rest_client] [140097149204224] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_1 [2021-02-12 22:49:56,389] - [basetestcase] [140097149204224] - INFO - sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_1 to complete bucket operations... Current : 1 Expected : 0 ... [2021-02-12 22:50:26,435] - [basetestcase] [140097149204224] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.97 port:8091 ssh_username:root] [2021-02-12 22:50:26,442] - [rest_client] [140097149204224] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_1 [2021-02-12 22:50:26,461] - [basetestcase] [140097149204224] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.97 port:8091 ssh_username:root] [2021-02-12 22:50:26,467] - [rest_client] [140097149204224] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_1 [2021-02-12 22:50:26,470] - [basetestcase] [140097149204224] - INFO - sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_1 to complete bucket operations... Current : 1 Expected : 0 ... [2021-02-12 22:50:56,518] - [basetestcase] [140097149204224] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.97 port:8091 ssh_username:root] [2021-02-12 22:50:56,524] - [rest_client] [140097149204224] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_1 [2021-02-12 22:50:56,542] - [basetestcase] [140097149204224] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.97 port:8091 ssh_username:root] [2021-02-12 22:50:56,549] - [rest_client] [140097149204224] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_1 [2021-02-12 22:50:56,552] - [basetestcase] [140097149204224] - INFO - sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_1 to complete bucket operations... Current : 1 Expected : 0 ... [2021-02-12 22:51:26,600] - [basetestcase] [140097149204224] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.97 port:8091 ssh_username:root] [2021-02-12 22:51:26,606] - [rest_client] [140097149204224] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_1 [2021-02-12 22:51:26,624] - [basetestcase] [140097149204224] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.97 port:8091 ssh_username:root] [2021-02-12 22:51:26,629] - [rest_client] [140097149204224] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_1 [2021-02-12 22:51:26,632] - [basetestcase] [140097149204224] - INFO - sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_1 to complete bucket operations... Current : 1 Expected : 0 ... [2021-02-12 22:51:56,675] - [basetestcase] [140097149204224] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.97 port:8091 ssh_username:root] [2021-02-12 22:51:56,681] - [rest_client] [140097149204224] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_1 [2021-02-12 22:51:56,700] - [basetestcase] [140097149204224] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.97 port:8091 ssh_username:root] [2021-02-12 22:51:56,705] - [rest_client] [140097149204224] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_1 [2021-02-12 22:51:56,708] - [basetestcase] [140097149204224] - INFO - sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_1 to complete bucket operations... Current : 1 Expected : 0 ... [2021-02-12 22:52:26,758] - [basetestcase] [140097149204224] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.97 port:8091 ssh_username:root] [2021-02-12 22:52:26,765] - [rest_client] [140097149204224] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_1 [2021-02-12 22:52:26,784] - [basetestcase] [140097149204224] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.97 port:8091 ssh_username:root] [2021-02-12 22:52:26,789] - [rest_client] [140097149204224] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_1 [2021-02-12 22:52:26,792] - [basetestcase] [140097149204224] - INFO - sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_1 to complete bucket operations... Current : 1 Expected : 0 ... [2021-02-12 22:52:56,812] - [basetestcase] [140097149204224] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.97 port:8091 ssh_username:root] [2021-02-12 22:52:56,818] - [rest_client] [140097149204224] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_1 [2021-02-12 22:52:56,834] - [eventing_base] [140097149204224] - INFO - Compacting bucket : dst_bucket [2021-02-12 22:52:56,839] - [rest_client] [140097149204224] - INFO - bucket compaction successful [2021-02-12 22:52:56,840] - [eventing_base] [140097149204224] - INFO - Compacting bucket : metadata [2021-02-12 22:52:56,844] - [rest_client] [140097149204224] - INFO - bucket compaction successful [2021-02-12 22:52:56,844] - [eventing_base] [140097149204224] - INFO - Compacting bucket : src_bucket [2021-02-12 22:52:56,849] - [rest_client] [140097149204224] - INFO - bucket compaction successful [2021-02-12 22:52:56,860] - [basetestcase] [140097149204224] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.97 port:8091 ssh_username:root] [2021-02-12 22:52:56,865] - [rest_client] [140097149204224] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_1 [2021-02-12 22:52:56,869] - [basetestcase] [140097149204224] - INFO - sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_1 to complete bucket operations... Current : 1 Expected : 0 ... [2021-02-12 22:53:26,920] - [basetestcase] [140097149204224] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.97 port:8091 ssh_username:root] [2021-02-12 22:53:26,926] - [rest_client] [140097149204224] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_1 [2021-02-12 22:53:26,947] - [basetestcase] [140097149204224] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.97 port:8091 ssh_username:root] [2021-02-12 22:53:26,951] - [rest_client] [140097149204224] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_1 [2021-02-12 22:53:26,955] - [basetestcase] [140097149204224] - INFO - sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_1 to complete bucket operations... Current : 1 Expected : 0 ... [2021-02-12 22:53:57,006] - [basetestcase] [140097149204224] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.97 port:8091 ssh_username:root] [2021-02-12 22:53:57,013] - [rest_client] [140097149204224] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_1 [2021-02-12 22:53:57,034] - [basetestcase] [140097149204224] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.97 port:8091 ssh_username:root] [2021-02-12 22:53:57,039] - [rest_client] [140097149204224] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_1 [2021-02-12 22:53:57,042] - [basetestcase] [140097149204224] - INFO - sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_1 to complete bucket operations... Current : 1 Expected : 0 ... [2021-02-12 22:54:27,092] - [basetestcase] [140097149204224] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.97 port:8091 ssh_username:root] [2021-02-12 22:54:27,098] - [rest_client] [140097149204224] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_1 [2021-02-12 22:54:27,117] - [basetestcase] [140097149204224] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.97 port:8091 ssh_username:root] [2021-02-12 22:54:27,122] - [rest_client] [140097149204224] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_1 [2021-02-12 22:54:27,125] - [basetestcase] [140097149204224] - INFO - sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_1 to complete bucket operations... Current : 1 Expected : 0 ... [2021-02-12 22:54:57,180] - [basetestcase] [140097149204224] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.97 port:8091 ssh_username:root] [2021-02-12 22:54:57,186] - [rest_client] [140097149204224] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_1 [2021-02-12 22:54:57,204] - [basetestcase] [140097149204224] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.97 port:8091 ssh_username:root] [2021-02-12 22:54:57,209] - [rest_client] [140097149204224] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_1 [2021-02-12 22:54:57,212] - [basetestcase] [140097149204224] - INFO - sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_1 to complete bucket operations... Current : 1 Expected : 0 ... [2021-02-12 22:55:27,262] - [basetestcase] [140097149204224] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.97 port:8091 ssh_username:root] [2021-02-12 22:55:27,270] - [rest_client] [140097149204224] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_1 [2021-02-12 22:55:27,288] - [basetestcase] [140097149204224] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.97 port:8091 ssh_username:root] [2021-02-12 22:55:27,294] - [rest_client] [140097149204224] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_1 [2021-02-12 22:55:27,297] - [basetestcase] [140097149204224] - INFO - sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_1 to complete bucket operations... Current : 1 Expected : 0 ... [2021-02-12 22:55:57,344] - [basetestcase] [140097149204224] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.97 port:8091 ssh_username:root] [2021-02-12 22:55:57,352] - [rest_client] [140097149204224] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_1 [2021-02-12 22:55:57,369] - [basetestcase] [140097149204224] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.97 port:8091 ssh_username:root] [2021-02-12 22:55:57,376] - [rest_client] [140097149204224] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_1 [2021-02-12 22:55:57,379] - [basetestcase] [140097149204224] - INFO - sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_1 to complete bucket operations... Current : 1 Expected : 0 ... [2021-02-12 22:56:27,434] - [basetestcase] [140097149204224] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.97 port:8091 ssh_username:root] [2021-02-12 22:56:27,446] - [rest_client] [140097149204224] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_1 [2021-02-12 22:56:27,535] - [basetestcase] [140097149204224] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.97 port:8091 ssh_username:root] [2021-02-12 22:56:27,540] - [rest_client] [140097149204224] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_1 [2021-02-12 22:56:27,584] - [basetestcase] [140097149204224] - INFO - sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_1 to complete bucket operations... Current : 1 Expected : 0 ... [2021-02-12 22:56:57,632] - [basetestcase] [140097149204224] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.97 port:8091 ssh_username:root] [2021-02-12 22:56:57,638] - [rest_client] [140097149204224] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_1 [2021-02-12 22:56:57,655] - [basetestcase] [140097149204224] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.97 port:8091 ssh_username:root] [2021-02-12 22:56:57,660] - [rest_client] [140097149204224] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_1 [2021-02-12 22:56:57,663] - [basetestcase] [140097149204224] - INFO - sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_1 to complete bucket operations... Current : 1 Expected : 0 ... [2021-02-12 22:57:27,712] - [basetestcase] [140097149204224] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.97 port:8091 ssh_username:root] [2021-02-12 22:57:27,718] - [rest_client] [140097149204224] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_1 [2021-02-12 22:57:27,736] - [basetestcase] [140097149204224] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.97 port:8091 ssh_username:root] [2021-02-12 22:57:27,741] - [rest_client] [140097149204224] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_1 [2021-02-12 22:57:27,752] - [basetestcase] [140097149204224] - INFO - list of eventing nodes in cluster: [ip:172.23.122.117 port:8091 ssh_username:root] [2021-02-12 22:57:27,797] - [eventing_base] [140097149204224] - INFO - Stats for Node 172.23.122.117 is [ { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 7, "agg_messages_sent_to_worker": 90147, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_deletion": 96915, "dcp_deletion_sent_to_worker": 40320, "dcp_mutation": 201600, "dcp_mutation_sent_to_worker": 40320, "dcp_snapshot": 184021, "dcp_stream_req_counter": 3336, "dcp_stream_req_err_counter": 247, "dcp_streamend": 1451, "dcp_streamreq": 3089, "dcp_system_event": 10240, "execution_stats": 1117, "failure_stats": 1117, "latency_stats": 1117, "lcb_exception_stats": 1117, "log_level": 1, "num_processed_events": 80640, "processed_events_size": 47060684, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 40320, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 40320, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 40320, "enqueued_dcp_mutation_msg_counter": 40320, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 90143, "num_processed_events": 80640, "on_delete_failure": 0, "on_delete_success": 40320, "on_update_failure": 0, "on_update_success": 40320, "processed_events_size": 47060684, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "84162": "2021-02-13T06:57:27Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "84162": "2021-02-13T06:57:27Z" }, "v8worker_events_lost": 0 }, "function_id": 3413234574, "function_name": "bucket_op", "gocb_creds_request_counter": 133, "internal_vb_distribution_stats": { "worker_bucket_op_0": "[0-1023]" }, "latency_percentile_stats": { "100": 262400, "50": 500, "80": 800, "90": 1000, "95": 1500, "99": 5200 }, "lcb_creds_request_counter": 80787, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 1024 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_bucket_op_0": "[0-1023]" } }, "worker_pids": { "worker_bucket_op_0": 84162 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 12, "agg_messages_sent_to_worker": 109370, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_deletion": 97933, "dcp_deletion_sent_to_worker": 41338, "dcp_mutation": 221975, "dcp_mutation_sent_to_worker": 58596, "dcp_snapshot": 183630, "dcp_stream_req_counter": 8318, "dcp_stream_req_err_counter": 5304, "dcp_streamend": 1460, "dcp_streamreq": 3014, "dcp_system_event": 10240, "execution_stats": 1096, "failure_stats": 1096, "latency_stats": 1096, "lcb_exception_stats": 1096, "log_level": 1, "num_processed_events": 99934, "processed_events_size": 63702196, "thr_count": 1, "thr_map": 1, "timer_events": 80640, "timer_responses_received": 80640, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 41338, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 58596, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 41338, "enqueued_dcp_mutation_msg_counter": 58596, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 19294, "lcb_retry_failure": 0, "messages_parsed": 109361, "num_processed_events": 99934, "on_delete_failure": 0, "on_delete_success": 40320, "on_update_failure": 0, "on_update_success": 40320, "processed_events_size": 63702196, "timer_callback_failure": 1, "timer_callback_success": 80639, "timer_cancel_counter": 0, "timer_create_counter": 80640, "timer_create_failure": 0, "timer_msg_counter": 80640, "timer_responses_sent": 0, "timestamp": { "84204": "2021-02-13T06:57:27Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 1, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "84204": "2021-02-13T06:57:27Z" }, "v8worker_events_lost": 0 }, "function_id": 1452344633, "function_name": "timers", "gocb_creds_request_counter": 133, "internal_vb_distribution_stats": { "worker_timers_0": "[0-1023]" }, "latency_percentile_stats": { "100": 511900, "50": 1300, "80": 1800, "90": 2400, "95": 3500, "99": 9800 }, "lcb_creds_request_counter": 80787, "lcb_exception_stats": { "210": 1 }, "metastore_stats": { "metastore_delete_err": 0, "metastore_deletes": 0, "metastore_not_found": 0, "metastore_scan": 0, "metastore_scan_due": 0, "metastore_scan_err": 0, "metastore_set": 0, "metastore_set_err": 0 }, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 1024 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_timers_0": "[0-1023]" } }, "worker_pids": { "worker_timers_0": 84204 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 2, "agg_messages_sent_to_worker": 90147, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_deletion": 80640, "dcp_deletion_sent_to_worker": 40320, "dcp_mutation": 146147, "dcp_mutation_sent_to_worker": 40320, "dcp_mutation_suppressed_counter": 65507, "dcp_snapshot": 183572, "dcp_stream_req_counter": 10409, "dcp_stream_req_err_counter": 7433, "dcp_streamend": 1668, "dcp_streamreq": 2976, "dcp_system_event": 10240, "execution_stats": 1074, "failure_stats": 1074, "latency_stats": 1074, "lcb_exception_stats": 1074, "log_level": 1, "num_processed_events": 80640, "processed_events_size": 47071176, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 40320, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 40320, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 40320, "enqueued_dcp_mutation_msg_counter": 40320, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 90143, "num_processed_events": 80640, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 40320, "processed_events_size": 47071176, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "84244": "2021-02-13T06:57:27Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "84244": "2021-02-13T06:57:27Z" }, "v8worker_events_lost": 0 }, "function_id": 5264693, "function_name": "sbm", "gocb_creds_request_counter": 133, "internal_vb_distribution_stats": { "worker_sbm_0": "[0-1023]" }, "latency_percentile_stats": { "100": 26500, "50": 500, "80": 800, "90": 1100, "95": 1600, "99": 3800 }, "lcb_creds_request_counter": 80787, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 1024 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_sbm_0": "[0-1023]" } }, "worker_pids": { "worker_sbm_0": 84244 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 10, "agg_messages_sent_to_worker": 92330, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_deletion": 97652, "dcp_deletion_sent_to_worker": 41057, "dcp_mutation": 223480, "dcp_mutation_sent_to_worker": 41564, "dcp_snapshot": 184800, "dcp_stream_req_counter": 9925, "dcp_stream_req_err_counter": 6896, "dcp_streamend": 1715, "dcp_streamreq": 3029, "dcp_system_event": 10240, "execution_stats": 1053, "failure_stats": 1053, "latency_stats": 1053, "lcb_exception_stats": 1053, "log_level": 1, "num_processed_events": 82621, "processed_events_size": 48375788, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 80640, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 80640, "dcp_delete_msg_counter": 41057, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 41564, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 41057, "enqueued_dcp_mutation_msg_counter": 41564, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 1981, "lcb_retry_failure": 0, "messages_parsed": 92326, "num_processed_events": 82621, "on_delete_failure": 0, "on_delete_success": 40320, "on_update_failure": 0, "on_update_success": 40320, "processed_events_size": 48375788, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "84330": "2021-02-13T06:57:27Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "84330": "2021-02-13T06:57:27Z" }, "v8worker_events_lost": 0 }, "function_id": 1051053557, "function_name": "curl", "gocb_creds_request_counter": 133, "internal_vb_distribution_stats": { "worker_curl_0": "[0-1023]" }, "latency_percentile_stats": { "100": 318900, "50": 1400, "80": 1900, "90": 2500, "95": 3500, "99": 9100 }, "lcb_creds_request_counter": 80787, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 1024 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_curl_0": "[0-1023]" } }, "worker_pids": { "worker_curl_0": 84330 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 6, "agg_messages_sent_to_worker": 97733, "agg_queue_memory": 0, "agg_queue_memory_cap": 20971520, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_deletion": 96915, "dcp_deletion_sent_to_worker": 40320, "dcp_mutation": 158727, "dcp_mutation_sent_to_worker": 49580, "dcp_snapshot": 172339, "dcp_stream_req_counter": 8643, "dcp_stream_req_err_counter": 6515, "dcp_streamend": 820, "dcp_streamreq": 2128, "dcp_system_event": 10240, "execution_stats": 1032, "failure_stats": 1032, "latency_stats": 1032, "lcb_exception_stats": 1032, "log_level": 1, "num_processed_events": 89900, "processed_events_size": 55352604, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 40320, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 49580, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 40320, "enqueued_dcp_mutation_msg_counter": 49580, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 9260, "lcb_retry_failure": 0, "messages_parsed": 97729, "num_processed_events": 89900, "on_delete_failure": 0, "on_delete_success": 40320, "on_update_failure": 0, "on_update_success": 40320, "processed_events_size": 55352604, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "84371": "2021-02-13T06:57:27Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "84371": "2021-02-13T06:57:27Z" }, "v8worker_events_lost": 0 }, "function_id": 1145501157, "function_name": "n1ql", "gocb_creds_request_counter": 133, "internal_vb_distribution_stats": { "worker_n1ql_0": "[0-1023]" }, "latency_percentile_stats": { "100": 333100, "50": 2200, "80": 3400, "90": 5200, "95": 7900, "99": 18100 }, "lcb_creds_request_counter": 80787, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 1024 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_n1ql_0": "[0-1023]" } }, "worker_pids": { "worker_n1ql_0": 84371 } } ] [2021-02-12 22:57:32,680] - [eventing_base] [140097149204224] - INFO - Stats for Node 172.23.122.117 is [ { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 7, "agg_messages_sent_to_worker": 90147, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_deletion": 96915, "dcp_deletion_sent_to_worker": 40320, "dcp_mutation": 201600, "dcp_mutation_sent_to_worker": 40320, "dcp_snapshot": 184021, "dcp_stream_req_counter": 3336, "dcp_stream_req_err_counter": 247, "dcp_streamend": 1451, "dcp_streamreq": 3089, "dcp_system_event": 10240, "execution_stats": 1117, "failure_stats": 1117, "latency_stats": 1117, "lcb_exception_stats": 1117, "log_level": 1, "num_processed_events": 80640, "processed_events_size": 47060684, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 40320, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 40320, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 40320, "enqueued_dcp_mutation_msg_counter": 40320, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 90143, "num_processed_events": 80640, "on_delete_failure": 0, "on_delete_success": 40320, "on_update_failure": 0, "on_update_success": 40320, "processed_events_size": 47060684, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "84162": "2021-02-13T06:57:27Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "84162": "2021-02-13T06:57:27Z" }, "v8worker_events_lost": 0 }, "function_id": 3413234574, "function_name": "bucket_op", "gocb_creds_request_counter": 133, "internal_vb_distribution_stats": { "worker_bucket_op_0": "[0-1023]" }, "latency_percentile_stats": { "100": 262400, "50": 500, "80": 800, "90": 1000, "95": 1500, "99": 5200 }, "lcb_creds_request_counter": 80787, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 1024 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_bucket_op_0": "[0-1023]" } }, "worker_pids": { "worker_bucket_op_0": 84162 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 12, "agg_messages_sent_to_worker": 109370, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_deletion": 97933, "dcp_deletion_sent_to_worker": 41338, "dcp_mutation": 221975, "dcp_mutation_sent_to_worker": 58596, "dcp_snapshot": 183630, "dcp_stream_req_counter": 8318, "dcp_stream_req_err_counter": 5304, "dcp_streamend": 1460, "dcp_streamreq": 3014, "dcp_system_event": 10240, "execution_stats": 1096, "failure_stats": 1096, "latency_stats": 1096, "lcb_exception_stats": 1096, "log_level": 1, "num_processed_events": 99934, "processed_events_size": 63702196, "thr_count": 1, "thr_map": 1, "timer_events": 80640, "timer_responses_received": 80640, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 41338, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 58596, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 41338, "enqueued_dcp_mutation_msg_counter": 58596, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 19294, "lcb_retry_failure": 0, "messages_parsed": 109361, "num_processed_events": 99934, "on_delete_failure": 0, "on_delete_success": 40320, "on_update_failure": 0, "on_update_success": 40320, "processed_events_size": 63702196, "timer_callback_failure": 1, "timer_callback_success": 80639, "timer_cancel_counter": 0, "timer_create_counter": 80640, "timer_create_failure": 0, "timer_msg_counter": 80640, "timer_responses_sent": 0, "timestamp": { "84204": "2021-02-13T06:57:27Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 1, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "84204": "2021-02-13T06:57:27Z" }, "v8worker_events_lost": 0 }, "function_id": 1452344633, "function_name": "timers", "gocb_creds_request_counter": 133, "internal_vb_distribution_stats": { "worker_timers_0": "[0-1023]" }, "latency_percentile_stats": { "100": 511900, "50": 1300, "80": 1800, "90": 2400, "95": 3500, "99": 9800 }, "lcb_creds_request_counter": 80787, "lcb_exception_stats": { "210": 1 }, "metastore_stats": { "metastore_delete_err": 0, "metastore_deletes": 0, "metastore_not_found": 0, "metastore_scan": 0, "metastore_scan_due": 0, "metastore_scan_err": 0, "metastore_set": 0, "metastore_set_err": 0 }, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 1024 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_timers_0": "[0-1023]" } }, "worker_pids": { "worker_timers_0": 84204 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 2, "agg_messages_sent_to_worker": 90147, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_deletion": 80640, "dcp_deletion_sent_to_worker": 40320, "dcp_mutation": 146147, "dcp_mutation_sent_to_worker": 40320, "dcp_mutation_suppressed_counter": 65507, "dcp_snapshot": 183572, "dcp_stream_req_counter": 10409, "dcp_stream_req_err_counter": 7433, "dcp_streamend": 1668, "dcp_streamreq": 2976, "dcp_system_event": 10240, "execution_stats": 1074, "failure_stats": 1074, "latency_stats": 1074, "lcb_exception_stats": 1074, "log_level": 1, "num_processed_events": 80640, "processed_events_size": 47071176, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 40320, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 40320, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 40320, "enqueued_dcp_mutation_msg_counter": 40320, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 90143, "num_processed_events": 80640, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 40320, "processed_events_size": 47071176, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "84244": "2021-02-13T06:57:27Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "84244": "2021-02-13T06:57:27Z" }, "v8worker_events_lost": 0 }, "function_id": 5264693, "function_name": "sbm", "gocb_creds_request_counter": 133, "internal_vb_distribution_stats": { "worker_sbm_0": "[0-1023]" }, "latency_percentile_stats": { "100": 26500, "50": 500, "80": 800, "90": 1100, "95": 1600, "99": 3800 }, "lcb_creds_request_counter": 80787, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 1024 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_sbm_0": "[0-1023]" } }, "worker_pids": { "worker_sbm_0": 84244 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 10, "agg_messages_sent_to_worker": 92330, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_deletion": 97652, "dcp_deletion_sent_to_worker": 41057, "dcp_mutation": 223480, "dcp_mutation_sent_to_worker": 41564, "dcp_snapshot": 184800, "dcp_stream_req_counter": 9925, "dcp_stream_req_err_counter": 6896, "dcp_streamend": 1715, "dcp_streamreq": 3029, "dcp_system_event": 10240, "execution_stats": 1053, "failure_stats": 1053, "latency_stats": 1053, "lcb_exception_stats": 1053, "log_level": 1, "num_processed_events": 82621, "processed_events_size": 48375788, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 80640, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 80640, "dcp_delete_msg_counter": 41057, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 41564, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 41057, "enqueued_dcp_mutation_msg_counter": 41564, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 1981, "lcb_retry_failure": 0, "messages_parsed": 92326, "num_processed_events": 82621, "on_delete_failure": 0, "on_delete_success": 40320, "on_update_failure": 0, "on_update_success": 40320, "processed_events_size": 48375788, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "84330": "2021-02-13T06:57:27Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "84330": "2021-02-13T06:57:27Z" }, "v8worker_events_lost": 0 }, "function_id": 1051053557, "function_name": "curl", "gocb_creds_request_counter": 133, "internal_vb_distribution_stats": { "worker_curl_0": "[0-1023]" }, "latency_percentile_stats": { "100": 318900, "50": 1400, "80": 1900, "90": 2500, "95": 3500, "99": 9100 }, "lcb_creds_request_counter": 80787, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 1024 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_curl_0": "[0-1023]" } }, "worker_pids": { "worker_curl_0": 84330 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 6, "agg_messages_sent_to_worker": 97733, "agg_queue_memory": 0, "agg_queue_memory_cap": 20971520, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_deletion": 96915, "dcp_deletion_sent_to_worker": 40320, "dcp_mutation": 158727, "dcp_mutation_sent_to_worker": 49580, "dcp_snapshot": 172339, "dcp_stream_req_counter": 8643, "dcp_stream_req_err_counter": 6515, "dcp_streamend": 820, "dcp_streamreq": 2128, "dcp_system_event": 10240, "execution_stats": 1032, "failure_stats": 1032, "latency_stats": 1032, "lcb_exception_stats": 1032, "log_level": 1, "num_processed_events": 89900, "processed_events_size": 55352604, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 40320, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 49580, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 40320, "enqueued_dcp_mutation_msg_counter": 49580, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 9260, "lcb_retry_failure": 0, "messages_parsed": 97729, "num_processed_events": 89900, "on_delete_failure": 0, "on_delete_success": 40320, "on_update_failure": 0, "on_update_success": 40320, "processed_events_size": 55352604, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "84371": "2021-02-13T06:57:27Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "84371": "2021-02-13T06:57:27Z" }, "v8worker_events_lost": 0 }, "function_id": 1145501157, "function_name": "n1ql", "gocb_creds_request_counter": 133, "internal_vb_distribution_stats": { "worker_n1ql_0": "[0-1023]" }, "latency_percentile_stats": { "100": 333100, "50": 2200, "80": 3400, "90": 5200, "95": 7900, "99": 18100 }, "lcb_creds_request_counter": 80787, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 1024 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_n1ql_0": "[0-1023]" } }, "worker_pids": { "worker_n1ql_0": 84371 } } ] [2021-02-12 22:57:33,690] - [eventing_base] [140097149204224] - INFO - Execution stats update_success: 40320 update_failure: 0 delete_success: 40320 delete_failure: 0 [2021-02-12 22:57:33,709] - [basetestcase] [140097149204224] - INFO - list of eventing nodes in cluster: [ip:172.23.122.117 port:8091 ssh_username:root] [2021-02-12 22:57:33,757] - [eventing_base] [140097149204224] - INFO - Stats for Node 172.23.122.117 is [ { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 7, "agg_messages_sent_to_worker": 90177, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_deletion": 96915, "dcp_deletion_sent_to_worker": 40320, "dcp_mutation": 201600, "dcp_mutation_sent_to_worker": 40320, "dcp_snapshot": 184021, "dcp_stream_req_counter": 3336, "dcp_stream_req_err_counter": 247, "dcp_streamend": 1451, "dcp_streamreq": 3089, "dcp_system_event": 10240, "execution_stats": 1123, "failure_stats": 1123, "latency_stats": 1123, "lcb_exception_stats": 1123, "log_level": 1, "num_processed_events": 80640, "processed_events_size": 47060684, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 40320, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 40320, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 40320, "enqueued_dcp_mutation_msg_counter": 40320, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 90173, "num_processed_events": 80640, "on_delete_failure": 0, "on_delete_success": 40320, "on_update_failure": 0, "on_update_success": 40320, "processed_events_size": 47060684, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "84162": "2021-02-13T06:57:33Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "84162": "2021-02-13T06:57:33Z" }, "v8worker_events_lost": 0 }, "function_id": 3413234574, "function_name": "bucket_op", "gocb_creds_request_counter": 133, "internal_vb_distribution_stats": { "worker_bucket_op_0": "[0-1023]" }, "latency_percentile_stats": { "100": 262400, "50": 500, "80": 800, "90": 1000, "95": 1500, "99": 5200 }, "lcb_creds_request_counter": 80787, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 1024 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_bucket_op_0": "[0-1023]" } }, "worker_pids": { "worker_bucket_op_0": 84162 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 12, "agg_messages_sent_to_worker": 109400, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_deletion": 97933, "dcp_deletion_sent_to_worker": 41338, "dcp_mutation": 221975, "dcp_mutation_sent_to_worker": 58596, "dcp_snapshot": 183630, "dcp_stream_req_counter": 8318, "dcp_stream_req_err_counter": 5304, "dcp_streamend": 1460, "dcp_streamreq": 3014, "dcp_system_event": 10240, "execution_stats": 1102, "failure_stats": 1102, "latency_stats": 1102, "lcb_exception_stats": 1102, "log_level": 1, "num_processed_events": 99934, "processed_events_size": 63702196, "thr_count": 1, "thr_map": 1, "timer_events": 80640, "timer_responses_received": 80640, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 41338, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 58596, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 41338, "enqueued_dcp_mutation_msg_counter": 58596, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 19294, "lcb_retry_failure": 0, "messages_parsed": 109391, "num_processed_events": 99934, "on_delete_failure": 0, "on_delete_success": 40320, "on_update_failure": 0, "on_update_success": 40320, "processed_events_size": 63702196, "timer_callback_failure": 1, "timer_callback_success": 80639, "timer_cancel_counter": 0, "timer_create_counter": 80640, "timer_create_failure": 0, "timer_msg_counter": 80640, "timer_responses_sent": 0, "timestamp": { "84204": "2021-02-13T06:57:33Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 1, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "84204": "2021-02-13T06:57:33Z" }, "v8worker_events_lost": 0 }, "function_id": 1452344633, "function_name": "timers", "gocb_creds_request_counter": 133, "internal_vb_distribution_stats": { "worker_timers_0": "[0-1023]" }, "latency_percentile_stats": { "100": 511900, "50": 1300, "80": 1800, "90": 2400, "95": 3500, "99": 9800 }, "lcb_creds_request_counter": 80787, "lcb_exception_stats": { "210": 1 }, "metastore_stats": { "metastore_delete_err": 0, "metastore_deletes": 0, "metastore_not_found": 0, "metastore_scan": 0, "metastore_scan_due": 0, "metastore_scan_err": 0, "metastore_set": 0, "metastore_set_err": 0 }, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 1024 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_timers_0": "[0-1023]" } }, "worker_pids": { "worker_timers_0": 84204 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 2, "agg_messages_sent_to_worker": 90177, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_deletion": 80640, "dcp_deletion_sent_to_worker": 40320, "dcp_mutation": 146147, "dcp_mutation_sent_to_worker": 40320, "dcp_mutation_suppressed_counter": 65507, "dcp_snapshot": 183572, "dcp_stream_req_counter": 10409, "dcp_stream_req_err_counter": 7433, "dcp_streamend": 1668, "dcp_streamreq": 2976, "dcp_system_event": 10240, "execution_stats": 1080, "failure_stats": 1080, "latency_stats": 1080, "lcb_exception_stats": 1080, "log_level": 1, "num_processed_events": 80640, "processed_events_size": 47071176, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 40320, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 40320, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 40320, "enqueued_dcp_mutation_msg_counter": 40320, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 90173, "num_processed_events": 80640, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 40320, "processed_events_size": 47071176, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "84244": "2021-02-13T06:57:33Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "84244": "2021-02-13T06:57:33Z" }, "v8worker_events_lost": 0 }, "function_id": 5264693, "function_name": "sbm", "gocb_creds_request_counter": 133, "internal_vb_distribution_stats": { "worker_sbm_0": "[0-1023]" }, "latency_percentile_stats": { "100": 26500, "50": 500, "80": 800, "90": 1100, "95": 1600, "99": 3800 }, "lcb_creds_request_counter": 80787, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 1024 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_sbm_0": "[0-1023]" } }, "worker_pids": { "worker_sbm_0": 84244 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 10, "agg_messages_sent_to_worker": 92360, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_deletion": 97652, "dcp_deletion_sent_to_worker": 41057, "dcp_mutation": 223480, "dcp_mutation_sent_to_worker": 41564, "dcp_snapshot": 184800, "dcp_stream_req_counter": 9925, "dcp_stream_req_err_counter": 6896, "dcp_streamend": 1715, "dcp_streamreq": 3029, "dcp_system_event": 10240, "execution_stats": 1059, "failure_stats": 1059, "latency_stats": 1059, "lcb_exception_stats": 1059, "log_level": 1, "num_processed_events": 82621, "processed_events_size": 48375788, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 80640, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 80640, "dcp_delete_msg_counter": 41057, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 41564, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 41057, "enqueued_dcp_mutation_msg_counter": 41564, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 1981, "lcb_retry_failure": 0, "messages_parsed": 92356, "num_processed_events": 82621, "on_delete_failure": 0, "on_delete_success": 40320, "on_update_failure": 0, "on_update_success": 40320, "processed_events_size": 48375788, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "84330": "2021-02-13T06:57:33Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "84330": "2021-02-13T06:57:33Z" }, "v8worker_events_lost": 0 }, "function_id": 1051053557, "function_name": "curl", "gocb_creds_request_counter": 133, "internal_vb_distribution_stats": { "worker_curl_0": "[0-1023]" }, "latency_percentile_stats": { "100": 318900, "50": 1400, "80": 1900, "90": 2500, "95": 3500, "99": 9100 }, "lcb_creds_request_counter": 80787, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 1024 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_curl_0": "[0-1023]" } }, "worker_pids": { "worker_curl_0": 84330 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 6, "agg_messages_sent_to_worker": 97763, "agg_queue_memory": 0, "agg_queue_memory_cap": 20971520, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_deletion": 96915, "dcp_deletion_sent_to_worker": 40320, "dcp_mutation": 158727, "dcp_mutation_sent_to_worker": 49580, "dcp_snapshot": 172339, "dcp_stream_req_counter": 8643, "dcp_stream_req_err_counter": 6515, "dcp_streamend": 820, "dcp_streamreq": 2128, "dcp_system_event": 10240, "execution_stats": 1038, "failure_stats": 1038, "latency_stats": 1038, "lcb_exception_stats": 1038, "log_level": 1, "num_processed_events": 89900, "processed_events_size": 55352604, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 40320, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 49580, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 40320, "enqueued_dcp_mutation_msg_counter": 49580, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 9260, "lcb_retry_failure": 0, "messages_parsed": 97759, "num_processed_events": 89900, "on_delete_failure": 0, "on_delete_success": 40320, "on_update_failure": 0, "on_update_success": 40320, "processed_events_size": 55352604, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "84371": "2021-02-13T06:57:33Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "84371": "2021-02-13T06:57:33Z" }, "v8worker_events_lost": 0 }, "function_id": 1145501157, "function_name": "n1ql", "gocb_creds_request_counter": 133, "internal_vb_distribution_stats": { "worker_n1ql_0": "[0-1023]" }, "latency_percentile_stats": { "100": 333100, "50": 2200, "80": 3400, "90": 5200, "95": 7900, "99": 18100 }, "lcb_creds_request_counter": 80787, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 1024 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_n1ql_0": "[0-1023]" } }, "worker_pids": { "worker_n1ql_0": 84371 } } ] [2021-02-12 22:57:33,771] - [basetestcase] [140097149204224] - INFO - list of eventing nodes in cluster: [ip:172.23.122.117 port:8091 ssh_username:root] [2021-02-12 22:57:33,773] - [remote_util] [140097149204224] - INFO - SSH Connecting to 172.23.122.117 with username:root, attempt#1 of 5 [2021-02-12 22:57:33,908] - [remote_util] [140097149204224] - INFO - SSH Connected to 172.23.122.117 as root [2021-02-12 22:57:34,167] - [remote_util] [140097149204224] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 22:57:34,459] - [remote_util] [140097149204224] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 22:57:34,467] - [rest_client] [140097149204224] - INFO - /diag/eval status on 172.23.122.117:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). [2021-02-12 22:57:34,467] - [remote_util] [140097149204224] - INFO - running command.raw on 172.23.122.117: zgrep "panic" "/opt/couchbase/var/lib/couchbase/logs"/eventing.log* | wc -l [2021-02-12 22:57:35,491] - [remote_util] [140097149204224] - INFO - command executed successfully with root [2021-02-12 22:57:35,491] - [remote_util] [140097149204224] - INFO - running command.raw on 172.23.122.117: ls "/opt/couchbase/var/lib/couchbase/logs"/../crash/| wc -l [2021-02-12 22:57:35,544] - [remote_util] [140097149204224] - INFO - command executed successfully with root [2021-02-12 22:57:35,835] - [basetestcase] [140097149204224] - INFO - ------- Cluster statistics ------- [2021-02-12 22:57:35,835] - [basetestcase] [140097149204224] - INFO - 172.23.122.117:8091 => {'services': ['eventing'], 'cpu_utilization': 11.68562564632885, 'mem_free': 2733871104, 'mem_total': 4201684992, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-02-12 22:57:35,835] - [basetestcase] [140097149204224] - INFO - 172.23.122.90:8091 => {'services': ['kv'], 'cpu_utilization': 4.550050556117291, 'mem_free': 2970234880, 'mem_total': 4201684992, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-02-12 22:57:35,835] - [basetestcase] [140097149204224] - INFO - 172.23.122.97:8091 => {'services': ['index', 'n1ql'], 'cpu_utilization': 1.778455284552846, 'mem_free': 3095359488, 'mem_total': 4201684992, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-02-12 22:57:35,835] - [basetestcase] [140097149204224] - INFO - 172.23.123.61:8091 => {'services': ['kv'], 'cpu_utilization': 5.556965237249429, 'mem_free': 2863013888, 'mem_total': 4201672704, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-02-12 22:57:35,835] - [basetestcase] [140097149204224] - INFO - 172.23.123.62:8091 => {'services': ['kv'], 'cpu_utilization': 5.360303413400758, 'mem_free': 2892578816, 'mem_total': 4201676800, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-02-12 22:57:35,835] - [basetestcase] [140097149204224] - INFO - --- End of cluster statistics --- [2021-02-12 22:57:35,836] - [remote_util] [140097149204224] - INFO - SSH Connecting to 172.23.123.61 with username:root, attempt#1 of 5 [2021-02-12 22:57:35,942] - [remote_util] [140097149204224] - INFO - SSH Connected to 172.23.123.61 as root [2021-02-12 22:57:36,213] - [remote_util] [140097149204224] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 22:57:36,512] - [remote_util] [140097149204224] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 22:57:36,514] - [remote_util] [140097149204224] - INFO - SSH Connecting to 172.23.123.62 with username:root, attempt#1 of 5 [2021-02-12 22:57:36,618] - [remote_util] [140097149204224] - INFO - SSH Connected to 172.23.123.62 as root [2021-02-12 22:57:36,899] - [remote_util] [140097149204224] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 22:57:37,197] - [remote_util] [140097149204224] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 22:57:37,198] - [remote_util] [140097149204224] - INFO - SSH Connecting to 172.23.122.90 with username:root, attempt#1 of 5 [2021-02-12 22:57:37,304] - [remote_util] [140097149204224] - INFO - SSH Connected to 172.23.122.90 as root [2021-02-12 22:57:37,598] - [remote_util] [140097149204224] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 22:57:37,896] - [remote_util] [140097149204224] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 22:57:37,898] - [remote_util] [140097149204224] - INFO - SSH Connecting to 172.23.122.117 with username:root, attempt#1 of 5 [2021-02-12 22:57:38,001] - [remote_util] [140097149204224] - INFO - SSH Connected to 172.23.122.117 as root [2021-02-12 22:57:38,255] - [remote_util] [140097149204224] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 22:57:38,549] - [remote_util] [140097149204224] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 22:57:38,551] - [remote_util] [140097149204224] - INFO - SSH Connecting to 172.23.122.97 with username:root, attempt#1 of 5 [2021-02-12 22:57:38,658] - [remote_util] [140097149204224] - INFO - SSH Connected to 172.23.122.97 as root [2021-02-12 22:57:38,935] - [remote_util] [140097149204224] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 22:57:39,239] - [remote_util] [140097149204224] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 22:57:39,240] - [remote_util] [140097149204224] - INFO - SSH Connecting to 172.23.122.82 with username:root, attempt#1 of 5 [2021-02-12 22:57:39,349] - [remote_util] [140097149204224] - INFO - SSH Connected to 172.23.122.82 as root [2021-02-12 22:57:39,637] - [remote_util] [140097149204224] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 22:57:39,936] - [remote_util] [140097149204224] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 22:57:39,938] - [remote_util] [140097149204224] - INFO - SSH Connecting to 172.23.122.118 with username:root, attempt#1 of 5 [2021-02-12 22:57:40,042] - [remote_util] [140097149204224] - INFO - SSH Connected to 172.23.122.118 as root [2021-02-12 22:57:40,293] - [remote_util] [140097149204224] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 22:57:40,586] - [remote_util] [140097149204224] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 22:57:40,588] - [remote_util] [140097149204224] - INFO - SSH Connecting to 172.23.122.116 with username:root, attempt#1 of 5 [2021-02-12 22:57:40,691] - [remote_util] [140097149204224] - INFO - SSH Connected to 172.23.122.116 as root [2021-02-12 22:57:40,972] - [remote_util] [140097149204224] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 22:57:41,269] - [remote_util] [140097149204224] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 22:59:03,282] - [basetestcase] [140097149204224] - INFO - ============== basetestcase cleanup was started for test #8 test_eventing_rebalance_with_multiple_kv_nodes ============== [2021-02-12 22:59:03,344] - [bucket_helper] [140097149204224] - INFO - deleting existing buckets ['dst_bucket', 'metadata', 'src_bucket'] on 172.23.123.61 [2021-02-12 22:59:04,539] - [bucket_helper] [140097149204224] - INFO - waiting for bucket deletion to complete.... [2021-02-12 22:59:04,561] - [rest_client] [140097149204224] - INFO - node 172.23.123.61 existing buckets : ['metadata', 'src_bucket'] [2021-02-12 22:59:04,562] - [bucket_helper] [140097149204224] - INFO - deleted bucket : dst_bucket from 172.23.123.61 [2021-02-12 22:59:04,993] - [bucket_helper] [140097149204224] - INFO - waiting for bucket deletion to complete.... [2021-02-12 22:59:05,005] - [rest_client] [140097149204224] - INFO - node 172.23.123.61 existing buckets : ['src_bucket'] [2021-02-12 22:59:05,005] - [bucket_helper] [140097149204224] - INFO - deleted bucket : metadata from 172.23.123.61 [2021-02-12 22:59:06,185] - [bucket_helper] [140097149204224] - INFO - waiting for bucket deletion to complete.... [2021-02-12 22:59:06,191] - [rest_client] [140097149204224] - INFO - node 172.23.123.61 existing buckets : [] [2021-02-12 22:59:06,192] - [bucket_helper] [140097149204224] - INFO - deleted bucket : src_bucket from 172.23.123.61 [2021-02-12 22:59:06,217] - [bucket_helper] [140097149204224] - INFO - Could not find any buckets for node 172.23.123.62, nothing to delete [2021-02-12 22:59:06,226] - [bucket_helper] [140097149204224] - INFO - Could not find any buckets for node 172.23.122.90, nothing to delete [2021-02-12 22:59:06,246] - [bucket_helper] [140097149204224] - INFO - Could not find any buckets for node 172.23.122.117, nothing to delete [2021-02-12 22:59:06,254] - [bucket_helper] [140097149204224] - INFO - Could not find any buckets for node 172.23.122.97, nothing to delete [2021-02-12 22:59:06,261] - [bucket_helper] [140097149204224] - INFO - Could not find any buckets for node 172.23.122.82, nothing to delete [2021-02-12 22:59:06,266] - [bucket_helper] [140097149204224] - INFO - Could not find any buckets for node 172.23.122.118, nothing to delete [2021-02-12 22:59:06,271] - [bucket_helper] [140097149204224] - INFO - Could not find any buckets for node 172.23.122.116, nothing to delete [2021-02-12 22:59:06,313] - [rest_client] [140097149204224] - INFO - -->is_ns_server_running? [2021-02-12 22:59:06,356] - [cluster_helper] [140097149204224] - INFO - rebalancing all nodes in order to remove nodes [2021-02-12 22:59:06,383] - [rest_client] [140097149204224] - INFO - rebalance params : {'knownNodes': 'ns_1@172.23.122.117,ns_1@172.23.122.90,ns_1@172.23.122.97,ns_1@172.23.123.61,ns_1@172.23.123.62', 'ejectedNodes': 'ns_1@172.23.122.117,ns_1@172.23.122.90,ns_1@172.23.122.97,ns_1@172.23.123.62', 'user': 'Administrator', 'password': 'password'} [2021-02-12 22:59:06,424] - [rest_client] [140097149204224] - INFO - rebalance operation started [2021-02-12 22:59:06,454] - [rest_client] [140097149204224] - INFO - rebalance percentage : 0.00 % [2021-02-12 22:59:16,464] - [rest_client] [140097149204224] - INFO - rebalance percentage : 70.00 % [2021-02-12 22:59:26,477] - [rest_client] [140097149204224] - INFO - rebalance percentage : 90.00 % [2021-02-12 22:59:46,494] - [rest_client] [140097149204224] - INFO - rebalance progress took 40.07 seconds [2021-02-12 22:59:46,494] - [rest_client] [140097149204224] - INFO - sleep for 10 seconds after rebalance... [2021-02-12 22:59:56,527] - [cluster_helper] [140097149204224] - INFO - removed all the nodes from cluster associated with ip:172.23.123.61 port:8091 ssh_username:root ? [('ns_1@172.23.122.117', 8091), ('ns_1@172.23.122.90', 8091), ('ns_1@172.23.122.97', 8091), ('ns_1@172.23.123.62', 8091)] [2021-02-12 22:59:56,533] - [cluster_helper] [140097149204224] - INFO - waiting for ns_server @ 172.23.123.61:8091 [2021-02-12 22:59:56,533] - [rest_client] [140097149204224] - INFO - -->is_ns_server_running? [2021-02-12 22:59:56,537] - [cluster_helper] [140097149204224] - INFO - ns_server @ 172.23.123.61:8091 is running [2021-02-12 22:59:56,540] - [cluster_helper] [140097149204224] - INFO - waiting for ns_server @ 172.23.123.62:8091 [2021-02-12 22:59:56,540] - [rest_client] [140097149204224] - INFO - -->is_ns_server_running? [2021-02-12 22:59:56,543] - [cluster_helper] [140097149204224] - INFO - ns_server @ 172.23.123.62:8091 is running [2021-02-12 22:59:56,546] - [cluster_helper] [140097149204224] - INFO - waiting for ns_server @ 172.23.122.90:8091 [2021-02-12 22:59:56,546] - [rest_client] [140097149204224] - INFO - -->is_ns_server_running? [2021-02-12 22:59:56,549] - [cluster_helper] [140097149204224] - INFO - ns_server @ 172.23.122.90:8091 is running [2021-02-12 22:59:56,552] - [cluster_helper] [140097149204224] - INFO - waiting for ns_server @ 172.23.122.117:8091 [2021-02-12 22:59:56,552] - [rest_client] [140097149204224] - INFO - -->is_ns_server_running? [2021-02-12 22:59:56,555] - [cluster_helper] [140097149204224] - INFO - ns_server @ 172.23.122.117:8091 is running [2021-02-12 22:59:56,558] - [cluster_helper] [140097149204224] - INFO - waiting for ns_server @ 172.23.122.97:8091 [2021-02-12 22:59:56,558] - [rest_client] [140097149204224] - INFO - -->is_ns_server_running? [2021-02-12 22:59:56,561] - [cluster_helper] [140097149204224] - INFO - ns_server @ 172.23.122.97:8091 is running [2021-02-12 22:59:56,564] - [cluster_helper] [140097149204224] - INFO - waiting for ns_server @ 172.23.122.82:8091 [2021-02-12 22:59:56,564] - [rest_client] [140097149204224] - INFO - -->is_ns_server_running? [2021-02-12 22:59:56,567] - [cluster_helper] [140097149204224] - INFO - ns_server @ 172.23.122.82:8091 is running [2021-02-12 22:59:56,571] - [cluster_helper] [140097149204224] - INFO - waiting for ns_server @ 172.23.122.118:8091 [2021-02-12 22:59:56,571] - [rest_client] [140097149204224] - INFO - -->is_ns_server_running? [2021-02-12 22:59:56,574] - [cluster_helper] [140097149204224] - INFO - ns_server @ 172.23.122.118:8091 is running [2021-02-12 22:59:56,577] - [cluster_helper] [140097149204224] - INFO - waiting for ns_server @ 172.23.122.116:8091 [2021-02-12 22:59:56,577] - [rest_client] [140097149204224] - INFO - -->is_ns_server_running? [2021-02-12 22:59:56,580] - [cluster_helper] [140097149204224] - INFO - ns_server @ 172.23.122.116:8091 is running [2021-02-12 22:59:56,580] - [basetestcase] [140097149204224] - INFO - ============== basetestcase cleanup was finished for test #8 test_eventing_rebalance_with_multiple_kv_nodes ============== [2021-02-12 22:59:56,581] - [basetestcase] [140097149204224] - INFO - closing all ssh connections [2021-02-12 22:59:56,583] - [basetestcase] [140097149204224] - INFO - closing all memcached connections [2021-02-12 22:59:56,584] - [remote_util] [140097149204224] - INFO - SSH Connecting to 172.23.123.61 with username:root, attempt#1 of 5 [2021-02-12 22:59:56,686] - [remote_util] [140097149204224] - INFO - SSH Connected to 172.23.123.61 as root [2021-02-12 22:59:56,956] - [remote_util] [140097149204224] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 22:59:57,252] - [remote_util] [140097149204224] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 22:59:57,264] - [remote_util] [140097149204224] - INFO - SSH Connecting to 172.23.123.61 with username:root, attempt#1 of 5 [2021-02-12 22:59:57,362] - [remote_util] [140097149204224] - INFO - SSH Connected to 172.23.123.61 as root [2021-02-12 22:59:57,621] - [remote_util] [140097149204224] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 22:59:57,911] - [remote_util] [140097149204224] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 22:59:57,970] - [remote_util] [140097149204224] - INFO - 172.23.123.61 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 22:59:58,021] - [remote_util] [140097149204224] - INFO - Running systemd command on this server [2021-02-12 22:59:58,021] - [remote_util] [140097149204224] - INFO - running command.raw on 172.23.123.61: systemctl stop couchbase-server.service [2021-02-12 22:59:59,852] - [remote_util] [140097149204224] - INFO - command executed successfully with root [2021-02-12 22:59:59,852] - [basetestcase] [140097149204224] - INFO - Couchbase stopped [2021-02-12 22:59:59,852] - [basetestcase] [140097149204224] - INFO - sleep for 5 secs. ... [2021-02-12 23:00:04,857] - [remote_util] [140097149204224] - INFO - running command.raw on 172.23.123.61: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-12 23:00:04,874] - [remote_util] [140097149204224] - INFO - command executed successfully with root [2021-02-12 23:00:04,874] - [remote_util] [140097149204224] - INFO - running command.raw on 172.23.123.61: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-12 23:00:04,934] - [remote_util] [140097149204224] - INFO - command executed successfully with root [2021-02-12 23:00:04,936] - [remote_util] [140097149204224] - INFO - SSH Connecting to 172.23.123.61 with username:root, attempt#1 of 5 [2021-02-12 23:00:05,037] - [remote_util] [140097149204224] - INFO - SSH Connected to 172.23.123.61 as root [2021-02-12 23:00:05,309] - [remote_util] [140097149204224] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:00:05,605] - [remote_util] [140097149204224] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:00:05,625] - [remote_util] [140097149204224] - INFO - 172.23.123.61 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 23:00:05,625] - [remote_util] [140097149204224] - INFO - Checking for process beam.smp on linux [2021-02-12 23:00:05,660] - [remote_util] [140097149204224] - INFO - Starting couchbase server [2021-02-12 23:00:05,755] - [remote_util] [140097149204224] - INFO - Running systemd command on this server [2021-02-12 23:00:05,755] - [remote_util] [140097149204224] - INFO - running command.raw on 172.23.123.61: systemctl start couchbase-server.service [2021-02-12 23:00:05,777] - [remote_util] [140097149204224] - INFO - command executed successfully with root [2021-02-12 23:00:05,777] - [remote_util] [140097149204224] - INFO - 172.23.123.61:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-12 23:00:10,782] - [remote_util] [140097149204224] - INFO - running command.raw on 172.23.123.61: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-12 23:00:10,809] - [remote_util] [140097149204224] - INFO - command executed successfully with root [2021-02-12 23:00:10,809] - [remote_util] [140097149204224] - INFO - Couchbase server status: [' Process: 129383 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-12 23:00:10,809] - [remote_util] [140097149204224] - INFO - Checking for process beam.smp on linux [2021-02-12 23:00:10,876] - [remote_util] [140097149204224] - INFO - process beam.smp is running on 172.23.123.61: with pid 129528 [2021-02-12 23:00:10,876] - [basetestcase] [140097149204224] - INFO - Couchbase started [2021-02-12 23:00:10,878] - [remote_util] [140097149204224] - INFO - SSH Connecting to 172.23.123.62 with username:root, attempt#1 of 5 [2021-02-12 23:00:10,987] - [remote_util] [140097149204224] - INFO - SSH Connected to 172.23.123.62 as root [2021-02-12 23:00:11,262] - [remote_util] [140097149204224] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:00:11,563] - [remote_util] [140097149204224] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:00:11,571] - [remote_util] [140097149204224] - INFO - SSH Connecting to 172.23.123.62 with username:root, attempt#1 of 5 [2021-02-12 23:00:11,674] - [remote_util] [140097149204224] - INFO - SSH Connected to 172.23.123.62 as root [2021-02-12 23:00:11,953] - [remote_util] [140097149204224] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:00:12,257] - [remote_util] [140097149204224] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:00:12,276] - [remote_util] [140097149204224] - INFO - 172.23.123.62 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 23:00:12,331] - [remote_util] [140097149204224] - INFO - Running systemd command on this server [2021-02-12 23:00:12,331] - [remote_util] [140097149204224] - INFO - running command.raw on 172.23.123.62: systemctl stop couchbase-server.service [2021-02-12 23:00:13,889] - [remote_util] [140097149204224] - INFO - command executed successfully with root [2021-02-12 23:00:13,889] - [basetestcase] [140097149204224] - INFO - Couchbase stopped [2021-02-12 23:00:13,890] - [basetestcase] [140097149204224] - INFO - sleep for 5 secs. ... [2021-02-12 23:00:18,895] - [remote_util] [140097149204224] - INFO - running command.raw on 172.23.123.62: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-12 23:00:18,911] - [remote_util] [140097149204224] - INFO - command executed successfully with root [2021-02-12 23:00:18,911] - [remote_util] [140097149204224] - INFO - running command.raw on 172.23.123.62: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-12 23:00:18,967] - [remote_util] [140097149204224] - INFO - command executed successfully with root [2021-02-12 23:00:18,968] - [remote_util] [140097149204224] - INFO - SSH Connecting to 172.23.123.62 with username:root, attempt#1 of 5 [2021-02-12 23:00:19,074] - [remote_util] [140097149204224] - INFO - SSH Connected to 172.23.123.62 as root [2021-02-12 23:00:19,355] - [remote_util] [140097149204224] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:00:19,653] - [remote_util] [140097149204224] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:00:19,712] - [remote_util] [140097149204224] - INFO - 172.23.123.62 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 23:00:19,712] - [remote_util] [140097149204224] - INFO - Checking for process beam.smp on linux [2021-02-12 23:00:19,748] - [remote_util] [140097149204224] - INFO - Starting couchbase server [2021-02-12 23:00:19,843] - [remote_util] [140097149204224] - INFO - Running systemd command on this server [2021-02-12 23:00:19,843] - [remote_util] [140097149204224] - INFO - running command.raw on 172.23.123.62: systemctl start couchbase-server.service [2021-02-12 23:00:19,864] - [remote_util] [140097149204224] - INFO - command executed successfully with root [2021-02-12 23:00:19,864] - [remote_util] [140097149204224] - INFO - 172.23.123.62:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-12 23:00:24,869] - [remote_util] [140097149204224] - INFO - running command.raw on 172.23.123.62: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-12 23:00:24,896] - [remote_util] [140097149204224] - INFO - command executed successfully with root [2021-02-12 23:00:24,896] - [remote_util] [140097149204224] - INFO - Couchbase server status: [' Process: 19349 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-12 23:00:24,896] - [remote_util] [140097149204224] - INFO - Checking for process beam.smp on linux [2021-02-12 23:00:24,966] - [remote_util] [140097149204224] - INFO - process beam.smp is running on 172.23.123.62: with pid 19536 [2021-02-12 23:00:24,967] - [basetestcase] [140097149204224] - INFO - Couchbase started [2021-02-12 23:00:24,969] - [remote_util] [140097149204224] - INFO - SSH Connecting to 172.23.122.90 with username:root, attempt#1 of 5 [2021-02-12 23:00:25,077] - [remote_util] [140097149204224] - INFO - SSH Connected to 172.23.122.90 as root [2021-02-12 23:00:25,358] - [remote_util] [140097149204224] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:00:25,651] - [remote_util] [140097149204224] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:00:25,659] - [remote_util] [140097149204224] - INFO - SSH Connecting to 172.23.122.90 with username:root, attempt#1 of 5 [2021-02-12 23:00:25,757] - [remote_util] [140097149204224] - INFO - SSH Connected to 172.23.122.90 as root [2021-02-12 23:00:26,024] - [remote_util] [140097149204224] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:00:26,312] - [remote_util] [140097149204224] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:00:26,332] - [remote_util] [140097149204224] - INFO - 172.23.122.90 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 23:00:26,384] - [remote_util] [140097149204224] - INFO - Running systemd command on this server [2021-02-12 23:00:26,384] - [remote_util] [140097149204224] - INFO - running command.raw on 172.23.122.90: systemctl stop couchbase-server.service [2021-02-12 23:00:28,039] - [remote_util] [140097149204224] - INFO - command executed successfully with root [2021-02-12 23:00:28,039] - [basetestcase] [140097149204224] - INFO - Couchbase stopped [2021-02-12 23:00:28,039] - [basetestcase] [140097149204224] - INFO - sleep for 5 secs. ... [2021-02-12 23:00:33,044] - [remote_util] [140097149204224] - INFO - running command.raw on 172.23.122.90: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-12 23:00:33,059] - [remote_util] [140097149204224] - INFO - command executed successfully with root [2021-02-12 23:00:33,059] - [remote_util] [140097149204224] - INFO - running command.raw on 172.23.122.90: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-12 23:00:33,113] - [remote_util] [140097149204224] - INFO - command executed successfully with root [2021-02-12 23:00:33,114] - [remote_util] [140097149204224] - INFO - SSH Connecting to 172.23.122.90 with username:root, attempt#1 of 5 [2021-02-12 23:00:33,217] - [remote_util] [140097149204224] - INFO - SSH Connected to 172.23.122.90 as root [2021-02-12 23:00:33,495] - [remote_util] [140097149204224] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:00:33,789] - [remote_util] [140097149204224] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:00:33,807] - [remote_util] [140097149204224] - INFO - 172.23.122.90 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 23:00:33,808] - [remote_util] [140097149204224] - INFO - Checking for process beam.smp on linux [2021-02-12 23:00:33,845] - [remote_util] [140097149204224] - INFO - Starting couchbase server [2021-02-12 23:00:33,937] - [remote_util] [140097149204224] - INFO - Running systemd command on this server [2021-02-12 23:00:33,937] - [remote_util] [140097149204224] - INFO - running command.raw on 172.23.122.90: systemctl start couchbase-server.service [2021-02-12 23:00:33,959] - [remote_util] [140097149204224] - INFO - command executed successfully with root [2021-02-12 23:00:33,959] - [remote_util] [140097149204224] - INFO - 172.23.122.90:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-12 23:00:38,964] - [remote_util] [140097149204224] - INFO - running command.raw on 172.23.122.90: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-12 23:00:38,989] - [remote_util] [140097149204224] - INFO - command executed successfully with root [2021-02-12 23:00:38,989] - [remote_util] [140097149204224] - INFO - Couchbase server status: [' Process: 128585 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-12 23:00:38,989] - [remote_util] [140097149204224] - INFO - Checking for process beam.smp on linux [2021-02-12 23:00:39,057] - [remote_util] [140097149204224] - INFO - process beam.smp is running on 172.23.122.90: with pid 128727 [2021-02-12 23:00:39,057] - [basetestcase] [140097149204224] - INFO - Couchbase started [2021-02-12 23:00:39,059] - [remote_util] [140097149204224] - INFO - SSH Connecting to 172.23.122.117 with username:root, attempt#1 of 5 [2021-02-12 23:00:39,161] - [remote_util] [140097149204224] - INFO - SSH Connected to 172.23.122.117 as root [2021-02-12 23:00:39,417] - [remote_util] [140097149204224] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:00:39,714] - [remote_util] [140097149204224] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:00:39,722] - [remote_util] [140097149204224] - INFO - SSH Connecting to 172.23.122.117 with username:root, attempt#1 of 5 [2021-02-12 23:00:39,822] - [remote_util] [140097149204224] - INFO - SSH Connected to 172.23.122.117 as root [2021-02-12 23:00:40,073] - [remote_util] [140097149204224] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:00:40,365] - [remote_util] [140097149204224] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:00:40,384] - [remote_util] [140097149204224] - INFO - 172.23.122.117 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 23:00:40,435] - [remote_util] [140097149204224] - INFO - Running systemd command on this server [2021-02-12 23:00:40,435] - [remote_util] [140097149204224] - INFO - running command.raw on 172.23.122.117: systemctl stop couchbase-server.service [2021-02-12 23:00:42,232] - [remote_util] [140097149204224] - INFO - command executed successfully with root [2021-02-12 23:00:42,232] - [basetestcase] [140097149204224] - INFO - Couchbase stopped [2021-02-12 23:00:42,233] - [basetestcase] [140097149204224] - INFO - sleep for 5 secs. ... [2021-02-12 23:00:47,238] - [remote_util] [140097149204224] - INFO - running command.raw on 172.23.122.117: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-12 23:00:47,278] - [remote_util] [140097149204224] - INFO - command executed successfully with root [2021-02-12 23:00:47,278] - [remote_util] [140097149204224] - INFO - running command.raw on 172.23.122.117: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-12 23:00:47,331] - [remote_util] [140097149204224] - INFO - command executed successfully with root [2021-02-12 23:00:47,333] - [remote_util] [140097149204224] - INFO - SSH Connecting to 172.23.122.117 with username:root, attempt#1 of 5 [2021-02-12 23:00:47,439] - [remote_util] [140097149204224] - INFO - SSH Connected to 172.23.122.117 as root [2021-02-12 23:00:47,705] - [remote_util] [140097149204224] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:00:48,006] - [remote_util] [140097149204224] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:00:48,026] - [remote_util] [140097149204224] - INFO - 172.23.122.117 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 23:00:48,026] - [remote_util] [140097149204224] - INFO - Checking for process beam.smp on linux [2021-02-12 23:00:48,063] - [remote_util] [140097149204224] - INFO - Starting couchbase server [2021-02-12 23:00:48,157] - [remote_util] [140097149204224] - INFO - Running systemd command on this server [2021-02-12 23:00:48,157] - [remote_util] [140097149204224] - INFO - running command.raw on 172.23.122.117: systemctl start couchbase-server.service [2021-02-12 23:00:48,179] - [remote_util] [140097149204224] - INFO - command executed successfully with root [2021-02-12 23:00:48,180] - [remote_util] [140097149204224] - INFO - 172.23.122.117:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-12 23:00:53,184] - [remote_util] [140097149204224] - INFO - running command.raw on 172.23.122.117: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-12 23:00:53,210] - [remote_util] [140097149204224] - INFO - command executed successfully with root [2021-02-12 23:00:53,210] - [remote_util] [140097149204224] - INFO - Couchbase server status: [' Process: 129348 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-12 23:00:53,211] - [remote_util] [140097149204224] - INFO - Checking for process beam.smp on linux [2021-02-12 23:00:53,279] - [remote_util] [140097149204224] - INFO - process beam.smp is running on 172.23.122.117: with pid 129493 [2021-02-12 23:00:53,279] - [basetestcase] [140097149204224] - INFO - Couchbase started [2021-02-12 23:00:53,281] - [remote_util] [140097149204224] - INFO - SSH Connecting to 172.23.122.97 with username:root, attempt#1 of 5 [2021-02-12 23:00:53,385] - [remote_util] [140097149204224] - INFO - SSH Connected to 172.23.122.97 as root [2021-02-12 23:00:53,647] - [remote_util] [140097149204224] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:00:53,945] - [remote_util] [140097149204224] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:00:53,953] - [remote_util] [140097149204224] - INFO - SSH Connecting to 172.23.122.97 with username:root, attempt#1 of 5 [2021-02-12 23:00:54,049] - [remote_util] [140097149204224] - INFO - SSH Connected to 172.23.122.97 as root [2021-02-12 23:00:54,300] - [remote_util] [140097149204224] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:00:54,594] - [remote_util] [140097149204224] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:00:54,653] - [remote_util] [140097149204224] - INFO - 172.23.122.97 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 23:00:54,704] - [remote_util] [140097149204224] - INFO - Running systemd command on this server [2021-02-12 23:00:54,704] - [remote_util] [140097149204224] - INFO - running command.raw on 172.23.122.97: systemctl stop couchbase-server.service [2021-02-12 23:00:56,464] - [remote_util] [140097149204224] - INFO - command executed successfully with root [2021-02-12 23:00:56,464] - [basetestcase] [140097149204224] - INFO - Couchbase stopped [2021-02-12 23:00:56,465] - [basetestcase] [140097149204224] - INFO - sleep for 5 secs. ... [2021-02-12 23:01:01,470] - [remote_util] [140097149204224] - INFO - running command.raw on 172.23.122.97: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-12 23:01:01,490] - [remote_util] [140097149204224] - INFO - command executed successfully with root [2021-02-12 23:01:01,490] - [remote_util] [140097149204224] - INFO - running command.raw on 172.23.122.97: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-12 23:01:01,545] - [remote_util] [140097149204224] - INFO - command executed successfully with root [2021-02-12 23:01:01,547] - [remote_util] [140097149204224] - INFO - SSH Connecting to 172.23.122.97 with username:root, attempt#1 of 5 [2021-02-12 23:01:01,651] - [remote_util] [140097149204224] - INFO - SSH Connected to 172.23.122.97 as root [2021-02-12 23:01:01,919] - [remote_util] [140097149204224] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:01:02,225] - [remote_util] [140097149204224] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:01:02,245] - [remote_util] [140097149204224] - INFO - 172.23.122.97 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 23:01:02,245] - [remote_util] [140097149204224] - INFO - Checking for process beam.smp on linux [2021-02-12 23:01:02,281] - [remote_util] [140097149204224] - INFO - Starting couchbase server [2021-02-12 23:01:02,378] - [remote_util] [140097149204224] - INFO - Running systemd command on this server [2021-02-12 23:01:02,378] - [remote_util] [140097149204224] - INFO - running command.raw on 172.23.122.97: systemctl start couchbase-server.service [2021-02-12 23:01:02,402] - [remote_util] [140097149204224] - INFO - command executed successfully with root [2021-02-12 23:01:02,402] - [remote_util] [140097149204224] - INFO - 172.23.122.97:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-12 23:01:07,407] - [remote_util] [140097149204224] - INFO - running command.raw on 172.23.122.97: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-12 23:01:07,433] - [remote_util] [140097149204224] - INFO - command executed successfully with root [2021-02-12 23:01:07,434] - [remote_util] [140097149204224] - INFO - Couchbase server status: [' Process: 110055 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-12 23:01:07,434] - [remote_util] [140097149204224] - INFO - Checking for process beam.smp on linux [2021-02-12 23:01:07,502] - [remote_util] [140097149204224] - INFO - process beam.smp is running on 172.23.122.97: with pid 110213 [2021-02-12 23:01:07,503] - [basetestcase] [140097149204224] - INFO - Couchbase started [2021-02-12 23:01:07,504] - [remote_util] [140097149204224] - INFO - SSH Connecting to 172.23.122.82 with username:root, attempt#1 of 5 [2021-02-12 23:01:07,604] - [remote_util] [140097149204224] - INFO - SSH Connected to 172.23.122.82 as root [2021-02-12 23:01:07,864] - [remote_util] [140097149204224] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:01:08,155] - [remote_util] [140097149204224] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:01:08,163] - [remote_util] [140097149204224] - INFO - SSH Connecting to 172.23.122.82 with username:root, attempt#1 of 5 [2021-02-12 23:01:08,259] - [remote_util] [140097149204224] - INFO - SSH Connected to 172.23.122.82 as root [2021-02-12 23:01:08,513] - [remote_util] [140097149204224] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:01:08,807] - [remote_util] [140097149204224] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:01:08,826] - [remote_util] [140097149204224] - INFO - 172.23.122.82 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 23:01:08,879] - [remote_util] [140097149204224] - INFO - Running systemd command on this server [2021-02-12 23:01:08,879] - [remote_util] [140097149204224] - INFO - running command.raw on 172.23.122.82: systemctl stop couchbase-server.service [2021-02-12 23:01:10,529] - [remote_util] [140097149204224] - INFO - command executed successfully with root [2021-02-12 23:01:10,529] - [basetestcase] [140097149204224] - INFO - Couchbase stopped [2021-02-12 23:01:10,530] - [basetestcase] [140097149204224] - INFO - sleep for 5 secs. ... [2021-02-12 23:01:15,532] - [remote_util] [140097149204224] - INFO - running command.raw on 172.23.122.82: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-12 23:01:15,552] - [remote_util] [140097149204224] - INFO - command executed successfully with root [2021-02-12 23:01:15,552] - [remote_util] [140097149204224] - INFO - running command.raw on 172.23.122.82: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-12 23:01:15,606] - [remote_util] [140097149204224] - INFO - command executed successfully with root [2021-02-12 23:01:15,608] - [remote_util] [140097149204224] - INFO - SSH Connecting to 172.23.122.82 with username:root, attempt#1 of 5 [2021-02-12 23:01:15,708] - [remote_util] [140097149204224] - INFO - SSH Connected to 172.23.122.82 as root [2021-02-12 23:01:15,971] - [remote_util] [140097149204224] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:01:16,263] - [remote_util] [140097149204224] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:01:16,281] - [remote_util] [140097149204224] - INFO - 172.23.122.82 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 23:01:16,281] - [remote_util] [140097149204224] - INFO - Checking for process beam.smp on linux [2021-02-12 23:01:16,318] - [remote_util] [140097149204224] - INFO - Starting couchbase server [2021-02-12 23:01:16,412] - [remote_util] [140097149204224] - INFO - Running systemd command on this server [2021-02-12 23:01:16,412] - [remote_util] [140097149204224] - INFO - running command.raw on 172.23.122.82: systemctl start couchbase-server.service [2021-02-12 23:01:16,434] - [remote_util] [140097149204224] - INFO - command executed successfully with root [2021-02-12 23:01:16,434] - [remote_util] [140097149204224] - INFO - 172.23.122.82:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-12 23:01:21,435] - [remote_util] [140097149204224] - INFO - running command.raw on 172.23.122.82: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-12 23:01:21,460] - [remote_util] [140097149204224] - INFO - command executed successfully with root [2021-02-12 23:01:21,461] - [remote_util] [140097149204224] - INFO - Couchbase server status: [' Process: 87248 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-12 23:01:21,461] - [remote_util] [140097149204224] - INFO - Checking for process beam.smp on linux [2021-02-12 23:01:21,529] - [remote_util] [140097149204224] - INFO - process beam.smp is running on 172.23.122.82: with pid 87432 [2021-02-12 23:01:21,529] - [basetestcase] [140097149204224] - INFO - Couchbase started [2021-02-12 23:01:21,531] - [remote_util] [140097149204224] - INFO - SSH Connecting to 172.23.122.118 with username:root, attempt#1 of 5 [2021-02-12 23:01:21,634] - [remote_util] [140097149204224] - INFO - SSH Connected to 172.23.122.118 as root [2021-02-12 23:01:21,899] - [remote_util] [140097149204224] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:01:22,202] - [remote_util] [140097149204224] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:01:22,211] - [remote_util] [140097149204224] - INFO - SSH Connecting to 172.23.122.118 with username:root, attempt#1 of 5 [2021-02-12 23:01:22,314] - [remote_util] [140097149204224] - INFO - SSH Connected to 172.23.122.118 as root [2021-02-12 23:01:22,561] - [remote_util] [140097149204224] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:01:22,862] - [remote_util] [140097149204224] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:01:22,882] - [remote_util] [140097149204224] - INFO - 172.23.122.118 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 23:01:22,937] - [remote_util] [140097149204224] - INFO - Running systemd command on this server [2021-02-12 23:01:22,937] - [remote_util] [140097149204224] - INFO - running command.raw on 172.23.122.118: systemctl stop couchbase-server.service [2021-02-12 23:01:24,619] - [remote_util] [140097149204224] - INFO - command executed successfully with root [2021-02-12 23:01:24,619] - [basetestcase] [140097149204224] - INFO - Couchbase stopped [2021-02-12 23:01:24,619] - [basetestcase] [140097149204224] - INFO - sleep for 5 secs. ... [2021-02-12 23:01:29,624] - [remote_util] [140097149204224] - INFO - running command.raw on 172.23.122.118: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-12 23:01:29,643] - [remote_util] [140097149204224] - INFO - command executed successfully with root [2021-02-12 23:01:29,643] - [remote_util] [140097149204224] - INFO - running command.raw on 172.23.122.118: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-12 23:01:29,697] - [remote_util] [140097149204224] - INFO - command executed successfully with root [2021-02-12 23:01:29,699] - [remote_util] [140097149204224] - INFO - SSH Connecting to 172.23.122.118 with username:root, attempt#1 of 5 [2021-02-12 23:01:29,804] - [remote_util] [140097149204224] - INFO - SSH Connected to 172.23.122.118 as root [2021-02-12 23:01:30,064] - [remote_util] [140097149204224] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:01:30,358] - [remote_util] [140097149204224] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:01:30,376] - [remote_util] [140097149204224] - INFO - 172.23.122.118 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 23:01:30,376] - [remote_util] [140097149204224] - INFO - Checking for process beam.smp on linux [2021-02-12 23:01:30,410] - [remote_util] [140097149204224] - INFO - Starting couchbase server [2021-02-12 23:01:30,498] - [remote_util] [140097149204224] - INFO - Running systemd command on this server [2021-02-12 23:01:30,498] - [remote_util] [140097149204224] - INFO - running command.raw on 172.23.122.118: systemctl start couchbase-server.service [2021-02-12 23:01:30,519] - [remote_util] [140097149204224] - INFO - command executed successfully with root [2021-02-12 23:01:30,519] - [remote_util] [140097149204224] - INFO - 172.23.122.118:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-12 23:01:35,524] - [remote_util] [140097149204224] - INFO - running command.raw on 172.23.122.118: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-12 23:01:35,550] - [remote_util] [140097149204224] - INFO - command executed successfully with root [2021-02-12 23:01:35,550] - [remote_util] [140097149204224] - INFO - Couchbase server status: [' Process: 113694 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-12 23:01:35,550] - [remote_util] [140097149204224] - INFO - Checking for process beam.smp on linux [2021-02-12 23:01:35,620] - [remote_util] [140097149204224] - INFO - process beam.smp is running on 172.23.122.118: with pid 113838 [2021-02-12 23:01:35,620] - [basetestcase] [140097149204224] - INFO - Couchbase started [2021-02-12 23:01:35,622] - [remote_util] [140097149204224] - INFO - SSH Connecting to 172.23.122.116 with username:root, attempt#1 of 5 [2021-02-12 23:01:35,735] - [remote_util] [140097149204224] - INFO - SSH Connected to 172.23.122.116 as root [2021-02-12 23:01:36,033] - [remote_util] [140097149204224] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:01:36,339] - [remote_util] [140097149204224] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:01:36,347] - [remote_util] [140097149204224] - INFO - SSH Connecting to 172.23.122.116 with username:root, attempt#1 of 5 [2021-02-12 23:01:36,449] - [remote_util] [140097149204224] - INFO - SSH Connected to 172.23.122.116 as root [2021-02-12 23:01:36,735] - [remote_util] [140097149204224] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:01:37,045] - [remote_util] [140097149204224] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:01:37,067] - [remote_util] [140097149204224] - INFO - 172.23.122.116 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 23:01:37,124] - [remote_util] [140097149204224] - INFO - Running systemd command on this server [2021-02-12 23:01:37,125] - [remote_util] [140097149204224] - INFO - running command.raw on 172.23.122.116: systemctl stop couchbase-server.service [2021-02-12 23:01:39,313] - [remote_util] [140097149204224] - INFO - command executed successfully with root [2021-02-12 23:01:39,313] - [basetestcase] [140097149204224] - INFO - Couchbase stopped [2021-02-12 23:01:39,314] - [basetestcase] [140097149204224] - INFO - sleep for 5 secs. ... [2021-02-12 23:01:44,319] - [remote_util] [140097149204224] - INFO - running command.raw on 172.23.122.116: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-12 23:01:44,334] - [remote_util] [140097149204224] - INFO - command executed successfully with root [2021-02-12 23:01:44,334] - [remote_util] [140097149204224] - INFO - running command.raw on 172.23.122.116: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-12 23:01:44,388] - [remote_util] [140097149204224] - INFO - command executed successfully with root [2021-02-12 23:01:44,389] - [remote_util] [140097149204224] - INFO - SSH Connecting to 172.23.122.116 with username:root, attempt#1 of 5 [2021-02-12 23:01:44,488] - [remote_util] [140097149204224] - INFO - SSH Connected to 172.23.122.116 as root [2021-02-12 23:01:44,768] - [remote_util] [140097149204224] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:01:45,065] - [remote_util] [140097149204224] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:01:45,083] - [remote_util] [140097149204224] - INFO - 172.23.122.116 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 23:01:45,083] - [remote_util] [140097149204224] - INFO - Checking for process beam.smp on linux [2021-02-12 23:01:45,119] - [remote_util] [140097149204224] - INFO - Starting couchbase server [2021-02-12 23:01:45,212] - [remote_util] [140097149204224] - INFO - Running systemd command on this server [2021-02-12 23:01:45,212] - [remote_util] [140097149204224] - INFO - running command.raw on 172.23.122.116: systemctl start couchbase-server.service [2021-02-12 23:01:45,232] - [remote_util] [140097149204224] - INFO - command executed successfully with root [2021-02-12 23:01:45,233] - [remote_util] [140097149204224] - INFO - 172.23.122.116:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-12 23:01:50,238] - [remote_util] [140097149204224] - INFO - running command.raw on 172.23.122.116: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-12 23:01:50,263] - [remote_util] [140097149204224] - INFO - command executed successfully with root [2021-02-12 23:01:50,263] - [remote_util] [140097149204224] - INFO - Couchbase server status: [' Process: 74000 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-12 23:01:50,263] - [remote_util] [140097149204224] - INFO - Checking for process beam.smp on linux [2021-02-12 23:01:50,330] - [remote_util] [140097149204224] - INFO - process beam.smp is running on 172.23.122.116: with pid 74145 [2021-02-12 23:01:50,330] - [basetestcase] [140097149204224] - INFO - Couchbase started [2021-02-12 23:01:50,330] - [basetestcase] [140097149204224] - INFO - sleep for 10 secs. ... [2021-02-12 23:02:00,377] - [remote_util] [140099567843072] - INFO - SSH Connecting to 172.23.123.62 with username:root, attempt#1 of 5 [2021-02-12 23:02:00,378] - [remote_util] [140097149204224] - INFO - SSH Connecting to 172.23.123.61 with username:root, attempt#1 of 5 [2021-02-12 23:02:00,380] - [remote_util] [140099350165248] - INFO - SSH Connecting to 172.23.122.90 with username:root, attempt#1 of 5 [2021-02-12 23:02:00,382] - [remote_util] [140099341772544] - INFO - SSH Connecting to 172.23.122.117 with username:root, attempt#1 of 5 [2021-02-12 23:02:00,385] - [remote_util] [140099324987136] - INFO - SSH Connecting to 172.23.122.97 with username:root, attempt#1 of 5 [2021-02-12 23:02:00,387] - [remote_util] [140098737792768] - INFO - SSH Connecting to 172.23.122.82 with username:root, attempt#1 of 5 [2021-02-12 23:02:00,388] - [remote_util] [140098712618752] - INFO - SSH Connecting to 172.23.122.118 with username:root, attempt#1 of 5 [2021-02-12 23:02:00,390] - [remote_util] [140098704226048] - INFO - SSH Connecting to 172.23.122.116 with username:root, attempt#1 of 5 [2021-02-12 23:02:00,487] - [remote_util] [140097149204224] - INFO - SSH Connected to 172.23.123.61 as root [2021-02-12 23:02:00,488] - [remote_util] [140099341772544] - INFO - SSH Connected to 172.23.122.117 as root [2021-02-12 23:02:00,489] - [remote_util] [140099567843072] - INFO - SSH Connected to 172.23.123.62 as root [2021-02-12 23:02:00,489] - [remote_util] [140099350165248] - INFO - SSH Connected to 172.23.122.90 as root [2021-02-12 23:02:00,491] - [remote_util] [140098737792768] - INFO - SSH Connected to 172.23.122.82 as root [2021-02-12 23:02:00,492] - [remote_util] [140099324987136] - INFO - SSH Connected to 172.23.122.97 as root [2021-02-12 23:02:00,493] - [remote_util] [140098704226048] - INFO - SSH Connected to 172.23.122.116 as root [2021-02-12 23:02:00,494] - [remote_util] [140098712618752] - INFO - SSH Connected to 172.23.122.118 as root [2021-02-12 23:02:00,770] - [remote_util] [140098712618752] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:02:00,781] - [remote_util] [140099341772544] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:02:00,798] - [remote_util] [140097149204224] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:02:00,802] - [remote_util] [140098737792768] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:02:00,812] - [remote_util] [140099324987136] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:02:00,816] - [remote_util] [140099567843072] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:02:00,822] - [remote_util] [140098704226048] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:02:00,825] - [remote_util] [140099350165248] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:02:01,087] - [remote_util] [140098712618752] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:02:01,088] - [remote_util] [140098712618752] - INFO - running command.raw on 172.23.122.118: /opt/couchbase/bin/cbcollect_info 172.23.122.118-20210212-2302-diag.zip [2021-02-12 23:02:01,116] - [remote_util] [140099324987136] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:02:01,116] - [remote_util] [140099324987136] - INFO - running command.raw on 172.23.122.97: /opt/couchbase/bin/cbcollect_info 172.23.122.97-20210212-2302-diag.zip [2021-02-12 23:02:01,117] - [remote_util] [140099567843072] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:02:01,118] - [remote_util] [140099567843072] - INFO - running command.raw on 172.23.123.62: /opt/couchbase/bin/cbcollect_info 172.23.123.62-20210212-2302-diag.zip [2021-02-12 23:02:01,119] - [remote_util] [140098704226048] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:02:01,120] - [remote_util] [140098704226048] - INFO - running command.raw on 172.23.122.116: /opt/couchbase/bin/cbcollect_info 172.23.122.116-20210212-2302-diag.zip [2021-02-12 23:02:01,135] - [remote_util] [140099341772544] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:02:01,135] - [remote_util] [140099341772544] - INFO - running command.raw on 172.23.122.117: /opt/couchbase/bin/cbcollect_info 172.23.122.117-20210212-2302-diag.zip [2021-02-12 23:02:01,143] - [remote_util] [140097149204224] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:02:01,144] - [remote_util] [140097149204224] - INFO - running command.raw on 172.23.123.61: /opt/couchbase/bin/cbcollect_info 172.23.123.61-20210212-2302-diag.zip [2021-02-12 23:02:01,146] - [remote_util] [140098737792768] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:02:01,146] - [remote_util] [140098737792768] - INFO - running command.raw on 172.23.122.82: /opt/couchbase/bin/cbcollect_info 172.23.122.82-20210212-2302-diag.zip [2021-02-12 23:02:01,166] - [remote_util] [140099350165248] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:02:01,166] - [remote_util] [140099350165248] - INFO - running command.raw on 172.23.122.90: /opt/couchbase/bin/cbcollect_info 172.23.122.90-20210212-2302-diag.zip [2021-02-12 23:03:29,734] - [remote_util] [140098704226048] - INFO - command executed successfully with root [2021-02-12 23:03:29,981] - [remote_util] [140098704226048] - INFO - running command.raw on 172.23.122.116: rm -f /root/172.23.122.116-20210212-2302-diag.zip [2021-02-12 23:03:29,998] - [remote_util] [140098704226048] - INFO - command executed successfully with root [2021-02-12 23:03:37,175] - [remote_util] [140098712618752] - INFO - command executed successfully with root [2021-02-12 23:03:37,670] - [remote_util] [140098712618752] - INFO - running command.raw on 172.23.122.118: rm -f /root/172.23.122.118-20210212-2302-diag.zip [2021-02-12 23:03:37,688] - [remote_util] [140098712618752] - INFO - command executed successfully with root [2021-02-12 23:03:39,033] - [remote_util] [140099341772544] - INFO - command executed successfully with root [2021-02-12 23:03:39,567] - [remote_util] [140099341772544] - INFO - running command.raw on 172.23.122.117: rm -f /root/172.23.122.117-20210212-2302-diag.zip [2021-02-12 23:03:39,588] - [remote_util] [140099341772544] - INFO - command executed successfully with root [2021-02-12 23:03:42,652] - [remote_util] [140099567843072] - INFO - command executed successfully with root [2021-02-12 23:03:43,355] - [remote_util] [140099567843072] - INFO - running command.raw on 172.23.123.62: rm -f /root/172.23.123.62-20210212-2302-diag.zip [2021-02-12 23:03:43,383] - [remote_util] [140099567843072] - INFO - command executed successfully with root [2021-02-12 23:03:45,536] - [remote_util] [140098737792768] - INFO - command executed successfully with root [2021-02-12 23:03:46,118] - [remote_util] [140098737792768] - INFO - running command.raw on 172.23.122.82: rm -f /root/172.23.122.82-20210212-2302-diag.zip [2021-02-12 23:03:46,137] - [remote_util] [140098737792768] - INFO - command executed successfully with root [2021-02-12 23:03:46,708] - [remote_util] [140097149204224] - INFO - command executed successfully with root [2021-02-12 23:03:47,369] - [remote_util] [140097149204224] - INFO - running command.raw on 172.23.123.61: rm -f /root/172.23.123.61-20210212-2302-diag.zip [2021-02-12 23:03:47,391] - [remote_util] [140097149204224] - INFO - command executed successfully with root [2021-02-12 23:03:47,827] - [remote_util] [140099350165248] - INFO - command executed successfully with root [2021-02-12 23:03:48,572] - [remote_util] [140099350165248] - INFO - running command.raw on 172.23.122.90: rm -f /root/172.23.122.90-20210212-2302-diag.zip [2021-02-12 23:03:48,593] - [remote_util] [140099350165248] - INFO - command executed successfully with root [2021-02-12 23:03:51,079] - [remote_util] [140099324987136] - INFO - command executed successfully with root [2021-02-12 23:03:51,829] - [remote_util] [140099324987136] - INFO - running command.raw on 172.23.122.97: rm -f /root/172.23.122.97-20210212-2302-diag.zip [2021-02-12 23:03:51,851] - [remote_util] [140099324987136] - INFO - command executed successfully with root