[2021-02-17 00:21:38,496] - [eventing_base] [139626970871552] - INFO - Starting Test: test_kv_rebalance_out_when_existing_eventing_node_is_processing_mutations [2021-02-17 00:21:38,498] - [remote_util] [139626970871552] - INFO - SSH Connecting to 172.23.122.91 with username:root, attempt#1 of 5 [2021-02-17 00:21:38,599] - [remote_util] [139626970871552] - INFO - SSH Connected to 172.23.122.91 as root [2021-02-17 00:21:38,880] - [remote_util] [139626970871552] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-17 00:21:39,177] - [remote_util] [139626970871552] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-17 00:21:39,182] - [rest_client] [139626970871552] - INFO - -->is_ns_server_running? [2021-02-17 00:21:39,193] - [rest_client] [139626970871552] - INFO - Node version in cluster 7.0.0-4476-enterprise [2021-02-17 00:21:39,198] - [rest_client] [139626970871552] - ERROR - GET http://172.23.122.91:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-02-17 00:21:39,198] - [rest_client] [139626970871552] - INFO - Node versions in cluster [] [2021-02-17 00:21:39,198] - [basetestcase] [139626970871552] - INFO - ============== basetestcase setup was started for test #5 test_kv_rebalance_out_when_existing_eventing_node_is_processing_mutations============== [2021-02-17 00:21:39,203] - [rest_client] [139626970871552] - ERROR - GET http://172.23.122.91:8091/pools/default/ body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-02-17 00:21:39,217] - [bucket_helper] [139626970871552] - INFO - Could not find any buckets for node 172.23.122.91, nothing to delete [2021-02-17 00:21:39,229] - [bucket_helper] [139626970871552] - INFO - Could not find any buckets for node 172.23.122.158, nothing to delete [2021-02-17 00:21:39,241] - [bucket_helper] [139626970871552] - INFO - Could not find any buckets for node 172.23.122.159, nothing to delete [2021-02-17 00:21:39,252] - [bucket_helper] [139626970871552] - INFO - Could not find any buckets for node 172.23.122.92, nothing to delete [2021-02-17 00:21:39,265] - [bucket_helper] [139626970871552] - INFO - Could not find any buckets for node 172.23.122.156, nothing to delete [2021-02-17 00:21:39,278] - [bucket_helper] [139626970871552] - INFO - Could not find any buckets for node 172.23.122.161, nothing to delete [2021-02-17 00:21:39,291] - [bucket_helper] [139626970871552] - INFO - Could not find any buckets for node 172.23.122.160, nothing to delete [2021-02-17 00:21:39,304] - [bucket_helper] [139626970871552] - INFO - Could not find any buckets for node 172.23.122.93, nothing to delete [2021-02-17 00:21:39,307] - [rest_client] [139626970871552] - INFO - -->is_ns_server_running? [2021-02-17 00:21:39,322] - [cluster_helper] [139626970871552] - INFO - waiting for ns_server @ 172.23.122.91:8091 [2021-02-17 00:21:39,322] - [rest_client] [139626970871552] - INFO - -->is_ns_server_running? [2021-02-17 00:21:39,325] - [cluster_helper] [139626970871552] - INFO - ns_server @ 172.23.122.91:8091 is running [2021-02-17 00:21:39,329] - [cluster_helper] [139626970871552] - INFO - waiting for ns_server @ 172.23.122.158:8091 [2021-02-17 00:21:39,329] - [rest_client] [139626970871552] - INFO - -->is_ns_server_running? [2021-02-17 00:21:39,331] - [cluster_helper] [139626970871552] - INFO - ns_server @ 172.23.122.158:8091 is running [2021-02-17 00:21:39,335] - [cluster_helper] [139626970871552] - INFO - waiting for ns_server @ 172.23.122.159:8091 [2021-02-17 00:21:39,335] - [rest_client] [139626970871552] - INFO - -->is_ns_server_running? [2021-02-17 00:21:39,338] - [cluster_helper] [139626970871552] - INFO - ns_server @ 172.23.122.159:8091 is running [2021-02-17 00:21:39,341] - [cluster_helper] [139626970871552] - INFO - waiting for ns_server @ 172.23.122.92:8091 [2021-02-17 00:21:39,341] - [rest_client] [139626970871552] - INFO - -->is_ns_server_running? [2021-02-17 00:21:39,344] - [cluster_helper] [139626970871552] - INFO - ns_server @ 172.23.122.92:8091 is running [2021-02-17 00:21:39,348] - [cluster_helper] [139626970871552] - INFO - waiting for ns_server @ 172.23.122.156:8091 [2021-02-17 00:21:39,348] - [rest_client] [139626970871552] - INFO - -->is_ns_server_running? [2021-02-17 00:21:39,351] - [cluster_helper] [139626970871552] - INFO - ns_server @ 172.23.122.156:8091 is running [2021-02-17 00:21:39,354] - [cluster_helper] [139626970871552] - INFO - waiting for ns_server @ 172.23.122.161:8091 [2021-02-17 00:21:39,354] - [rest_client] [139626970871552] - INFO - -->is_ns_server_running? [2021-02-17 00:21:39,357] - [cluster_helper] [139626970871552] - INFO - ns_server @ 172.23.122.161:8091 is running [2021-02-17 00:21:39,361] - [cluster_helper] [139626970871552] - INFO - waiting for ns_server @ 172.23.122.160:8091 [2021-02-17 00:21:39,361] - [rest_client] [139626970871552] - INFO - -->is_ns_server_running? [2021-02-17 00:21:39,363] - [cluster_helper] [139626970871552] - INFO - ns_server @ 172.23.122.160:8091 is running [2021-02-17 00:21:39,367] - [cluster_helper] [139626970871552] - INFO - waiting for ns_server @ 172.23.122.93:8091 [2021-02-17 00:21:39,367] - [rest_client] [139626970871552] - INFO - -->is_ns_server_running? [2021-02-17 00:21:39,370] - [cluster_helper] [139626970871552] - INFO - ns_server @ 172.23.122.93:8091 is running [2021-02-17 00:21:39,371] - [basetestcase] [139626970871552] - INFO - Building docker image with java sdk client [2021-02-17 00:21:43,310] - [basetestcase] [139626970871552] - INFO - initializing cluster [2021-02-17 00:21:43,313] - [remote_util] [139626970871552] - INFO - SSH Connecting to 172.23.122.91 with username:root, attempt#1 of 5 [2021-02-17 00:21:43,415] - [remote_util] [139626970871552] - INFO - SSH Connected to 172.23.122.91 as root [2021-02-17 00:21:43,696] - [remote_util] [139626970871552] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-17 00:21:44,000] - [remote_util] [139626970871552] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-17 00:21:44,010] - [remote_util] [139626970871552] - INFO - SSH Connecting to 172.23.122.91 with username:root, attempt#1 of 5 [2021-02-17 00:21:44,111] - [remote_util] [139626970871552] - INFO - SSH Connected to 172.23.122.91 as root [2021-02-17 00:21:44,395] - [remote_util] [139626970871552] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-17 00:21:44,702] - [remote_util] [139626970871552] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-17 00:21:44,724] - [remote_util] [139626970871552] - INFO - 172.23.122.91 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-17 00:21:44,787] - [remote_util] [139626970871552] - INFO - Running systemd command on this server [2021-02-17 00:21:44,787] - [remote_util] [139626970871552] - INFO - running command.raw on 172.23.122.91: systemctl stop couchbase-server.service [2021-02-17 00:21:46,448] - [remote_util] [139626970871552] - INFO - command executed successfully with root [2021-02-17 00:21:46,449] - [basetestcase] [139626970871552] - INFO - Couchbase stopped [2021-02-17 00:21:46,449] - [basetestcase] [139626970871552] - INFO - sleep for 5 secs. ... [2021-02-17 00:21:51,454] - [remote_util] [139626970871552] - INFO - running command.raw on 172.23.122.91: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-17 00:21:51,471] - [remote_util] [139626970871552] - INFO - command executed successfully with root [2021-02-17 00:21:51,471] - [remote_util] [139626970871552] - INFO - running command.raw on 172.23.122.91: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-17 00:21:51,526] - [remote_util] [139626970871552] - INFO - command executed successfully with root [2021-02-17 00:21:51,528] - [remote_util] [139626970871552] - INFO - SSH Connecting to 172.23.122.91 with username:root, attempt#1 of 5 [2021-02-17 00:21:51,630] - [remote_util] [139626970871552] - INFO - SSH Connected to 172.23.122.91 as root [2021-02-17 00:21:51,917] - [remote_util] [139626970871552] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-17 00:21:52,215] - [remote_util] [139626970871552] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-17 00:21:52,275] - [remote_util] [139626970871552] - INFO - 172.23.122.91 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-17 00:21:52,275] - [remote_util] [139626970871552] - INFO - Checking for process beam.smp on linux [2021-02-17 00:21:52,352] - [remote_util] [139626970871552] - INFO - Starting couchbase server [2021-02-17 00:21:52,452] - [remote_util] [139626970871552] - INFO - Running systemd command on this server [2021-02-17 00:21:52,452] - [remote_util] [139626970871552] - INFO - running command.raw on 172.23.122.91: systemctl start couchbase-server.service [2021-02-17 00:21:52,474] - [remote_util] [139626970871552] - INFO - command executed successfully with root [2021-02-17 00:21:52,475] - [remote_util] [139626970871552] - INFO - 172.23.122.91:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-17 00:21:57,478] - [remote_util] [139626970871552] - INFO - running command.raw on 172.23.122.91: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-17 00:21:57,504] - [remote_util] [139626970871552] - INFO - command executed successfully with root [2021-02-17 00:21:57,505] - [remote_util] [139626970871552] - INFO - Couchbase server status: [' Process: 14435 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-17 00:21:57,505] - [remote_util] [139626970871552] - INFO - Checking for process beam.smp on linux [2021-02-17 00:21:57,572] - [remote_util] [139626970871552] - INFO - process beam.smp is running on 172.23.122.91: with pid 14610 [2021-02-17 00:21:57,572] - [basetestcase] [139626970871552] - INFO - Couchbase started [2021-02-17 00:21:57,574] - [remote_util] [139626970871552] - INFO - SSH Connecting to 172.23.122.158 with username:root, attempt#1 of 5 [2021-02-17 00:21:57,675] - [remote_util] [139626970871552] - INFO - SSH Connected to 172.23.122.158 as root [2021-02-17 00:21:57,923] - [remote_util] [139626970871552] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-17 00:21:58,214] - [remote_util] [139626970871552] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-17 00:21:58,223] - [remote_util] [139626970871552] - INFO - SSH Connecting to 172.23.122.158 with username:root, attempt#1 of 5 [2021-02-17 00:21:58,315] - [remote_util] [139626970871552] - INFO - SSH Connected to 172.23.122.158 as root [2021-02-17 00:21:58,561] - [remote_util] [139626970871552] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-17 00:21:58,855] - [remote_util] [139626970871552] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-17 00:21:58,874] - [remote_util] [139626970871552] - INFO - 172.23.122.158 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-17 00:21:58,932] - [remote_util] [139626970871552] - INFO - Running systemd command on this server [2021-02-17 00:21:58,932] - [remote_util] [139626970871552] - INFO - running command.raw on 172.23.122.158: systemctl stop couchbase-server.service [2021-02-17 00:22:00,625] - [remote_util] [139626970871552] - INFO - command executed successfully with root [2021-02-17 00:22:00,626] - [basetestcase] [139626970871552] - INFO - Couchbase stopped [2021-02-17 00:22:00,626] - [basetestcase] [139626970871552] - INFO - sleep for 5 secs. ... [2021-02-17 00:22:05,629] - [remote_util] [139626970871552] - INFO - running command.raw on 172.23.122.158: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-17 00:22:05,644] - [remote_util] [139626970871552] - INFO - command executed successfully with root [2021-02-17 00:22:05,644] - [remote_util] [139626970871552] - INFO - running command.raw on 172.23.122.158: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-17 00:22:05,697] - [remote_util] [139626970871552] - INFO - command executed successfully with root [2021-02-17 00:22:05,699] - [remote_util] [139626970871552] - INFO - SSH Connecting to 172.23.122.158 with username:root, attempt#1 of 5 [2021-02-17 00:22:05,800] - [remote_util] [139626970871552] - INFO - SSH Connected to 172.23.122.158 as root [2021-02-17 00:22:06,047] - [remote_util] [139626970871552] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-17 00:22:06,346] - [remote_util] [139626970871552] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-17 00:22:06,406] - [remote_util] [139626970871552] - INFO - 172.23.122.158 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-17 00:22:06,406] - [remote_util] [139626970871552] - INFO - Checking for process beam.smp on linux [2021-02-17 00:22:06,443] - [remote_util] [139626970871552] - INFO - Starting couchbase server [2021-02-17 00:22:06,540] - [remote_util] [139626970871552] - INFO - Running systemd command on this server [2021-02-17 00:22:06,540] - [remote_util] [139626970871552] - INFO - running command.raw on 172.23.122.158: systemctl start couchbase-server.service [2021-02-17 00:22:06,562] - [remote_util] [139626970871552] - INFO - command executed successfully with root [2021-02-17 00:22:06,563] - [remote_util] [139626970871552] - INFO - 172.23.122.158:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-17 00:22:11,565] - [remote_util] [139626970871552] - INFO - running command.raw on 172.23.122.158: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-17 00:22:11,589] - [remote_util] [139626970871552] - INFO - command executed successfully with root [2021-02-17 00:22:11,589] - [remote_util] [139626970871552] - INFO - Couchbase server status: [' Process: 41615 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-17 00:22:11,589] - [remote_util] [139626970871552] - INFO - Checking for process beam.smp on linux [2021-02-17 00:22:11,657] - [remote_util] [139626970871552] - INFO - process beam.smp is running on 172.23.122.158: with pid 41759 [2021-02-17 00:22:11,657] - [basetestcase] [139626970871552] - INFO - Couchbase started [2021-02-17 00:22:11,659] - [remote_util] [139626970871552] - INFO - SSH Connecting to 172.23.122.159 with username:root, attempt#1 of 5 [2021-02-17 00:22:11,764] - [remote_util] [139626970871552] - INFO - SSH Connected to 172.23.122.159 as root [2021-02-17 00:22:12,042] - [remote_util] [139626970871552] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-17 00:22:12,332] - [remote_util] [139626970871552] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-17 00:22:12,341] - [remote_util] [139626970871552] - INFO - SSH Connecting to 172.23.122.159 with username:root, attempt#1 of 5 [2021-02-17 00:22:12,439] - [remote_util] [139626970871552] - INFO - SSH Connected to 172.23.122.159 as root [2021-02-17 00:22:12,705] - [remote_util] [139626970871552] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-17 00:22:12,992] - [remote_util] [139626970871552] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-17 00:22:13,052] - [remote_util] [139626970871552] - INFO - 172.23.122.159 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-17 00:22:13,108] - [remote_util] [139626970871552] - INFO - Running systemd command on this server [2021-02-17 00:22:13,108] - [remote_util] [139626970871552] - INFO - running command.raw on 172.23.122.159: systemctl stop couchbase-server.service [2021-02-17 00:22:14,760] - [remote_util] [139626970871552] - INFO - command executed successfully with root [2021-02-17 00:22:14,760] - [basetestcase] [139626970871552] - INFO - Couchbase stopped [2021-02-17 00:22:14,760] - [basetestcase] [139626970871552] - INFO - sleep for 5 secs. ... [2021-02-17 00:22:19,763] - [remote_util] [139626970871552] - INFO - running command.raw on 172.23.122.159: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-17 00:22:19,777] - [remote_util] [139626970871552] - INFO - command executed successfully with root [2021-02-17 00:22:19,778] - [remote_util] [139626970871552] - INFO - running command.raw on 172.23.122.159: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-17 00:22:19,830] - [remote_util] [139626970871552] - INFO - command executed successfully with root [2021-02-17 00:22:19,832] - [remote_util] [139626970871552] - INFO - SSH Connecting to 172.23.122.159 with username:root, attempt#1 of 5 [2021-02-17 00:22:19,928] - [remote_util] [139626970871552] - INFO - SSH Connected to 172.23.122.159 as root [2021-02-17 00:22:20,200] - [remote_util] [139626970871552] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-17 00:22:20,494] - [remote_util] [139626970871552] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-17 00:22:20,515] - [remote_util] [139626970871552] - INFO - 172.23.122.159 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-17 00:22:20,515] - [remote_util] [139626970871552] - INFO - Checking for process beam.smp on linux [2021-02-17 00:22:20,551] - [remote_util] [139626970871552] - INFO - Starting couchbase server [2021-02-17 00:22:20,648] - [remote_util] [139626970871552] - INFO - Running systemd command on this server [2021-02-17 00:22:20,649] - [remote_util] [139626970871552] - INFO - running command.raw on 172.23.122.159: systemctl start couchbase-server.service [2021-02-17 00:22:20,672] - [remote_util] [139626970871552] - INFO - command executed successfully with root [2021-02-17 00:22:20,672] - [remote_util] [139626970871552] - INFO - 172.23.122.159:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-17 00:22:25,676] - [remote_util] [139626970871552] - INFO - running command.raw on 172.23.122.159: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-17 00:22:25,700] - [remote_util] [139626970871552] - INFO - command executed successfully with root [2021-02-17 00:22:25,700] - [remote_util] [139626970871552] - INFO - Couchbase server status: [' Process: 48380 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-17 00:22:25,701] - [remote_util] [139626970871552] - INFO - Checking for process beam.smp on linux [2021-02-17 00:22:25,768] - [remote_util] [139626970871552] - INFO - process beam.smp is running on 172.23.122.159: with pid 48524 [2021-02-17 00:22:25,769] - [basetestcase] [139626970871552] - INFO - Couchbase started [2021-02-17 00:22:25,771] - [remote_util] [139626970871552] - INFO - SSH Connecting to 172.23.122.92 with username:root, attempt#1 of 5 [2021-02-17 00:22:25,876] - [remote_util] [139626970871552] - INFO - SSH Connected to 172.23.122.92 as root [2021-02-17 00:22:26,145] - [remote_util] [139626970871552] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-17 00:22:26,451] - [remote_util] [139626970871552] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-17 00:22:26,460] - [remote_util] [139626970871552] - INFO - SSH Connecting to 172.23.122.92 with username:root, attempt#1 of 5 [2021-02-17 00:22:26,559] - [remote_util] [139626970871552] - INFO - SSH Connected to 172.23.122.92 as root [2021-02-17 00:22:26,817] - [remote_util] [139626970871552] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-17 00:22:27,117] - [remote_util] [139626970871552] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-17 00:22:27,177] - [remote_util] [139626970871552] - INFO - 172.23.122.92 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-17 00:22:27,275] - [remote_util] [139626970871552] - INFO - Running systemd command on this server [2021-02-17 00:22:27,275] - [remote_util] [139626970871552] - INFO - running command.raw on 172.23.122.92: systemctl stop couchbase-server.service [2021-02-17 00:22:28,903] - [remote_util] [139626970871552] - INFO - command executed successfully with root [2021-02-17 00:22:28,903] - [basetestcase] [139626970871552] - INFO - Couchbase stopped [2021-02-17 00:22:28,903] - [basetestcase] [139626970871552] - INFO - sleep for 5 secs. ... [2021-02-17 00:22:33,908] - [remote_util] [139626970871552] - INFO - running command.raw on 172.23.122.92: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-17 00:22:33,924] - [remote_util] [139626970871552] - INFO - command executed successfully with root [2021-02-17 00:22:33,924] - [remote_util] [139626970871552] - INFO - running command.raw on 172.23.122.92: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-17 00:22:33,979] - [remote_util] [139626970871552] - INFO - command executed successfully with root [2021-02-17 00:22:33,981] - [remote_util] [139626970871552] - INFO - SSH Connecting to 172.23.122.92 with username:root, attempt#1 of 5 [2021-02-17 00:22:34,084] - [remote_util] [139626970871552] - INFO - SSH Connected to 172.23.122.92 as root [2021-02-17 00:22:34,352] - [remote_util] [139626970871552] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-17 00:22:34,655] - [remote_util] [139626970871552] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-17 00:22:34,716] - [remote_util] [139626970871552] - INFO - 172.23.122.92 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-17 00:22:34,716] - [remote_util] [139626970871552] - INFO - Checking for process beam.smp on linux [2021-02-17 00:22:34,754] - [remote_util] [139626970871552] - INFO - Starting couchbase server [2021-02-17 00:22:34,853] - [remote_util] [139626970871552] - INFO - Running systemd command on this server [2021-02-17 00:22:34,853] - [remote_util] [139626970871552] - INFO - running command.raw on 172.23.122.92: systemctl start couchbase-server.service [2021-02-17 00:22:34,875] - [remote_util] [139626970871552] - INFO - command executed successfully with root [2021-02-17 00:22:34,875] - [remote_util] [139626970871552] - INFO - 172.23.122.92:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-17 00:22:39,877] - [remote_util] [139626970871552] - INFO - running command.raw on 172.23.122.92: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-17 00:22:39,908] - [remote_util] [139626970871552] - INFO - command executed successfully with root [2021-02-17 00:22:39,908] - [remote_util] [139626970871552] - INFO - Couchbase server status: [' Process: 55193 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-17 00:22:39,908] - [remote_util] [139626970871552] - INFO - Checking for process beam.smp on linux [2021-02-17 00:22:39,980] - [remote_util] [139626970871552] - INFO - process beam.smp is running on 172.23.122.92: with pid 55338 [2021-02-17 00:22:39,980] - [basetestcase] [139626970871552] - INFO - Couchbase started [2021-02-17 00:22:39,982] - [remote_util] [139626970871552] - INFO - SSH Connecting to 172.23.122.156 with username:root, attempt#1 of 5 [2021-02-17 00:22:40,087] - [remote_util] [139626970871552] - INFO - SSH Connected to 172.23.122.156 as root [2021-02-17 00:22:40,378] - [remote_util] [139626970871552] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-17 00:22:40,680] - [remote_util] [139626970871552] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-17 00:22:40,690] - [remote_util] [139626970871552] - INFO - SSH Connecting to 172.23.122.156 with username:root, attempt#1 of 5 [2021-02-17 00:22:40,791] - [remote_util] [139626970871552] - INFO - SSH Connected to 172.23.122.156 as root [2021-02-17 00:22:41,068] - [remote_util] [139626970871552] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-17 00:22:41,369] - [remote_util] [139626970871552] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-17 00:22:41,430] - [remote_util] [139626970871552] - INFO - 172.23.122.156 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-17 00:22:41,486] - [remote_util] [139626970871552] - INFO - Running systemd command on this server [2021-02-17 00:22:41,487] - [remote_util] [139626970871552] - INFO - running command.raw on 172.23.122.156: systemctl stop couchbase-server.service [2021-02-17 00:22:43,218] - [remote_util] [139626970871552] - INFO - command executed successfully with root [2021-02-17 00:22:43,218] - [basetestcase] [139626970871552] - INFO - Couchbase stopped [2021-02-17 00:22:43,219] - [basetestcase] [139626970871552] - INFO - sleep for 5 secs. ... [2021-02-17 00:22:48,224] - [remote_util] [139626970871552] - INFO - running command.raw on 172.23.122.156: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-17 00:22:48,239] - [remote_util] [139626970871552] - INFO - command executed successfully with root [2021-02-17 00:22:48,239] - [remote_util] [139626970871552] - INFO - running command.raw on 172.23.122.156: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-17 00:22:48,292] - [remote_util] [139626970871552] - INFO - command executed successfully with root [2021-02-17 00:22:48,294] - [remote_util] [139626970871552] - INFO - SSH Connecting to 172.23.122.156 with username:root, attempt#1 of 5 [2021-02-17 00:22:48,397] - [remote_util] [139626970871552] - INFO - SSH Connected to 172.23.122.156 as root [2021-02-17 00:22:48,676] - [remote_util] [139626970871552] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-17 00:22:48,974] - [remote_util] [139626970871552] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-17 00:22:49,034] - [remote_util] [139626970871552] - INFO - 172.23.122.156 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-17 00:22:49,034] - [remote_util] [139626970871552] - INFO - Checking for process beam.smp on linux [2021-02-17 00:22:49,073] - [remote_util] [139626970871552] - INFO - Starting couchbase server [2021-02-17 00:22:49,171] - [remote_util] [139626970871552] - INFO - Running systemd command on this server [2021-02-17 00:22:49,172] - [remote_util] [139626970871552] - INFO - running command.raw on 172.23.122.156: systemctl start couchbase-server.service [2021-02-17 00:22:49,193] - [remote_util] [139626970871552] - INFO - command executed successfully with root [2021-02-17 00:22:49,194] - [remote_util] [139626970871552] - INFO - 172.23.122.156:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-17 00:22:54,199] - [remote_util] [139626970871552] - INFO - running command.raw on 172.23.122.156: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-17 00:22:54,227] - [remote_util] [139626970871552] - INFO - command executed successfully with root [2021-02-17 00:22:54,227] - [remote_util] [139626970871552] - INFO - Couchbase server status: [' Process: 101998 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-17 00:22:54,227] - [remote_util] [139626970871552] - INFO - Checking for process beam.smp on linux [2021-02-17 00:22:54,298] - [remote_util] [139626970871552] - INFO - process beam.smp is running on 172.23.122.156: with pid 102142 [2021-02-17 00:22:54,298] - [basetestcase] [139626970871552] - INFO - Couchbase started [2021-02-17 00:22:54,300] - [remote_util] [139626970871552] - INFO - SSH Connecting to 172.23.122.161 with username:root, attempt#1 of 5 [2021-02-17 00:22:54,405] - [remote_util] [139626970871552] - INFO - SSH Connected to 172.23.122.161 as root [2021-02-17 00:22:54,687] - [remote_util] [139626970871552] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-17 00:22:54,985] - [remote_util] [139626970871552] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-17 00:22:54,994] - [remote_util] [139626970871552] - INFO - SSH Connecting to 172.23.122.161 with username:root, attempt#1 of 5 [2021-02-17 00:22:55,094] - [remote_util] [139626970871552] - INFO - SSH Connected to 172.23.122.161 as root [2021-02-17 00:22:55,364] - [remote_util] [139626970871552] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-17 00:22:55,660] - [remote_util] [139626970871552] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-17 00:22:55,721] - [remote_util] [139626970871552] - INFO - 172.23.122.161 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-17 00:22:55,780] - [remote_util] [139626970871552] - INFO - Running systemd command on this server [2021-02-17 00:22:55,780] - [remote_util] [139626970871552] - INFO - running command.raw on 172.23.122.161: systemctl stop couchbase-server.service [2021-02-17 00:22:57,388] - [remote_util] [139626970871552] - INFO - command executed successfully with root [2021-02-17 00:22:57,388] - [basetestcase] [139626970871552] - INFO - Couchbase stopped [2021-02-17 00:22:57,389] - [basetestcase] [139626970871552] - INFO - sleep for 5 secs. ... [2021-02-17 00:23:02,394] - [remote_util] [139626970871552] - INFO - running command.raw on 172.23.122.161: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-17 00:23:02,408] - [remote_util] [139626970871552] - INFO - command executed successfully with root [2021-02-17 00:23:02,409] - [remote_util] [139626970871552] - INFO - running command.raw on 172.23.122.161: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-17 00:23:02,461] - [remote_util] [139626970871552] - INFO - command executed successfully with root [2021-02-17 00:23:02,463] - [remote_util] [139626970871552] - INFO - SSH Connecting to 172.23.122.161 with username:root, attempt#1 of 5 [2021-02-17 00:23:02,561] - [remote_util] [139626970871552] - INFO - SSH Connected to 172.23.122.161 as root [2021-02-17 00:23:02,829] - [remote_util] [139626970871552] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-17 00:23:03,123] - [remote_util] [139626970871552] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-17 00:23:03,183] - [remote_util] [139626970871552] - INFO - 172.23.122.161 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-17 00:23:03,183] - [remote_util] [139626970871552] - INFO - Checking for process beam.smp on linux [2021-02-17 00:23:03,220] - [remote_util] [139626970871552] - INFO - Starting couchbase server [2021-02-17 00:23:03,315] - [remote_util] [139626970871552] - INFO - Running systemd command on this server [2021-02-17 00:23:03,315] - [remote_util] [139626970871552] - INFO - running command.raw on 172.23.122.161: systemctl start couchbase-server.service [2021-02-17 00:23:03,337] - [remote_util] [139626970871552] - INFO - command executed successfully with root [2021-02-17 00:23:03,337] - [remote_util] [139626970871552] - INFO - 172.23.122.161:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-17 00:23:08,343] - [remote_util] [139626970871552] - INFO - running command.raw on 172.23.122.161: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-17 00:23:08,370] - [remote_util] [139626970871552] - INFO - command executed successfully with root [2021-02-17 00:23:08,370] - [remote_util] [139626970871552] - INFO - Couchbase server status: [' Process: 82720 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-17 00:23:08,370] - [remote_util] [139626970871552] - INFO - Checking for process beam.smp on linux [2021-02-17 00:23:08,441] - [remote_util] [139626970871552] - INFO - process beam.smp is running on 172.23.122.161: with pid 82865 [2021-02-17 00:23:08,441] - [basetestcase] [139626970871552] - INFO - Couchbase started [2021-02-17 00:23:08,443] - [remote_util] [139626970871552] - INFO - SSH Connecting to 172.23.122.160 with username:root, attempt#1 of 5 [2021-02-17 00:23:08,549] - [remote_util] [139626970871552] - INFO - SSH Connected to 172.23.122.160 as root [2021-02-17 00:23:08,839] - [remote_util] [139626970871552] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-17 00:23:09,142] - [remote_util] [139626970871552] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-17 00:23:09,151] - [remote_util] [139626970871552] - INFO - SSH Connecting to 172.23.122.160 with username:root, attempt#1 of 5 [2021-02-17 00:23:09,252] - [remote_util] [139626970871552] - INFO - SSH Connected to 172.23.122.160 as root [2021-02-17 00:23:09,535] - [remote_util] [139626970871552] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-17 00:23:09,841] - [remote_util] [139626970871552] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-17 00:23:09,903] - [remote_util] [139626970871552] - INFO - 172.23.122.160 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-17 00:23:09,959] - [remote_util] [139626970871552] - INFO - Running systemd command on this server [2021-02-17 00:23:09,960] - [remote_util] [139626970871552] - INFO - running command.raw on 172.23.122.160: systemctl stop couchbase-server.service [2021-02-17 00:23:11,715] - [remote_util] [139626970871552] - INFO - command executed successfully with root [2021-02-17 00:23:11,716] - [basetestcase] [139626970871552] - INFO - Couchbase stopped [2021-02-17 00:23:11,716] - [basetestcase] [139626970871552] - INFO - sleep for 5 secs. ... [2021-02-17 00:23:16,720] - [remote_util] [139626970871552] - INFO - running command.raw on 172.23.122.160: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-17 00:23:16,736] - [remote_util] [139626970871552] - INFO - command executed successfully with root [2021-02-17 00:23:16,736] - [remote_util] [139626970871552] - INFO - running command.raw on 172.23.122.160: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-17 00:23:16,789] - [remote_util] [139626970871552] - INFO - command executed successfully with root [2021-02-17 00:23:16,791] - [remote_util] [139626970871552] - INFO - SSH Connecting to 172.23.122.160 with username:root, attempt#1 of 5 [2021-02-17 00:23:16,890] - [remote_util] [139626970871552] - INFO - SSH Connected to 172.23.122.160 as root [2021-02-17 00:23:17,166] - [remote_util] [139626970871552] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-17 00:23:17,459] - [remote_util] [139626970871552] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-17 00:23:17,479] - [remote_util] [139626970871552] - INFO - 172.23.122.160 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-17 00:23:17,479] - [remote_util] [139626970871552] - INFO - Checking for process beam.smp on linux [2021-02-17 00:23:17,517] - [remote_util] [139626970871552] - INFO - Starting couchbase server [2021-02-17 00:23:17,609] - [remote_util] [139626970871552] - INFO - Running systemd command on this server [2021-02-17 00:23:17,610] - [remote_util] [139626970871552] - INFO - running command.raw on 172.23.122.160: systemctl start couchbase-server.service [2021-02-17 00:23:17,631] - [remote_util] [139626970871552] - INFO - command executed successfully with root [2021-02-17 00:23:17,631] - [remote_util] [139626970871552] - INFO - 172.23.122.160:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-17 00:23:22,633] - [remote_util] [139626970871552] - INFO - running command.raw on 172.23.122.160: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-17 00:23:22,659] - [remote_util] [139626970871552] - INFO - command executed successfully with root [2021-02-17 00:23:22,660] - [remote_util] [139626970871552] - INFO - Couchbase server status: [' Process: 37848 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-17 00:23:22,660] - [remote_util] [139626970871552] - INFO - Checking for process beam.smp on linux [2021-02-17 00:23:22,731] - [remote_util] [139626970871552] - INFO - process beam.smp is running on 172.23.122.160: with pid 37992 [2021-02-17 00:23:22,732] - [basetestcase] [139626970871552] - INFO - Couchbase started [2021-02-17 00:23:22,733] - [remote_util] [139626970871552] - INFO - SSH Connecting to 172.23.122.93 with username:root, attempt#1 of 5 [2021-02-17 00:23:22,844] - [remote_util] [139626970871552] - INFO - SSH Connected to 172.23.122.93 as root [2021-02-17 00:23:23,101] - [remote_util] [139626970871552] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-17 00:23:23,402] - [remote_util] [139626970871552] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-17 00:23:23,412] - [remote_util] [139626970871552] - INFO - SSH Connecting to 172.23.122.93 with username:root, attempt#1 of 5 [2021-02-17 00:23:23,512] - [remote_util] [139626970871552] - INFO - SSH Connected to 172.23.122.93 as root [2021-02-17 00:23:23,756] - [remote_util] [139626970871552] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-17 00:23:24,056] - [remote_util] [139626970871552] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-17 00:23:24,077] - [remote_util] [139626970871552] - INFO - 172.23.122.93 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-17 00:23:24,135] - [remote_util] [139626970871552] - INFO - Running systemd command on this server [2021-02-17 00:23:24,136] - [remote_util] [139626970871552] - INFO - running command.raw on 172.23.122.93: systemctl stop couchbase-server.service [2021-02-17 00:23:25,769] - [remote_util] [139626970871552] - INFO - command executed successfully with root [2021-02-17 00:23:25,770] - [basetestcase] [139626970871552] - INFO - Couchbase stopped [2021-02-17 00:23:25,770] - [basetestcase] [139626970871552] - INFO - sleep for 5 secs. ... [2021-02-17 00:23:30,775] - [remote_util] [139626970871552] - INFO - running command.raw on 172.23.122.93: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-17 00:23:30,793] - [remote_util] [139626970871552] - INFO - command executed successfully with root [2021-02-17 00:23:30,793] - [remote_util] [139626970871552] - INFO - running command.raw on 172.23.122.93: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-17 00:23:30,848] - [remote_util] [139626970871552] - INFO - command executed successfully with root [2021-02-17 00:23:30,850] - [remote_util] [139626970871552] - INFO - SSH Connecting to 172.23.122.93 with username:root, attempt#1 of 5 [2021-02-17 00:23:30,949] - [remote_util] [139626970871552] - INFO - SSH Connected to 172.23.122.93 as root [2021-02-17 00:23:31,195] - [remote_util] [139626970871552] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-17 00:23:31,497] - [remote_util] [139626970871552] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-17 00:23:31,560] - [remote_util] [139626970871552] - INFO - 172.23.122.93 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-17 00:23:31,560] - [remote_util] [139626970871552] - INFO - Checking for process beam.smp on linux [2021-02-17 00:23:31,639] - [remote_util] [139626970871552] - INFO - Starting couchbase server [2021-02-17 00:23:31,740] - [remote_util] [139626970871552] - INFO - Running systemd command on this server [2021-02-17 00:23:31,740] - [remote_util] [139626970871552] - INFO - running command.raw on 172.23.122.93: systemctl start couchbase-server.service [2021-02-17 00:23:31,764] - [remote_util] [139626970871552] - INFO - command executed successfully with root [2021-02-17 00:23:31,764] - [remote_util] [139626970871552] - INFO - 172.23.122.93:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-17 00:23:36,769] - [remote_util] [139626970871552] - INFO - running command.raw on 172.23.122.93: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-17 00:23:36,796] - [remote_util] [139626970871552] - INFO - command executed successfully with root [2021-02-17 00:23:36,796] - [remote_util] [139626970871552] - INFO - Couchbase server status: [' Process: 88722 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-17 00:23:36,796] - [remote_util] [139626970871552] - INFO - Checking for process beam.smp on linux [2021-02-17 00:23:36,865] - [remote_util] [139626970871552] - INFO - process beam.smp is running on 172.23.122.93: with pid 88867 [2021-02-17 00:23:36,865] - [basetestcase] [139626970871552] - INFO - Couchbase started [2021-02-17 00:23:36,866] - [basetestcase] [139626970871552] - INFO - sleep for 10 secs. ... [2021-02-17 00:23:47,619] - [task] [139626962478848] - INFO - server: ip:172.23.122.91 port:8091 ssh_username:root, nodes/self [2021-02-17 00:23:47,623] - [task] [139626962478848] - INFO - {'uptime': '110', 'memoryTotal': 4201684992, 'memoryFree': 3587334144, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.122.91:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4476-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.122.91', 'ip': '172.23.122.91', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} [2021-02-17 00:23:47,625] - [rest_client] [139626962478848] - ERROR - GET http://172.23.122.91:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-02-17 00:23:47,625] - [rest_client] [139626962478848] - INFO - pools/default params : memoryQuota=2147 [2021-02-17 00:23:47,631] - [rest_client] [139626962478848] - INFO - --> init_node_services(Administrator,password,172.23.122.91,8091,['kv']) [2021-02-17 00:23:47,631] - [rest_client] [139626962478848] - INFO - /node/controller/setupServices params on 172.23.122.91: 8091:hostname=172.23.122.91&user=Administrator&password=password&services=kv [2021-02-17 00:23:47,660] - [rest_client] [139626962478848] - INFO - --> in init_cluster...Administrator,password,8091 [2021-02-17 00:23:47,660] - [rest_client] [139626962478848] - INFO - settings/web params on 172.23.122.91:8091:port=8091&username=Administrator&password=password [2021-02-17 00:23:47,716] - [rest_client] [139626962478848] - INFO - --> status:True [2021-02-17 00:23:47,718] - [remote_util] [139626962478848] - INFO - SSH Connecting to 172.23.122.91 with username:root, attempt#1 of 5 [2021-02-17 00:23:47,820] - [remote_util] [139626962478848] - INFO - SSH Connected to 172.23.122.91 as root [2021-02-17 00:23:48,104] - [remote_util] [139626962478848] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-17 00:23:48,401] - [remote_util] [139626962478848] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-17 00:23:48,411] - [remote_util] [139626962478848] - INFO - running command.raw on 172.23.122.91: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-02-17 00:23:48,465] - [remote_util] [139626962478848] - INFO - command executed successfully with root [2021-02-17 00:23:48,465] - [remote_util] [139626962478848] - INFO - ['ok'] [2021-02-17 00:23:48,468] - [rest_client] [139626962478848] - INFO - /diag/eval status on 172.23.122.91:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-17 00:23:48,470] - [rest_client] [139626962478848] - INFO - /diag/eval status on 172.23.122.91:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-17 00:23:48,476] - [rest_client] [139626962478848] - INFO - settings/indexes params : storageMode=plasma [2021-02-17 00:23:48,495] - [task] [139626962478848] - INFO - server: ip:172.23.122.158 port:8091 ssh_username:root, nodes/self [2021-02-17 00:23:48,498] - [task] [139626962478848] - INFO - {'uptime': '95', 'memoryTotal': 4201684992, 'memoryFree': 3623051264, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.122.158:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4476-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.122.158', 'ip': '172.23.122.158', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} [2021-02-17 00:23:48,500] - [rest_client] [139626962478848] - ERROR - GET http://172.23.122.158:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-02-17 00:23:48,501] - [rest_client] [139626962478848] - INFO - pools/default params : memoryQuota=2147 [2021-02-17 00:23:48,506] - [rest_client] [139626962478848] - INFO - --> in init_cluster...Administrator,password,8091 [2021-02-17 00:23:48,506] - [rest_client] [139626962478848] - INFO - settings/web params on 172.23.122.158:8091:port=8091&username=Administrator&password=password [2021-02-17 00:23:48,562] - [rest_client] [139626962478848] - INFO - --> status:True [2021-02-17 00:23:48,564] - [remote_util] [139626962478848] - INFO - SSH Connecting to 172.23.122.158 with username:root, attempt#1 of 5 [2021-02-17 00:23:48,668] - [remote_util] [139626962478848] - INFO - SSH Connected to 172.23.122.158 as root [2021-02-17 00:23:48,918] - [remote_util] [139626962478848] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-17 00:23:49,215] - [remote_util] [139626962478848] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-17 00:23:49,224] - [remote_util] [139626962478848] - INFO - running command.raw on 172.23.122.158: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-02-17 00:23:49,280] - [remote_util] [139626962478848] - INFO - command executed successfully with root [2021-02-17 00:23:49,280] - [remote_util] [139626962478848] - INFO - ['ok'] [2021-02-17 00:23:49,282] - [rest_client] [139626962478848] - INFO - /diag/eval status on 172.23.122.158:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-17 00:23:49,285] - [rest_client] [139626962478848] - INFO - /diag/eval status on 172.23.122.158:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-17 00:23:49,289] - [rest_client] [139626962478848] - INFO - settings/indexes params : storageMode=plasma [2021-02-17 00:23:49,307] - [task] [139626962478848] - INFO - server: ip:172.23.122.159 port:8091 ssh_username:root, nodes/self [2021-02-17 00:23:49,310] - [task] [139626962478848] - INFO - {'uptime': '84', 'memoryTotal': 4201684992, 'memoryFree': 3623829504, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.122.159:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4476-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.122.159', 'ip': '172.23.122.159', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} [2021-02-17 00:23:49,312] - [rest_client] [139626962478848] - ERROR - GET http://172.23.122.159:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-02-17 00:23:49,312] - [rest_client] [139626962478848] - INFO - pools/default params : memoryQuota=2147 [2021-02-17 00:23:49,318] - [rest_client] [139626962478848] - INFO - --> in init_cluster...Administrator,password,8091 [2021-02-17 00:23:49,318] - [rest_client] [139626962478848] - INFO - settings/web params on 172.23.122.159:8091:port=8091&username=Administrator&password=password [2021-02-17 00:23:49,374] - [rest_client] [139626962478848] - INFO - --> status:True [2021-02-17 00:23:49,375] - [remote_util] [139626962478848] - INFO - SSH Connecting to 172.23.122.159 with username:root, attempt#1 of 5 [2021-02-17 00:23:49,481] - [remote_util] [139626962478848] - INFO - SSH Connected to 172.23.122.159 as root [2021-02-17 00:23:49,764] - [remote_util] [139626962478848] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-17 00:23:50,056] - [remote_util] [139626962478848] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-17 00:23:50,066] - [remote_util] [139626962478848] - INFO - running command.raw on 172.23.122.159: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-02-17 00:23:50,120] - [remote_util] [139626962478848] - INFO - command executed successfully with root [2021-02-17 00:23:50,120] - [remote_util] [139626962478848] - INFO - ['ok'] [2021-02-17 00:23:50,123] - [rest_client] [139626962478848] - INFO - /diag/eval status on 172.23.122.159:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-17 00:23:50,125] - [rest_client] [139626962478848] - INFO - /diag/eval status on 172.23.122.159:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-17 00:23:50,128] - [rest_client] [139626962478848] - INFO - settings/indexes params : storageMode=plasma [2021-02-17 00:23:50,148] - [task] [139626962478848] - INFO - server: ip:172.23.122.92 port:8091 ssh_username:root, nodes/self [2021-02-17 00:23:50,151] - [task] [139626962478848] - INFO - {'uptime': '70', 'memoryTotal': 4201684992, 'memoryFree': 3641872384, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.122.92:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4476-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.122.92', 'ip': '172.23.122.92', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} [2021-02-17 00:23:50,153] - [rest_client] [139626962478848] - ERROR - GET http://172.23.122.92:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-02-17 00:23:50,154] - [rest_client] [139626962478848] - INFO - pools/default params : memoryQuota=2147 [2021-02-17 00:23:50,160] - [rest_client] [139626962478848] - INFO - --> in init_cluster...Administrator,password,8091 [2021-02-17 00:23:50,160] - [rest_client] [139626962478848] - INFO - settings/web params on 172.23.122.92:8091:port=8091&username=Administrator&password=password [2021-02-17 00:23:50,219] - [rest_client] [139626962478848] - INFO - --> status:True [2021-02-17 00:23:50,221] - [remote_util] [139626962478848] - INFO - SSH Connecting to 172.23.122.92 with username:root, attempt#1 of 5 [2021-02-17 00:23:50,324] - [remote_util] [139626962478848] - INFO - SSH Connected to 172.23.122.92 as root [2021-02-17 00:23:50,590] - [remote_util] [139626962478848] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-17 00:23:50,892] - [remote_util] [139626962478848] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-17 00:23:50,901] - [remote_util] [139626962478848] - INFO - running command.raw on 172.23.122.92: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-02-17 00:23:50,955] - [remote_util] [139626962478848] - INFO - command executed successfully with root [2021-02-17 00:23:50,956] - [remote_util] [139626962478848] - INFO - ['ok'] [2021-02-17 00:23:50,958] - [rest_client] [139626962478848] - INFO - /diag/eval status on 172.23.122.92:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-17 00:23:50,960] - [rest_client] [139626962478848] - INFO - /diag/eval status on 172.23.122.92:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-17 00:23:50,964] - [rest_client] [139626962478848] - INFO - settings/indexes params : storageMode=plasma [2021-02-17 00:23:50,982] - [task] [139626962478848] - INFO - server: ip:172.23.122.156 port:8091 ssh_username:root, nodes/self [2021-02-17 00:23:50,985] - [task] [139626962478848] - INFO - {'uptime': '54', 'memoryTotal': 4201684992, 'memoryFree': 3593981952, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.122.156:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4476-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.122.156', 'ip': '172.23.122.156', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} [2021-02-17 00:23:50,986] - [rest_client] [139626962478848] - ERROR - GET http://172.23.122.156:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-02-17 00:23:50,987] - [rest_client] [139626962478848] - INFO - pools/default params : memoryQuota=2147 [2021-02-17 00:23:50,992] - [rest_client] [139626962478848] - INFO - --> in init_cluster...Administrator,password,8091 [2021-02-17 00:23:50,992] - [rest_client] [139626962478848] - INFO - settings/web params on 172.23.122.156:8091:port=8091&username=Administrator&password=password [2021-02-17 00:23:51,050] - [rest_client] [139626962478848] - INFO - --> status:True [2021-02-17 00:23:51,052] - [remote_util] [139626962478848] - INFO - SSH Connecting to 172.23.122.156 with username:root, attempt#1 of 5 [2021-02-17 00:23:51,155] - [remote_util] [139626962478848] - INFO - SSH Connected to 172.23.122.156 as root [2021-02-17 00:23:51,442] - [remote_util] [139626962478848] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-17 00:23:51,740] - [remote_util] [139626962478848] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-17 00:23:51,749] - [remote_util] [139626962478848] - INFO - running command.raw on 172.23.122.156: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-02-17 00:23:51,804] - [remote_util] [139626962478848] - INFO - command executed successfully with root [2021-02-17 00:23:51,804] - [remote_util] [139626962478848] - INFO - ['ok'] [2021-02-17 00:23:51,809] - [rest_client] [139626962478848] - INFO - /diag/eval status on 172.23.122.156:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-17 00:23:51,811] - [rest_client] [139626962478848] - INFO - /diag/eval status on 172.23.122.156:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-17 00:23:51,816] - [rest_client] [139626962478848] - INFO - settings/indexes params : storageMode=plasma [2021-02-17 00:23:51,835] - [task] [139626962478848] - INFO - server: ip:172.23.122.161 port:8091 ssh_username:root, nodes/self [2021-02-17 00:23:51,838] - [task] [139626962478848] - INFO - {'uptime': '44', 'memoryTotal': 4201684992, 'memoryFree': 3625037824, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.122.161:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4476-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.122.161', 'ip': '172.23.122.161', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} [2021-02-17 00:23:51,840] - [rest_client] [139626962478848] - ERROR - GET http://172.23.122.161:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-02-17 00:23:51,840] - [rest_client] [139626962478848] - INFO - pools/default params : memoryQuota=2147 [2021-02-17 00:23:51,846] - [rest_client] [139626962478848] - INFO - --> in init_cluster...Administrator,password,8091 [2021-02-17 00:23:51,846] - [rest_client] [139626962478848] - INFO - settings/web params on 172.23.122.161:8091:port=8091&username=Administrator&password=password [2021-02-17 00:23:51,906] - [rest_client] [139626962478848] - INFO - --> status:True [2021-02-17 00:23:51,908] - [remote_util] [139626962478848] - INFO - SSH Connecting to 172.23.122.161 with username:root, attempt#1 of 5 [2021-02-17 00:23:52,015] - [remote_util] [139626962478848] - INFO - SSH Connected to 172.23.122.161 as root [2021-02-17 00:23:52,291] - [remote_util] [139626962478848] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-17 00:23:52,587] - [remote_util] [139626962478848] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-17 00:23:52,596] - [remote_util] [139626962478848] - INFO - running command.raw on 172.23.122.161: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-02-17 00:23:52,652] - [remote_util] [139626962478848] - INFO - command executed successfully with root [2021-02-17 00:23:52,652] - [remote_util] [139626962478848] - INFO - ['ok'] [2021-02-17 00:23:52,654] - [rest_client] [139626962478848] - INFO - /diag/eval status on 172.23.122.161:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-17 00:23:52,657] - [rest_client] [139626962478848] - INFO - /diag/eval status on 172.23.122.161:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-17 00:23:52,661] - [rest_client] [139626962478848] - INFO - settings/indexes params : storageMode=plasma [2021-02-17 00:23:52,684] - [task] [139626962478848] - INFO - server: ip:172.23.122.160 port:8091 ssh_username:root, nodes/self [2021-02-17 00:23:52,687] - [task] [139626962478848] - INFO - {'uptime': '29', 'memoryTotal': 4201684992, 'memoryFree': 3630473216, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.122.160:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4476-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.122.160', 'ip': '172.23.122.160', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} [2021-02-17 00:23:52,689] - [rest_client] [139626962478848] - ERROR - GET http://172.23.122.160:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-02-17 00:23:52,689] - [rest_client] [139626962478848] - INFO - pools/default params : memoryQuota=2147 [2021-02-17 00:23:52,695] - [rest_client] [139626962478848] - INFO - --> in init_cluster...Administrator,password,8091 [2021-02-17 00:23:52,695] - [rest_client] [139626962478848] - INFO - settings/web params on 172.23.122.160:8091:port=8091&username=Administrator&password=password [2021-02-17 00:23:52,751] - [rest_client] [139626962478848] - INFO - --> status:True [2021-02-17 00:23:52,752] - [remote_util] [139626962478848] - INFO - SSH Connecting to 172.23.122.160 with username:root, attempt#1 of 5 [2021-02-17 00:23:52,854] - [remote_util] [139626962478848] - INFO - SSH Connected to 172.23.122.160 as root [2021-02-17 00:23:53,137] - [remote_util] [139626962478848] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-17 00:23:53,432] - [remote_util] [139626962478848] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-17 00:23:53,441] - [remote_util] [139626962478848] - INFO - running command.raw on 172.23.122.160: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-02-17 00:23:53,496] - [remote_util] [139626962478848] - INFO - command executed successfully with root [2021-02-17 00:23:53,497] - [remote_util] [139626962478848] - INFO - ['ok'] [2021-02-17 00:23:53,499] - [rest_client] [139626962478848] - INFO - /diag/eval status on 172.23.122.160:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-17 00:23:53,500] - [rest_client] [139626962478848] - INFO - /diag/eval status on 172.23.122.160:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-17 00:23:53,505] - [rest_client] [139626962478848] - INFO - settings/indexes params : storageMode=plasma [2021-02-17 00:23:53,522] - [task] [139626962478848] - INFO - server: ip:172.23.122.93 port:8091 ssh_username:root, nodes/self [2021-02-17 00:23:53,525] - [task] [139626962478848] - INFO - {'uptime': '19', 'memoryTotal': 4201689088, 'memoryFree': 3555332096, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.122.93:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4476-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@cb.local', 'ip': '172.23.122.93', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} [2021-02-17 00:23:53,527] - [rest_client] [139626962478848] - ERROR - GET http://172.23.122.93:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-02-17 00:23:53,528] - [rest_client] [139626962478848] - INFO - pools/default params : memoryQuota=2147 [2021-02-17 00:23:53,533] - [rest_client] [139626962478848] - INFO - --> in init_cluster...Administrator,password,8091 [2021-02-17 00:23:53,533] - [rest_client] [139626962478848] - INFO - settings/web params on 172.23.122.93:8091:port=8091&username=Administrator&password=password [2021-02-17 00:23:53,589] - [rest_client] [139626962478848] - INFO - --> status:True [2021-02-17 00:23:53,591] - [remote_util] [139626962478848] - INFO - SSH Connecting to 172.23.122.93 with username:root, attempt#1 of 5 [2021-02-17 00:23:53,692] - [remote_util] [139626962478848] - INFO - SSH Connected to 172.23.122.93 as root [2021-02-17 00:23:53,937] - [remote_util] [139626962478848] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-17 00:23:54,235] - [remote_util] [139626962478848] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-17 00:23:54,245] - [remote_util] [139626962478848] - INFO - running command.raw on 172.23.122.93: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-02-17 00:23:54,299] - [remote_util] [139626962478848] - INFO - command executed successfully with root [2021-02-17 00:23:54,299] - [remote_util] [139626962478848] - INFO - ['ok'] [2021-02-17 00:23:54,302] - [rest_client] [139626962478848] - INFO - /diag/eval status on 172.23.122.93:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-17 00:23:54,304] - [rest_client] [139626962478848] - INFO - /diag/eval status on 172.23.122.93:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-17 00:23:54,308] - [rest_client] [139626962478848] - INFO - settings/indexes params : storageMode=plasma [2021-02-17 00:23:54,330] - [basetestcase] [139626970871552] - INFO - **** add built-in 'cbadminbucket' user to node 172.23.122.91 **** [2021-02-17 00:23:54,340] - [rest_client] [139626970871552] - ERROR - DELETE http://172.23.122.91:8091/settings/rbac/users/local/cbadminbucket body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:password [2021-02-17 00:23:54,341] - [internal_user] [139626970871552] - INFO - Exception while deleting user. Exception is -b'"User was not found."' [2021-02-17 00:23:54,400] - [basetestcase] [139626970871552] - INFO - sleep for 5 secs. ... [2021-02-17 00:23:59,405] - [basetestcase] [139626970871552] - INFO - **** add 'admin' role to 'cbadminbucket' user **** [2021-02-17 00:23:59,417] - [basetestcase] [139626970871552] - INFO - done initializing cluster [2021-02-17 00:24:00,328] - [task] [139626962478848] - INFO - adding node 172.23.122.158:8091 to cluster [2021-02-17 00:24:00,329] - [rest_client] [139626962478848] - INFO - adding remote node @172.23.122.158:8091 to this cluster @172.23.122.91:8091 [2021-02-17 00:24:02,436] - [task] [139626962478848] - INFO - adding node 172.23.122.159:8091 to cluster [2021-02-17 00:24:02,436] - [rest_client] [139626962478848] - INFO - adding remote node @172.23.122.159:8091 to this cluster @172.23.122.91:8091 [2021-02-17 00:24:04,663] - [task] [139626962478848] - INFO - adding node 172.23.122.92:8091 to cluster [2021-02-17 00:24:04,663] - [rest_client] [139626962478848] - INFO - adding remote node @172.23.122.92:8091 to this cluster @172.23.122.91:8091 [2021-02-17 00:24:06,820] - [task] [139626962478848] - INFO - adding node 172.23.122.156:8091 to cluster [2021-02-17 00:24:06,820] - [rest_client] [139626962478848] - INFO - adding remote node @172.23.122.156:8091 to this cluster @172.23.122.91:8091 [2021-02-17 00:24:09,137] - [task] [139626962478848] - INFO - adding node 172.23.122.161:8091 to cluster [2021-02-17 00:24:09,137] - [rest_client] [139626962478848] - INFO - adding remote node @172.23.122.161:8091 to this cluster @172.23.122.91:8091 [2021-02-17 00:24:11,378] - [task] [139626962478848] - INFO - adding node 172.23.122.160:8091 to cluster [2021-02-17 00:24:11,378] - [rest_client] [139626962478848] - INFO - adding remote node @172.23.122.160:8091 to this cluster @172.23.122.91:8091 [2021-02-17 00:24:13,610] - [rest_client] [139626962478848] - INFO - rebalance params : {'knownNodes': 'ns_1@172.23.122.156,ns_1@172.23.122.158,ns_1@172.23.122.159,ns_1@172.23.122.160,ns_1@172.23.122.161,ns_1@172.23.122.91,ns_1@172.23.122.92', 'ejectedNodes': '', 'user': 'Administrator', 'password': 'password'} [2021-02-17 00:24:13,639] - [rest_client] [139626962478848] - INFO - rebalance operation started [2021-02-17 00:24:13,641] - [rest_client] [139626962478848] - INFO - rebalance percentage : 0.00 % [2021-02-17 00:24:13,641] - [task] [139626962478848] - INFO - Rebalance - status: running, progress: 0.00% [2021-02-17 00:24:23,661] - [task] [139626962478848] - INFO - Rebalance - status: none, progress: 100.00% [2021-02-17 00:24:23,670] - [task] [139626962478848] - INFO - rebalancing was completed with progress: 100% in 10.031243562698364 sec [2021-02-17 00:24:23,672] - [remote_util] [139626970871552] - INFO - SSH Connecting to 172.23.122.91 with username:root, attempt#1 of 5 [2021-02-17 00:24:23,777] - [remote_util] [139626970871552] - INFO - SSH Connected to 172.23.122.91 as root [2021-02-17 00:24:24,062] - [remote_util] [139626970871552] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-17 00:24:24,367] - [remote_util] [139626970871552] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-17 00:24:24,376] - [remote_util] [139626970871552] - INFO - running command.raw on 172.23.122.91: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-02-17 00:24:24,432] - [remote_util] [139626970871552] - INFO - command executed successfully with root [2021-02-17 00:24:24,433] - [remote_util] [139626970871552] - INFO - ['ok'] [2021-02-17 00:24:24,433] - [basetestcase] [139626970871552] - INFO - Enabled diag/eval for non-local hosts from 172.23.122.91 [2021-02-17 00:24:24,433] - [basetestcase] [139626970871552] - INFO - ============== basetestcase setup was finished for test #5 test_kv_rebalance_out_when_existing_eventing_node_is_processing_mutations ============== [2021-02-17 00:24:24,442] - [remote_util] [139626970871552] - INFO - SSH Connecting to 172.23.122.91 with username:root, attempt#1 of 5 [2021-02-17 00:24:24,545] - [remote_util] [139626970871552] - INFO - SSH Connected to 172.23.122.91 as root [2021-02-17 00:24:24,831] - [remote_util] [139626970871552] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-17 00:24:25,133] - [remote_util] [139626970871552] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-17 00:24:25,135] - [remote_util] [139626970871552] - INFO - SSH Connecting to 172.23.122.158 with username:root, attempt#1 of 5 [2021-02-17 00:24:25,236] - [remote_util] [139626970871552] - INFO - SSH Connected to 172.23.122.158 as root [2021-02-17 00:24:25,491] - [remote_util] [139626970871552] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-17 00:24:25,792] - [remote_util] [139626970871552] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-17 00:24:25,796] - [remote_util] [139626970871552] - INFO - SSH Connecting to 172.23.122.159 with username:root, attempt#1 of 5 [2021-02-17 00:24:25,899] - [remote_util] [139626970871552] - INFO - SSH Connected to 172.23.122.159 as root [2021-02-17 00:24:26,173] - [remote_util] [139626970871552] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-17 00:24:26,470] - [remote_util] [139626970871552] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-17 00:24:26,472] - [remote_util] [139626970871552] - INFO - SSH Connecting to 172.23.122.92 with username:root, attempt#1 of 5 [2021-02-17 00:24:26,574] - [remote_util] [139626970871552] - INFO - SSH Connected to 172.23.122.92 as root [2021-02-17 00:24:26,839] - [remote_util] [139626970871552] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-17 00:24:27,146] - [remote_util] [139626970871552] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-17 00:24:27,148] - [remote_util] [139626970871552] - INFO - SSH Connecting to 172.23.122.156 with username:root, attempt#1 of 5 [2021-02-17 00:24:27,251] - [remote_util] [139626970871552] - INFO - SSH Connected to 172.23.122.156 as root [2021-02-17 00:24:27,537] - [remote_util] [139626970871552] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-17 00:24:27,839] - [remote_util] [139626970871552] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-17 00:24:27,841] - [remote_util] [139626970871552] - INFO - SSH Connecting to 172.23.122.161 with username:root, attempt#1 of 5 [2021-02-17 00:24:27,942] - [remote_util] [139626970871552] - INFO - SSH Connected to 172.23.122.161 as root [2021-02-17 00:24:28,219] - [remote_util] [139626970871552] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-17 00:24:28,515] - [remote_util] [139626970871552] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-17 00:24:28,518] - [remote_util] [139626970871552] - INFO - SSH Connecting to 172.23.122.160 with username:root, attempt#1 of 5 [2021-02-17 00:24:28,622] - [remote_util] [139626970871552] - INFO - SSH Connected to 172.23.122.160 as root [2021-02-17 00:24:28,906] - [remote_util] [139626970871552] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-17 00:24:29,202] - [remote_util] [139626970871552] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-17 00:24:29,204] - [remote_util] [139626970871552] - INFO - SSH Connecting to 172.23.122.93 with username:root, attempt#1 of 5 [2021-02-17 00:24:29,307] - [remote_util] [139626970871552] - INFO - SSH Connected to 172.23.122.93 as root [2021-02-17 00:24:29,563] - [remote_util] [139626970871552] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-17 00:24:29,867] - [remote_util] [139626970871552] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-17 00:24:54,309] - [basetestcase] [139626970871552] - INFO - ------- Cluster statistics ------- [2021-02-17 00:24:54,309] - [basetestcase] [139626970871552] - INFO - 172.23.122.156:8091 => {'services': ['eventing'], 'cpu_utilization': 3.29476861167002, 'mem_free': 3549057024, 'mem_total': 4201684992, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-02-17 00:24:54,309] - [basetestcase] [139626970871552] - INFO - 172.23.122.158:8091 => {'services': ['kv'], 'cpu_utilization': 0.7014028056112225, 'mem_free': 3586334720, 'mem_total': 4201684992, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-02-17 00:24:54,309] - [basetestcase] [139626970871552] - INFO - 172.23.122.159:8091 => {'services': ['kv'], 'cpu_utilization': 14.86280487804878, 'mem_free': 3597762560, 'mem_total': 4201684992, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-02-17 00:24:54,309] - [basetestcase] [139626970871552] - INFO - 172.23.122.160:8091 => {'services': ['index', 'n1ql'], 'cpu_utilization': 0.3772635814889336, 'mem_free': 3556720640, 'mem_total': 4201684992, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-02-17 00:24:54,309] - [basetestcase] [139626970871552] - INFO - 172.23.122.161:8091 => {'services': ['eventing'], 'cpu_utilization': 1.078775715002509, 'mem_free': 3581816832, 'mem_total': 4201684992, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-02-17 00:24:54,310] - [basetestcase] [139626970871552] - INFO - 172.23.122.91:8091 => {'services': ['kv'], 'cpu_utilization': 0.5527638190954773, 'mem_free': 3547136000, 'mem_total': 4201684992, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-02-17 00:24:54,310] - [basetestcase] [139626970871552] - INFO - 172.23.122.92:8091 => {'services': ['eventing'], 'cpu_utilization': 10.10356150543066, 'mem_free': 3604111360, 'mem_total': 4201684992, 'swap_mem_used': 96731136, 'swap_mem_total': 3758092288} [2021-02-17 00:24:54,310] - [basetestcase] [139626970871552] - INFO - --- End of cluster statistics --- [2021-02-17 00:24:54,311] - [remote_util] [139626970871552] - INFO - SSH Connecting to 172.23.122.91 with username:root, attempt#1 of 5 [2021-02-17 00:24:54,412] - [remote_util] [139626970871552] - INFO - SSH Connected to 172.23.122.91 as root [2021-02-17 00:24:54,697] - [remote_util] [139626970871552] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-17 00:24:55,000] - [remote_util] [139626970871552] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-17 00:24:58,535] - [basetestcase] [139626970871552] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.160 port:8091 ssh_username:root] [2021-02-17 00:24:58,542] - [basetestcase] [139626970871552] - INFO - list of eventing nodes in cluster: [ip:172.23.122.156 port:8091 ssh_username:root, ip:172.23.122.161 port:8091 ssh_username:root, ip:172.23.122.92 port:8091 ssh_username:root] [2021-02-17 00:24:58,549] - [rest_client] [139626970871552] - INFO - settings/indexes params : storageMode=plasma [2021-02-17 00:24:58,554] - [eventing_base] [139626970871552] - INFO - Setting the min possible memory quota so that adding mode nodes to the cluster wouldn't be a problem. [2021-02-17 00:24:58,555] - [rest_client] [139626970871552] - INFO - pools/default params : memoryQuota=330 [2021-02-17 00:24:58,564] - [rest_client] [139626970871552] - INFO - pools/default params : indexMemoryQuota=256 [2021-02-17 00:24:58,572] - [rest_client] [139626970871552] - INFO - pools/default params : eventingMemoryQuota=512 [2021-02-17 00:24:58,598] - [basetestcase] [139626970871552] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.160 port:8091 ssh_username:root] [2021-02-17 00:24:58,603] - [rest_client] [139626970871552] - INFO - pools/default params : memoryQuota=700 [2021-02-17 00:24:58,612] - [eventing_rebalance_collection] [139626970871552] - INFO - 200 [2021-02-17 00:24:59,384] - [rest_client] [139626962478848] - INFO - http://172.23.122.91:8091/pools/default/buckets with param: name=src_bucket&ramQuotaMB=200&authType=none&replicaNumber=0&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore [2021-02-17 00:24:59,413] - [rest_client] [139626962478848] - INFO - 0.03 seconds to create bucket src_bucket [2021-02-17 00:24:59,414] - [bucket_helper] [139626962478848] - INFO - waiting for memcached bucket : src_bucket in 172.23.122.91 to accept set ops [2021-02-17 00:25:00,178] - [data_helper] [139626962478848] - INFO - creating direct client 172.23.122.158:11210 src_bucket [2021-02-17 00:25:00,262] - [data_helper] [139626962478848] - INFO - creating direct client 172.23.122.159:11210 src_bucket [2021-02-17 00:25:00,350] - [data_helper] [139626962478848] - INFO - creating direct client 172.23.122.91:11210 src_bucket [2021-02-17 00:25:00,623] - [data_helper] [139626962478848] - INFO - creating direct client 172.23.122.158:11210 src_bucket [2021-02-17 00:25:00,719] - [data_helper] [139626962478848] - INFO - creating direct client 172.23.122.159:11210 src_bucket [2021-02-17 00:25:00,803] - [data_helper] [139626962478848] - INFO - creating direct client 172.23.122.91:11210 src_bucket [2021-02-17 00:25:00,997] - [task] [139626962478848] - WARNING - vbucket map not ready after try 0 [2021-02-17 00:25:00,998] - [bucket_helper] [139626962478848] - INFO - waiting for memcached bucket : src_bucket in 172.23.122.91 to accept set ops [2021-02-17 00:25:01,107] - [data_helper] [139626962478848] - INFO - creating direct client 172.23.122.158:11210 src_bucket [2021-02-17 00:25:01,184] - [data_helper] [139626962478848] - INFO - creating direct client 172.23.122.159:11210 src_bucket [2021-02-17 00:25:01,264] - [data_helper] [139626962478848] - INFO - creating direct client 172.23.122.91:11210 src_bucket [2021-02-17 00:25:01,498] - [data_helper] [139626962478848] - INFO - creating direct client 172.23.122.158:11210 src_bucket [2021-02-17 00:25:01,579] - [data_helper] [139626962478848] - INFO - creating direct client 172.23.122.159:11210 src_bucket [2021-02-17 00:25:01,655] - [data_helper] [139626962478848] - INFO - creating direct client 172.23.122.91:11210 src_bucket [2021-02-17 00:25:01,845] - [task] [139626962478848] - WARNING - vbucket map not ready after try 1 [2021-02-17 00:25:01,845] - [bucket_helper] [139626962478848] - INFO - waiting for memcached bucket : src_bucket in 172.23.122.91 to accept set ops [2021-02-17 00:25:01,955] - [data_helper] [139626962478848] - INFO - creating direct client 172.23.122.158:11210 src_bucket [2021-02-17 00:25:02,035] - [data_helper] [139626962478848] - INFO - creating direct client 172.23.122.159:11210 src_bucket [2021-02-17 00:25:02,231] - [data_helper] [139626962478848] - INFO - creating direct client 172.23.122.91:11210 src_bucket [2021-02-17 00:25:02,357] - [data_helper] [139626962478848] - INFO - creating direct client 172.23.122.158:11210 src_bucket [2021-02-17 00:25:02,450] - [data_helper] [139626962478848] - INFO - creating direct client 172.23.122.159:11210 src_bucket [2021-02-17 00:25:02,649] - [data_helper] [139626962478848] - INFO - creating direct client 172.23.122.91:11210 src_bucket [2021-02-17 00:25:02,714] - [task] [139626962478848] - WARNING - vbucket map not ready after try 2 [2021-02-17 00:25:02,714] - [bucket_helper] [139626962478848] - INFO - waiting for memcached bucket : src_bucket in 172.23.122.91 to accept set ops [2021-02-17 00:25:02,825] - [data_helper] [139626962478848] - INFO - creating direct client 172.23.122.158:11210 src_bucket [2021-02-17 00:25:03,016] - [data_helper] [139626962478848] - INFO - creating direct client 172.23.122.159:11210 src_bucket [2021-02-17 00:25:03,095] - [data_helper] [139626962478848] - INFO - creating direct client 172.23.122.91:11210 src_bucket [2021-02-17 00:25:03,217] - [data_helper] [139626962478848] - INFO - creating direct client 172.23.122.158:11210 src_bucket [2021-02-17 00:25:03,418] - [data_helper] [139626962478848] - INFO - creating direct client 172.23.122.159:11210 src_bucket [2021-02-17 00:25:03,505] - [data_helper] [139626962478848] - INFO - creating direct client 172.23.122.91:11210 src_bucket [2021-02-17 00:25:03,575] - [task] [139626962478848] - WARNING - vbucket map not ready after try 3 [2021-02-17 00:25:03,575] - [bucket_helper] [139626962478848] - INFO - waiting for memcached bucket : src_bucket in 172.23.122.91 to accept set ops [2021-02-17 00:25:03,821] - [data_helper] [139626962478848] - INFO - creating direct client 172.23.122.158:11210 src_bucket [2021-02-17 00:25:03,906] - [data_helper] [139626962478848] - INFO - creating direct client 172.23.122.159:11210 src_bucket [2021-02-17 00:25:03,990] - [data_helper] [139626962478848] - INFO - creating direct client 172.23.122.91:11210 src_bucket [2021-02-17 00:25:04,222] - [data_helper] [139626962478848] - INFO - creating direct client 172.23.122.158:11210 src_bucket [2021-02-17 00:25:04,298] - [data_helper] [139626962478848] - INFO - creating direct client 172.23.122.159:11210 src_bucket [2021-02-17 00:25:04,375] - [data_helper] [139626962478848] - INFO - creating direct client 172.23.122.91:11210 src_bucket [2021-02-17 00:25:04,559] - [task] [139626962478848] - WARNING - vbucket map not ready after try 4 [2021-02-17 00:25:04,559] - [bucket_helper] [139626962478848] - INFO - waiting for memcached bucket : src_bucket in 172.23.122.91 to accept set ops [2021-02-17 00:25:04,686] - [data_helper] [139626962478848] - INFO - creating direct client 172.23.122.158:11210 src_bucket [2021-02-17 00:25:04,773] - [data_helper] [139626962478848] - INFO - creating direct client 172.23.122.159:11210 src_bucket [2021-02-17 00:25:04,973] - [data_helper] [139626962478848] - INFO - creating direct client 172.23.122.91:11210 src_bucket [2021-02-17 00:25:05,088] - [data_helper] [139626962478848] - INFO - creating direct client 172.23.122.158:11210 src_bucket [2021-02-17 00:25:05,167] - [data_helper] [139626962478848] - INFO - creating direct client 172.23.122.159:11210 src_bucket [2021-02-17 00:25:05,244] - [data_helper] [139626962478848] - INFO - creating direct client 172.23.122.91:11210 src_bucket [2021-02-17 00:25:05,428] - [task] [139626962478848] - WARNING - vbucket map not ready after try 5 [2021-02-17 00:25:06,455] - [rest_client] [139626962478848] - INFO - http://172.23.122.91:8091/pools/default/buckets with param: name=dst_bucket&ramQuotaMB=200&authType=none&replicaNumber=0&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore [2021-02-17 00:25:06,483] - [rest_client] [139626962478848] - INFO - 0.03 seconds to create bucket dst_bucket [2021-02-17 00:25:06,483] - [bucket_helper] [139626962478848] - INFO - waiting for memcached bucket : dst_bucket in 172.23.122.91 to accept set ops [2021-02-17 00:25:07,123] - [data_helper] [139626962478848] - INFO - creating direct client 172.23.122.158:11210 dst_bucket [2021-02-17 00:25:07,328] - [data_helper] [139626962478848] - INFO - creating direct client 172.23.122.159:11210 dst_bucket [2021-02-17 00:25:07,450] - [data_helper] [139626962478848] - INFO - creating direct client 172.23.122.91:11210 dst_bucket [2021-02-17 00:25:07,568] - [data_helper] [139626962478848] - INFO - creating direct client 172.23.122.158:11210 dst_bucket [2021-02-17 00:25:07,765] - [data_helper] [139626962478848] - INFO - creating direct client 172.23.122.159:11210 dst_bucket [2021-02-17 00:25:07,858] - [data_helper] [139626962478848] - INFO - creating direct client 172.23.122.91:11210 dst_bucket [2021-02-17 00:25:07,926] - [task] [139626962478848] - WARNING - vbucket map not ready after try 0 [2021-02-17 00:25:07,926] - [bucket_helper] [139626962478848] - INFO - waiting for memcached bucket : dst_bucket in 172.23.122.91 to accept set ops [2021-02-17 00:25:08,043] - [data_helper] [139626962478848] - INFO - creating direct client 172.23.122.158:11210 dst_bucket [2021-02-17 00:25:08,260] - [data_helper] [139626962478848] - INFO - creating direct client 172.23.122.159:11210 dst_bucket [2021-02-17 00:25:08,341] - [data_helper] [139626962478848] - INFO - creating direct client 172.23.122.91:11210 dst_bucket [2021-02-17 00:25:08,453] - [data_helper] [139626962478848] - INFO - creating direct client 172.23.122.158:11210 dst_bucket [2021-02-17 00:25:08,649] - [data_helper] [139626962478848] - INFO - creating direct client 172.23.122.159:11210 dst_bucket [2021-02-17 00:25:08,728] - [data_helper] [139626962478848] - INFO - creating direct client 172.23.122.91:11210 dst_bucket [2021-02-17 00:25:08,797] - [task] [139626962478848] - WARNING - vbucket map not ready after try 1 [2021-02-17 00:25:08,797] - [bucket_helper] [139626962478848] - INFO - waiting for memcached bucket : dst_bucket in 172.23.122.91 to accept set ops [2021-02-17 00:25:08,920] - [data_helper] [139626962478848] - INFO - creating direct client 172.23.122.158:11210 dst_bucket [2021-02-17 00:25:09,132] - [data_helper] [139626962478848] - INFO - creating direct client 172.23.122.159:11210 dst_bucket [2021-02-17 00:25:09,217] - [data_helper] [139626962478848] - INFO - creating direct client 172.23.122.91:11210 dst_bucket [2021-02-17 00:25:09,334] - [data_helper] [139626962478848] - INFO - creating direct client 172.23.122.158:11210 dst_bucket [2021-02-17 00:25:09,535] - [data_helper] [139626962478848] - INFO - creating direct client 172.23.122.159:11210 dst_bucket [2021-02-17 00:25:09,615] - [data_helper] [139626962478848] - INFO - creating direct client 172.23.122.91:11210 dst_bucket [2021-02-17 00:25:09,684] - [task] [139626962478848] - WARNING - vbucket map not ready after try 2 [2021-02-17 00:25:09,684] - [bucket_helper] [139626962478848] - INFO - waiting for memcached bucket : dst_bucket in 172.23.122.91 to accept set ops [2021-02-17 00:25:09,922] - [data_helper] [139626962478848] - INFO - creating direct client 172.23.122.158:11210 dst_bucket [2021-02-17 00:25:10,006] - [data_helper] [139626962478848] - INFO - creating direct client 172.23.122.159:11210 dst_bucket [2021-02-17 00:25:10,089] - [data_helper] [139626962478848] - INFO - creating direct client 172.23.122.91:11210 dst_bucket [2021-02-17 00:25:10,333] - [data_helper] [139626962478848] - INFO - creating direct client 172.23.122.158:11210 dst_bucket [2021-02-17 00:25:10,420] - [data_helper] [139626962478848] - INFO - creating direct client 172.23.122.159:11210 dst_bucket [2021-02-17 00:25:10,506] - [data_helper] [139626962478848] - INFO - creating direct client 172.23.122.91:11210 dst_bucket [2021-02-17 00:25:10,572] - [task] [139626962478848] - WARNING - vbucket map not ready after try 3 [2021-02-17 00:25:10,572] - [bucket_helper] [139626962478848] - INFO - waiting for memcached bucket : dst_bucket in 172.23.122.91 to accept set ops [2021-02-17 00:25:10,809] - [data_helper] [139626962478848] - INFO - creating direct client 172.23.122.158:11210 dst_bucket [2021-02-17 00:25:10,892] - [data_helper] [139626962478848] - INFO - creating direct client 172.23.122.159:11210 dst_bucket [2021-02-17 00:25:10,972] - [data_helper] [139626962478848] - INFO - creating direct client 172.23.122.91:11210 dst_bucket [2021-02-17 00:25:11,214] - [data_helper] [139626962478848] - INFO - creating direct client 172.23.122.158:11210 dst_bucket [2021-02-17 00:25:11,315] - [data_helper] [139626962478848] - INFO - creating direct client 172.23.122.159:11210 dst_bucket [2021-02-17 00:25:11,408] - [data_helper] [139626962478848] - INFO - creating direct client 172.23.122.91:11210 dst_bucket [2021-02-17 00:25:11,590] - [task] [139626962478848] - WARNING - vbucket map not ready after try 4 [2021-02-17 00:25:11,591] - [bucket_helper] [139626962478848] - INFO - waiting for memcached bucket : dst_bucket in 172.23.122.91 to accept set ops [2021-02-17 00:25:11,706] - [data_helper] [139626962478848] - INFO - creating direct client 172.23.122.158:11210 dst_bucket [2021-02-17 00:25:11,785] - [data_helper] [139626962478848] - INFO - creating direct client 172.23.122.159:11210 dst_bucket [2021-02-17 00:25:11,866] - [data_helper] [139626962478848] - INFO - creating direct client 172.23.122.91:11210 dst_bucket [2021-02-17 00:25:12,105] - [data_helper] [139626962478848] - INFO - creating direct client 172.23.122.158:11210 dst_bucket [2021-02-17 00:25:12,184] - [data_helper] [139626962478848] - INFO - creating direct client 172.23.122.159:11210 dst_bucket [2021-02-17 00:25:12,263] - [data_helper] [139626962478848] - INFO - creating direct client 172.23.122.91:11210 dst_bucket [2021-02-17 00:25:12,452] - [task] [139626962478848] - WARNING - vbucket map not ready after try 5 [2021-02-17 00:25:13,482] - [rest_client] [139626962478848] - INFO - http://172.23.122.91:8091/pools/default/buckets with param: name=metadata&ramQuotaMB=200&authType=none&replicaNumber=0&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore [2021-02-17 00:25:13,517] - [rest_client] [139626962478848] - INFO - 0.04 seconds to create bucket metadata [2021-02-17 00:25:13,518] - [bucket_helper] [139626962478848] - INFO - waiting for memcached bucket : metadata in 172.23.122.91 to accept set ops [2021-02-17 00:25:14,175] - [data_helper] [139626962478848] - INFO - creating direct client 172.23.122.158:11210 metadata [2021-02-17 00:25:14,267] - [data_helper] [139626962478848] - INFO - creating direct client 172.23.122.159:11210 metadata [2021-02-17 00:25:14,476] - [data_helper] [139626962478848] - INFO - creating direct client 172.23.122.91:11210 metadata [2021-02-17 00:25:14,600] - [data_helper] [139626962478848] - INFO - creating direct client 172.23.122.158:11210 metadata [2021-02-17 00:25:14,683] - [data_helper] [139626962478848] - INFO - creating direct client 172.23.122.159:11210 metadata [2021-02-17 00:25:14,776] - [data_helper] [139626962478848] - INFO - creating direct client 172.23.122.91:11210 metadata [2021-02-17 00:25:14,984] - [task] [139626962478848] - WARNING - vbucket map not ready after try 0 [2021-02-17 00:25:14,984] - [bucket_helper] [139626962478848] - INFO - waiting for memcached bucket : metadata in 172.23.122.91 to accept set ops [2021-02-17 00:25:15,148] - [data_helper] [139626962478848] - INFO - creating direct client 172.23.122.158:11210 metadata [2021-02-17 00:25:15,243] - [data_helper] [139626962478848] - INFO - creating direct client 172.23.122.159:11210 metadata [2021-02-17 00:25:15,447] - [data_helper] [139626962478848] - INFO - creating direct client 172.23.122.91:11210 metadata [2021-02-17 00:25:15,576] - [data_helper] [139626962478848] - INFO - creating direct client 172.23.122.158:11210 metadata [2021-02-17 00:25:15,663] - [data_helper] [139626962478848] - INFO - creating direct client 172.23.122.159:11210 metadata [2021-02-17 00:25:15,862] - [data_helper] [139626962478848] - INFO - creating direct client 172.23.122.91:11210 metadata [2021-02-17 00:25:15,941] - [task] [139626962478848] - WARNING - vbucket map not ready after try 1 [2021-02-17 00:25:15,941] - [bucket_helper] [139626962478848] - INFO - waiting for memcached bucket : metadata in 172.23.122.91 to accept set ops [2021-02-17 00:25:16,077] - [data_helper] [139626962478848] - INFO - creating direct client 172.23.122.158:11210 metadata [2021-02-17 00:25:16,163] - [data_helper] [139626962478848] - INFO - creating direct client 172.23.122.159:11210 metadata [2021-02-17 00:25:16,371] - [data_helper] [139626962478848] - INFO - creating direct client 172.23.122.91:11210 metadata [2021-02-17 00:25:16,506] - [data_helper] [139626962478848] - INFO - creating direct client 172.23.122.158:11210 metadata [2021-02-17 00:25:16,598] - [data_helper] [139626962478848] - INFO - creating direct client 172.23.122.159:11210 metadata [2021-02-17 00:25:16,810] - [data_helper] [139626962478848] - INFO - creating direct client 172.23.122.91:11210 metadata [2021-02-17 00:25:16,885] - [task] [139626962478848] - WARNING - vbucket map not ready after try 2 [2021-02-17 00:25:16,885] - [bucket_helper] [139626962478848] - INFO - waiting for memcached bucket : metadata in 172.23.122.91 to accept set ops [2021-02-17 00:25:17,023] - [data_helper] [139626962478848] - INFO - creating direct client 172.23.122.158:11210 metadata [2021-02-17 00:25:17,245] - [data_helper] [139626962478848] - INFO - creating direct client 172.23.122.159:11210 metadata [2021-02-17 00:25:17,330] - [data_helper] [139626962478848] - INFO - creating direct client 172.23.122.91:11210 metadata [2021-02-17 00:25:17,467] - [data_helper] [139626962478848] - INFO - creating direct client 172.23.122.158:11210 metadata [2021-02-17 00:25:17,678] - [data_helper] [139626962478848] - INFO - creating direct client 172.23.122.159:11210 metadata [2021-02-17 00:25:17,763] - [data_helper] [139626962478848] - INFO - creating direct client 172.23.122.91:11210 metadata [2021-02-17 00:25:17,837] - [task] [139626962478848] - WARNING - vbucket map not ready after try 3 [2021-02-17 00:25:17,837] - [bucket_helper] [139626962478848] - INFO - waiting for memcached bucket : metadata in 172.23.122.91 to accept set ops [2021-02-17 00:25:17,967] - [data_helper] [139626962478848] - INFO - creating direct client 172.23.122.158:11210 metadata [2021-02-17 00:25:18,169] - [data_helper] [139626962478848] - INFO - creating direct client 172.23.122.159:11210 metadata [2021-02-17 00:25:18,268] - [data_helper] [139626962478848] - INFO - creating direct client 172.23.122.91:11210 metadata [2021-02-17 00:25:18,403] - [data_helper] [139626962478848] - INFO - creating direct client 172.23.122.158:11210 metadata [2021-02-17 00:25:18,610] - [data_helper] [139626962478848] - INFO - creating direct client 172.23.122.159:11210 metadata [2021-02-17 00:25:18,696] - [data_helper] [139626962478848] - INFO - creating direct client 172.23.122.91:11210 metadata [2021-02-17 00:25:18,772] - [task] [139626962478848] - WARNING - vbucket map not ready after try 4 [2021-02-17 00:25:18,772] - [bucket_helper] [139626962478848] - INFO - waiting for memcached bucket : metadata in 172.23.122.91 to accept set ops [2021-02-17 00:25:19,026] - [data_helper] [139626962478848] - INFO - creating direct client 172.23.122.158:11210 metadata [2021-02-17 00:25:19,116] - [data_helper] [139626962478848] - INFO - creating direct client 172.23.122.159:11210 metadata [2021-02-17 00:25:19,201] - [data_helper] [139626962478848] - INFO - creating direct client 172.23.122.91:11210 metadata [2021-02-17 00:25:19,461] - [data_helper] [139626962478848] - INFO - creating direct client 172.23.122.158:11210 metadata [2021-02-17 00:25:19,553] - [data_helper] [139626962478848] - INFO - creating direct client 172.23.122.159:11210 metadata [2021-02-17 00:25:19,648] - [data_helper] [139626962478848] - INFO - creating direct client 172.23.122.91:11210 metadata [2021-02-17 00:25:19,724] - [task] [139626962478848] - WARNING - vbucket map not ready after try 5 [2021-02-17 00:25:19,932] - [rest_client] [139626970871552] - INFO - http://172.23.122.156:8091/pools/default/buckets/dst_bucket/scopes with params: name=scope_0 [2021-02-17 00:25:19,932] - [rest_client] [139626970871552] - INFO - Scope created dst_bucket->scope_0 {'uid': '1'} [2021-02-17 00:25:19,961] - [rest_client] [139626970871552] - INFO - http://172.23.122.156:8091/pools/default/buckets/dst_bucket/scopes with params: name=scope_1 [2021-02-17 00:25:19,961] - [rest_client] [139626970871552] - INFO - Scope created dst_bucket->scope_1 {'uid': '2'} [2021-02-17 00:25:19,990] - [rest_client] [139626970871552] - INFO - http://172.23.122.156:8091/pools/default/buckets/dst_bucket/scopes with params: name=scope_2 [2021-02-17 00:25:19,990] - [rest_client] [139626970871552] - INFO - Scope created dst_bucket->scope_2 {'uid': '3'} [2021-02-17 00:25:20,019] - [rest_client] [139626970871552] - INFO - http://172.23.122.156:8091/pools/default/buckets/dst_bucket/scopes with params: name=scope_3 [2021-02-17 00:25:20,019] - [rest_client] [139626970871552] - INFO - Scope created dst_bucket->scope_3 {'uid': '4'} [2021-02-17 00:25:20,047] - [rest_client] [139626970871552] - INFO - http://172.23.122.156:8091/pools/default/buckets/dst_bucket/scopes with params: name=scope_4 [2021-02-17 00:25:20,048] - [rest_client] [139626970871552] - INFO - Scope created dst_bucket->scope_4 {'uid': '5'} [2021-02-17 00:25:20,076] - [rest_client] [139626970871552] - INFO - http://172.23.122.156:8091/pools/default/buckets/src_bucket/scopes with params: name=scope_0 [2021-02-17 00:25:20,076] - [rest_client] [139626970871552] - INFO - Scope created src_bucket->scope_0 {'uid': '1'} [2021-02-17 00:25:20,107] - [rest_client] [139626970871552] - INFO - http://172.23.122.156:8091/pools/default/buckets/src_bucket/scopes with params: name=scope_1 [2021-02-17 00:25:20,107] - [rest_client] [139626970871552] - INFO - Scope created src_bucket->scope_1 {'uid': '2'} [2021-02-17 00:25:20,136] - [rest_client] [139626970871552] - INFO - http://172.23.122.156:8091/pools/default/buckets/src_bucket/scopes with params: name=scope_2 [2021-02-17 00:25:20,136] - [rest_client] [139626970871552] - INFO - Scope created src_bucket->scope_2 {'uid': '3'} [2021-02-17 00:25:20,165] - [rest_client] [139626970871552] - INFO - http://172.23.122.156:8091/pools/default/buckets/src_bucket/scopes with params: name=scope_3 [2021-02-17 00:25:20,165] - [rest_client] [139626970871552] - INFO - Scope created src_bucket->scope_3 {'uid': '4'} [2021-02-17 00:25:20,194] - [rest_client] [139626970871552] - INFO - http://172.23.122.156:8091/pools/default/buckets/src_bucket/scopes with params: name=scope_4 [2021-02-17 00:25:20,194] - [rest_client] [139626970871552] - INFO - Scope created src_bucket->scope_4 {'uid': '5'} [2021-02-17 00:25:20,225] - [rest_client] [139626970871552] - INFO - http://172.23.122.156:8091/pools/default/buckets/dst_bucket/scopes/scope_1/collections with params: name=coll_0 [2021-02-17 00:25:20,225] - [rest_client] [139626970871552] - INFO - Collection created dst_bucket->scope_1->coll_0 manifest:{'uid': '6'} [2021-02-17 00:25:20,256] - [rest_client] [139626970871552] - INFO - http://172.23.122.156:8091/pools/default/buckets/dst_bucket/scopes/scope_1/collections with params: name=coll_1 [2021-02-17 00:25:20,256] - [rest_client] [139626970871552] - INFO - Collection created dst_bucket->scope_1->coll_1 manifest:{'uid': '7'} [2021-02-17 00:25:20,286] - [rest_client] [139626970871552] - INFO - http://172.23.122.156:8091/pools/default/buckets/dst_bucket/scopes/scope_1/collections with params: name=coll_2 [2021-02-17 00:25:20,286] - [rest_client] [139626970871552] - INFO - Collection created dst_bucket->scope_1->coll_2 manifest:{'uid': '8'} [2021-02-17 00:25:20,316] - [rest_client] [139626970871552] - INFO - http://172.23.122.156:8091/pools/default/buckets/dst_bucket/scopes/scope_1/collections with params: name=coll_3 [2021-02-17 00:25:20,316] - [rest_client] [139626970871552] - INFO - Collection created dst_bucket->scope_1->coll_3 manifest:{'uid': '9'} [2021-02-17 00:25:20,345] - [rest_client] [139626970871552] - INFO - http://172.23.122.156:8091/pools/default/buckets/dst_bucket/scopes/scope_1/collections with params: name=coll_4 [2021-02-17 00:25:20,345] - [rest_client] [139626970871552] - INFO - Collection created dst_bucket->scope_1->coll_4 manifest:{'uid': 'a'} [2021-02-17 00:25:20,373] - [rest_client] [139626970871552] - INFO - http://172.23.122.156:8091/pools/default/buckets/src_bucket/scopes/scope_1/collections with params: name=coll_0 [2021-02-17 00:25:20,373] - [rest_client] [139626970871552] - INFO - Collection created src_bucket->scope_1->coll_0 manifest:{'uid': '6'} [2021-02-17 00:25:20,406] - [rest_client] [139626970871552] - INFO - http://172.23.122.156:8091/pools/default/buckets/src_bucket/scopes/scope_1/collections with params: name=coll_1 [2021-02-17 00:25:20,406] - [rest_client] [139626970871552] - INFO - Collection created src_bucket->scope_1->coll_1 manifest:{'uid': '7'} [2021-02-17 00:25:20,433] - [rest_client] [139626970871552] - INFO - http://172.23.122.156:8091/pools/default/buckets/src_bucket/scopes/scope_1/collections with params: name=coll_2 [2021-02-17 00:25:20,434] - [rest_client] [139626970871552] - INFO - Collection created src_bucket->scope_1->coll_2 manifest:{'uid': '8'} [2021-02-17 00:25:20,463] - [rest_client] [139626970871552] - INFO - http://172.23.122.156:8091/pools/default/buckets/src_bucket/scopes/scope_1/collections with params: name=coll_3 [2021-02-17 00:25:20,464] - [rest_client] [139626970871552] - INFO - Collection created src_bucket->scope_1->coll_3 manifest:{'uid': '9'} [2021-02-17 00:25:20,494] - [rest_client] [139626970871552] - INFO - http://172.23.122.156:8091/pools/default/buckets/src_bucket/scopes/scope_1/collections with params: name=coll_4 [2021-02-17 00:25:20,494] - [rest_client] [139626970871552] - INFO - Collection created src_bucket->scope_1->coll_4 manifest:{'uid': 'a'} [2021-02-17 00:25:21,729] - [eventing_base] [139626970871552] - INFO - saving function bucket_op [2021-02-17 00:25:22,919] - [eventing_base] [139626970871552] - INFO - saving function timers [2021-02-17 00:25:24,188] - [eventing_base] [139626970871552] - INFO - saving function sbm [2021-02-17 00:25:25,408] - [eventing_base] [139626970871552] - INFO - saving function curl [2021-02-17 00:25:26,634] - [eventing_base] [139626970871552] - INFO - saving function n1ql [2021-02-17 00:25:27,905] - [basetestcase] [139626970871552] - INFO - sleep for 5 secs. Waiting for bucket_op to deployed... ... [2021-02-17 00:25:32,949] - [basetestcase] [139626970871552] - INFO - sleep for 5 secs. Waiting for bucket_op to deployed... ... [2021-02-17 00:25:37,995] - [basetestcase] [139626970871552] - INFO - sleep for 5 secs. Waiting for bucket_op to deployed... ... [2021-02-17 00:25:43,068] - [basetestcase] [139626970871552] - INFO - sleep for 5 secs. Waiting for bucket_op to deployed... ... [2021-02-17 00:25:49,364] - [basetestcase] [139626970871552] - INFO - sleep for 5 secs. Waiting for timers to deployed... ... [2021-02-17 00:25:54,406] - [basetestcase] [139626970871552] - INFO - sleep for 5 secs. Waiting for timers to deployed... ... [2021-02-17 00:25:59,483] - [basetestcase] [139626970871552] - INFO - sleep for 5 secs. Waiting for timers to deployed... ... [2021-02-17 00:26:04,524] - [basetestcase] [139626970871552] - INFO - sleep for 5 secs. Waiting for timers to deployed... ... [2021-02-17 00:26:10,899] - [basetestcase] [139626970871552] - INFO - sleep for 5 secs. Waiting for sbm to deployed... ... [2021-02-17 00:26:15,943] - [basetestcase] [139626970871552] - INFO - sleep for 5 secs. Waiting for sbm to deployed... ... [2021-02-17 00:26:21,008] - [basetestcase] [139626970871552] - INFO - sleep for 5 secs. Waiting for sbm to deployed... ... [2021-02-17 00:26:26,048] - [basetestcase] [139626970871552] - INFO - sleep for 5 secs. Waiting for sbm to deployed... ... [2021-02-17 00:26:32,321] - [basetestcase] [139626970871552] - INFO - sleep for 5 secs. Waiting for curl to deployed... ... [2021-02-17 00:26:37,362] - [basetestcase] [139626970871552] - INFO - sleep for 5 secs. Waiting for curl to deployed... ... [2021-02-17 00:26:42,429] - [basetestcase] [139626970871552] - INFO - sleep for 5 secs. Waiting for curl to deployed... ... [2021-02-17 00:26:47,477] - [basetestcase] [139626970871552] - INFO - sleep for 5 secs. Waiting for curl to deployed... ... [2021-02-17 00:26:53,769] - [basetestcase] [139626970871552] - INFO - sleep for 5 secs. Waiting for n1ql to deployed... ... [2021-02-17 00:26:58,808] - [basetestcase] [139626970871552] - INFO - sleep for 5 secs. Waiting for n1ql to deployed... ... [2021-02-17 00:27:03,865] - [basetestcase] [139626970871552] - INFO - sleep for 5 secs. Waiting for n1ql to deployed... ... [2021-02-17 00:27:08,907] - [basetestcase] [139626970871552] - INFO - sleep for 5 secs. Waiting for n1ql to deployed... ... [2021-02-17 00:27:14,843] - [task] [139626962478848] - INFO - java -jar java_sdk_client/collections/target/javaclient/javaclient.jar -i 172.23.122.91 -u Administrator -p password -b src_bucket -s _default -c _default -n 40320 -pc 100 -pu 0 -pd 0 -l uniform -dsn 1 -dpx doc_ -dt Person -de 0 -ds 500 -ac False -st 0 -en 40319 -o False -sd False [2021-02-17 00:27:16,979] - [rest_client] [139626970871552] - INFO - rebalance reached >100% in 3.005150318145752 seconds [2021-02-17 00:27:27,402] - [task] [139626962478848] - INFO - java -jar java_sdk_client/collections/target/javaclient/javaclient.jar -i 172.23.122.91 -u Administrator -p password -b src_bucket -s scope_1 -c coll_1 -n 40320 -pc 100 -pu 0 -pd 0 -l uniform -dsn 1 -dpx doc_ -dt Person -de 0 -ds 500 -ac False -st 0 -en 40319 -o False -sd False [2021-02-17 00:27:40,082] - [rest_client] [139626962478848] - INFO - rebalance params : {'knownNodes': 'ns_1@172.23.122.156,ns_1@172.23.122.158,ns_1@172.23.122.159,ns_1@172.23.122.160,ns_1@172.23.122.161,ns_1@172.23.122.91,ns_1@172.23.122.92', 'ejectedNodes': 'ns_1@172.23.122.158', 'user': 'Administrator', 'password': 'password'} [2021-02-17 00:27:40,111] - [rest_client] [139626962478848] - INFO - rebalance operation started [2021-02-17 00:27:40,114] - [rest_client] [139626962478848] - INFO - rebalance percentage : 0.00 % [2021-02-17 00:27:40,114] - [task] [139626962478848] - INFO - Rebalance - status: running, progress: 0.00% [2021-02-17 00:27:50,141] - [rest_client] [139626962478848] - INFO - rebalance percentage : 14.00 % [2021-02-17 00:27:50,141] - [task] [139626962478848] - INFO - Rebalance - status: running, progress: 14.00% [2021-02-17 00:28:00,169] - [rest_client] [139626962478848] - INFO - rebalance percentage : 27.00 % [2021-02-17 00:28:00,169] - [task] [139626962478848] - INFO - Rebalance - status: running, progress: 27.00% [2021-02-17 00:28:10,198] - [rest_client] [139626962478848] - INFO - rebalance percentage : 28.00 % [2021-02-17 00:28:10,198] - [task] [139626962478848] - INFO - Rebalance - status: running, progress: 28.00% [2021-02-17 00:28:20,223] - [rest_client] [139626962478848] - INFO - rebalance percentage : 42.00 % [2021-02-17 00:28:20,224] - [task] [139626962478848] - INFO - Rebalance - status: running, progress: 42.00% [2021-02-17 00:28:30,248] - [rest_client] [139626962478848] - INFO - rebalance percentage : 50.00 % [2021-02-17 00:28:30,248] - [task] [139626962478848] - INFO - Rebalance - status: running, progress: 50.00% [2021-02-17 00:28:40,274] - [rest_client] [139626962478848] - INFO - rebalance percentage : 50.00 % [2021-02-17 00:28:40,274] - [task] [139626962478848] - INFO - Rebalance - status: running, progress: 50.00% [2021-02-17 00:28:50,303] - [rest_client] [139626962478848] - INFO - rebalance percentage : 62.00 % [2021-02-17 00:28:50,303] - [task] [139626962478848] - INFO - Rebalance - status: running, progress: 62.00% [2021-02-17 00:29:00,332] - [rest_client] [139626962478848] - INFO - rebalance percentage : 71.00 % [2021-02-17 00:29:00,332] - [task] [139626962478848] - INFO - Rebalance - status: running, progress: 71.00% [2021-02-17 00:29:10,358] - [rest_client] [139626962478848] - INFO - rebalance percentage : 79.00 % [2021-02-17 00:29:10,359] - [task] [139626962478848] - INFO - Rebalance - status: running, progress: 79.00% [2021-02-17 00:29:20,387] - [rest_client] [139626962478848] - INFO - rebalance percentage : 87.00 % [2021-02-17 00:29:20,388] - [task] [139626962478848] - INFO - Rebalance - status: running, progress: 87.00% [2021-02-17 00:29:30,418] - [rest_client] [139626962478848] - INFO - rebalance percentage : 91.00 % [2021-02-17 00:29:30,418] - [task] [139626962478848] - INFO - Rebalance - status: running, progress: 91.00% [2021-02-17 00:29:40,449] - [task] [139626962478848] - INFO - Rebalance - status: none, progress: 100.00% [2021-02-17 00:29:40,467] - [task] [139626962478848] - INFO - rebalancing was completed with progress: 100% in 120.35618805885315 sec [2021-02-17 00:29:40,476] - [basetestcase] [139626970871552] - INFO - list of eventing nodes in cluster: [ip:172.23.122.156 port:8091 ssh_username:root, ip:172.23.122.161 port:8091 ssh_username:root, ip:172.23.122.92 port:8091 ssh_username:root] [2021-02-17 00:29:40,482] - [tuq_helper] [139626970871552] - INFO - RUN QUERY create primary index on dst_bucket.scope_1.coll_0 [2021-02-17 00:29:40,487] - [rest_client] [139626970871552] - INFO - query params : statement=create+primary+index+on+dst_bucket.scope_1.coll_0 [2021-02-17 00:29:42,862] - [tuq_helper] [139626970871552] - INFO - TOTAL ELAPSED TIME: 2.372963808s [2021-02-17 00:29:42,882] - [basetestcase] [139626970871552] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.160 port:8091 ssh_username:root] [2021-02-17 00:29:42,888] - [rest_client] [139626970871552] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_0 [2021-02-17 00:29:42,898] - [eventing_base] [139626970871552] - INFO - Final docs count... Current : 40320 Expected : 40320 [2021-02-17 00:29:42,909] - [basetestcase] [139626970871552] - INFO - list of eventing nodes in cluster: [ip:172.23.122.156 port:8091 ssh_username:root, ip:172.23.122.161 port:8091 ssh_username:root, ip:172.23.122.92 port:8091 ssh_username:root] [2021-02-17 00:29:42,914] - [tuq_helper] [139626970871552] - INFO - RUN QUERY create primary index on dst_bucket.scope_1.coll_1 [2021-02-17 00:29:42,919] - [rest_client] [139626970871552] - INFO - query params : statement=create+primary+index+on+dst_bucket.scope_1.coll_1 [2021-02-17 00:29:45,967] - [tuq_helper] [139626970871552] - INFO - TOTAL ELAPSED TIME: 3.046137423s [2021-02-17 00:29:45,986] - [basetestcase] [139626970871552] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.160 port:8091 ssh_username:root] [2021-02-17 00:29:45,992] - [rest_client] [139626970871552] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_1 [2021-02-17 00:29:46,003] - [eventing_base] [139626970871552] - INFO - Final docs count... Current : 40320 Expected : 40320 [2021-02-17 00:29:46,011] - [basetestcase] [139626970871552] - INFO - list of eventing nodes in cluster: [ip:172.23.122.156 port:8091 ssh_username:root, ip:172.23.122.161 port:8091 ssh_username:root, ip:172.23.122.92 port:8091 ssh_username:root] [2021-02-17 00:29:46,017] - [tuq_helper] [139626970871552] - INFO - RUN QUERY create primary index on dst_bucket.scope_1.coll_3 [2021-02-17 00:29:46,022] - [rest_client] [139626970871552] - INFO - query params : statement=create+primary+index+on+dst_bucket.scope_1.coll_3 [2021-02-17 00:29:48,812] - [tuq_helper] [139626970871552] - INFO - TOTAL ELAPSED TIME: 2.788735956s [2021-02-17 00:29:48,831] - [basetestcase] [139626970871552] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.160 port:8091 ssh_username:root] [2021-02-17 00:29:48,837] - [rest_client] [139626970871552] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_3 [2021-02-17 00:29:48,849] - [eventing_base] [139626970871552] - INFO - Final docs count... Current : 40320 Expected : 40320 [2021-02-17 00:29:48,860] - [basetestcase] [139626970871552] - INFO - list of eventing nodes in cluster: [ip:172.23.122.156 port:8091 ssh_username:root, ip:172.23.122.161 port:8091 ssh_username:root, ip:172.23.122.92 port:8091 ssh_username:root] [2021-02-17 00:29:48,865] - [tuq_helper] [139626970871552] - INFO - RUN QUERY create primary index on dst_bucket.scope_1.coll_4 [2021-02-17 00:29:48,870] - [rest_client] [139626970871552] - INFO - query params : statement=create+primary+index+on+dst_bucket.scope_1.coll_4 [2021-02-17 00:29:51,674] - [tuq_helper] [139626970871552] - INFO - TOTAL ELAPSED TIME: 2.801590342s [2021-02-17 00:29:51,698] - [basetestcase] [139626970871552] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.160 port:8091 ssh_username:root] [2021-02-17 00:29:51,704] - [rest_client] [139626970871552] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-17 00:29:51,714] - [basetestcase] [139626970871552] - INFO - sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_4 to complete bucket operations... Current : 40316 Expected : 40320 ... [2021-02-17 00:30:21,739] - [basetestcase] [139626970871552] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.160 port:8091 ssh_username:root] [2021-02-17 00:30:21,745] - [rest_client] [139626970871552] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-17 00:30:21,762] - [basetestcase] [139626970871552] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.160 port:8091 ssh_username:root] [2021-02-17 00:30:21,768] - [rest_client] [139626970871552] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-17 00:30:21,771] - [basetestcase] [139626970871552] - INFO - sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_4 to complete bucket operations... Current : 40316 Expected : 40320 ... [2021-02-17 00:30:51,821] - [basetestcase] [139626970871552] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.160 port:8091 ssh_username:root] [2021-02-17 00:30:51,827] - [rest_client] [139626970871552] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-17 00:30:51,846] - [basetestcase] [139626970871552] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.160 port:8091 ssh_username:root] [2021-02-17 00:30:51,852] - [rest_client] [139626970871552] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-17 00:30:51,856] - [basetestcase] [139626970871552] - INFO - sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_4 to complete bucket operations... Current : 40316 Expected : 40320 ... [2021-02-17 00:31:21,906] - [basetestcase] [139626970871552] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.160 port:8091 ssh_username:root] [2021-02-17 00:31:21,913] - [rest_client] [139626970871552] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-17 00:31:21,930] - [basetestcase] [139626970871552] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.160 port:8091 ssh_username:root] [2021-02-17 00:31:21,934] - [rest_client] [139626970871552] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-17 00:31:21,938] - [basetestcase] [139626970871552] - INFO - sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_4 to complete bucket operations... Current : 40316 Expected : 40320 ... [2021-02-17 00:31:51,987] - [basetestcase] [139626970871552] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.160 port:8091 ssh_username:root] [2021-02-17 00:31:51,994] - [rest_client] [139626970871552] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-17 00:31:52,013] - [basetestcase] [139626970871552] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.160 port:8091 ssh_username:root] [2021-02-17 00:31:52,018] - [rest_client] [139626970871552] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-17 00:31:52,021] - [basetestcase] [139626970871552] - INFO - sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_4 to complete bucket operations... Current : 40316 Expected : 40320 ... [2021-02-17 00:32:22,072] - [basetestcase] [139626970871552] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.160 port:8091 ssh_username:root] [2021-02-17 00:32:22,077] - [rest_client] [139626970871552] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-17 00:32:22,095] - [basetestcase] [139626970871552] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.160 port:8091 ssh_username:root] [2021-02-17 00:32:22,100] - [rest_client] [139626970871552] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-17 00:32:22,103] - [basetestcase] [139626970871552] - INFO - sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_4 to complete bucket operations... Current : 40316 Expected : 40320 ... [2021-02-17 00:32:52,148] - [basetestcase] [139626970871552] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.160 port:8091 ssh_username:root] [2021-02-17 00:32:52,155] - [rest_client] [139626970871552] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-17 00:32:52,172] - [basetestcase] [139626970871552] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.160 port:8091 ssh_username:root] [2021-02-17 00:32:52,177] - [rest_client] [139626970871552] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-17 00:32:52,180] - [basetestcase] [139626970871552] - INFO - sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_4 to complete bucket operations... Current : 40316 Expected : 40320 ... [2021-02-17 00:33:22,231] - [basetestcase] [139626970871552] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.160 port:8091 ssh_username:root] [2021-02-17 00:33:22,239] - [rest_client] [139626970871552] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-17 00:33:22,259] - [basetestcase] [139626970871552] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.160 port:8091 ssh_username:root] [2021-02-17 00:33:22,264] - [rest_client] [139626970871552] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-17 00:33:22,267] - [basetestcase] [139626970871552] - INFO - sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_4 to complete bucket operations... Current : 40316 Expected : 40320 ... [2021-02-17 00:33:52,317] - [basetestcase] [139626970871552] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.160 port:8091 ssh_username:root] [2021-02-17 00:33:52,323] - [rest_client] [139626970871552] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-17 00:33:52,342] - [basetestcase] [139626970871552] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.160 port:8091 ssh_username:root] [2021-02-17 00:33:52,347] - [rest_client] [139626970871552] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-17 00:33:52,351] - [basetestcase] [139626970871552] - INFO - sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_4 to complete bucket operations... Current : 40316 Expected : 40320 ... [2021-02-17 00:34:22,398] - [basetestcase] [139626970871552] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.160 port:8091 ssh_username:root] [2021-02-17 00:34:22,405] - [rest_client] [139626970871552] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-17 00:34:22,421] - [basetestcase] [139626970871552] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.160 port:8091 ssh_username:root] [2021-02-17 00:34:22,426] - [rest_client] [139626970871552] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-17 00:34:22,429] - [basetestcase] [139626970871552] - INFO - sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_4 to complete bucket operations... Current : 40316 Expected : 40320 ... [2021-02-17 00:34:52,475] - [basetestcase] [139626970871552] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.160 port:8091 ssh_username:root] [2021-02-17 00:34:52,482] - [rest_client] [139626970871552] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-17 00:34:52,500] - [basetestcase] [139626970871552] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.160 port:8091 ssh_username:root] [2021-02-17 00:34:52,506] - [rest_client] [139626970871552] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-17 00:34:52,509] - [basetestcase] [139626970871552] - INFO - sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_4 to complete bucket operations... Current : 40316 Expected : 40320 ... [2021-02-17 00:35:22,562] - [basetestcase] [139626970871552] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.160 port:8091 ssh_username:root] [2021-02-17 00:35:22,567] - [rest_client] [139626970871552] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-17 00:35:22,581] - [eventing_base] [139626970871552] - INFO - Compacting bucket : dst_bucket [2021-02-17 00:35:22,587] - [rest_client] [139626970871552] - INFO - bucket compaction successful [2021-02-17 00:35:22,588] - [eventing_base] [139626970871552] - INFO - Compacting bucket : metadata [2021-02-17 00:35:22,592] - [rest_client] [139626970871552] - INFO - bucket compaction successful [2021-02-17 00:35:22,593] - [eventing_base] [139626970871552] - INFO - Compacting bucket : src_bucket [2021-02-17 00:35:22,597] - [rest_client] [139626970871552] - INFO - bucket compaction successful [2021-02-17 00:35:22,616] - [basetestcase] [139626970871552] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.160 port:8091 ssh_username:root] [2021-02-17 00:35:22,621] - [rest_client] [139626970871552] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-17 00:35:22,625] - [basetestcase] [139626970871552] - INFO - sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_4 to complete bucket operations... Current : 40316 Expected : 40320 ... [2021-02-17 00:35:52,676] - [basetestcase] [139626970871552] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.160 port:8091 ssh_username:root] [2021-02-17 00:35:52,682] - [rest_client] [139626970871552] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-17 00:35:52,702] - [basetestcase] [139626970871552] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.160 port:8091 ssh_username:root] [2021-02-17 00:35:52,707] - [rest_client] [139626970871552] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-17 00:35:52,711] - [basetestcase] [139626970871552] - INFO - sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_4 to complete bucket operations... Current : 40316 Expected : 40320 ... [2021-02-17 00:36:22,763] - [basetestcase] [139626970871552] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.160 port:8091 ssh_username:root] [2021-02-17 00:36:22,770] - [rest_client] [139626970871552] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-17 00:36:22,789] - [basetestcase] [139626970871552] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.160 port:8091 ssh_username:root] [2021-02-17 00:36:22,795] - [rest_client] [139626970871552] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-17 00:36:22,799] - [basetestcase] [139626970871552] - INFO - sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_4 to complete bucket operations... Current : 40316 Expected : 40320 ... [2021-02-17 00:36:52,849] - [basetestcase] [139626970871552] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.160 port:8091 ssh_username:root] [2021-02-17 00:36:52,856] - [rest_client] [139626970871552] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-17 00:36:52,874] - [basetestcase] [139626970871552] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.160 port:8091 ssh_username:root] [2021-02-17 00:36:52,880] - [rest_client] [139626970871552] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-17 00:36:52,883] - [basetestcase] [139626970871552] - INFO - sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_4 to complete bucket operations... Current : 40316 Expected : 40320 ... [2021-02-17 00:37:22,926] - [basetestcase] [139626970871552] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.160 port:8091 ssh_username:root] [2021-02-17 00:37:22,933] - [rest_client] [139626970871552] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-17 00:37:22,950] - [basetestcase] [139626970871552] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.160 port:8091 ssh_username:root] [2021-02-17 00:37:22,955] - [rest_client] [139626970871552] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-17 00:37:22,959] - [basetestcase] [139626970871552] - INFO - sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_4 to complete bucket operations... Current : 40316 Expected : 40320 ... [2021-02-17 00:37:53,005] - [basetestcase] [139626970871552] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.160 port:8091 ssh_username:root] [2021-02-17 00:37:53,012] - [rest_client] [139626970871552] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-17 00:37:53,030] - [basetestcase] [139626970871552] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.160 port:8091 ssh_username:root] [2021-02-17 00:37:53,036] - [rest_client] [139626970871552] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-17 00:37:53,039] - [basetestcase] [139626970871552] - INFO - sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_4 to complete bucket operations... Current : 40316 Expected : 40320 ... [2021-02-17 00:38:23,088] - [basetestcase] [139626970871552] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.160 port:8091 ssh_username:root] [2021-02-17 00:38:23,095] - [rest_client] [139626970871552] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-17 00:38:23,114] - [basetestcase] [139626970871552] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.160 port:8091 ssh_username:root] [2021-02-17 00:38:23,119] - [rest_client] [139626970871552] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-17 00:38:23,123] - [basetestcase] [139626970871552] - INFO - sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_4 to complete bucket operations... Current : 40316 Expected : 40320 ... [2021-02-17 00:38:53,172] - [basetestcase] [139626970871552] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.160 port:8091 ssh_username:root] [2021-02-17 00:38:53,180] - [rest_client] [139626970871552] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-17 00:38:53,198] - [basetestcase] [139626970871552] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.160 port:8091 ssh_username:root] [2021-02-17 00:38:53,203] - [rest_client] [139626970871552] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-17 00:38:53,206] - [basetestcase] [139626970871552] - INFO - sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_4 to complete bucket operations... Current : 40316 Expected : 40320 ... [2021-02-17 00:39:23,252] - [basetestcase] [139626970871552] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.160 port:8091 ssh_username:root] [2021-02-17 00:39:23,258] - [rest_client] [139626970871552] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-17 00:39:23,276] - [basetestcase] [139626970871552] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.160 port:8091 ssh_username:root] [2021-02-17 00:39:23,281] - [rest_client] [139626970871552] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-17 00:39:23,284] - [basetestcase] [139626970871552] - INFO - sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_4 to complete bucket operations... Current : 40316 Expected : 40320 ... [2021-02-17 00:39:53,335] - [basetestcase] [139626970871552] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.160 port:8091 ssh_username:root] [2021-02-17 00:39:53,342] - [rest_client] [139626970871552] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-17 00:39:53,360] - [basetestcase] [139626970871552] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.160 port:8091 ssh_username:root] [2021-02-17 00:39:53,365] - [rest_client] [139626970871552] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-17 00:39:53,376] - [basetestcase] [139626970871552] - INFO - list of eventing nodes in cluster: [ip:172.23.122.156 port:8091 ssh_username:root, ip:172.23.122.161 port:8091 ssh_username:root, ip:172.23.122.92 port:8091 ssh_username:root] [2021-02-17 00:39:53,425] - [eventing_base] [139626970871552] - INFO - Stats for Node 172.23.122.156 is [ { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 1, "agg_messages_sent_to_worker": 18710, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 67159, "dcp_mutation_sent_to_worker": 13499, "dcp_snapshot": 40865, "dcp_stream_req_counter": 8868, "dcp_stream_req_err_counter": 8049, "dcp_streamend": 351, "dcp_streamreq": 819, "dcp_system_event": 3420, "execution_stats": 833, "failure_stats": 833, "latency_stats": 833, "lcb_exception_stats": 833, "log_level": 1, "num_processed_events": 13499, "processed_events_size": 12082884, "thr_count": 1, "thr_map": 1, "timer_events": 13528, "timer_responses_received": 13499, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13499, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13499, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 18706, "num_processed_events": 13499, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13499, "processed_events_size": 12082884, "timer_callback_failure": 0, "timer_callback_success": 13528, "timer_cancel_counter": 0, "timer_create_counter": 13499, "timer_create_failure": 0, "timer_msg_counter": 13528, "timer_responses_sent": 0, "timestamp": { "104712": "2021-02-17T08:39:52Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "104712": "2021-02-17T08:39:52Z" }, "v8worker_events_lost": 0 }, "function_id": 1236009705, "function_name": "timers", "gocb_creds_request_counter": 105, "internal_vb_distribution_stats": { "worker_timers_0": "[0-341]" }, "latency_percentile_stats": { "100": 19200, "50": 1500, "80": 2000, "90": 2300, "95": 2700, "99": 4000 }, "lcb_creds_request_counter": 27077, "lcb_exception_stats": {}, "metastore_stats": { "metastore_delete_err": 0, "metastore_deletes": 0, "metastore_not_found": 0, "metastore_scan": 0, "metastore_scan_due": 0, "metastore_scan_err": 0, "metastore_set": 0, "metastore_set_err": 0 }, "planner_stats": [ { "host_name": "172.23.122.156:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.161:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.92:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.156:8096": { "worker_timers_0": "[0-341]" }, "172.23.122.161:8096": { "worker_timers_0": "[342-682]" }, "172.23.122.92:8096": { "worker_timers_0": "[683-1023]" } }, "worker_pids": { "worker_timers_0": 104712 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "agg_messages_sent_to_worker": 18604, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 40712, "dcp_mutation_sent_to_worker": 13499, "dcp_mutation_suppressed_counter": 13714, "dcp_snapshot": 40712, "dcp_stream_req_counter": 8213, "dcp_stream_req_err_counter": 7353, "dcp_streamend": 351, "dcp_streamreq": 860, "dcp_system_event": 3420, "execution_stats": 812, "failure_stats": 812, "latency_stats": 812, "lcb_exception_stats": 812, "log_level": 1, "num_processed_events": 13499, "processed_events_size": 12086816, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13499, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13499, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 18600, "num_processed_events": 13499, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13499, "processed_events_size": 12086816, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "104754": "2021-02-17T08:39:52Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "104754": "2021-02-17T08:39:52Z" }, "v8worker_events_lost": 0 }, "function_id": 717983186, "function_name": "sbm", "gocb_creds_request_counter": 105, "internal_vb_distribution_stats": { "worker_sbm_0": "[0-341]" }, "latency_percentile_stats": { "100": 22500, "50": 700, "80": 1000, "90": 1300, "95": 1600, "99": 2700 }, "lcb_creds_request_counter": 27077, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.156:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.161:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.92:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.156:8096": { "worker_sbm_0": "[0-341]" }, "172.23.122.161:8096": { "worker_sbm_0": "[342-682]" }, "172.23.122.92:8096": { "worker_sbm_0": "[683-1023]" } }, "worker_pids": { "worker_sbm_0": 104754 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 2, "agg_messages_sent_to_worker": 18493, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 67159, "dcp_mutation_sent_to_worker": 13499, "dcp_snapshot": 40866, "dcp_stream_req_counter": 7847, "dcp_stream_req_err_counter": 6946, "dcp_streamend": 351, "dcp_streamreq": 901, "dcp_system_event": 3420, "execution_stats": 790, "failure_stats": 790, "latency_stats": 790, "lcb_exception_stats": 790, "log_level": 1, "num_processed_events": 13499, "processed_events_size": 12082884, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 13499, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 13499, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13499, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13499, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 18489, "num_processed_events": 13499, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13499, "processed_events_size": 12082884, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "104827": "2021-02-17T08:39:52Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "104827": "2021-02-17T08:39:52Z" }, "v8worker_events_lost": 0 }, "function_id": 613640013, "function_name": "curl", "gocb_creds_request_counter": 105, "internal_vb_distribution_stats": { "worker_curl_0": "[0-341]" }, "latency_percentile_stats": { "100": 26400, "50": 1400, "80": 1800, "90": 2000, "95": 2400, "99": 4000 }, "lcb_creds_request_counter": 27077, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.156:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.161:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.92:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.156:8096": { "worker_curl_0": "[0-341]" }, "172.23.122.161:8096": { "worker_curl_0": "[342-682]" }, "172.23.122.92:8096": { "worker_curl_0": "[683-1023]" } }, "worker_pids": { "worker_curl_0": 104827 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 3, "agg_messages_sent_to_worker": 18387, "agg_queue_memory": 0, "agg_queue_memory_cap": 20971520, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 67159, "dcp_mutation_sent_to_worker": 13499, "dcp_snapshot": 40847, "dcp_stream_req_counter": 7450, "dcp_stream_req_err_counter": 6515, "dcp_streamend": 351, "dcp_streamreq": 935, "dcp_system_event": 3420, "execution_stats": 769, "failure_stats": 769, "latency_stats": 769, "lcb_exception_stats": 769, "log_level": 1, "num_processed_events": 13499, "processed_events_size": 12082884, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13499, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13499, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 18383, "num_processed_events": 13499, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 4, "on_update_success": 13495, "processed_events_size": 12082884, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "104867": "2021-02-17T08:39:52Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 4, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "104867": "2021-02-17T08:39:52Z" }, "v8worker_events_lost": 0 }, "function_id": 22036014, "function_name": "n1ql", "gocb_creds_request_counter": 105, "internal_vb_distribution_stats": { "worker_n1ql_0": "[0-341]" }, "latency_percentile_stats": { "100": 125600, "50": 3400, "80": 4700, "90": 5700, "95": 7100, "99": 12200 }, "lcb_creds_request_counter": 27077, "lcb_exception_stats": { "12009": 4, "209": 4 }, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.156:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.161:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.92:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.156:8096": { "worker_n1ql_0": "[0-341]" }, "172.23.122.161:8096": { "worker_n1ql_0": "[342-682]" }, "172.23.122.92:8096": { "worker_n1ql_0": "[683-1023]" } }, "worker_pids": { "worker_n1ql_0": 104867 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 3, "agg_messages_sent_to_worker": 18821, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 67159, "dcp_mutation_sent_to_worker": 13499, "dcp_snapshot": 40847, "dcp_stream_req_counter": 9026, "dcp_stream_req_err_counter": 8233, "dcp_streamend": 351, "dcp_streamreq": 793, "dcp_system_event": 3420, "execution_stats": 855, "failure_stats": 855, "latency_stats": 855, "lcb_exception_stats": 855, "log_level": 1, "num_processed_events": 13499, "processed_events_size": 12082884, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13499, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13499, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 18817, "num_processed_events": 13499, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13499, "processed_events_size": 12082884, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "104671": "2021-02-17T08:39:53Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "104671": "2021-02-17T08:39:53Z" }, "v8worker_events_lost": 0 }, "function_id": 3349188981, "function_name": "bucket_op", "gocb_creds_request_counter": 105, "internal_vb_distribution_stats": { "worker_bucket_op_0": "[0-341]" }, "latency_percentile_stats": { "100": 14800, "50": 600, "80": 900, "90": 1100, "95": 1300, "99": 2100 }, "lcb_creds_request_counter": 27077, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.156:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.161:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.92:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.156:8096": { "worker_bucket_op_0": "[0-341]" }, "172.23.122.161:8096": { "worker_bucket_op_0": "[342-682]" }, "172.23.122.92:8096": { "worker_bucket_op_0": "[683-1023]" } }, "worker_pids": { "worker_bucket_op_0": 104671 } } ] [2021-02-17 00:39:53,476] - [eventing_base] [139626970871552] - INFO - Stats for Node 172.23.122.161 is [ { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 2, "agg_messages_sent_to_worker": 18060, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 40376, "dcp_mutation_sent_to_worker": 13432, "dcp_snapshot": 40593, "dcp_stream_req_counter": 341, "dcp_streamreq": 341, "dcp_system_event": 3410, "execution_stats": 855, "failure_stats": 855, "latency_stats": 855, "lcb_exception_stats": 855, "log_level": 1, "num_processed_events": 13432, "processed_events_size": 12025248, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13432, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13432, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 18056, "num_processed_events": 13432, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13432, "processed_events_size": 12025248, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "85465": "2021-02-17T08:39:53Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "85465": "2021-02-17T08:39:53Z" }, "v8worker_events_lost": 0 }, "function_id": 3349188981, "function_name": "bucket_op", "gocb_creds_request_counter": 83, "internal_vb_distribution_stats": { "worker_bucket_op_0": "[342-682]" }, "latency_percentile_stats": { "100": 18600, "50": 500, "80": 800, "90": 900, "95": 1100, "99": 1800 }, "lcb_creds_request_counter": 26952, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.156:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.161:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.92:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.156:8096": { "worker_bucket_op_0": "[0-341]" }, "172.23.122.161:8096": { "worker_bucket_op_0": "[342-682]" }, "172.23.122.92:8096": { "worker_bucket_op_0": "[683-1023]" } }, "worker_pids": { "worker_bucket_op_0": 85465 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 2, "agg_messages_sent_to_worker": 17949, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 40376, "dcp_mutation_sent_to_worker": 13432, "dcp_snapshot": 40587, "dcp_stream_req_counter": 341, "dcp_streamreq": 341, "dcp_system_event": 3410, "execution_stats": 833, "failure_stats": 833, "latency_stats": 833, "lcb_exception_stats": 833, "log_level": 1, "num_processed_events": 13432, "processed_events_size": 12025248, "thr_count": 1, "thr_map": 1, "timer_events": 13558, "timer_responses_received": 13432, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13432, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13432, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 17945, "num_processed_events": 13432, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13432, "processed_events_size": 12025248, "timer_callback_failure": 0, "timer_callback_success": 13558, "timer_cancel_counter": 0, "timer_create_counter": 13432, "timer_create_failure": 0, "timer_msg_counter": 13558, "timer_responses_sent": 0, "timestamp": { "85496": "2021-02-17T08:39:52Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "85496": "2021-02-17T08:39:52Z" }, "v8worker_events_lost": 0 }, "function_id": 1236009705, "function_name": "timers", "gocb_creds_request_counter": 83, "internal_vb_distribution_stats": { "worker_timers_0": "[342-682]" }, "latency_percentile_stats": { "100": 19800, "50": 1500, "80": 1900, "90": 2200, "95": 2500, "99": 3900 }, "lcb_creds_request_counter": 26952, "lcb_exception_stats": {}, "metastore_stats": { "metastore_delete_err": 0, "metastore_deletes": 0, "metastore_not_found": 0, "metastore_scan": 0, "metastore_scan_due": 0, "metastore_scan_err": 0, "metastore_set": 0, "metastore_set_err": 0 }, "planner_stats": [ { "host_name": "172.23.122.156:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.161:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.92:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.156:8096": { "worker_timers_0": "[0-341]" }, "172.23.122.161:8096": { "worker_timers_0": "[342-682]" }, "172.23.122.92:8096": { "worker_timers_0": "[683-1023]" } }, "worker_pids": { "worker_timers_0": 85496 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 2, "agg_messages_sent_to_worker": 17843, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 40376, "dcp_mutation_sent_to_worker": 13432, "dcp_mutation_suppressed_counter": 13512, "dcp_snapshot": 40579, "dcp_stream_req_counter": 341, "dcp_streamreq": 341, "dcp_system_event": 3410, "execution_stats": 812, "failure_stats": 812, "latency_stats": 812, "lcb_exception_stats": 812, "log_level": 1, "num_processed_events": 13432, "processed_events_size": 12022944, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13432, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13432, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 17839, "num_processed_events": 13432, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13432, "processed_events_size": 12022944, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "85574": "2021-02-17T08:39:53Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "85574": "2021-02-17T08:39:53Z" }, "v8worker_events_lost": 0 }, "function_id": 717983186, "function_name": "sbm", "gocb_creds_request_counter": 83, "internal_vb_distribution_stats": { "worker_sbm_0": "[342-682]" }, "latency_percentile_stats": { "100": 15200, "50": 600, "80": 1000, "90": 1200, "95": 1600, "99": 2800 }, "lcb_creds_request_counter": 26952, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.156:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.161:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.92:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.156:8096": { "worker_sbm_0": "[0-341]" }, "172.23.122.161:8096": { "worker_sbm_0": "[342-682]" }, "172.23.122.92:8096": { "worker_sbm_0": "[683-1023]" } }, "worker_pids": { "worker_sbm_0": 85574 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 3, "agg_messages_sent_to_worker": 17737, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 40376, "dcp_mutation_sent_to_worker": 13432, "dcp_snapshot": 40558, "dcp_stream_req_counter": 341, "dcp_streamreq": 341, "dcp_system_event": 3410, "execution_stats": 791, "failure_stats": 791, "latency_stats": 791, "lcb_exception_stats": 791, "log_level": 1, "num_processed_events": 13432, "processed_events_size": 12025248, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 13432, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 13432, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13432, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13432, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 17733, "num_processed_events": 13432, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13432, "processed_events_size": 12025248, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "85605": "2021-02-17T08:39:53Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "85605": "2021-02-17T08:39:53Z" }, "v8worker_events_lost": 0 }, "function_id": 613640013, "function_name": "curl", "gocb_creds_request_counter": 83, "internal_vb_distribution_stats": { "worker_curl_0": "[342-682]" }, "latency_percentile_stats": { "100": 26400, "50": 1400, "80": 1700, "90": 2000, "95": 2200, "99": 3400 }, "lcb_creds_request_counter": 26952, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.156:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.161:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.92:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.156:8096": { "worker_curl_0": "[0-341]" }, "172.23.122.161:8096": { "worker_curl_0": "[342-682]" }, "172.23.122.92:8096": { "worker_curl_0": "[683-1023]" } }, "worker_pids": { "worker_curl_0": 85605 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 3, "agg_messages_sent_to_worker": 17626, "agg_queue_memory": 0, "agg_queue_memory_cap": 20971520, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 40376, "dcp_mutation_sent_to_worker": 13432, "dcp_snapshot": 40573, "dcp_stream_req_counter": 341, "dcp_streamreq": 341, "dcp_system_event": 3410, "execution_stats": 769, "failure_stats": 769, "latency_stats": 769, "lcb_exception_stats": 769, "log_level": 1, "num_processed_events": 13432, "processed_events_size": 12025248, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13432, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13432, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 17622, "num_processed_events": 13432, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13432, "processed_events_size": 12025248, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "85637": "2021-02-17T08:39:52Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "85637": "2021-02-17T08:39:52Z" }, "v8worker_events_lost": 0 }, "function_id": 22036014, "function_name": "n1ql", "gocb_creds_request_counter": 83, "internal_vb_distribution_stats": { "worker_n1ql_0": "[342-682]" }, "latency_percentile_stats": { "100": 88800, "50": 3500, "80": 4700, "90": 5600, "95": 6800, "99": 11200 }, "lcb_creds_request_counter": 26952, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.156:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.161:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.92:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.156:8096": { "worker_n1ql_0": "[0-341]" }, "172.23.122.161:8096": { "worker_n1ql_0": "[342-682]" }, "172.23.122.92:8096": { "worker_n1ql_0": "[683-1023]" } }, "worker_pids": { "worker_n1ql_0": 85637 } } ] [2021-02-17 00:39:53,522] - [eventing_base] [139626970871552] - INFO - Stats for Node 172.23.122.92 is [ { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 2, "agg_messages_sent_to_worker": 17911, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 40255, "dcp_mutation_sent_to_worker": 13389, "dcp_snapshot": 40435, "dcp_stream_req_counter": 341, "dcp_streamreq": 341, "dcp_system_event": 3410, "execution_stats": 834, "failure_stats": 834, "latency_stats": 834, "lcb_exception_stats": 834, "log_level": 1, "num_processed_events": 13389, "processed_events_size": 11986280, "thr_count": 1, "thr_map": 1, "timer_events": 13234, "timer_responses_received": 13389, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13389, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13389, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 17907, "num_processed_events": 13389, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13389, "processed_events_size": 11986280, "timer_callback_failure": 0, "timer_callback_success": 13234, "timer_cancel_counter": 0, "timer_create_counter": 13389, "timer_create_failure": 0, "timer_msg_counter": 13234, "timer_responses_sent": 0, "timestamp": { "58130": "2021-02-17T08:39:53Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "58130": "2021-02-17T08:39:53Z" }, "v8worker_events_lost": 0 }, "function_id": 1236009705, "function_name": "timers", "gocb_creds_request_counter": 89, "internal_vb_distribution_stats": { "worker_timers_0": "[683-1023]" }, "latency_percentile_stats": { "100": 22400, "50": 1500, "80": 1900, "90": 2200, "95": 2600, "99": 3900 }, "lcb_creds_request_counter": 26865, "lcb_exception_stats": {}, "metastore_stats": { "metastore_delete_err": 0, "metastore_deletes": 0, "metastore_not_found": 0, "metastore_scan": 0, "metastore_scan_due": 0, "metastore_scan_err": 0, "metastore_set": 0, "metastore_set_err": 0 }, "planner_stats": [ { "host_name": "172.23.122.156:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.161:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.92:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.156:8096": { "worker_timers_0": "[0-341]" }, "172.23.122.161:8096": { "worker_timers_0": "[342-682]" }, "172.23.122.92:8096": { "worker_timers_0": "[683-1023]" } }, "worker_pids": { "worker_timers_0": 58130 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 1, "agg_messages_sent_to_worker": 17800, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 40255, "dcp_mutation_sent_to_worker": 13389, "dcp_mutation_suppressed_counter": 13477, "dcp_snapshot": 40422, "dcp_stream_req_counter": 341, "dcp_streamreq": 341, "dcp_system_event": 3410, "execution_stats": 812, "failure_stats": 812, "latency_stats": 812, "lcb_exception_stats": 812, "log_level": 1, "num_processed_events": 13389, "processed_events_size": 11990148, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13389, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13389, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 17796, "num_processed_events": 13389, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13389, "processed_events_size": 11990148, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "58163": "2021-02-17T08:39:53Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "58163": "2021-02-17T08:39:53Z" }, "v8worker_events_lost": 0 }, "function_id": 717983186, "function_name": "sbm", "gocb_creds_request_counter": 89, "internal_vb_distribution_stats": { "worker_sbm_0": "[683-1023]" }, "latency_percentile_stats": { "100": 16800, "50": 700, "80": 1000, "90": 1200, "95": 1500, "99": 2600 }, "lcb_creds_request_counter": 26865, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.156:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.161:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.92:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.156:8096": { "worker_sbm_0": "[0-341]" }, "172.23.122.161:8096": { "worker_sbm_0": "[342-682]" }, "172.23.122.92:8096": { "worker_sbm_0": "[683-1023]" } }, "worker_pids": { "worker_sbm_0": 58163 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 1, "agg_messages_sent_to_worker": 17694, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 40255, "dcp_mutation_sent_to_worker": 13389, "dcp_snapshot": 40418, "dcp_stream_req_counter": 341, "dcp_streamreq": 341, "dcp_system_event": 3410, "execution_stats": 791, "failure_stats": 791, "latency_stats": 791, "lcb_exception_stats": 791, "log_level": 1, "num_processed_events": 13389, "processed_events_size": 11986280, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 13389, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 13389, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13389, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13389, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 17690, "num_processed_events": 13389, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13389, "processed_events_size": 11986280, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "58195": "2021-02-17T08:39:53Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "58195": "2021-02-17T08:39:53Z" }, "v8worker_events_lost": 0 }, "function_id": 613640013, "function_name": "curl", "gocb_creds_request_counter": 89, "internal_vb_distribution_stats": { "worker_curl_0": "[683-1023]" }, "latency_percentile_stats": { "100": 29300, "50": 1400, "80": 1700, "90": 2000, "95": 2300, "99": 3500 }, "lcb_creds_request_counter": 26865, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.156:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.161:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.92:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.156:8096": { "worker_curl_0": "[0-341]" }, "172.23.122.161:8096": { "worker_curl_0": "[342-682]" }, "172.23.122.92:8096": { "worker_curl_0": "[683-1023]" } }, "worker_pids": { "worker_curl_0": 58195 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 1, "agg_messages_sent_to_worker": 17583, "agg_queue_memory": 0, "agg_queue_memory_cap": 20971520, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 40255, "dcp_mutation_sent_to_worker": 13389, "dcp_snapshot": 40419, "dcp_stream_req_counter": 341, "dcp_streamreq": 341, "dcp_system_event": 3410, "execution_stats": 769, "failure_stats": 769, "latency_stats": 769, "lcb_exception_stats": 769, "log_level": 1, "num_processed_events": 13389, "processed_events_size": 11986280, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13389, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13389, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 17579, "num_processed_events": 13389, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13389, "processed_events_size": 11986280, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "58257": "2021-02-17T08:39:52Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "58257": "2021-02-17T08:39:52Z" }, "v8worker_events_lost": 0 }, "function_id": 22036014, "function_name": "n1ql", "gocb_creds_request_counter": 89, "internal_vb_distribution_stats": { "worker_n1ql_0": "[683-1023]" }, "latency_percentile_stats": { "100": 143000, "50": 3500, "80": 4600, "90": 5700, "95": 6900, "99": 11300 }, "lcb_creds_request_counter": 26865, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.156:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.161:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.92:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.156:8096": { "worker_n1ql_0": "[0-341]" }, "172.23.122.161:8096": { "worker_n1ql_0": "[342-682]" }, "172.23.122.92:8096": { "worker_n1ql_0": "[683-1023]" } }, "worker_pids": { "worker_n1ql_0": 58257 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 1, "agg_messages_sent_to_worker": 18017, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 40255, "dcp_mutation_sent_to_worker": 13389, "dcp_snapshot": 40417, "dcp_stream_req_counter": 341, "dcp_streamreq": 341, "dcp_system_event": 3410, "execution_stats": 855, "failure_stats": 855, "latency_stats": 855, "lcb_exception_stats": 855, "log_level": 1, "num_processed_events": 13389, "processed_events_size": 11986280, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13389, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13389, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 18013, "num_processed_events": 13389, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13389, "processed_events_size": 11986280, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "58055": "2021-02-17T08:39:53Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "58055": "2021-02-17T08:39:53Z" }, "v8worker_events_lost": 0 }, "function_id": 3349188981, "function_name": "bucket_op", "gocb_creds_request_counter": 89, "internal_vb_distribution_stats": { "worker_bucket_op_0": "[683-1023]" }, "latency_percentile_stats": { "100": 30800, "50": 600, "80": 800, "90": 1000, "95": 1200, "99": 1900 }, "lcb_creds_request_counter": 26865, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.156:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.161:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.92:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.156:8096": { "worker_bucket_op_0": "[0-341]" }, "172.23.122.161:8096": { "worker_bucket_op_0": "[342-682]" }, "172.23.122.92:8096": { "worker_bucket_op_0": "[683-1023]" } }, "worker_pids": { "worker_bucket_op_0": 58055 } } ] [2021-02-17 00:39:57,860] - [eventing_base] [139626970871552] - INFO - Stats for Node 172.23.122.156 is [ { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 3, "agg_messages_sent_to_worker": 18821, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 67159, "dcp_mutation_sent_to_worker": 13499, "dcp_snapshot": 40847, "dcp_stream_req_counter": 9026, "dcp_stream_req_err_counter": 8233, "dcp_streamend": 351, "dcp_streamreq": 793, "dcp_system_event": 3420, "execution_stats": 855, "failure_stats": 855, "latency_stats": 855, "lcb_exception_stats": 855, "log_level": 1, "num_processed_events": 13499, "processed_events_size": 12082884, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13499, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13499, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 18817, "num_processed_events": 13499, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13499, "processed_events_size": 12082884, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "104671": "2021-02-17T08:39:53Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "104671": "2021-02-17T08:39:53Z" }, "v8worker_events_lost": 0 }, "function_id": 3349188981, "function_name": "bucket_op", "gocb_creds_request_counter": 105, "internal_vb_distribution_stats": { "worker_bucket_op_0": "[0-341]" }, "latency_percentile_stats": { "100": 14800, "50": 600, "80": 900, "90": 1100, "95": 1300, "99": 2100 }, "lcb_creds_request_counter": 27077, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.156:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.161:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.92:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.156:8096": { "worker_bucket_op_0": "[0-341]" }, "172.23.122.161:8096": { "worker_bucket_op_0": "[342-682]" }, "172.23.122.92:8096": { "worker_bucket_op_0": "[683-1023]" } }, "worker_pids": { "worker_bucket_op_0": 104671 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 1, "agg_messages_sent_to_worker": 18715, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 67159, "dcp_mutation_sent_to_worker": 13499, "dcp_snapshot": 40865, "dcp_stream_req_counter": 8868, "dcp_stream_req_err_counter": 8049, "dcp_streamend": 351, "dcp_streamreq": 819, "dcp_system_event": 3420, "execution_stats": 834, "failure_stats": 834, "latency_stats": 834, "lcb_exception_stats": 834, "log_level": 1, "num_processed_events": 13499, "processed_events_size": 12082884, "thr_count": 1, "thr_map": 1, "timer_events": 13528, "timer_responses_received": 13499, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13499, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13499, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 18711, "num_processed_events": 13499, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13499, "processed_events_size": 12082884, "timer_callback_failure": 0, "timer_callback_success": 13528, "timer_cancel_counter": 0, "timer_create_counter": 13499, "timer_create_failure": 0, "timer_msg_counter": 13528, "timer_responses_sent": 0, "timestamp": { "104712": "2021-02-17T08:39:53Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "104712": "2021-02-17T08:39:53Z" }, "v8worker_events_lost": 0 }, "function_id": 1236009705, "function_name": "timers", "gocb_creds_request_counter": 105, "internal_vb_distribution_stats": { "worker_timers_0": "[0-341]" }, "latency_percentile_stats": { "100": 19200, "50": 1500, "80": 2000, "90": 2300, "95": 2700, "99": 4000 }, "lcb_creds_request_counter": 27077, "lcb_exception_stats": {}, "metastore_stats": { "metastore_delete_err": 0, "metastore_deletes": 0, "metastore_not_found": 0, "metastore_scan": 0, "metastore_scan_due": 0, "metastore_scan_err": 0, "metastore_set": 0, "metastore_set_err": 0 }, "planner_stats": [ { "host_name": "172.23.122.156:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.161:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.92:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.156:8096": { "worker_timers_0": "[0-341]" }, "172.23.122.161:8096": { "worker_timers_0": "[342-682]" }, "172.23.122.92:8096": { "worker_timers_0": "[683-1023]" } }, "worker_pids": { "worker_timers_0": 104712 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "agg_messages_sent_to_worker": 18604, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 40712, "dcp_mutation_sent_to_worker": 13499, "dcp_mutation_suppressed_counter": 13714, "dcp_snapshot": 40712, "dcp_stream_req_counter": 8213, "dcp_stream_req_err_counter": 7353, "dcp_streamend": 351, "dcp_streamreq": 860, "dcp_system_event": 3420, "execution_stats": 812, "failure_stats": 812, "latency_stats": 812, "lcb_exception_stats": 812, "log_level": 1, "num_processed_events": 13499, "processed_events_size": 12086816, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13499, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13499, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 18600, "num_processed_events": 13499, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13499, "processed_events_size": 12086816, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "104754": "2021-02-17T08:39:52Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "104754": "2021-02-17T08:39:52Z" }, "v8worker_events_lost": 0 }, "function_id": 717983186, "function_name": "sbm", "gocb_creds_request_counter": 105, "internal_vb_distribution_stats": { "worker_sbm_0": "[0-341]" }, "latency_percentile_stats": { "100": 22500, "50": 700, "80": 1000, "90": 1300, "95": 1600, "99": 2700 }, "lcb_creds_request_counter": 27077, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.156:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.161:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.92:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.156:8096": { "worker_sbm_0": "[0-341]" }, "172.23.122.161:8096": { "worker_sbm_0": "[342-682]" }, "172.23.122.92:8096": { "worker_sbm_0": "[683-1023]" } }, "worker_pids": { "worker_sbm_0": 104754 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 2, "agg_messages_sent_to_worker": 18498, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 67159, "dcp_mutation_sent_to_worker": 13499, "dcp_snapshot": 40866, "dcp_stream_req_counter": 7847, "dcp_stream_req_err_counter": 6946, "dcp_streamend": 351, "dcp_streamreq": 901, "dcp_system_event": 3420, "execution_stats": 791, "failure_stats": 791, "latency_stats": 791, "lcb_exception_stats": 791, "log_level": 1, "num_processed_events": 13499, "processed_events_size": 12082884, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 13499, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 13499, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13499, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13499, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 18494, "num_processed_events": 13499, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13499, "processed_events_size": 12082884, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "104827": "2021-02-17T08:39:53Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "104827": "2021-02-17T08:39:53Z" }, "v8worker_events_lost": 0 }, "function_id": 613640013, "function_name": "curl", "gocb_creds_request_counter": 105, "internal_vb_distribution_stats": { "worker_curl_0": "[0-341]" }, "latency_percentile_stats": { "100": 26400, "50": 1400, "80": 1800, "90": 2000, "95": 2400, "99": 4000 }, "lcb_creds_request_counter": 27077, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.156:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.161:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.92:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.156:8096": { "worker_curl_0": "[0-341]" }, "172.23.122.161:8096": { "worker_curl_0": "[342-682]" }, "172.23.122.92:8096": { "worker_curl_0": "[683-1023]" } }, "worker_pids": { "worker_curl_0": 104827 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 3, "agg_messages_sent_to_worker": 18387, "agg_queue_memory": 0, "agg_queue_memory_cap": 20971520, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 67159, "dcp_mutation_sent_to_worker": 13499, "dcp_snapshot": 40847, "dcp_stream_req_counter": 7450, "dcp_stream_req_err_counter": 6515, "dcp_streamend": 351, "dcp_streamreq": 935, "dcp_system_event": 3420, "execution_stats": 769, "failure_stats": 769, "latency_stats": 769, "lcb_exception_stats": 769, "log_level": 1, "num_processed_events": 13499, "processed_events_size": 12082884, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13499, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13499, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 18383, "num_processed_events": 13499, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 4, "on_update_success": 13495, "processed_events_size": 12082884, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "104867": "2021-02-17T08:39:52Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 4, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "104867": "2021-02-17T08:39:52Z" }, "v8worker_events_lost": 0 }, "function_id": 22036014, "function_name": "n1ql", "gocb_creds_request_counter": 105, "internal_vb_distribution_stats": { "worker_n1ql_0": "[0-341]" }, "latency_percentile_stats": { "100": 125600, "50": 3400, "80": 4700, "90": 5700, "95": 7100, "99": 12200 }, "lcb_creds_request_counter": 27077, "lcb_exception_stats": { "12009": 4, "209": 4 }, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.156:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.161:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.92:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.156:8096": { "worker_n1ql_0": "[0-341]" }, "172.23.122.161:8096": { "worker_n1ql_0": "[342-682]" }, "172.23.122.92:8096": { "worker_n1ql_0": "[683-1023]" } }, "worker_pids": { "worker_n1ql_0": 104867 } } ] [2021-02-17 00:40:02,922] - [eventing_base] [139626970871552] - INFO - Stats for Node 172.23.122.161 is [ { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 3, "agg_messages_sent_to_worker": 17651, "agg_queue_memory": 0, "agg_queue_memory_cap": 20971520, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 40376, "dcp_mutation_sent_to_worker": 13432, "dcp_snapshot": 40573, "dcp_stream_req_counter": 341, "dcp_streamreq": 341, "dcp_system_event": 3410, "execution_stats": 774, "failure_stats": 774, "latency_stats": 774, "lcb_exception_stats": 774, "log_level": 1, "num_processed_events": 13432, "processed_events_size": 12025248, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13432, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13432, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 17647, "num_processed_events": 13432, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13432, "processed_events_size": 12025248, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "85637": "2021-02-17T08:39:57Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "85637": "2021-02-17T08:39:57Z" }, "v8worker_events_lost": 0 }, "function_id": 22036014, "function_name": "n1ql", "gocb_creds_request_counter": 83, "internal_vb_distribution_stats": { "worker_n1ql_0": "[342-682]" }, "latency_percentile_stats": { "100": 88800, "50": 3500, "80": 4700, "90": 5600, "95": 6800, "99": 11200 }, "lcb_creds_request_counter": 26952, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.156:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.161:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.92:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.156:8096": { "worker_n1ql_0": "[0-341]" }, "172.23.122.161:8096": { "worker_n1ql_0": "[342-682]" }, "172.23.122.92:8096": { "worker_n1ql_0": "[683-1023]" } }, "worker_pids": { "worker_n1ql_0": 85637 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 2, "agg_messages_sent_to_worker": 18085, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 40376, "dcp_mutation_sent_to_worker": 13432, "dcp_snapshot": 40593, "dcp_stream_req_counter": 341, "dcp_streamreq": 341, "dcp_system_event": 3410, "execution_stats": 860, "failure_stats": 860, "latency_stats": 860, "lcb_exception_stats": 860, "log_level": 1, "num_processed_events": 13432, "processed_events_size": 12025248, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13432, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13432, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 18081, "num_processed_events": 13432, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13432, "processed_events_size": 12025248, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "85465": "2021-02-17T08:39:58Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "85465": "2021-02-17T08:39:58Z" }, "v8worker_events_lost": 0 }, "function_id": 3349188981, "function_name": "bucket_op", "gocb_creds_request_counter": 83, "internal_vb_distribution_stats": { "worker_bucket_op_0": "[342-682]" }, "latency_percentile_stats": { "100": 18600, "50": 500, "80": 800, "90": 900, "95": 1100, "99": 1800 }, "lcb_creds_request_counter": 26952, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.156:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.161:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.92:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.156:8096": { "worker_bucket_op_0": "[0-341]" }, "172.23.122.161:8096": { "worker_bucket_op_0": "[342-682]" }, "172.23.122.92:8096": { "worker_bucket_op_0": "[683-1023]" } }, "worker_pids": { "worker_bucket_op_0": 85465 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 2, "agg_messages_sent_to_worker": 17979, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 40376, "dcp_mutation_sent_to_worker": 13432, "dcp_snapshot": 40587, "dcp_stream_req_counter": 341, "dcp_streamreq": 341, "dcp_system_event": 3410, "execution_stats": 839, "failure_stats": 839, "latency_stats": 839, "lcb_exception_stats": 839, "log_level": 1, "num_processed_events": 13432, "processed_events_size": 12025248, "thr_count": 1, "thr_map": 1, "timer_events": 13558, "timer_responses_received": 13432, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13432, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13432, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 17975, "num_processed_events": 13432, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13432, "processed_events_size": 12025248, "timer_callback_failure": 0, "timer_callback_success": 13558, "timer_cancel_counter": 0, "timer_create_counter": 13432, "timer_create_failure": 0, "timer_msg_counter": 13558, "timer_responses_sent": 0, "timestamp": { "85496": "2021-02-17T08:39:58Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "85496": "2021-02-17T08:39:58Z" }, "v8worker_events_lost": 0 }, "function_id": 1236009705, "function_name": "timers", "gocb_creds_request_counter": 83, "internal_vb_distribution_stats": { "worker_timers_0": "[342-682]" }, "latency_percentile_stats": { "100": 19800, "50": 1500, "80": 1900, "90": 2200, "95": 2500, "99": 3900 }, "lcb_creds_request_counter": 26952, "lcb_exception_stats": {}, "metastore_stats": { "metastore_delete_err": 0, "metastore_deletes": 0, "metastore_not_found": 0, "metastore_scan": 0, "metastore_scan_due": 0, "metastore_scan_err": 0, "metastore_set": 0, "metastore_set_err": 0 }, "planner_stats": [ { "host_name": "172.23.122.156:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.161:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.92:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.156:8096": { "worker_timers_0": "[0-341]" }, "172.23.122.161:8096": { "worker_timers_0": "[342-682]" }, "172.23.122.92:8096": { "worker_timers_0": "[683-1023]" } }, "worker_pids": { "worker_timers_0": 85496 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 2, "agg_messages_sent_to_worker": 17868, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 40376, "dcp_mutation_sent_to_worker": 13432, "dcp_mutation_suppressed_counter": 13512, "dcp_snapshot": 40579, "dcp_stream_req_counter": 341, "dcp_streamreq": 341, "dcp_system_event": 3410, "execution_stats": 817, "failure_stats": 817, "latency_stats": 817, "lcb_exception_stats": 817, "log_level": 1, "num_processed_events": 13432, "processed_events_size": 12022944, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13432, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13432, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 17864, "num_processed_events": 13432, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13432, "processed_events_size": 12022944, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "85574": "2021-02-17T08:39:58Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "85574": "2021-02-17T08:39:58Z" }, "v8worker_events_lost": 0 }, "function_id": 717983186, "function_name": "sbm", "gocb_creds_request_counter": 83, "internal_vb_distribution_stats": { "worker_sbm_0": "[342-682]" }, "latency_percentile_stats": { "100": 15200, "50": 600, "80": 1000, "90": 1200, "95": 1600, "99": 2800 }, "lcb_creds_request_counter": 26952, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.156:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.161:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.92:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.156:8096": { "worker_sbm_0": "[0-341]" }, "172.23.122.161:8096": { "worker_sbm_0": "[342-682]" }, "172.23.122.92:8096": { "worker_sbm_0": "[683-1023]" } }, "worker_pids": { "worker_sbm_0": 85574 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 3, "agg_messages_sent_to_worker": 17762, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 40376, "dcp_mutation_sent_to_worker": 13432, "dcp_snapshot": 40558, "dcp_stream_req_counter": 341, "dcp_streamreq": 341, "dcp_system_event": 3410, "execution_stats": 796, "failure_stats": 796, "latency_stats": 796, "lcb_exception_stats": 796, "log_level": 1, "num_processed_events": 13432, "processed_events_size": 12025248, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 13432, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 13432, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13432, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13432, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 17758, "num_processed_events": 13432, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13432, "processed_events_size": 12025248, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "85605": "2021-02-17T08:39:58Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "85605": "2021-02-17T08:39:58Z" }, "v8worker_events_lost": 0 }, "function_id": 613640013, "function_name": "curl", "gocb_creds_request_counter": 83, "internal_vb_distribution_stats": { "worker_curl_0": "[342-682]" }, "latency_percentile_stats": { "100": 26400, "50": 1400, "80": 1700, "90": 2000, "95": 2200, "99": 3400 }, "lcb_creds_request_counter": 26952, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.156:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.161:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.92:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.156:8096": { "worker_curl_0": "[0-341]" }, "172.23.122.161:8096": { "worker_curl_0": "[342-682]" }, "172.23.122.92:8096": { "worker_curl_0": "[683-1023]" } }, "worker_pids": { "worker_curl_0": 85605 } } ] [2021-02-17 00:40:08,023] - [eventing_base] [139626970871552] - INFO - Stats for Node 172.23.122.92 is [ { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 1, "agg_messages_sent_to_worker": 17633, "agg_queue_memory": 0, "agg_queue_memory_cap": 20971520, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 40255, "dcp_mutation_sent_to_worker": 13389, "dcp_snapshot": 40419, "dcp_stream_req_counter": 341, "dcp_streamreq": 341, "dcp_system_event": 3410, "execution_stats": 779, "failure_stats": 779, "latency_stats": 779, "lcb_exception_stats": 779, "log_level": 1, "num_processed_events": 13389, "processed_events_size": 11986280, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13389, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13389, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 17629, "num_processed_events": 13389, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13389, "processed_events_size": 11986280, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "58257": "2021-02-17T08:40:02Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "58257": "2021-02-17T08:40:02Z" }, "v8worker_events_lost": 0 }, "function_id": 22036014, "function_name": "n1ql", "gocb_creds_request_counter": 89, "internal_vb_distribution_stats": { "worker_n1ql_0": "[683-1023]" }, "latency_percentile_stats": { "100": 143000, "50": 3500, "80": 4600, "90": 5700, "95": 6900, "99": 11300 }, "lcb_creds_request_counter": 26865, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.156:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.161:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.92:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.156:8096": { "worker_n1ql_0": "[0-341]" }, "172.23.122.161:8096": { "worker_n1ql_0": "[342-682]" }, "172.23.122.92:8096": { "worker_n1ql_0": "[683-1023]" } }, "worker_pids": { "worker_n1ql_0": 58257 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 1, "agg_messages_sent_to_worker": 18067, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 40255, "dcp_mutation_sent_to_worker": 13389, "dcp_snapshot": 40417, "dcp_stream_req_counter": 341, "dcp_streamreq": 341, "dcp_system_event": 3410, "execution_stats": 865, "failure_stats": 865, "latency_stats": 865, "lcb_exception_stats": 865, "log_level": 1, "num_processed_events": 13389, "processed_events_size": 11986280, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13389, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13389, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 18063, "num_processed_events": 13389, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13389, "processed_events_size": 11986280, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "58055": "2021-02-17T08:40:03Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "58055": "2021-02-17T08:40:03Z" }, "v8worker_events_lost": 0 }, "function_id": 3349188981, "function_name": "bucket_op", "gocb_creds_request_counter": 89, "internal_vb_distribution_stats": { "worker_bucket_op_0": "[683-1023]" }, "latency_percentile_stats": { "100": 30800, "50": 600, "80": 800, "90": 1000, "95": 1200, "99": 1900 }, "lcb_creds_request_counter": 26865, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.156:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.161:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.92:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.156:8096": { "worker_bucket_op_0": "[0-341]" }, "172.23.122.161:8096": { "worker_bucket_op_0": "[342-682]" }, "172.23.122.92:8096": { "worker_bucket_op_0": "[683-1023]" } }, "worker_pids": { "worker_bucket_op_0": 58055 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 2, "agg_messages_sent_to_worker": 17961, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 40255, "dcp_mutation_sent_to_worker": 13389, "dcp_snapshot": 40435, "dcp_stream_req_counter": 341, "dcp_streamreq": 341, "dcp_system_event": 3410, "execution_stats": 844, "failure_stats": 844, "latency_stats": 844, "lcb_exception_stats": 844, "log_level": 1, "num_processed_events": 13389, "processed_events_size": 11986280, "thr_count": 1, "thr_map": 1, "timer_events": 13234, "timer_responses_received": 13389, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13389, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13389, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 17957, "num_processed_events": 13389, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13389, "processed_events_size": 11986280, "timer_callback_failure": 0, "timer_callback_success": 13234, "timer_cancel_counter": 0, "timer_create_counter": 13389, "timer_create_failure": 0, "timer_msg_counter": 13234, "timer_responses_sent": 0, "timestamp": { "58130": "2021-02-17T08:40:03Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "58130": "2021-02-17T08:40:03Z" }, "v8worker_events_lost": 0 }, "function_id": 1236009705, "function_name": "timers", "gocb_creds_request_counter": 89, "internal_vb_distribution_stats": { "worker_timers_0": "[683-1023]" }, "latency_percentile_stats": { "100": 22400, "50": 1500, "80": 1900, "90": 2200, "95": 2600, "99": 3900 }, "lcb_creds_request_counter": 26865, "lcb_exception_stats": {}, "metastore_stats": { "metastore_delete_err": 0, "metastore_deletes": 0, "metastore_not_found": 0, "metastore_scan": 0, "metastore_scan_due": 0, "metastore_scan_err": 0, "metastore_set": 0, "metastore_set_err": 0 }, "planner_stats": [ { "host_name": "172.23.122.156:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.161:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.92:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.156:8096": { "worker_timers_0": "[0-341]" }, "172.23.122.161:8096": { "worker_timers_0": "[342-682]" }, "172.23.122.92:8096": { "worker_timers_0": "[683-1023]" } }, "worker_pids": { "worker_timers_0": 58130 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 1, "agg_messages_sent_to_worker": 17850, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 40255, "dcp_mutation_sent_to_worker": 13389, "dcp_mutation_suppressed_counter": 13477, "dcp_snapshot": 40422, "dcp_stream_req_counter": 341, "dcp_streamreq": 341, "dcp_system_event": 3410, "execution_stats": 822, "failure_stats": 822, "latency_stats": 822, "lcb_exception_stats": 822, "log_level": 1, "num_processed_events": 13389, "processed_events_size": 11990148, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13389, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13389, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 17846, "num_processed_events": 13389, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13389, "processed_events_size": 11990148, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "58163": "2021-02-17T08:40:03Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "58163": "2021-02-17T08:40:03Z" }, "v8worker_events_lost": 0 }, "function_id": 717983186, "function_name": "sbm", "gocb_creds_request_counter": 89, "internal_vb_distribution_stats": { "worker_sbm_0": "[683-1023]" }, "latency_percentile_stats": { "100": 16800, "50": 700, "80": 1000, "90": 1200, "95": 1500, "99": 2600 }, "lcb_creds_request_counter": 26865, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.156:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.161:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.92:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.156:8096": { "worker_sbm_0": "[0-341]" }, "172.23.122.161:8096": { "worker_sbm_0": "[342-682]" }, "172.23.122.92:8096": { "worker_sbm_0": "[683-1023]" } }, "worker_pids": { "worker_sbm_0": 58163 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 1, "agg_messages_sent_to_worker": 17744, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 40255, "dcp_mutation_sent_to_worker": 13389, "dcp_snapshot": 40418, "dcp_stream_req_counter": 341, "dcp_streamreq": 341, "dcp_system_event": 3410, "execution_stats": 801, "failure_stats": 801, "latency_stats": 801, "lcb_exception_stats": 801, "log_level": 1, "num_processed_events": 13389, "processed_events_size": 11986280, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 13389, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 13389, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13389, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13389, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 17740, "num_processed_events": 13389, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13389, "processed_events_size": 11986280, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "58195": "2021-02-17T08:40:03Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "58195": "2021-02-17T08:40:03Z" }, "v8worker_events_lost": 0 }, "function_id": 613640013, "function_name": "curl", "gocb_creds_request_counter": 89, "internal_vb_distribution_stats": { "worker_curl_0": "[683-1023]" }, "latency_percentile_stats": { "100": 29300, "50": 1400, "80": 1700, "90": 2000, "95": 2300, "99": 3500 }, "lcb_creds_request_counter": 26865, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.156:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.161:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.92:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.156:8096": { "worker_curl_0": "[0-341]" }, "172.23.122.161:8096": { "worker_curl_0": "[342-682]" }, "172.23.122.92:8096": { "worker_curl_0": "[683-1023]" } }, "worker_pids": { "worker_curl_0": 58195 } } ] [2021-02-17 00:40:08,832] - [eventing_base] [139626970871552] - INFO - Execution stats update_success: 40320 update_failure: 0 delete_success: 0 delete_failure: 0 [2021-02-17 00:40:08,852] - [basetestcase] [139626970871552] - INFO - list of eventing nodes in cluster: [ip:172.23.122.156 port:8091 ssh_username:root, ip:172.23.122.161 port:8091 ssh_username:root, ip:172.23.122.92 port:8091 ssh_username:root] [2021-02-17 00:40:08,898] - [eventing_base] [139626970871552] - INFO - Stats for Node 172.23.122.156 is [ { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 3, "agg_messages_sent_to_worker": 18896, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 67159, "dcp_mutation_sent_to_worker": 13499, "dcp_snapshot": 40847, "dcp_stream_req_counter": 9026, "dcp_stream_req_err_counter": 8233, "dcp_streamend": 351, "dcp_streamreq": 793, "dcp_system_event": 3420, "execution_stats": 870, "failure_stats": 870, "latency_stats": 870, "lcb_exception_stats": 870, "log_level": 1, "num_processed_events": 13499, "processed_events_size": 12082884, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13499, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13499, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 18892, "num_processed_events": 13499, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13499, "processed_events_size": 12082884, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "104671": "2021-02-17T08:40:08Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "104671": "2021-02-17T08:40:08Z" }, "v8worker_events_lost": 0 }, "function_id": 3349188981, "function_name": "bucket_op", "gocb_creds_request_counter": 105, "internal_vb_distribution_stats": { "worker_bucket_op_0": "[0-341]" }, "latency_percentile_stats": { "100": 14800, "50": 600, "80": 900, "90": 1100, "95": 1300, "99": 2100 }, "lcb_creds_request_counter": 27077, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.156:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.161:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.92:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.156:8096": { "worker_bucket_op_0": "[0-341]" }, "172.23.122.161:8096": { "worker_bucket_op_0": "[342-682]" }, "172.23.122.92:8096": { "worker_bucket_op_0": "[683-1023]" } }, "worker_pids": { "worker_bucket_op_0": 104671 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 1, "agg_messages_sent_to_worker": 18790, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 67159, "dcp_mutation_sent_to_worker": 13499, "dcp_snapshot": 40865, "dcp_stream_req_counter": 8868, "dcp_stream_req_err_counter": 8049, "dcp_streamend": 351, "dcp_streamreq": 819, "dcp_system_event": 3420, "execution_stats": 849, "failure_stats": 849, "latency_stats": 849, "lcb_exception_stats": 849, "log_level": 1, "num_processed_events": 13499, "processed_events_size": 12082884, "thr_count": 1, "thr_map": 1, "timer_events": 13528, "timer_responses_received": 13499, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13499, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13499, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 18786, "num_processed_events": 13499, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13499, "processed_events_size": 12082884, "timer_callback_failure": 0, "timer_callback_success": 13528, "timer_cancel_counter": 0, "timer_create_counter": 13499, "timer_create_failure": 0, "timer_msg_counter": 13528, "timer_responses_sent": 0, "timestamp": { "104712": "2021-02-17T08:40:08Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "104712": "2021-02-17T08:40:08Z" }, "v8worker_events_lost": 0 }, "function_id": 1236009705, "function_name": "timers", "gocb_creds_request_counter": 105, "internal_vb_distribution_stats": { "worker_timers_0": "[0-341]" }, "latency_percentile_stats": { "100": 19200, "50": 1500, "80": 2000, "90": 2300, "95": 2700, "99": 4000 }, "lcb_creds_request_counter": 27077, "lcb_exception_stats": {}, "metastore_stats": { "metastore_delete_err": 0, "metastore_deletes": 0, "metastore_not_found": 0, "metastore_scan": 0, "metastore_scan_due": 0, "metastore_scan_err": 0, "metastore_set": 0, "metastore_set_err": 0 }, "planner_stats": [ { "host_name": "172.23.122.156:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.161:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.92:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.156:8096": { "worker_timers_0": "[0-341]" }, "172.23.122.161:8096": { "worker_timers_0": "[342-682]" }, "172.23.122.92:8096": { "worker_timers_0": "[683-1023]" } }, "worker_pids": { "worker_timers_0": 104712 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "agg_messages_sent_to_worker": 18679, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 40712, "dcp_mutation_sent_to_worker": 13499, "dcp_mutation_suppressed_counter": 13714, "dcp_snapshot": 40712, "dcp_stream_req_counter": 8213, "dcp_stream_req_err_counter": 7353, "dcp_streamend": 351, "dcp_streamreq": 860, "dcp_system_event": 3420, "execution_stats": 827, "failure_stats": 827, "latency_stats": 827, "lcb_exception_stats": 827, "log_level": 1, "num_processed_events": 13499, "processed_events_size": 12086816, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13499, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13499, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 18675, "num_processed_events": 13499, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13499, "processed_events_size": 12086816, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "104754": "2021-02-17T08:40:07Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "104754": "2021-02-17T08:40:07Z" }, "v8worker_events_lost": 0 }, "function_id": 717983186, "function_name": "sbm", "gocb_creds_request_counter": 105, "internal_vb_distribution_stats": { "worker_sbm_0": "[0-341]" }, "latency_percentile_stats": { "100": 22500, "50": 700, "80": 1000, "90": 1300, "95": 1600, "99": 2700 }, "lcb_creds_request_counter": 27077, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.156:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.161:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.92:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.156:8096": { "worker_sbm_0": "[0-341]" }, "172.23.122.161:8096": { "worker_sbm_0": "[342-682]" }, "172.23.122.92:8096": { "worker_sbm_0": "[683-1023]" } }, "worker_pids": { "worker_sbm_0": 104754 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 2, "agg_messages_sent_to_worker": 18573, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 67159, "dcp_mutation_sent_to_worker": 13499, "dcp_snapshot": 40866, "dcp_stream_req_counter": 7847, "dcp_stream_req_err_counter": 6946, "dcp_streamend": 351, "dcp_streamreq": 901, "dcp_system_event": 3420, "execution_stats": 806, "failure_stats": 806, "latency_stats": 806, "lcb_exception_stats": 806, "log_level": 1, "num_processed_events": 13499, "processed_events_size": 12082884, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 13499, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 13499, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13499, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13499, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 18569, "num_processed_events": 13499, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13499, "processed_events_size": 12082884, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "104827": "2021-02-17T08:40:08Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "104827": "2021-02-17T08:40:08Z" }, "v8worker_events_lost": 0 }, "function_id": 613640013, "function_name": "curl", "gocb_creds_request_counter": 105, "internal_vb_distribution_stats": { "worker_curl_0": "[0-341]" }, "latency_percentile_stats": { "100": 26400, "50": 1400, "80": 1800, "90": 2000, "95": 2400, "99": 4000 }, "lcb_creds_request_counter": 27077, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.156:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.161:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.92:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.156:8096": { "worker_curl_0": "[0-341]" }, "172.23.122.161:8096": { "worker_curl_0": "[342-682]" }, "172.23.122.92:8096": { "worker_curl_0": "[683-1023]" } }, "worker_pids": { "worker_curl_0": 104827 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 3, "agg_messages_sent_to_worker": 18467, "agg_queue_memory": 0, "agg_queue_memory_cap": 20971520, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 67159, "dcp_mutation_sent_to_worker": 13499, "dcp_snapshot": 40847, "dcp_stream_req_counter": 7450, "dcp_stream_req_err_counter": 6515, "dcp_streamend": 351, "dcp_streamreq": 935, "dcp_system_event": 3420, "execution_stats": 785, "failure_stats": 785, "latency_stats": 785, "lcb_exception_stats": 785, "log_level": 1, "num_processed_events": 13499, "processed_events_size": 12082884, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13499, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13499, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 18463, "num_processed_events": 13499, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 4, "on_update_success": 13495, "processed_events_size": 12082884, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "104867": "2021-02-17T08:40:08Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 4, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "104867": "2021-02-17T08:40:08Z" }, "v8worker_events_lost": 0 }, "function_id": 22036014, "function_name": "n1ql", "gocb_creds_request_counter": 105, "internal_vb_distribution_stats": { "worker_n1ql_0": "[0-341]" }, "latency_percentile_stats": { "100": 125600, "50": 3400, "80": 4700, "90": 5700, "95": 7100, "99": 12200 }, "lcb_creds_request_counter": 27077, "lcb_exception_stats": { "12009": 4, "209": 4 }, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.156:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.161:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.92:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.156:8096": { "worker_n1ql_0": "[0-341]" }, "172.23.122.161:8096": { "worker_n1ql_0": "[342-682]" }, "172.23.122.92:8096": { "worker_n1ql_0": "[683-1023]" } }, "worker_pids": { "worker_n1ql_0": 104867 } } ] [2021-02-17 00:40:08,942] - [eventing_base] [139626970871552] - INFO - Stats for Node 172.23.122.161 is [ { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 2, "agg_messages_sent_to_worker": 18029, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 40376, "dcp_mutation_sent_to_worker": 13432, "dcp_snapshot": 40587, "dcp_stream_req_counter": 341, "dcp_streamreq": 341, "dcp_system_event": 3410, "execution_stats": 849, "failure_stats": 849, "latency_stats": 849, "lcb_exception_stats": 849, "log_level": 1, "num_processed_events": 13432, "processed_events_size": 12025248, "thr_count": 1, "thr_map": 1, "timer_events": 13558, "timer_responses_received": 13432, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13432, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13432, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 18025, "num_processed_events": 13432, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13432, "processed_events_size": 12025248, "timer_callback_failure": 0, "timer_callback_success": 13558, "timer_cancel_counter": 0, "timer_create_counter": 13432, "timer_create_failure": 0, "timer_msg_counter": 13558, "timer_responses_sent": 0, "timestamp": { "85496": "2021-02-17T08:40:08Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "85496": "2021-02-17T08:40:08Z" }, "v8worker_events_lost": 0 }, "function_id": 1236009705, "function_name": "timers", "gocb_creds_request_counter": 83, "internal_vb_distribution_stats": { "worker_timers_0": "[342-682]" }, "latency_percentile_stats": { "100": 19800, "50": 1500, "80": 1900, "90": 2200, "95": 2500, "99": 3900 }, "lcb_creds_request_counter": 26952, "lcb_exception_stats": {}, "metastore_stats": { "metastore_delete_err": 0, "metastore_deletes": 0, "metastore_not_found": 0, "metastore_scan": 0, "metastore_scan_due": 0, "metastore_scan_err": 0, "metastore_set": 0, "metastore_set_err": 0 }, "planner_stats": [ { "host_name": "172.23.122.156:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.161:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.92:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.156:8096": { "worker_timers_0": "[0-341]" }, "172.23.122.161:8096": { "worker_timers_0": "[342-682]" }, "172.23.122.92:8096": { "worker_timers_0": "[683-1023]" } }, "worker_pids": { "worker_timers_0": 85496 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 2, "agg_messages_sent_to_worker": 17918, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 40376, "dcp_mutation_sent_to_worker": 13432, "dcp_mutation_suppressed_counter": 13512, "dcp_snapshot": 40579, "dcp_stream_req_counter": 341, "dcp_streamreq": 341, "dcp_system_event": 3410, "execution_stats": 827, "failure_stats": 827, "latency_stats": 827, "lcb_exception_stats": 827, "log_level": 1, "num_processed_events": 13432, "processed_events_size": 12022944, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13432, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13432, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 17914, "num_processed_events": 13432, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13432, "processed_events_size": 12022944, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "85574": "2021-02-17T08:40:08Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "85574": "2021-02-17T08:40:08Z" }, "v8worker_events_lost": 0 }, "function_id": 717983186, "function_name": "sbm", "gocb_creds_request_counter": 83, "internal_vb_distribution_stats": { "worker_sbm_0": "[342-682]" }, "latency_percentile_stats": { "100": 15200, "50": 600, "80": 1000, "90": 1200, "95": 1600, "99": 2800 }, "lcb_creds_request_counter": 26952, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.156:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.161:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.92:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.156:8096": { "worker_sbm_0": "[0-341]" }, "172.23.122.161:8096": { "worker_sbm_0": "[342-682]" }, "172.23.122.92:8096": { "worker_sbm_0": "[683-1023]" } }, "worker_pids": { "worker_sbm_0": 85574 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 3, "agg_messages_sent_to_worker": 17812, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 40376, "dcp_mutation_sent_to_worker": 13432, "dcp_snapshot": 40558, "dcp_stream_req_counter": 341, "dcp_streamreq": 341, "dcp_system_event": 3410, "execution_stats": 806, "failure_stats": 806, "latency_stats": 806, "lcb_exception_stats": 806, "log_level": 1, "num_processed_events": 13432, "processed_events_size": 12025248, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 13432, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 13432, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13432, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13432, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 17808, "num_processed_events": 13432, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13432, "processed_events_size": 12025248, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "85605": "2021-02-17T08:40:08Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "85605": "2021-02-17T08:40:08Z" }, "v8worker_events_lost": 0 }, "function_id": 613640013, "function_name": "curl", "gocb_creds_request_counter": 83, "internal_vb_distribution_stats": { "worker_curl_0": "[342-682]" }, "latency_percentile_stats": { "100": 26400, "50": 1400, "80": 1700, "90": 2000, "95": 2200, "99": 3400 }, "lcb_creds_request_counter": 26952, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.156:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.161:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.92:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.156:8096": { "worker_curl_0": "[0-341]" }, "172.23.122.161:8096": { "worker_curl_0": "[342-682]" }, "172.23.122.92:8096": { "worker_curl_0": "[683-1023]" } }, "worker_pids": { "worker_curl_0": 85605 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 3, "agg_messages_sent_to_worker": 17706, "agg_queue_memory": 0, "agg_queue_memory_cap": 20971520, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 40376, "dcp_mutation_sent_to_worker": 13432, "dcp_snapshot": 40573, "dcp_stream_req_counter": 341, "dcp_streamreq": 341, "dcp_system_event": 3410, "execution_stats": 785, "failure_stats": 785, "latency_stats": 785, "lcb_exception_stats": 785, "log_level": 1, "num_processed_events": 13432, "processed_events_size": 12025248, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13432, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13432, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 17702, "num_processed_events": 13432, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13432, "processed_events_size": 12025248, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "85637": "2021-02-17T08:40:08Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "85637": "2021-02-17T08:40:08Z" }, "v8worker_events_lost": 0 }, "function_id": 22036014, "function_name": "n1ql", "gocb_creds_request_counter": 83, "internal_vb_distribution_stats": { "worker_n1ql_0": "[342-682]" }, "latency_percentile_stats": { "100": 88800, "50": 3500, "80": 4700, "90": 5600, "95": 6800, "99": 11200 }, "lcb_creds_request_counter": 26952, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.156:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.161:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.92:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.156:8096": { "worker_n1ql_0": "[0-341]" }, "172.23.122.161:8096": { "worker_n1ql_0": "[342-682]" }, "172.23.122.92:8096": { "worker_n1ql_0": "[683-1023]" } }, "worker_pids": { "worker_n1ql_0": 85637 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 2, "agg_messages_sent_to_worker": 18135, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 40376, "dcp_mutation_sent_to_worker": 13432, "dcp_snapshot": 40593, "dcp_stream_req_counter": 341, "dcp_streamreq": 341, "dcp_system_event": 3410, "execution_stats": 870, "failure_stats": 870, "latency_stats": 870, "lcb_exception_stats": 870, "log_level": 1, "num_processed_events": 13432, "processed_events_size": 12025248, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13432, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13432, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 18131, "num_processed_events": 13432, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13432, "processed_events_size": 12025248, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "85465": "2021-02-17T08:40:08Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "85465": "2021-02-17T08:40:08Z" }, "v8worker_events_lost": 0 }, "function_id": 3349188981, "function_name": "bucket_op", "gocb_creds_request_counter": 83, "internal_vb_distribution_stats": { "worker_bucket_op_0": "[342-682]" }, "latency_percentile_stats": { "100": 18600, "50": 500, "80": 800, "90": 900, "95": 1100, "99": 1800 }, "lcb_creds_request_counter": 26952, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.156:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.161:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.92:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.156:8096": { "worker_bucket_op_0": "[0-341]" }, "172.23.122.161:8096": { "worker_bucket_op_0": "[342-682]" }, "172.23.122.92:8096": { "worker_bucket_op_0": "[683-1023]" } }, "worker_pids": { "worker_bucket_op_0": 85465 } } ] [2021-02-17 00:40:08,984] - [eventing_base] [139626970871552] - INFO - Stats for Node 172.23.122.92 is [ { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 1, "agg_messages_sent_to_worker": 17663, "agg_queue_memory": 0, "agg_queue_memory_cap": 20971520, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 40255, "dcp_mutation_sent_to_worker": 13389, "dcp_snapshot": 40419, "dcp_stream_req_counter": 341, "dcp_streamreq": 341, "dcp_system_event": 3410, "execution_stats": 785, "failure_stats": 785, "latency_stats": 785, "lcb_exception_stats": 785, "log_level": 1, "num_processed_events": 13389, "processed_events_size": 11986280, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13389, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13389, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 17659, "num_processed_events": 13389, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13389, "processed_events_size": 11986280, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "58257": "2021-02-17T08:40:08Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "58257": "2021-02-17T08:40:08Z" }, "v8worker_events_lost": 0 }, "function_id": 22036014, "function_name": "n1ql", "gocb_creds_request_counter": 89, "internal_vb_distribution_stats": { "worker_n1ql_0": "[683-1023]" }, "latency_percentile_stats": { "100": 143000, "50": 3500, "80": 4600, "90": 5700, "95": 6900, "99": 11300 }, "lcb_creds_request_counter": 26865, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.156:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.161:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.92:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.156:8096": { "worker_n1ql_0": "[0-341]" }, "172.23.122.161:8096": { "worker_n1ql_0": "[342-682]" }, "172.23.122.92:8096": { "worker_n1ql_0": "[683-1023]" } }, "worker_pids": { "worker_n1ql_0": 58257 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 1, "agg_messages_sent_to_worker": 18092, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 40255, "dcp_mutation_sent_to_worker": 13389, "dcp_snapshot": 40417, "dcp_stream_req_counter": 341, "dcp_streamreq": 341, "dcp_system_event": 3410, "execution_stats": 870, "failure_stats": 870, "latency_stats": 870, "lcb_exception_stats": 870, "log_level": 1, "num_processed_events": 13389, "processed_events_size": 11986280, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13389, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13389, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 18088, "num_processed_events": 13389, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13389, "processed_events_size": 11986280, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "58055": "2021-02-17T08:40:08Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "58055": "2021-02-17T08:40:08Z" }, "v8worker_events_lost": 0 }, "function_id": 3349188981, "function_name": "bucket_op", "gocb_creds_request_counter": 89, "internal_vb_distribution_stats": { "worker_bucket_op_0": "[683-1023]" }, "latency_percentile_stats": { "100": 30800, "50": 600, "80": 800, "90": 1000, "95": 1200, "99": 1900 }, "lcb_creds_request_counter": 26865, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.156:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.161:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.92:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.156:8096": { "worker_bucket_op_0": "[0-341]" }, "172.23.122.161:8096": { "worker_bucket_op_0": "[342-682]" }, "172.23.122.92:8096": { "worker_bucket_op_0": "[683-1023]" } }, "worker_pids": { "worker_bucket_op_0": 58055 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 2, "agg_messages_sent_to_worker": 17986, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 40255, "dcp_mutation_sent_to_worker": 13389, "dcp_snapshot": 40435, "dcp_stream_req_counter": 341, "dcp_streamreq": 341, "dcp_system_event": 3410, "execution_stats": 849, "failure_stats": 849, "latency_stats": 849, "lcb_exception_stats": 849, "log_level": 1, "num_processed_events": 13389, "processed_events_size": 11986280, "thr_count": 1, "thr_map": 1, "timer_events": 13234, "timer_responses_received": 13389, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13389, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13389, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 17982, "num_processed_events": 13389, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13389, "processed_events_size": 11986280, "timer_callback_failure": 0, "timer_callback_success": 13234, "timer_cancel_counter": 0, "timer_create_counter": 13389, "timer_create_failure": 0, "timer_msg_counter": 13234, "timer_responses_sent": 0, "timestamp": { "58130": "2021-02-17T08:40:08Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "58130": "2021-02-17T08:40:08Z" }, "v8worker_events_lost": 0 }, "function_id": 1236009705, "function_name": "timers", "gocb_creds_request_counter": 89, "internal_vb_distribution_stats": { "worker_timers_0": "[683-1023]" }, "latency_percentile_stats": { "100": 22400, "50": 1500, "80": 1900, "90": 2200, "95": 2600, "99": 3900 }, "lcb_creds_request_counter": 26865, "lcb_exception_stats": {}, "metastore_stats": { "metastore_delete_err": 0, "metastore_deletes": 0, "metastore_not_found": 0, "metastore_scan": 0, "metastore_scan_due": 0, "metastore_scan_err": 0, "metastore_set": 0, "metastore_set_err": 0 }, "planner_stats": [ { "host_name": "172.23.122.156:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.161:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.92:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.156:8096": { "worker_timers_0": "[0-341]" }, "172.23.122.161:8096": { "worker_timers_0": "[342-682]" }, "172.23.122.92:8096": { "worker_timers_0": "[683-1023]" } }, "worker_pids": { "worker_timers_0": 58130 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 1, "agg_messages_sent_to_worker": 17875, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 40255, "dcp_mutation_sent_to_worker": 13389, "dcp_mutation_suppressed_counter": 13477, "dcp_snapshot": 40422, "dcp_stream_req_counter": 341, "dcp_streamreq": 341, "dcp_system_event": 3410, "execution_stats": 827, "failure_stats": 827, "latency_stats": 827, "lcb_exception_stats": 827, "log_level": 1, "num_processed_events": 13389, "processed_events_size": 11990148, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13389, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13389, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 17871, "num_processed_events": 13389, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13389, "processed_events_size": 11990148, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "58163": "2021-02-17T08:40:08Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "58163": "2021-02-17T08:40:08Z" }, "v8worker_events_lost": 0 }, "function_id": 717983186, "function_name": "sbm", "gocb_creds_request_counter": 89, "internal_vb_distribution_stats": { "worker_sbm_0": "[683-1023]" }, "latency_percentile_stats": { "100": 16800, "50": 700, "80": 1000, "90": 1200, "95": 1500, "99": 2600 }, "lcb_creds_request_counter": 26865, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.156:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.161:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.92:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.156:8096": { "worker_sbm_0": "[0-341]" }, "172.23.122.161:8096": { "worker_sbm_0": "[342-682]" }, "172.23.122.92:8096": { "worker_sbm_0": "[683-1023]" } }, "worker_pids": { "worker_sbm_0": 58163 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 1, "agg_messages_sent_to_worker": 17769, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 40255, "dcp_mutation_sent_to_worker": 13389, "dcp_snapshot": 40418, "dcp_stream_req_counter": 341, "dcp_streamreq": 341, "dcp_system_event": 3410, "execution_stats": 806, "failure_stats": 806, "latency_stats": 806, "lcb_exception_stats": 806, "log_level": 1, "num_processed_events": 13389, "processed_events_size": 11986280, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 13389, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 13389, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13389, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13389, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 17765, "num_processed_events": 13389, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13389, "processed_events_size": 11986280, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "58195": "2021-02-17T08:40:08Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "58195": "2021-02-17T08:40:08Z" }, "v8worker_events_lost": 0 }, "function_id": 613640013, "function_name": "curl", "gocb_creds_request_counter": 89, "internal_vb_distribution_stats": { "worker_curl_0": "[683-1023]" }, "latency_percentile_stats": { "100": 29300, "50": 1400, "80": 1700, "90": 2000, "95": 2300, "99": 3500 }, "lcb_creds_request_counter": 26865, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.156:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.161:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.92:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.156:8096": { "worker_curl_0": "[0-341]" }, "172.23.122.161:8096": { "worker_curl_0": "[342-682]" }, "172.23.122.92:8096": { "worker_curl_0": "[683-1023]" } }, "worker_pids": { "worker_curl_0": 58195 } } ] [2021-02-17 00:40:09,001] - [basetestcase] [139626970871552] - INFO - list of eventing nodes in cluster: [ip:172.23.122.156 port:8091 ssh_username:root, ip:172.23.122.161 port:8091 ssh_username:root, ip:172.23.122.92 port:8091 ssh_username:root] [2021-02-17 00:40:09,003] - [remote_util] [139626970871552] - INFO - SSH Connecting to 172.23.122.156 with username:root, attempt#1 of 5 [2021-02-17 00:40:09,109] - [remote_util] [139626970871552] - INFO - SSH Connected to 172.23.122.156 as root [2021-02-17 00:40:09,399] - [remote_util] [139626970871552] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-17 00:40:09,702] - [remote_util] [139626970871552] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-17 00:40:09,712] - [rest_client] [139626970871552] - INFO - /diag/eval status on 172.23.122.156:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). [2021-02-17 00:40:09,712] - [remote_util] [139626970871552] - INFO - running command.raw on 172.23.122.156: zgrep "panic" "/opt/couchbase/var/lib/couchbase/logs"/eventing.log* | wc -l [2021-02-17 00:40:10,360] - [remote_util] [139626970871552] - INFO - command executed successfully with root [2021-02-17 00:40:10,361] - [remote_util] [139626970871552] - INFO - running command.raw on 172.23.122.156: ls "/opt/couchbase/var/lib/couchbase/logs"/../crash/| wc -l [2021-02-17 00:40:10,376] - [remote_util] [139626970871552] - INFO - command executed successfully with root [2021-02-17 00:40:10,378] - [remote_util] [139626970871552] - INFO - SSH Connecting to 172.23.122.161 with username:root, attempt#1 of 5 [2021-02-17 00:40:10,481] - [remote_util] [139626970871552] - INFO - SSH Connected to 172.23.122.161 as root [2021-02-17 00:40:10,755] - [remote_util] [139626970871552] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-17 00:40:11,050] - [remote_util] [139626970871552] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-17 00:40:11,059] - [rest_client] [139626970871552] - INFO - /diag/eval status on 172.23.122.161:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). [2021-02-17 00:40:11,059] - [remote_util] [139626970871552] - INFO - running command.raw on 172.23.122.161: zgrep "panic" "/opt/couchbase/var/lib/couchbase/logs"/eventing.log* | wc -l [2021-02-17 00:40:11,149] - [remote_util] [139626970871552] - INFO - command executed successfully with root [2021-02-17 00:40:11,149] - [remote_util] [139626970871552] - INFO - running command.raw on 172.23.122.161: ls "/opt/couchbase/var/lib/couchbase/logs"/../crash/| wc -l [2021-02-17 00:40:11,163] - [remote_util] [139626970871552] - INFO - command executed successfully with root [2021-02-17 00:40:11,165] - [remote_util] [139626970871552] - INFO - SSH Connecting to 172.23.122.92 with username:root, attempt#1 of 5 [2021-02-17 00:40:11,269] - [remote_util] [139626970871552] - INFO - SSH Connected to 172.23.122.92 as root [2021-02-17 00:40:11,549] - [remote_util] [139626970871552] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-17 00:40:11,856] - [remote_util] [139626970871552] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-17 00:40:11,866] - [rest_client] [139626970871552] - INFO - /diag/eval status on 172.23.122.92:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). [2021-02-17 00:40:11,866] - [remote_util] [139626970871552] - INFO - running command.raw on 172.23.122.92: zgrep "panic" "/opt/couchbase/var/lib/couchbase/logs"/eventing.log* | wc -l [2021-02-17 00:40:12,113] - [remote_util] [139626970871552] - INFO - command executed successfully with root [2021-02-17 00:40:12,114] - [remote_util] [139626970871552] - INFO - running command.raw on 172.23.122.92: ls "/opt/couchbase/var/lib/couchbase/logs"/../crash/| wc -l [2021-02-17 00:40:12,129] - [remote_util] [139626970871552] - INFO - command executed successfully with root [2021-02-17 00:40:12,341] - [basetestcase] [139626970871552] - INFO - ------- Cluster statistics ------- [2021-02-17 00:40:12,341] - [basetestcase] [139626970871552] - INFO - 172.23.122.156:8091 => {'services': ['eventing'], 'cpu_utilization': 8.326908249807248, 'mem_free': 3072221184, 'mem_total': 4201684992, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-02-17 00:40:12,342] - [basetestcase] [139626970871552] - INFO - 172.23.122.159:8091 => {'services': ['kv'], 'cpu_utilization': 7.963917525773196, 'mem_free': 3010912256, 'mem_total': 4201684992, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-02-17 00:40:12,342] - [basetestcase] [139626970871552] - INFO - 172.23.122.160:8091 => {'services': ['index', 'n1ql'], 'cpu_utilization': 1.498602997205994, 'mem_free': 3276382208, 'mem_total': 4201684992, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-02-17 00:40:12,342] - [basetestcase] [139626970871552] - INFO - 172.23.122.161:8091 => {'services': ['eventing'], 'cpu_utilization': 9.997403271877435, 'mem_free': 3117604864, 'mem_total': 4201684992, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-02-17 00:40:12,342] - [basetestcase] [139626970871552] - INFO - 172.23.122.91:8091 => {'services': ['kv'], 'cpu_utilization': 4.985717995325889, 'mem_free': 2965188608, 'mem_total': 4201684992, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-02-17 00:40:12,342] - [basetestcase] [139626970871552] - INFO - 172.23.122.92:8091 => {'services': ['eventing'], 'cpu_utilization': 7.879101007491604, 'mem_free': 3234832384, 'mem_total': 4201684992, 'swap_mem_used': 96731136, 'swap_mem_total': 3758092288} [2021-02-17 00:40:12,342] - [basetestcase] [139626970871552] - INFO - --- End of cluster statistics --- [2021-02-17 00:40:12,344] - [remote_util] [139626970871552] - INFO - SSH Connecting to 172.23.122.91 with username:root, attempt#1 of 5 [2021-02-17 00:40:12,446] - [remote_util] [139626970871552] - INFO - SSH Connected to 172.23.122.91 as root [2021-02-17 00:40:12,735] - [remote_util] [139626970871552] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-17 00:40:13,035] - [remote_util] [139626970871552] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-17 00:40:13,038] - [remote_util] [139626970871552] - INFO - SSH Connecting to 172.23.122.158 with username:root, attempt#1 of 5 [2021-02-17 00:40:13,142] - [remote_util] [139626970871552] - INFO - SSH Connected to 172.23.122.158 as root [2021-02-17 00:40:13,398] - [remote_util] [139626970871552] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-17 00:40:13,697] - [remote_util] [139626970871552] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-17 00:40:13,700] - [remote_util] [139626970871552] - INFO - SSH Connecting to 172.23.122.159 with username:root, attempt#1 of 5 [2021-02-17 00:40:13,806] - [remote_util] [139626970871552] - INFO - SSH Connected to 172.23.122.159 as root [2021-02-17 00:40:14,092] - [remote_util] [139626970871552] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-17 00:40:14,392] - [remote_util] [139626970871552] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-17 00:40:14,394] - [remote_util] [139626970871552] - INFO - SSH Connecting to 172.23.122.92 with username:root, attempt#1 of 5 [2021-02-17 00:40:14,495] - [remote_util] [139626970871552] - INFO - SSH Connected to 172.23.122.92 as root [2021-02-17 00:40:14,762] - [remote_util] [139626970871552] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-17 00:40:15,067] - [remote_util] [139626970871552] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-17 00:40:15,070] - [remote_util] [139626970871552] - INFO - SSH Connecting to 172.23.122.156 with username:root, attempt#1 of 5 [2021-02-17 00:40:15,172] - [remote_util] [139626970871552] - INFO - SSH Connected to 172.23.122.156 as root [2021-02-17 00:40:15,465] - [remote_util] [139626970871552] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-17 00:40:15,768] - [remote_util] [139626970871552] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-17 00:40:15,770] - [remote_util] [139626970871552] - INFO - SSH Connecting to 172.23.122.161 with username:root, attempt#1 of 5 [2021-02-17 00:40:15,873] - [remote_util] [139626970871552] - INFO - SSH Connected to 172.23.122.161 as root [2021-02-17 00:40:16,165] - [remote_util] [139626970871552] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-17 00:40:16,465] - [remote_util] [139626970871552] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-17 00:40:16,467] - [remote_util] [139626970871552] - INFO - SSH Connecting to 172.23.122.160 with username:root, attempt#1 of 5 [2021-02-17 00:40:16,573] - [remote_util] [139626970871552] - INFO - SSH Connected to 172.23.122.160 as root [2021-02-17 00:40:16,865] - [remote_util] [139626970871552] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-17 00:40:17,171] - [remote_util] [139626970871552] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-17 00:40:17,174] - [remote_util] [139626970871552] - INFO - SSH Connecting to 172.23.122.93 with username:root, attempt#1 of 5 [2021-02-17 00:40:17,279] - [remote_util] [139626970871552] - INFO - SSH Connected to 172.23.122.93 as root [2021-02-17 00:40:17,534] - [remote_util] [139626970871552] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-17 00:40:17,839] - [remote_util] [139626970871552] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-17 00:40:49,650] - [basetestcase] [139626970871552] - INFO - ============== basetestcase cleanup was started for test #5 test_kv_rebalance_out_when_existing_eventing_node_is_processing_mutations ============== [2021-02-17 00:40:49,711] - [bucket_helper] [139626970871552] - INFO - deleting existing buckets ['dst_bucket', 'metadata', 'src_bucket'] on 172.23.122.91 [2021-02-17 00:40:50,470] - [bucket_helper] [139626970871552] - INFO - waiting for bucket deletion to complete.... [2021-02-17 00:40:50,493] - [rest_client] [139626970871552] - INFO - node 172.23.122.91 existing buckets : ['metadata', 'src_bucket'] [2021-02-17 00:40:50,494] - [bucket_helper] [139626970871552] - INFO - deleted bucket : dst_bucket from 172.23.122.91 [2021-02-17 00:40:50,927] - [bucket_helper] [139626970871552] - INFO - waiting for bucket deletion to complete.... [2021-02-17 00:40:50,939] - [rest_client] [139626970871552] - INFO - node 172.23.122.91 existing buckets : ['src_bucket'] [2021-02-17 00:40:50,940] - [bucket_helper] [139626970871552] - INFO - deleted bucket : metadata from 172.23.122.91 [2021-02-17 00:40:51,715] - [bucket_helper] [139626970871552] - INFO - waiting for bucket deletion to complete.... [2021-02-17 00:40:51,718] - [rest_client] [139626970871552] - INFO - node 172.23.122.91 existing buckets : [] [2021-02-17 00:40:51,718] - [bucket_helper] [139626970871552] - INFO - deleted bucket : src_bucket from 172.23.122.91 [2021-02-17 00:40:51,726] - [bucket_helper] [139626970871552] - INFO - Could not find any buckets for node 172.23.122.158, nothing to delete [2021-02-17 00:40:51,733] - [bucket_helper] [139626970871552] - INFO - Could not find any buckets for node 172.23.122.159, nothing to delete [2021-02-17 00:40:51,740] - [bucket_helper] [139626970871552] - INFO - Could not find any buckets for node 172.23.122.92, nothing to delete [2021-02-17 00:40:51,747] - [bucket_helper] [139626970871552] - INFO - Could not find any buckets for node 172.23.122.156, nothing to delete [2021-02-17 00:40:51,757] - [bucket_helper] [139626970871552] - INFO - Could not find any buckets for node 172.23.122.161, nothing to delete [2021-02-17 00:40:51,764] - [bucket_helper] [139626970871552] - INFO - Could not find any buckets for node 172.23.122.160, nothing to delete [2021-02-17 00:40:51,769] - [bucket_helper] [139626970871552] - INFO - Could not find any buckets for node 172.23.122.93, nothing to delete [2021-02-17 00:40:51,775] - [rest_client] [139626970871552] - INFO - -->is_ns_server_running? [2021-02-17 00:40:51,787] - [cluster_helper] [139626970871552] - INFO - rebalancing all nodes in order to remove nodes [2021-02-17 00:40:51,790] - [rest_client] [139626970871552] - INFO - rebalance params : {'knownNodes': 'ns_1@172.23.122.156,ns_1@172.23.122.159,ns_1@172.23.122.160,ns_1@172.23.122.161,ns_1@172.23.122.91,ns_1@172.23.122.92', 'ejectedNodes': 'ns_1@172.23.122.156,ns_1@172.23.122.159,ns_1@172.23.122.160,ns_1@172.23.122.161,ns_1@172.23.122.92', 'user': 'Administrator', 'password': 'password'} [2021-02-17 00:40:51,816] - [rest_client] [139626970871552] - INFO - rebalance operation started [2021-02-17 00:40:51,819] - [rest_client] [139626970871552] - INFO - rebalance percentage : 0.00 % [2021-02-17 00:41:01,833] - [rest_client] [139626970871552] - INFO - rebalance percentage : 41.00 % [2021-02-17 00:41:11,846] - [rest_client] [139626970871552] - INFO - rebalance percentage : 91.00 % [2021-02-17 00:41:31,861] - [rest_client] [139626970871552] - INFO - rebalance progress took 40.04 seconds [2021-02-17 00:41:31,862] - [rest_client] [139626970871552] - INFO - sleep for 10 seconds after rebalance... [2021-02-17 00:41:41,896] - [cluster_helper] [139626970871552] - INFO - removed all the nodes from cluster associated with ip:172.23.122.91 port:8091 ssh_username:root ? [('ns_1@172.23.122.156', 8091), ('ns_1@172.23.122.159', 8091), ('ns_1@172.23.122.160', 8091), ('ns_1@172.23.122.161', 8091), ('ns_1@172.23.122.92', 8091)] [2021-02-17 00:41:41,902] - [cluster_helper] [139626970871552] - INFO - waiting for ns_server @ 172.23.122.91:8091 [2021-02-17 00:41:41,902] - [rest_client] [139626970871552] - INFO - -->is_ns_server_running? [2021-02-17 00:41:41,906] - [cluster_helper] [139626970871552] - INFO - ns_server @ 172.23.122.91:8091 is running [2021-02-17 00:41:41,909] - [cluster_helper] [139626970871552] - INFO - waiting for ns_server @ 172.23.122.158:8091 [2021-02-17 00:41:41,909] - [rest_client] [139626970871552] - INFO - -->is_ns_server_running? [2021-02-17 00:41:41,912] - [cluster_helper] [139626970871552] - INFO - ns_server @ 172.23.122.158:8091 is running [2021-02-17 00:41:41,915] - [cluster_helper] [139626970871552] - INFO - waiting for ns_server @ 172.23.122.159:8091 [2021-02-17 00:41:41,915] - [rest_client] [139626970871552] - INFO - -->is_ns_server_running? [2021-02-17 00:41:41,918] - [cluster_helper] [139626970871552] - INFO - ns_server @ 172.23.122.159:8091 is running [2021-02-17 00:41:41,921] - [cluster_helper] [139626970871552] - INFO - waiting for ns_server @ 172.23.122.92:8091 [2021-02-17 00:41:41,921] - [rest_client] [139626970871552] - INFO - -->is_ns_server_running? [2021-02-17 00:41:41,924] - [cluster_helper] [139626970871552] - INFO - ns_server @ 172.23.122.92:8091 is running [2021-02-17 00:41:41,926] - [cluster_helper] [139626970871552] - INFO - waiting for ns_server @ 172.23.122.156:8091 [2021-02-17 00:41:41,926] - [rest_client] [139626970871552] - INFO - -->is_ns_server_running? [2021-02-17 00:41:41,929] - [cluster_helper] [139626970871552] - INFO - ns_server @ 172.23.122.156:8091 is running [2021-02-17 00:41:41,932] - [cluster_helper] [139626970871552] - INFO - waiting for ns_server @ 172.23.122.161:8091 [2021-02-17 00:41:41,932] - [rest_client] [139626970871552] - INFO - -->is_ns_server_running? [2021-02-17 00:41:41,935] - [cluster_helper] [139626970871552] - INFO - ns_server @ 172.23.122.161:8091 is running [2021-02-17 00:41:41,938] - [cluster_helper] [139626970871552] - INFO - waiting for ns_server @ 172.23.122.160:8091 [2021-02-17 00:41:41,938] - [rest_client] [139626970871552] - INFO - -->is_ns_server_running? [2021-02-17 00:41:41,941] - [cluster_helper] [139626970871552] - INFO - ns_server @ 172.23.122.160:8091 is running [2021-02-17 00:41:41,945] - [cluster_helper] [139626970871552] - INFO - waiting for ns_server @ 172.23.122.93:8091 [2021-02-17 00:41:41,945] - [rest_client] [139626970871552] - INFO - -->is_ns_server_running? [2021-02-17 00:41:41,948] - [cluster_helper] [139626970871552] - INFO - ns_server @ 172.23.122.93:8091 is running [2021-02-17 00:41:41,948] - [basetestcase] [139626970871552] - INFO - ============== basetestcase cleanup was finished for test #5 test_kv_rebalance_out_when_existing_eventing_node_is_processing_mutations ============== [2021-02-17 00:41:41,948] - [basetestcase] [139626970871552] - INFO - closing all ssh connections [2021-02-17 00:41:41,949] - [basetestcase] [139626970871552] - INFO - closing all memcached connections [2021-02-17 00:41:41,951] - [remote_util] [139626970871552] - INFO - SSH Connecting to 172.23.122.91 with username:root, attempt#1 of 5 [2021-02-17 00:41:42,055] - [remote_util] [139626970871552] - INFO - SSH Connected to 172.23.122.91 as root [2021-02-17 00:41:42,334] - [remote_util] [139626970871552] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-17 00:41:42,631] - [remote_util] [139626970871552] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-17 00:41:42,642] - [remote_util] [139626970871552] - INFO - SSH Connecting to 172.23.122.91 with username:root, attempt#1 of 5 [2021-02-17 00:41:42,740] - [remote_util] [139626970871552] - INFO - SSH Connected to 172.23.122.91 as root [2021-02-17 00:41:43,005] - [remote_util] [139626970871552] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-17 00:41:43,296] - [remote_util] [139626970871552] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-17 00:41:43,357] - [remote_util] [139626970871552] - INFO - 172.23.122.91 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-17 00:41:43,414] - [remote_util] [139626970871552] - INFO - Running systemd command on this server [2021-02-17 00:41:43,414] - [remote_util] [139626970871552] - INFO - running command.raw on 172.23.122.91: systemctl stop couchbase-server.service [2021-02-17 00:41:45,532] - [remote_util] [139626970871552] - INFO - command executed successfully with root [2021-02-17 00:41:45,532] - [basetestcase] [139626970871552] - INFO - Couchbase stopped [2021-02-17 00:41:45,533] - [basetestcase] [139626970871552] - INFO - sleep for 5 secs. ... [2021-02-17 00:41:50,536] - [remote_util] [139626970871552] - INFO - running command.raw on 172.23.122.91: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-17 00:41:50,551] - [remote_util] [139626970871552] - INFO - command executed successfully with root [2021-02-17 00:41:50,551] - [remote_util] [139626970871552] - INFO - running command.raw on 172.23.122.91: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-17 00:41:50,611] - [remote_util] [139626970871552] - INFO - command executed successfully with root [2021-02-17 00:41:50,613] - [remote_util] [139626970871552] - INFO - SSH Connecting to 172.23.122.91 with username:root, attempt#1 of 5 [2021-02-17 00:41:50,715] - [remote_util] [139626970871552] - INFO - SSH Connected to 172.23.122.91 as root [2021-02-17 00:41:50,995] - [remote_util] [139626970871552] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-17 00:41:51,293] - [remote_util] [139626970871552] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-17 00:41:51,353] - [remote_util] [139626970871552] - INFO - 172.23.122.91 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-17 00:41:51,353] - [remote_util] [139626970871552] - INFO - Checking for process beam.smp on linux [2021-02-17 00:41:51,392] - [remote_util] [139626970871552] - INFO - Starting couchbase server [2021-02-17 00:41:51,489] - [remote_util] [139626970871552] - INFO - Running systemd command on this server [2021-02-17 00:41:51,490] - [remote_util] [139626970871552] - INFO - running command.raw on 172.23.122.91: systemctl start couchbase-server.service [2021-02-17 00:41:51,513] - [remote_util] [139626970871552] - INFO - command executed successfully with root [2021-02-17 00:41:51,513] - [remote_util] [139626970871552] - INFO - 172.23.122.91:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-17 00:41:56,518] - [remote_util] [139626970871552] - INFO - running command.raw on 172.23.122.91: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-17 00:41:56,544] - [remote_util] [139626970871552] - INFO - command executed successfully with root [2021-02-17 00:41:56,544] - [remote_util] [139626970871552] - INFO - Couchbase server status: [' Process: 21807 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-17 00:41:56,545] - [remote_util] [139626970871552] - INFO - Checking for process beam.smp on linux [2021-02-17 00:41:56,613] - [remote_util] [139626970871552] - INFO - process beam.smp is running on 172.23.122.91: with pid 21951 [2021-02-17 00:41:56,613] - [basetestcase] [139626970871552] - INFO - Couchbase started [2021-02-17 00:41:56,615] - [remote_util] [139626970871552] - INFO - SSH Connecting to 172.23.122.158 with username:root, attempt#1 of 5 [2021-02-17 00:41:56,716] - [remote_util] [139626970871552] - INFO - SSH Connected to 172.23.122.158 as root [2021-02-17 00:41:56,966] - [remote_util] [139626970871552] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-17 00:41:57,259] - [remote_util] [139626970871552] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-17 00:41:57,268] - [remote_util] [139626970871552] - INFO - SSH Connecting to 172.23.122.158 with username:root, attempt#1 of 5 [2021-02-17 00:41:57,365] - [remote_util] [139626970871552] - INFO - SSH Connected to 172.23.122.158 as root [2021-02-17 00:41:57,609] - [remote_util] [139626970871552] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-17 00:41:57,900] - [remote_util] [139626970871552] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-17 00:41:57,920] - [remote_util] [139626970871552] - INFO - 172.23.122.158 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-17 00:41:57,976] - [remote_util] [139626970871552] - INFO - Running systemd command on this server [2021-02-17 00:41:57,976] - [remote_util] [139626970871552] - INFO - running command.raw on 172.23.122.158: systemctl stop couchbase-server.service [2021-02-17 00:41:59,724] - [remote_util] [139626970871552] - INFO - command executed successfully with root [2021-02-17 00:41:59,724] - [basetestcase] [139626970871552] - INFO - Couchbase stopped [2021-02-17 00:41:59,725] - [basetestcase] [139626970871552] - INFO - sleep for 5 secs. ... [2021-02-17 00:42:04,730] - [remote_util] [139626970871552] - INFO - running command.raw on 172.23.122.158: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-17 00:42:04,750] - [remote_util] [139626970871552] - INFO - command executed successfully with root [2021-02-17 00:42:04,750] - [remote_util] [139626970871552] - INFO - running command.raw on 172.23.122.158: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-17 00:42:04,803] - [remote_util] [139626970871552] - INFO - command executed successfully with root [2021-02-17 00:42:04,805] - [remote_util] [139626970871552] - INFO - SSH Connecting to 172.23.122.158 with username:root, attempt#1 of 5 [2021-02-17 00:42:04,904] - [remote_util] [139626970871552] - INFO - SSH Connected to 172.23.122.158 as root [2021-02-17 00:42:05,154] - [remote_util] [139626970871552] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-17 00:42:05,452] - [remote_util] [139626970871552] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-17 00:42:05,512] - [remote_util] [139626970871552] - INFO - 172.23.122.158 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-17 00:42:05,512] - [remote_util] [139626970871552] - INFO - Checking for process beam.smp on linux [2021-02-17 00:42:05,552] - [remote_util] [139626970871552] - INFO - Starting couchbase server [2021-02-17 00:42:05,652] - [remote_util] [139626970871552] - INFO - Running systemd command on this server [2021-02-17 00:42:05,652] - [remote_util] [139626970871552] - INFO - running command.raw on 172.23.122.158: systemctl start couchbase-server.service [2021-02-17 00:42:05,674] - [remote_util] [139626970871552] - INFO - command executed successfully with root [2021-02-17 00:42:05,674] - [remote_util] [139626970871552] - INFO - 172.23.122.158:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-17 00:42:10,677] - [remote_util] [139626970871552] - INFO - running command.raw on 172.23.122.158: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-17 00:42:10,704] - [remote_util] [139626970871552] - INFO - command executed successfully with root [2021-02-17 00:42:10,704] - [remote_util] [139626970871552] - INFO - Couchbase server status: [' Process: 48278 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-17 00:42:10,704] - [remote_util] [139626970871552] - INFO - Checking for process beam.smp on linux [2021-02-17 00:42:10,771] - [remote_util] [139626970871552] - INFO - process beam.smp is running on 172.23.122.158: with pid 48423 [2021-02-17 00:42:10,772] - [basetestcase] [139626970871552] - INFO - Couchbase started [2021-02-17 00:42:10,774] - [remote_util] [139626970871552] - INFO - SSH Connecting to 172.23.122.159 with username:root, attempt#1 of 5 [2021-02-17 00:42:10,873] - [remote_util] [139626970871552] - INFO - SSH Connected to 172.23.122.159 as root [2021-02-17 00:42:11,157] - [remote_util] [139626970871552] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-17 00:42:11,451] - [remote_util] [139626970871552] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-17 00:42:11,459] - [remote_util] [139626970871552] - INFO - SSH Connecting to 172.23.122.159 with username:root, attempt#1 of 5 [2021-02-17 00:42:11,555] - [remote_util] [139626970871552] - INFO - SSH Connected to 172.23.122.159 as root [2021-02-17 00:42:11,833] - [remote_util] [139626970871552] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-17 00:42:12,129] - [remote_util] [139626970871552] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-17 00:42:12,188] - [remote_util] [139626970871552] - INFO - 172.23.122.159 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-17 00:42:12,244] - [remote_util] [139626970871552] - INFO - Running systemd command on this server [2021-02-17 00:42:12,245] - [remote_util] [139626970871552] - INFO - running command.raw on 172.23.122.159: systemctl stop couchbase-server.service [2021-02-17 00:42:14,001] - [remote_util] [139626970871552] - INFO - command executed successfully with root [2021-02-17 00:42:14,001] - [basetestcase] [139626970871552] - INFO - Couchbase stopped [2021-02-17 00:42:14,002] - [basetestcase] [139626970871552] - INFO - sleep for 5 secs. ... [2021-02-17 00:42:19,007] - [remote_util] [139626970871552] - INFO - running command.raw on 172.23.122.159: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-17 00:42:19,021] - [remote_util] [139626970871552] - INFO - command executed successfully with root [2021-02-17 00:42:19,022] - [remote_util] [139626970871552] - INFO - running command.raw on 172.23.122.159: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-17 00:42:19,075] - [remote_util] [139626970871552] - INFO - command executed successfully with root [2021-02-17 00:42:19,077] - [remote_util] [139626970871552] - INFO - SSH Connecting to 172.23.122.159 with username:root, attempt#1 of 5 [2021-02-17 00:42:19,176] - [remote_util] [139626970871552] - INFO - SSH Connected to 172.23.122.159 as root [2021-02-17 00:42:19,450] - [remote_util] [139626970871552] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-17 00:42:19,745] - [remote_util] [139626970871552] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-17 00:42:19,765] - [remote_util] [139626970871552] - INFO - 172.23.122.159 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-17 00:42:19,765] - [remote_util] [139626970871552] - INFO - Checking for process beam.smp on linux [2021-02-17 00:42:19,800] - [remote_util] [139626970871552] - INFO - Starting couchbase server [2021-02-17 00:42:19,894] - [remote_util] [139626970871552] - INFO - Running systemd command on this server [2021-02-17 00:42:19,894] - [remote_util] [139626970871552] - INFO - running command.raw on 172.23.122.159: systemctl start couchbase-server.service [2021-02-17 00:42:19,916] - [remote_util] [139626970871552] - INFO - command executed successfully with root [2021-02-17 00:42:19,916] - [remote_util] [139626970871552] - INFO - 172.23.122.159:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-17 00:42:24,919] - [remote_util] [139626970871552] - INFO - running command.raw on 172.23.122.159: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-17 00:42:24,942] - [remote_util] [139626970871552] - INFO - command executed successfully with root [2021-02-17 00:42:24,942] - [remote_util] [139626970871552] - INFO - Couchbase server status: [' Process: 55221 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-17 00:42:24,942] - [remote_util] [139626970871552] - INFO - Checking for process beam.smp on linux [2021-02-17 00:42:25,009] - [remote_util] [139626970871552] - INFO - process beam.smp is running on 172.23.122.159: with pid 55365 [2021-02-17 00:42:25,009] - [basetestcase] [139626970871552] - INFO - Couchbase started [2021-02-17 00:42:25,011] - [remote_util] [139626970871552] - INFO - SSH Connecting to 172.23.122.92 with username:root, attempt#1 of 5 [2021-02-17 00:42:25,111] - [remote_util] [139626970871552] - INFO - SSH Connected to 172.23.122.92 as root [2021-02-17 00:42:25,375] - [remote_util] [139626970871552] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-17 00:42:25,676] - [remote_util] [139626970871552] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-17 00:42:25,686] - [remote_util] [139626970871552] - INFO - SSH Connecting to 172.23.122.92 with username:root, attempt#1 of 5 [2021-02-17 00:42:25,781] - [remote_util] [139626970871552] - INFO - SSH Connected to 172.23.122.92 as root [2021-02-17 00:42:26,039] - [remote_util] [139626970871552] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-17 00:42:26,336] - [remote_util] [139626970871552] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-17 00:42:26,396] - [remote_util] [139626970871552] - INFO - 172.23.122.92 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-17 00:42:26,454] - [remote_util] [139626970871552] - INFO - Running systemd command on this server [2021-02-17 00:42:26,454] - [remote_util] [139626970871552] - INFO - running command.raw on 172.23.122.92: systemctl stop couchbase-server.service [2021-02-17 00:42:28,128] - [remote_util] [139626970871552] - INFO - command executed successfully with root [2021-02-17 00:42:28,129] - [basetestcase] [139626970871552] - INFO - Couchbase stopped [2021-02-17 00:42:28,129] - [basetestcase] [139626970871552] - INFO - sleep for 5 secs. ... [2021-02-17 00:42:33,131] - [remote_util] [139626970871552] - INFO - running command.raw on 172.23.122.92: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-17 00:42:33,151] - [remote_util] [139626970871552] - INFO - command executed successfully with root [2021-02-17 00:42:33,151] - [remote_util] [139626970871552] - INFO - running command.raw on 172.23.122.92: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-17 00:42:33,205] - [remote_util] [139626970871552] - INFO - command executed successfully with root [2021-02-17 00:42:33,207] - [remote_util] [139626970871552] - INFO - SSH Connecting to 172.23.122.92 with username:root, attempt#1 of 5 [2021-02-17 00:42:33,308] - [remote_util] [139626970871552] - INFO - SSH Connected to 172.23.122.92 as root [2021-02-17 00:42:33,572] - [remote_util] [139626970871552] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-17 00:42:33,869] - [remote_util] [139626970871552] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-17 00:42:33,930] - [remote_util] [139626970871552] - INFO - 172.23.122.92 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-17 00:42:33,930] - [remote_util] [139626970871552] - INFO - Checking for process beam.smp on linux [2021-02-17 00:42:33,968] - [remote_util] [139626970871552] - INFO - Starting couchbase server [2021-02-17 00:42:34,066] - [remote_util] [139626970871552] - INFO - Running systemd command on this server [2021-02-17 00:42:34,066] - [remote_util] [139626970871552] - INFO - running command.raw on 172.23.122.92: systemctl start couchbase-server.service [2021-02-17 00:42:34,088] - [remote_util] [139626970871552] - INFO - command executed successfully with root [2021-02-17 00:42:34,088] - [remote_util] [139626970871552] - INFO - 172.23.122.92:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-17 00:42:39,091] - [remote_util] [139626970871552] - INFO - running command.raw on 172.23.122.92: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-17 00:42:39,118] - [remote_util] [139626970871552] - INFO - command executed successfully with root [2021-02-17 00:42:39,118] - [remote_util] [139626970871552] - INFO - Couchbase server status: [' Process: 75538 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-17 00:42:39,119] - [remote_util] [139626970871552] - INFO - Checking for process beam.smp on linux [2021-02-17 00:42:39,188] - [remote_util] [139626970871552] - INFO - process beam.smp is running on 172.23.122.92: with pid 75683 [2021-02-17 00:42:39,189] - [basetestcase] [139626970871552] - INFO - Couchbase started [2021-02-17 00:42:39,191] - [remote_util] [139626970871552] - INFO - SSH Connecting to 172.23.122.156 with username:root, attempt#1 of 5 [2021-02-17 00:42:39,295] - [remote_util] [139626970871552] - INFO - SSH Connected to 172.23.122.156 as root [2021-02-17 00:42:39,582] - [remote_util] [139626970871552] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-17 00:42:39,881] - [remote_util] [139626970871552] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-17 00:42:39,890] - [remote_util] [139626970871552] - INFO - SSH Connecting to 172.23.122.156 with username:root, attempt#1 of 5 [2021-02-17 00:42:39,989] - [remote_util] [139626970871552] - INFO - SSH Connected to 172.23.122.156 as root [2021-02-17 00:42:40,261] - [remote_util] [139626970871552] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-17 00:42:40,555] - [remote_util] [139626970871552] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-17 00:42:40,614] - [remote_util] [139626970871552] - INFO - 172.23.122.156 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-17 00:42:40,669] - [remote_util] [139626970871552] - INFO - Running systemd command on this server [2021-02-17 00:42:40,670] - [remote_util] [139626970871552] - INFO - running command.raw on 172.23.122.156: systemctl stop couchbase-server.service [2021-02-17 00:42:42,303] - [remote_util] [139626970871552] - INFO - command executed successfully with root [2021-02-17 00:42:42,304] - [basetestcase] [139626970871552] - INFO - Couchbase stopped [2021-02-17 00:42:42,304] - [basetestcase] [139626970871552] - INFO - sleep for 5 secs. ... [2021-02-17 00:42:47,308] - [remote_util] [139626970871552] - INFO - running command.raw on 172.23.122.156: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-17 00:42:47,328] - [remote_util] [139626970871552] - INFO - command executed successfully with root [2021-02-17 00:42:47,328] - [remote_util] [139626970871552] - INFO - running command.raw on 172.23.122.156: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-17 00:42:47,381] - [remote_util] [139626970871552] - INFO - command executed successfully with root [2021-02-17 00:42:47,383] - [remote_util] [139626970871552] - INFO - SSH Connecting to 172.23.122.156 with username:root, attempt#1 of 5 [2021-02-17 00:42:47,483] - [remote_util] [139626970871552] - INFO - SSH Connected to 172.23.122.156 as root [2021-02-17 00:42:47,763] - [remote_util] [139626970871552] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-17 00:42:48,058] - [remote_util] [139626970871552] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-17 00:42:48,119] - [remote_util] [139626970871552] - INFO - 172.23.122.156 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-17 00:42:48,119] - [remote_util] [139626970871552] - INFO - Checking for process beam.smp on linux [2021-02-17 00:42:48,158] - [remote_util] [139626970871552] - INFO - Starting couchbase server [2021-02-17 00:42:48,254] - [remote_util] [139626970871552] - INFO - Running systemd command on this server [2021-02-17 00:42:48,254] - [remote_util] [139626970871552] - INFO - running command.raw on 172.23.122.156: systemctl start couchbase-server.service [2021-02-17 00:42:48,276] - [remote_util] [139626970871552] - INFO - command executed successfully with root [2021-02-17 00:42:48,276] - [remote_util] [139626970871552] - INFO - 172.23.122.156:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-17 00:42:53,280] - [remote_util] [139626970871552] - INFO - running command.raw on 172.23.122.156: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-17 00:42:53,309] - [remote_util] [139626970871552] - INFO - command executed successfully with root [2021-02-17 00:42:53,310] - [remote_util] [139626970871552] - INFO - Couchbase server status: [' Process: 122134 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-17 00:42:53,310] - [remote_util] [139626970871552] - INFO - Checking for process beam.smp on linux [2021-02-17 00:42:53,380] - [remote_util] [139626970871552] - INFO - process beam.smp is running on 172.23.122.156: with pid 122279 [2021-02-17 00:42:53,380] - [basetestcase] [139626970871552] - INFO - Couchbase started [2021-02-17 00:42:53,382] - [remote_util] [139626970871552] - INFO - SSH Connecting to 172.23.122.161 with username:root, attempt#1 of 5 [2021-02-17 00:42:53,483] - [remote_util] [139626970871552] - INFO - SSH Connected to 172.23.122.161 as root [2021-02-17 00:42:53,756] - [remote_util] [139626970871552] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-17 00:42:54,054] - [remote_util] [139626970871552] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-17 00:42:54,063] - [remote_util] [139626970871552] - INFO - SSH Connecting to 172.23.122.161 with username:root, attempt#1 of 5 [2021-02-17 00:42:54,162] - [remote_util] [139626970871552] - INFO - SSH Connected to 172.23.122.161 as root [2021-02-17 00:42:54,427] - [remote_util] [139626970871552] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-17 00:42:54,721] - [remote_util] [139626970871552] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-17 00:42:54,781] - [remote_util] [139626970871552] - INFO - 172.23.122.161 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-17 00:42:54,838] - [remote_util] [139626970871552] - INFO - Running systemd command on this server [2021-02-17 00:42:54,838] - [remote_util] [139626970871552] - INFO - running command.raw on 172.23.122.161: systemctl stop couchbase-server.service [2021-02-17 00:42:56,615] - [remote_util] [139626970871552] - INFO - command executed successfully with root [2021-02-17 00:42:56,615] - [basetestcase] [139626970871552] - INFO - Couchbase stopped [2021-02-17 00:42:56,616] - [basetestcase] [139626970871552] - INFO - sleep for 5 secs. ... [2021-02-17 00:43:01,621] - [remote_util] [139626970871552] - INFO - running command.raw on 172.23.122.161: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-17 00:43:01,641] - [remote_util] [139626970871552] - INFO - command executed successfully with root [2021-02-17 00:43:01,641] - [remote_util] [139626970871552] - INFO - running command.raw on 172.23.122.161: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-17 00:43:01,694] - [remote_util] [139626970871552] - INFO - command executed successfully with root [2021-02-17 00:43:01,696] - [remote_util] [139626970871552] - INFO - SSH Connecting to 172.23.122.161 with username:root, attempt#1 of 5 [2021-02-17 00:43:01,796] - [remote_util] [139626970871552] - INFO - SSH Connected to 172.23.122.161 as root [2021-02-17 00:43:02,071] - [remote_util] [139626970871552] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-17 00:43:02,366] - [remote_util] [139626970871552] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-17 00:43:02,386] - [remote_util] [139626970871552] - INFO - 172.23.122.161 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-17 00:43:02,387] - [remote_util] [139626970871552] - INFO - Checking for process beam.smp on linux [2021-02-17 00:43:02,425] - [remote_util] [139626970871552] - INFO - Starting couchbase server [2021-02-17 00:43:02,521] - [remote_util] [139626970871552] - INFO - Running systemd command on this server [2021-02-17 00:43:02,521] - [remote_util] [139626970871552] - INFO - running command.raw on 172.23.122.161: systemctl start couchbase-server.service [2021-02-17 00:43:02,545] - [remote_util] [139626970871552] - INFO - command executed successfully with root [2021-02-17 00:43:02,545] - [remote_util] [139626970871552] - INFO - 172.23.122.161:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-17 00:43:07,550] - [remote_util] [139626970871552] - INFO - running command.raw on 172.23.122.161: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-17 00:43:07,576] - [remote_util] [139626970871552] - INFO - command executed successfully with root [2021-02-17 00:43:07,576] - [remote_util] [139626970871552] - INFO - Couchbase server status: [' Process: 102824 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-17 00:43:07,576] - [remote_util] [139626970871552] - INFO - Checking for process beam.smp on linux [2021-02-17 00:43:07,646] - [remote_util] [139626970871552] - INFO - process beam.smp is running on 172.23.122.161: with pid 102969 [2021-02-17 00:43:07,646] - [basetestcase] [139626970871552] - INFO - Couchbase started [2021-02-17 00:43:07,648] - [remote_util] [139626970871552] - INFO - SSH Connecting to 172.23.122.160 with username:root, attempt#1 of 5 [2021-02-17 00:43:07,753] - [remote_util] [139626970871552] - INFO - SSH Connected to 172.23.122.160 as root [2021-02-17 00:43:08,036] - [remote_util] [139626970871552] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-17 00:43:08,336] - [remote_util] [139626970871552] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-17 00:43:08,345] - [remote_util] [139626970871552] - INFO - SSH Connecting to 172.23.122.160 with username:root, attempt#1 of 5 [2021-02-17 00:43:08,450] - [remote_util] [139626970871552] - INFO - SSH Connected to 172.23.122.160 as root [2021-02-17 00:43:08,726] - [remote_util] [139626970871552] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-17 00:43:09,027] - [remote_util] [139626970871552] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-17 00:43:09,048] - [remote_util] [139626970871552] - INFO - 172.23.122.160 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-17 00:43:09,151] - [remote_util] [139626970871552] - INFO - Running systemd command on this server [2021-02-17 00:43:09,151] - [remote_util] [139626970871552] - INFO - running command.raw on 172.23.122.160: systemctl stop couchbase-server.service [2021-02-17 00:43:10,958] - [remote_util] [139626970871552] - INFO - command executed successfully with root [2021-02-17 00:43:10,959] - [basetestcase] [139626970871552] - INFO - Couchbase stopped [2021-02-17 00:43:10,959] - [basetestcase] [139626970871552] - INFO - sleep for 5 secs. ... [2021-02-17 00:43:15,962] - [remote_util] [139626970871552] - INFO - running command.raw on 172.23.122.160: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-17 00:43:15,980] - [remote_util] [139626970871552] - INFO - command executed successfully with root [2021-02-17 00:43:15,980] - [remote_util] [139626970871552] - INFO - running command.raw on 172.23.122.160: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-17 00:43:16,033] - [remote_util] [139626970871552] - INFO - command executed successfully with root [2021-02-17 00:43:16,035] - [remote_util] [139626970871552] - INFO - SSH Connecting to 172.23.122.160 with username:root, attempt#1 of 5 [2021-02-17 00:43:16,134] - [remote_util] [139626970871552] - INFO - SSH Connected to 172.23.122.160 as root [2021-02-17 00:43:16,411] - [remote_util] [139626970871552] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-17 00:43:16,706] - [remote_util] [139626970871552] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-17 00:43:16,728] - [remote_util] [139626970871552] - INFO - 172.23.122.160 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-17 00:43:16,728] - [remote_util] [139626970871552] - INFO - Checking for process beam.smp on linux [2021-02-17 00:43:16,765] - [remote_util] [139626970871552] - INFO - Starting couchbase server [2021-02-17 00:43:16,861] - [remote_util] [139626970871552] - INFO - Running systemd command on this server [2021-02-17 00:43:16,862] - [remote_util] [139626970871552] - INFO - running command.raw on 172.23.122.160: systemctl start couchbase-server.service [2021-02-17 00:43:16,885] - [remote_util] [139626970871552] - INFO - command executed successfully with root [2021-02-17 00:43:16,886] - [remote_util] [139626970871552] - INFO - 172.23.122.160:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-17 00:43:21,890] - [remote_util] [139626970871552] - INFO - running command.raw on 172.23.122.160: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-17 00:43:21,918] - [remote_util] [139626970871552] - INFO - command executed successfully with root [2021-02-17 00:43:21,918] - [remote_util] [139626970871552] - INFO - Couchbase server status: [' Process: 44728 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-17 00:43:21,918] - [remote_util] [139626970871552] - INFO - Checking for process beam.smp on linux [2021-02-17 00:43:21,986] - [remote_util] [139626970871552] - INFO - process beam.smp is running on 172.23.122.160: with pid 44872 [2021-02-17 00:43:21,986] - [basetestcase] [139626970871552] - INFO - Couchbase started [2021-02-17 00:43:21,988] - [remote_util] [139626970871552] - INFO - SSH Connecting to 172.23.122.93 with username:root, attempt#1 of 5 [2021-02-17 00:43:22,089] - [remote_util] [139626970871552] - INFO - SSH Connected to 172.23.122.93 as root [2021-02-17 00:43:22,339] - [remote_util] [139626970871552] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-17 00:43:22,633] - [remote_util] [139626970871552] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-17 00:43:22,643] - [remote_util] [139626970871552] - INFO - SSH Connecting to 172.23.122.93 with username:root, attempt#1 of 5 [2021-02-17 00:43:22,738] - [remote_util] [139626970871552] - INFO - SSH Connected to 172.23.122.93 as root [2021-02-17 00:43:22,980] - [remote_util] [139626970871552] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-17 00:43:23,275] - [remote_util] [139626970871552] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-17 00:43:23,339] - [remote_util] [139626970871552] - INFO - 172.23.122.93 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-17 00:43:23,402] - [remote_util] [139626970871552] - INFO - Running systemd command on this server [2021-02-17 00:43:23,402] - [remote_util] [139626970871552] - INFO - running command.raw on 172.23.122.93: systemctl stop couchbase-server.service [2021-02-17 00:43:25,071] - [remote_util] [139626970871552] - INFO - command executed successfully with root [2021-02-17 00:43:25,071] - [basetestcase] [139626970871552] - INFO - Couchbase stopped [2021-02-17 00:43:25,071] - [basetestcase] [139626970871552] - INFO - sleep for 5 secs. ... [2021-02-17 00:43:30,077] - [remote_util] [139626970871552] - INFO - running command.raw on 172.23.122.93: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-17 00:43:30,092] - [remote_util] [139626970871552] - INFO - command executed successfully with root [2021-02-17 00:43:30,092] - [remote_util] [139626970871552] - INFO - running command.raw on 172.23.122.93: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-17 00:43:30,145] - [remote_util] [139626970871552] - INFO - command executed successfully with root [2021-02-17 00:43:30,148] - [remote_util] [139626970871552] - INFO - SSH Connecting to 172.23.122.93 with username:root, attempt#1 of 5 [2021-02-17 00:43:30,244] - [remote_util] [139626970871552] - INFO - SSH Connected to 172.23.122.93 as root [2021-02-17 00:43:30,483] - [remote_util] [139626970871552] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-17 00:43:30,780] - [remote_util] [139626970871552] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-17 00:43:30,839] - [remote_util] [139626970871552] - INFO - 172.23.122.93 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-17 00:43:30,840] - [remote_util] [139626970871552] - INFO - Checking for process beam.smp on linux [2021-02-17 00:43:30,876] - [remote_util] [139626970871552] - INFO - Starting couchbase server [2021-02-17 00:43:30,974] - [remote_util] [139626970871552] - INFO - Running systemd command on this server [2021-02-17 00:43:30,974] - [remote_util] [139626970871552] - INFO - running command.raw on 172.23.122.93: systemctl start couchbase-server.service [2021-02-17 00:43:30,996] - [remote_util] [139626970871552] - INFO - command executed successfully with root [2021-02-17 00:43:30,996] - [remote_util] [139626970871552] - INFO - 172.23.122.93:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-17 00:43:36,001] - [remote_util] [139626970871552] - INFO - running command.raw on 172.23.122.93: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-17 00:43:36,029] - [remote_util] [139626970871552] - INFO - command executed successfully with root [2021-02-17 00:43:36,029] - [remote_util] [139626970871552] - INFO - Couchbase server status: [' Process: 94390 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-17 00:43:36,030] - [remote_util] [139626970871552] - INFO - Checking for process beam.smp on linux [2021-02-17 00:43:36,098] - [remote_util] [139626970871552] - INFO - process beam.smp is running on 172.23.122.93: with pid 94535 [2021-02-17 00:43:36,099] - [basetestcase] [139626970871552] - INFO - Couchbase started [2021-02-17 00:43:36,099] - [basetestcase] [139626970871552] - INFO - sleep for 10 secs. ... [2021-02-17 00:43:46,130] - [remote_util] [139626970871552] - INFO - SSH Connecting to 172.23.122.91 with username:root, attempt#1 of 5 [2021-02-17 00:43:46,133] - [remote_util] [139627316946688] - INFO - SSH Connecting to 172.23.122.159 with username:root, attempt#1 of 5 [2021-02-17 00:43:46,135] - [remote_util] [139627325339392] - INFO - SSH Connecting to 172.23.122.158 with username:root, attempt#1 of 5 [2021-02-17 00:43:46,138] - [remote_util] [139626928908032] - INFO - SSH Connecting to 172.23.122.92 with username:root, attempt#1 of 5 [2021-02-17 00:43:46,140] - [remote_util] [139626920515328] - INFO - SSH Connecting to 172.23.122.156 with username:root, attempt#1 of 5 [2021-02-17 00:43:46,142] - [remote_util] [139626425607936] - INFO - SSH Connecting to 172.23.122.161 with username:root, attempt#1 of 5 [2021-02-17 00:43:46,144] - [remote_util] [139626408822528] - INFO - SSH Connecting to 172.23.122.160 with username:root, attempt#1 of 5 [2021-02-17 00:43:46,147] - [remote_util] [139626400429824] - INFO - SSH Connecting to 172.23.122.93 with username:root, attempt#1 of 5 [2021-02-17 00:43:46,233] - [remote_util] [139627316946688] - INFO - SSH Connected to 172.23.122.159 as root [2021-02-17 00:43:46,234] - [remote_util] [139626970871552] - INFO - SSH Connected to 172.23.122.91 as root [2021-02-17 00:43:46,241] - [remote_util] [139627325339392] - INFO - SSH Connected to 172.23.122.158 as root [2021-02-17 00:43:46,247] - [remote_util] [139626928908032] - INFO - SSH Connected to 172.23.122.92 as root [2021-02-17 00:43:46,248] - [remote_util] [139626400429824] - INFO - SSH Connected to 172.23.122.93 as root [2021-02-17 00:43:46,250] - [remote_util] [139626920515328] - INFO - SSH Connected to 172.23.122.156 as root [2021-02-17 00:43:46,251] - [remote_util] [139626408822528] - INFO - SSH Connected to 172.23.122.160 as root [2021-02-17 00:43:46,252] - [remote_util] [139626425607936] - INFO - SSH Connected to 172.23.122.161 as root [2021-02-17 00:43:46,499] - [remote_util] [139627325339392] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-17 00:43:46,518] - [remote_util] [139626400429824] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-17 00:43:46,539] - [remote_util] [139626928908032] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-17 00:43:46,559] - [remote_util] [139627316946688] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-17 00:43:46,566] - [remote_util] [139626970871552] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-17 00:43:46,578] - [remote_util] [139626425607936] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-17 00:43:46,579] - [remote_util] [139626920515328] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-17 00:43:46,587] - [remote_util] [139626408822528] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-17 00:43:46,834] - [remote_util] [139626400429824] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-17 00:43:46,835] - [remote_util] [139626400429824] - INFO - running command.raw on 172.23.122.93: /opt/couchbase/bin/cbcollect_info 172.23.122.93-20210217-0043-diag.zip [2021-02-17 00:43:46,856] - [remote_util] [139627325339392] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-17 00:43:46,857] - [remote_util] [139627325339392] - INFO - running command.raw on 172.23.122.158: /opt/couchbase/bin/cbcollect_info 172.23.122.158-20210217-0043-diag.zip [2021-02-17 00:43:46,867] - [remote_util] [139626970871552] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-17 00:43:46,867] - [remote_util] [139626970871552] - INFO - running command.raw on 172.23.122.91: /opt/couchbase/bin/cbcollect_info 172.23.122.91-20210217-0043-diag.zip [2021-02-17 00:43:46,880] - [remote_util] [139626425607936] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-17 00:43:46,881] - [remote_util] [139626425607936] - INFO - running command.raw on 172.23.122.161: /opt/couchbase/bin/cbcollect_info 172.23.122.161-20210217-0043-diag.zip [2021-02-17 00:43:46,885] - [remote_util] [139626920515328] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-17 00:43:46,886] - [remote_util] [139626920515328] - INFO - running command.raw on 172.23.122.156: /opt/couchbase/bin/cbcollect_info 172.23.122.156-20210217-0043-diag.zip [2021-02-17 00:43:46,891] - [remote_util] [139626408822528] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-17 00:43:46,892] - [remote_util] [139626408822528] - INFO - running command.raw on 172.23.122.160: /opt/couchbase/bin/cbcollect_info 172.23.122.160-20210217-0043-diag.zip [2021-02-17 00:43:46,894] - [remote_util] [139626928908032] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-17 00:43:46,895] - [remote_util] [139627316946688] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-17 00:43:46,895] - [remote_util] [139626928908032] - INFO - running command.raw on 172.23.122.92: /opt/couchbase/bin/cbcollect_info 172.23.122.92-20210217-0043-diag.zip [2021-02-17 00:43:46,898] - [remote_util] [139627316946688] - INFO - running command.raw on 172.23.122.159: /opt/couchbase/bin/cbcollect_info 172.23.122.159-20210217-0043-diag.zip [2021-02-17 00:45:19,808] - [remote_util] [139627316946688] - INFO - command executed successfully with root [2021-02-17 00:45:20,308] - [remote_util] [139627325339392] - INFO - command executed successfully with root [2021-02-17 00:45:20,342] - [remote_util] [139627316946688] - INFO - running command.raw on 172.23.122.159: rm -f /root/172.23.122.159-20210217-0043-diag.zip [2021-02-17 00:45:20,361] - [remote_util] [139627316946688] - INFO - command executed successfully with root [2021-02-17 00:45:20,901] - [remote_util] [139627325339392] - INFO - running command.raw on 172.23.122.158: rm -f /root/172.23.122.158-20210217-0043-diag.zip [2021-02-17 00:45:20,920] - [remote_util] [139627325339392] - INFO - command executed successfully with root [2021-02-17 00:45:26,400] - [remote_util] [139626920515328] - INFO - command executed successfully with root [2021-02-17 00:45:26,737] - [remote_util] [139626400429824] - INFO - command executed successfully with root [2021-02-17 00:45:26,853] - [remote_util] [139626425607936] - INFO - command executed successfully with root [2021-02-17 00:45:27,554] - [remote_util] [139626920515328] - INFO - running command.raw on 172.23.122.156: rm -f /root/172.23.122.156-20210217-0043-diag.zip [2021-02-17 00:45:27,616] - [remote_util] [139626920515328] - INFO - command executed successfully with root [2021-02-17 00:45:27,872] - [remote_util] [139626400429824] - INFO - running command.raw on 172.23.122.93: rm -f /root/172.23.122.93-20210217-0043-diag.zip [2021-02-17 00:45:27,929] - [remote_util] [139626400429824] - INFO - command executed successfully with root [2021-02-17 00:45:27,973] - [remote_util] [139626425607936] - INFO - running command.raw on 172.23.122.161: rm -f /root/172.23.122.161-20210217-0043-diag.zip [2021-02-17 00:45:27,993] - [remote_util] [139626425607936] - INFO - command executed successfully with root [2021-02-17 00:45:28,801] - [remote_util] [139626970871552] - INFO - command executed successfully with root [2021-02-17 00:45:29,567] - [remote_util] [139626970871552] - INFO - running command.raw on 172.23.122.91: rm -f /root/172.23.122.91-20210217-0043-diag.zip [2021-02-17 00:45:29,589] - [remote_util] [139626970871552] - INFO - command executed successfully with root [2021-02-17 00:45:29,819] - [remote_util] [139626928908032] - INFO - command executed successfully with root [2021-02-17 00:45:30,347] - [remote_util] [139626928908032] - INFO - running command.raw on 172.23.122.92: rm -f /root/172.23.122.92-20210217-0043-diag.zip [2021-02-17 00:45:30,366] - [remote_util] [139626928908032] - INFO - command executed successfully with root [2021-02-17 00:45:41,458] - [remote_util] [139626408822528] - INFO - command executed successfully with root [2021-02-17 00:45:41,881] - [remote_util] [139626408822528] - INFO - running command.raw on 172.23.122.160: rm -f /root/172.23.122.160-20210217-0043-diag.zip [2021-02-17 00:45:41,899] - [remote_util] [139626408822528] - INFO - command executed successfully with root