[2021-02-12 21:07:26,451] - [eventing_base] [140099333379840] - INFO - Starting Test: test_kv_rebalance_in_when_existing_eventing_node_is_processing_mutations [2021-02-12 21:07:26,453] - [remote_util] [140099333379840] - INFO - SSH Connecting to 172.23.123.61 with username:root, attempt#1 of 5 [2021-02-12 21:07:26,558] - [remote_util] [140099333379840] - INFO - SSH Connected to 172.23.123.61 as root [2021-02-12 21:07:26,827] - [remote_util] [140099333379840] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:07:27,125] - [remote_util] [140099333379840] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:07:27,130] - [rest_client] [140099333379840] - INFO - -->is_ns_server_running? [2021-02-12 21:07:27,138] - [rest_client] [140099333379840] - INFO - Node version in cluster 7.0.0-4454-enterprise [2021-02-12 21:07:27,143] - [rest_client] [140099333379840] - ERROR - GET http://172.23.123.61:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-02-12 21:07:27,143] - [rest_client] [140099333379840] - INFO - Node versions in cluster [] [2021-02-12 21:07:27,143] - [basetestcase] [140099333379840] - INFO - ============== basetestcase setup was started for test #4 test_kv_rebalance_in_when_existing_eventing_node_is_processing_mutations============== [2021-02-12 21:07:27,147] - [rest_client] [140099333379840] - ERROR - GET http://172.23.123.61:8091/pools/default/ body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-02-12 21:07:27,161] - [bucket_helper] [140099333379840] - INFO - Could not find any buckets for node 172.23.123.61, nothing to delete [2021-02-12 21:07:27,173] - [bucket_helper] [140099333379840] - INFO - Could not find any buckets for node 172.23.123.62, nothing to delete [2021-02-12 21:07:27,185] - [bucket_helper] [140099333379840] - INFO - Could not find any buckets for node 172.23.122.90, nothing to delete [2021-02-12 21:07:27,197] - [bucket_helper] [140099333379840] - INFO - Could not find any buckets for node 172.23.122.117, nothing to delete [2021-02-12 21:07:27,208] - [bucket_helper] [140099333379840] - INFO - Could not find any buckets for node 172.23.122.97, nothing to delete [2021-02-12 21:07:27,221] - [bucket_helper] [140099333379840] - INFO - Could not find any buckets for node 172.23.122.82, nothing to delete [2021-02-12 21:07:27,232] - [bucket_helper] [140099333379840] - INFO - Could not find any buckets for node 172.23.122.118, nothing to delete [2021-02-12 21:07:27,244] - [bucket_helper] [140099333379840] - INFO - Could not find any buckets for node 172.23.122.116, nothing to delete [2021-02-12 21:07:27,248] - [rest_client] [140099333379840] - INFO - -->is_ns_server_running? [2021-02-12 21:07:27,262] - [cluster_helper] [140099333379840] - INFO - waiting for ns_server @ 172.23.123.61:8091 [2021-02-12 21:07:27,262] - [rest_client] [140099333379840] - INFO - -->is_ns_server_running? [2021-02-12 21:07:27,264] - [cluster_helper] [140099333379840] - INFO - ns_server @ 172.23.123.61:8091 is running [2021-02-12 21:07:27,268] - [cluster_helper] [140099333379840] - INFO - waiting for ns_server @ 172.23.123.62:8091 [2021-02-12 21:07:27,268] - [rest_client] [140099333379840] - INFO - -->is_ns_server_running? [2021-02-12 21:07:27,271] - [cluster_helper] [140099333379840] - INFO - ns_server @ 172.23.123.62:8091 is running [2021-02-12 21:07:27,273] - [cluster_helper] [140099333379840] - INFO - waiting for ns_server @ 172.23.122.90:8091 [2021-02-12 21:07:27,273] - [rest_client] [140099333379840] - INFO - -->is_ns_server_running? [2021-02-12 21:07:27,276] - [cluster_helper] [140099333379840] - INFO - ns_server @ 172.23.122.90:8091 is running [2021-02-12 21:07:27,279] - [cluster_helper] [140099333379840] - INFO - waiting for ns_server @ 172.23.122.117:8091 [2021-02-12 21:07:27,280] - [rest_client] [140099333379840] - INFO - -->is_ns_server_running? [2021-02-12 21:07:27,282] - [cluster_helper] [140099333379840] - INFO - ns_server @ 172.23.122.117:8091 is running [2021-02-12 21:07:27,285] - [cluster_helper] [140099333379840] - INFO - waiting for ns_server @ 172.23.122.97:8091 [2021-02-12 21:07:27,285] - [rest_client] [140099333379840] - INFO - -->is_ns_server_running? [2021-02-12 21:07:27,288] - [cluster_helper] [140099333379840] - INFO - ns_server @ 172.23.122.97:8091 is running [2021-02-12 21:07:27,291] - [cluster_helper] [140099333379840] - INFO - waiting for ns_server @ 172.23.122.82:8091 [2021-02-12 21:07:27,291] - [rest_client] [140099333379840] - INFO - -->is_ns_server_running? [2021-02-12 21:07:27,293] - [cluster_helper] [140099333379840] - INFO - ns_server @ 172.23.122.82:8091 is running [2021-02-12 21:07:27,296] - [cluster_helper] [140099333379840] - INFO - waiting for ns_server @ 172.23.122.118:8091 [2021-02-12 21:07:27,296] - [rest_client] [140099333379840] - INFO - -->is_ns_server_running? [2021-02-12 21:07:27,299] - [cluster_helper] [140099333379840] - INFO - ns_server @ 172.23.122.118:8091 is running [2021-02-12 21:07:27,302] - [cluster_helper] [140099333379840] - INFO - waiting for ns_server @ 172.23.122.116:8091 [2021-02-12 21:07:27,302] - [rest_client] [140099333379840] - INFO - -->is_ns_server_running? [2021-02-12 21:07:27,305] - [cluster_helper] [140099333379840] - INFO - ns_server @ 172.23.122.116:8091 is running [2021-02-12 21:07:27,305] - [basetestcase] [140099333379840] - INFO - Building docker image with java sdk client [2021-02-12 21:07:30,842] - [basetestcase] [140099333379840] - INFO - initializing cluster [2021-02-12 21:07:30,845] - [remote_util] [140099333379840] - INFO - SSH Connecting to 172.23.123.61 with username:root, attempt#1 of 5 [2021-02-12 21:07:30,950] - [remote_util] [140099333379840] - INFO - SSH Connected to 172.23.123.61 as root [2021-02-12 21:07:31,214] - [remote_util] [140099333379840] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:07:31,510] - [remote_util] [140099333379840] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:07:31,519] - [remote_util] [140099333379840] - INFO - SSH Connecting to 172.23.123.61 with username:root, attempt#1 of 5 [2021-02-12 21:07:31,618] - [remote_util] [140099333379840] - INFO - SSH Connected to 172.23.123.61 as root [2021-02-12 21:07:31,882] - [remote_util] [140099333379840] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:07:32,178] - [remote_util] [140099333379840] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:07:32,198] - [remote_util] [140099333379840] - INFO - 172.23.123.61 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 21:07:32,252] - [remote_util] [140099333379840] - INFO - Running systemd command on this server [2021-02-12 21:07:32,252] - [remote_util] [140099333379840] - INFO - running command.raw on 172.23.123.61: systemctl stop couchbase-server.service [2021-02-12 21:07:34,083] - [remote_util] [140099333379840] - INFO - command executed successfully with root [2021-02-12 21:07:34,083] - [basetestcase] [140099333379840] - INFO - Couchbase stopped [2021-02-12 21:07:34,084] - [basetestcase] [140099333379840] - INFO - sleep for 5 secs. ... [2021-02-12 21:07:39,089] - [remote_util] [140099333379840] - INFO - running command.raw on 172.23.123.61: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-12 21:07:39,105] - [remote_util] [140099333379840] - INFO - command executed successfully with root [2021-02-12 21:07:39,105] - [remote_util] [140099333379840] - INFO - running command.raw on 172.23.123.61: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-12 21:07:39,160] - [remote_util] [140099333379840] - INFO - command executed successfully with root [2021-02-12 21:07:39,162] - [remote_util] [140099333379840] - INFO - SSH Connecting to 172.23.123.61 with username:root, attempt#1 of 5 [2021-02-12 21:07:39,271] - [remote_util] [140099333379840] - INFO - SSH Connected to 172.23.123.61 as root [2021-02-12 21:07:39,562] - [remote_util] [140099333379840] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:07:39,873] - [remote_util] [140099333379840] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:07:39,895] - [remote_util] [140099333379840] - INFO - 172.23.123.61 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 21:07:39,895] - [remote_util] [140099333379840] - INFO - Checking for process beam.smp on linux [2021-02-12 21:07:39,936] - [remote_util] [140099333379840] - INFO - Starting couchbase server [2021-02-12 21:07:40,038] - [remote_util] [140099333379840] - INFO - Running systemd command on this server [2021-02-12 21:07:40,038] - [remote_util] [140099333379840] - INFO - running command.raw on 172.23.123.61: systemctl start couchbase-server.service [2021-02-12 21:07:40,065] - [remote_util] [140099333379840] - INFO - command executed successfully with root [2021-02-12 21:07:40,065] - [remote_util] [140099333379840] - INFO - 172.23.123.61:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-12 21:07:45,069] - [remote_util] [140099333379840] - INFO - running command.raw on 172.23.123.61: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-12 21:07:45,096] - [remote_util] [140099333379840] - INFO - command executed successfully with root [2021-02-12 21:07:45,096] - [remote_util] [140099333379840] - INFO - Couchbase server status: [' Process: 78731 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-12 21:07:45,096] - [remote_util] [140099333379840] - INFO - Checking for process beam.smp on linux [2021-02-12 21:07:45,165] - [remote_util] [140099333379840] - INFO - process beam.smp is running on 172.23.123.61: with pid 78874 [2021-02-12 21:07:45,165] - [basetestcase] [140099333379840] - INFO - Couchbase started [2021-02-12 21:07:45,167] - [remote_util] [140099333379840] - INFO - SSH Connecting to 172.23.123.62 with username:root, attempt#1 of 5 [2021-02-12 21:07:45,278] - [remote_util] [140099333379840] - INFO - SSH Connected to 172.23.123.62 as root [2021-02-12 21:07:45,577] - [remote_util] [140099333379840] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:07:45,893] - [remote_util] [140099333379840] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:07:45,901] - [remote_util] [140099333379840] - INFO - SSH Connecting to 172.23.123.62 with username:root, attempt#1 of 5 [2021-02-12 21:07:46,009] - [remote_util] [140099333379840] - INFO - SSH Connected to 172.23.123.62 as root [2021-02-12 21:07:46,300] - [remote_util] [140099333379840] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:07:46,621] - [remote_util] [140099333379840] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:07:46,644] - [remote_util] [140099333379840] - INFO - 172.23.123.62 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 21:07:46,706] - [remote_util] [140099333379840] - INFO - Running systemd command on this server [2021-02-12 21:07:46,706] - [remote_util] [140099333379840] - INFO - running command.raw on 172.23.123.62: systemctl stop couchbase-server.service [2021-02-12 21:07:48,290] - [remote_util] [140099333379840] - INFO - command executed successfully with root [2021-02-12 21:07:48,290] - [basetestcase] [140099333379840] - INFO - Couchbase stopped [2021-02-12 21:07:48,291] - [basetestcase] [140099333379840] - INFO - sleep for 5 secs. ... [2021-02-12 21:07:53,291] - [remote_util] [140099333379840] - INFO - running command.raw on 172.23.123.62: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-12 21:07:53,307] - [remote_util] [140099333379840] - INFO - command executed successfully with root [2021-02-12 21:07:53,307] - [remote_util] [140099333379840] - INFO - running command.raw on 172.23.123.62: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-12 21:07:53,366] - [remote_util] [140099333379840] - INFO - command executed successfully with root [2021-02-12 21:07:53,368] - [remote_util] [140099333379840] - INFO - SSH Connecting to 172.23.123.62 with username:root, attempt#1 of 5 [2021-02-12 21:07:53,471] - [remote_util] [140099333379840] - INFO - SSH Connected to 172.23.123.62 as root [2021-02-12 21:07:53,744] - [remote_util] [140099333379840] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:07:54,043] - [remote_util] [140099333379840] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:07:54,062] - [remote_util] [140099333379840] - INFO - 172.23.123.62 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 21:07:54,062] - [remote_util] [140099333379840] - INFO - Checking for process beam.smp on linux [2021-02-12 21:07:54,102] - [remote_util] [140099333379840] - INFO - Starting couchbase server [2021-02-12 21:07:54,195] - [remote_util] [140099333379840] - INFO - Running systemd command on this server [2021-02-12 21:07:54,195] - [remote_util] [140099333379840] - INFO - running command.raw on 172.23.123.62: systemctl start couchbase-server.service [2021-02-12 21:07:54,218] - [remote_util] [140099333379840] - INFO - command executed successfully with root [2021-02-12 21:07:54,219] - [remote_util] [140099333379840] - INFO - 172.23.123.62:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-12 21:07:59,224] - [remote_util] [140099333379840] - INFO - running command.raw on 172.23.123.62: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-12 21:07:59,252] - [remote_util] [140099333379840] - INFO - command executed successfully with root [2021-02-12 21:07:59,252] - [remote_util] [140099333379840] - INFO - Couchbase server status: [' Process: 92661 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-12 21:07:59,252] - [remote_util] [140099333379840] - INFO - Checking for process beam.smp on linux [2021-02-12 21:07:59,320] - [remote_util] [140099333379840] - INFO - process beam.smp is running on 172.23.123.62: with pid 92804 [2021-02-12 21:07:59,320] - [basetestcase] [140099333379840] - INFO - Couchbase started [2021-02-12 21:07:59,322] - [remote_util] [140099333379840] - INFO - SSH Connecting to 172.23.122.90 with username:root, attempt#1 of 5 [2021-02-12 21:07:59,424] - [remote_util] [140099333379840] - INFO - SSH Connected to 172.23.122.90 as root [2021-02-12 21:07:59,705] - [remote_util] [140099333379840] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:08:00,002] - [remote_util] [140099333379840] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:08:00,010] - [remote_util] [140099333379840] - INFO - SSH Connecting to 172.23.122.90 with username:root, attempt#1 of 5 [2021-02-12 21:08:00,107] - [remote_util] [140099333379840] - INFO - SSH Connected to 172.23.122.90 as root [2021-02-12 21:08:00,378] - [remote_util] [140099333379840] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:08:00,671] - [remote_util] [140099333379840] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:08:00,689] - [remote_util] [140099333379840] - INFO - 172.23.122.90 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 21:08:00,739] - [remote_util] [140099333379840] - INFO - Running systemd command on this server [2021-02-12 21:08:00,739] - [remote_util] [140099333379840] - INFO - running command.raw on 172.23.122.90: systemctl stop couchbase-server.service [2021-02-12 21:08:02,482] - [remote_util] [140099333379840] - INFO - command executed successfully with root [2021-02-12 21:08:02,482] - [basetestcase] [140099333379840] - INFO - Couchbase stopped [2021-02-12 21:08:02,483] - [basetestcase] [140099333379840] - INFO - sleep for 5 secs. ... [2021-02-12 21:08:07,488] - [remote_util] [140099333379840] - INFO - running command.raw on 172.23.122.90: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-12 21:08:07,503] - [remote_util] [140099333379840] - INFO - command executed successfully with root [2021-02-12 21:08:07,503] - [remote_util] [140099333379840] - INFO - running command.raw on 172.23.122.90: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-12 21:08:07,556] - [remote_util] [140099333379840] - INFO - command executed successfully with root [2021-02-12 21:08:07,558] - [remote_util] [140099333379840] - INFO - SSH Connecting to 172.23.122.90 with username:root, attempt#1 of 5 [2021-02-12 21:08:07,659] - [remote_util] [140099333379840] - INFO - SSH Connected to 172.23.122.90 as root [2021-02-12 21:08:07,942] - [remote_util] [140099333379840] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:08:08,237] - [remote_util] [140099333379840] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:08:08,257] - [remote_util] [140099333379840] - INFO - 172.23.122.90 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 21:08:08,257] - [remote_util] [140099333379840] - INFO - Checking for process beam.smp on linux [2021-02-12 21:08:08,297] - [remote_util] [140099333379840] - INFO - Starting couchbase server [2021-02-12 21:08:08,390] - [remote_util] [140099333379840] - INFO - Running systemd command on this server [2021-02-12 21:08:08,390] - [remote_util] [140099333379840] - INFO - running command.raw on 172.23.122.90: systemctl start couchbase-server.service [2021-02-12 21:08:08,412] - [remote_util] [140099333379840] - INFO - command executed successfully with root [2021-02-12 21:08:08,412] - [remote_util] [140099333379840] - INFO - 172.23.122.90:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-12 21:08:13,418] - [remote_util] [140099333379840] - INFO - running command.raw on 172.23.122.90: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-12 21:08:13,439] - [remote_util] [140099333379840] - INFO - command executed successfully with root [2021-02-12 21:08:13,439] - [remote_util] [140099333379840] - INFO - Couchbase server status: [' Process: 44585 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-12 21:08:13,440] - [remote_util] [140099333379840] - INFO - Checking for process beam.smp on linux [2021-02-12 21:08:13,507] - [remote_util] [140099333379840] - INFO - process beam.smp is running on 172.23.122.90: with pid 44729 [2021-02-12 21:08:13,507] - [basetestcase] [140099333379840] - INFO - Couchbase started [2021-02-12 21:08:13,509] - [remote_util] [140099333379840] - INFO - SSH Connecting to 172.23.122.117 with username:root, attempt#1 of 5 [2021-02-12 21:08:13,609] - [remote_util] [140099333379840] - INFO - SSH Connected to 172.23.122.117 as root [2021-02-12 21:08:13,863] - [remote_util] [140099333379840] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:08:14,154] - [remote_util] [140099333379840] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:08:14,162] - [remote_util] [140099333379840] - INFO - SSH Connecting to 172.23.122.117 with username:root, attempt#1 of 5 [2021-02-12 21:08:14,259] - [remote_util] [140099333379840] - INFO - SSH Connected to 172.23.122.117 as root [2021-02-12 21:08:14,507] - [remote_util] [140099333379840] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:08:14,802] - [remote_util] [140099333379840] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:08:14,822] - [remote_util] [140099333379840] - INFO - 172.23.122.117 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 21:08:14,876] - [remote_util] [140099333379840] - INFO - Running systemd command on this server [2021-02-12 21:08:14,876] - [remote_util] [140099333379840] - INFO - running command.raw on 172.23.122.117: systemctl stop couchbase-server.service [2021-02-12 21:08:16,613] - [remote_util] [140099333379840] - INFO - command executed successfully with root [2021-02-12 21:08:16,613] - [basetestcase] [140099333379840] - INFO - Couchbase stopped [2021-02-12 21:08:16,614] - [basetestcase] [140099333379840] - INFO - sleep for 5 secs. ... [2021-02-12 21:08:21,619] - [remote_util] [140099333379840] - INFO - running command.raw on 172.23.122.117: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-12 21:08:21,635] - [remote_util] [140099333379840] - INFO - command executed successfully with root [2021-02-12 21:08:21,635] - [remote_util] [140099333379840] - INFO - running command.raw on 172.23.122.117: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-12 21:08:21,690] - [remote_util] [140099333379840] - INFO - command executed successfully with root [2021-02-12 21:08:21,691] - [remote_util] [140099333379840] - INFO - SSH Connecting to 172.23.122.117 with username:root, attempt#1 of 5 [2021-02-12 21:08:21,793] - [remote_util] [140099333379840] - INFO - SSH Connected to 172.23.122.117 as root [2021-02-12 21:08:22,056] - [remote_util] [140099333379840] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:08:22,353] - [remote_util] [140099333379840] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:08:22,372] - [remote_util] [140099333379840] - INFO - 172.23.122.117 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 21:08:22,372] - [remote_util] [140099333379840] - INFO - Checking for process beam.smp on linux [2021-02-12 21:08:22,409] - [remote_util] [140099333379840] - INFO - Starting couchbase server [2021-02-12 21:08:22,505] - [remote_util] [140099333379840] - INFO - Running systemd command on this server [2021-02-12 21:08:22,506] - [remote_util] [140099333379840] - INFO - running command.raw on 172.23.122.117: systemctl start couchbase-server.service [2021-02-12 21:08:22,529] - [remote_util] [140099333379840] - INFO - command executed successfully with root [2021-02-12 21:08:22,529] - [remote_util] [140099333379840] - INFO - 172.23.122.117:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-12 21:08:27,534] - [remote_util] [140099333379840] - INFO - running command.raw on 172.23.122.117: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-12 21:08:27,561] - [remote_util] [140099333379840] - INFO - command executed successfully with root [2021-02-12 21:08:27,561] - [remote_util] [140099333379840] - INFO - Couchbase server status: [' Process: 121436 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-12 21:08:27,561] - [remote_util] [140099333379840] - INFO - Checking for process beam.smp on linux [2021-02-12 21:08:27,628] - [remote_util] [140099333379840] - INFO - process beam.smp is running on 172.23.122.117: with pid 121581 [2021-02-12 21:08:27,628] - [basetestcase] [140099333379840] - INFO - Couchbase started [2021-02-12 21:08:27,630] - [remote_util] [140099333379840] - INFO - SSH Connecting to 172.23.122.97 with username:root, attempt#1 of 5 [2021-02-12 21:08:27,733] - [remote_util] [140099333379840] - INFO - SSH Connected to 172.23.122.97 as root [2021-02-12 21:08:27,992] - [remote_util] [140099333379840] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:08:28,287] - [remote_util] [140099333379840] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:08:28,297] - [remote_util] [140099333379840] - INFO - SSH Connecting to 172.23.122.97 with username:root, attempt#1 of 5 [2021-02-12 21:08:28,396] - [remote_util] [140099333379840] - INFO - SSH Connected to 172.23.122.97 as root [2021-02-12 21:08:28,644] - [remote_util] [140099333379840] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:08:28,940] - [remote_util] [140099333379840] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:08:28,959] - [remote_util] [140099333379840] - INFO - 172.23.122.97 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 21:08:29,015] - [remote_util] [140099333379840] - INFO - Running systemd command on this server [2021-02-12 21:08:29,015] - [remote_util] [140099333379840] - INFO - running command.raw on 172.23.122.97: systemctl stop couchbase-server.service [2021-02-12 21:08:30,642] - [remote_util] [140099333379840] - INFO - command executed successfully with root [2021-02-12 21:08:30,642] - [basetestcase] [140099333379840] - INFO - Couchbase stopped [2021-02-12 21:08:30,642] - [basetestcase] [140099333379840] - INFO - sleep for 5 secs. ... [2021-02-12 21:08:35,647] - [remote_util] [140099333379840] - INFO - running command.raw on 172.23.122.97: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-12 21:08:35,662] - [remote_util] [140099333379840] - INFO - command executed successfully with root [2021-02-12 21:08:35,663] - [remote_util] [140099333379840] - INFO - running command.raw on 172.23.122.97: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-12 21:08:35,717] - [remote_util] [140099333379840] - INFO - command executed successfully with root [2021-02-12 21:08:35,718] - [remote_util] [140099333379840] - INFO - SSH Connecting to 172.23.122.97 with username:root, attempt#1 of 5 [2021-02-12 21:08:35,821] - [remote_util] [140099333379840] - INFO - SSH Connected to 172.23.122.97 as root [2021-02-12 21:08:36,093] - [remote_util] [140099333379840] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:08:36,393] - [remote_util] [140099333379840] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:08:36,414] - [remote_util] [140099333379840] - INFO - 172.23.122.97 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 21:08:36,414] - [remote_util] [140099333379840] - INFO - Checking for process beam.smp on linux [2021-02-12 21:08:36,452] - [remote_util] [140099333379840] - INFO - Starting couchbase server [2021-02-12 21:08:36,547] - [remote_util] [140099333379840] - INFO - Running systemd command on this server [2021-02-12 21:08:36,548] - [remote_util] [140099333379840] - INFO - running command.raw on 172.23.122.97: systemctl start couchbase-server.service [2021-02-12 21:08:36,570] - [remote_util] [140099333379840] - INFO - command executed successfully with root [2021-02-12 21:08:36,570] - [remote_util] [140099333379840] - INFO - 172.23.122.97:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-12 21:08:41,576] - [remote_util] [140099333379840] - INFO - running command.raw on 172.23.122.97: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-12 21:08:41,603] - [remote_util] [140099333379840] - INFO - command executed successfully with root [2021-02-12 21:08:41,603] - [remote_util] [140099333379840] - INFO - Couchbase server status: [' Process: 23987 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-12 21:08:41,603] - [remote_util] [140099333379840] - INFO - Checking for process beam.smp on linux [2021-02-12 21:08:41,673] - [remote_util] [140099333379840] - INFO - process beam.smp is running on 172.23.122.97: with pid 24131 [2021-02-12 21:08:41,673] - [basetestcase] [140099333379840] - INFO - Couchbase started [2021-02-12 21:08:41,675] - [remote_util] [140099333379840] - INFO - SSH Connecting to 172.23.122.82 with username:root, attempt#1 of 5 [2021-02-12 21:08:41,778] - [remote_util] [140099333379840] - INFO - SSH Connected to 172.23.122.82 as root [2021-02-12 21:08:42,053] - [remote_util] [140099333379840] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:08:42,352] - [remote_util] [140099333379840] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:08:42,362] - [remote_util] [140099333379840] - INFO - SSH Connecting to 172.23.122.82 with username:root, attempt#1 of 5 [2021-02-12 21:08:42,461] - [remote_util] [140099333379840] - INFO - SSH Connected to 172.23.122.82 as root [2021-02-12 21:08:42,721] - [remote_util] [140099333379840] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:08:43,015] - [remote_util] [140099333379840] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:08:43,075] - [remote_util] [140099333379840] - INFO - 172.23.122.82 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 21:08:43,128] - [remote_util] [140099333379840] - INFO - Running systemd command on this server [2021-02-12 21:08:43,129] - [remote_util] [140099333379840] - INFO - running command.raw on 172.23.122.82: systemctl stop couchbase-server.service [2021-02-12 21:08:44,854] - [remote_util] [140099333379840] - INFO - command executed successfully with root [2021-02-12 21:08:44,854] - [basetestcase] [140099333379840] - INFO - Couchbase stopped [2021-02-12 21:08:44,855] - [basetestcase] [140099333379840] - INFO - sleep for 5 secs. ... [2021-02-12 21:08:49,860] - [remote_util] [140099333379840] - INFO - running command.raw on 172.23.122.82: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-12 21:08:49,875] - [remote_util] [140099333379840] - INFO - command executed successfully with root [2021-02-12 21:08:49,875] - [remote_util] [140099333379840] - INFO - running command.raw on 172.23.122.82: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-12 21:08:49,928] - [remote_util] [140099333379840] - INFO - command executed successfully with root [2021-02-12 21:08:49,930] - [remote_util] [140099333379840] - INFO - SSH Connecting to 172.23.122.82 with username:root, attempt#1 of 5 [2021-02-12 21:08:50,028] - [remote_util] [140099333379840] - INFO - SSH Connected to 172.23.122.82 as root [2021-02-12 21:08:50,293] - [remote_util] [140099333379840] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:08:50,588] - [remote_util] [140099333379840] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:08:50,607] - [remote_util] [140099333379840] - INFO - 172.23.122.82 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 21:08:50,607] - [remote_util] [140099333379840] - INFO - Checking for process beam.smp on linux [2021-02-12 21:08:50,647] - [remote_util] [140099333379840] - INFO - Starting couchbase server [2021-02-12 21:08:50,742] - [remote_util] [140099333379840] - INFO - Running systemd command on this server [2021-02-12 21:08:50,743] - [remote_util] [140099333379840] - INFO - running command.raw on 172.23.122.82: systemctl start couchbase-server.service [2021-02-12 21:08:50,766] - [remote_util] [140099333379840] - INFO - command executed successfully with root [2021-02-12 21:08:50,766] - [remote_util] [140099333379840] - INFO - 172.23.122.82:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-12 21:08:55,771] - [remote_util] [140099333379840] - INFO - running command.raw on 172.23.122.82: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-12 21:08:55,799] - [remote_util] [140099333379840] - INFO - command executed successfully with root [2021-02-12 21:08:55,799] - [remote_util] [140099333379840] - INFO - Couchbase server status: [' Process: 29244 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-12 21:08:55,799] - [remote_util] [140099333379840] - INFO - Checking for process beam.smp on linux [2021-02-12 21:08:55,869] - [remote_util] [140099333379840] - INFO - process beam.smp is running on 172.23.122.82: with pid 29389 [2021-02-12 21:08:55,869] - [basetestcase] [140099333379840] - INFO - Couchbase started [2021-02-12 21:08:55,871] - [remote_util] [140099333379840] - INFO - SSH Connecting to 172.23.122.118 with username:root, attempt#1 of 5 [2021-02-12 21:08:55,969] - [remote_util] [140099333379840] - INFO - SSH Connected to 172.23.122.118 as root [2021-02-12 21:08:56,211] - [remote_util] [140099333379840] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:08:56,503] - [remote_util] [140099333379840] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:08:56,511] - [remote_util] [140099333379840] - INFO - SSH Connecting to 172.23.122.118 with username:root, attempt#1 of 5 [2021-02-12 21:08:56,606] - [remote_util] [140099333379840] - INFO - SSH Connected to 172.23.122.118 as root [2021-02-12 21:08:56,841] - [remote_util] [140099333379840] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:08:57,130] - [remote_util] [140099333379840] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:08:57,148] - [remote_util] [140099333379840] - INFO - 172.23.122.118 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 21:08:57,198] - [remote_util] [140099333379840] - INFO - Running systemd command on this server [2021-02-12 21:08:57,198] - [remote_util] [140099333379840] - INFO - running command.raw on 172.23.122.118: systemctl stop couchbase-server.service [2021-02-12 21:08:58,845] - [remote_util] [140099333379840] - INFO - command executed successfully with root [2021-02-12 21:08:58,845] - [basetestcase] [140099333379840] - INFO - Couchbase stopped [2021-02-12 21:08:58,845] - [basetestcase] [140099333379840] - INFO - sleep for 5 secs. ... [2021-02-12 21:09:03,846] - [remote_util] [140099333379840] - INFO - running command.raw on 172.23.122.118: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-12 21:09:03,861] - [remote_util] [140099333379840] - INFO - command executed successfully with root [2021-02-12 21:09:03,861] - [remote_util] [140099333379840] - INFO - running command.raw on 172.23.122.118: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-12 21:09:03,914] - [remote_util] [140099333379840] - INFO - command executed successfully with root [2021-02-12 21:09:03,915] - [remote_util] [140099333379840] - INFO - SSH Connecting to 172.23.122.118 with username:root, attempt#1 of 5 [2021-02-12 21:09:04,014] - [remote_util] [140099333379840] - INFO - SSH Connected to 172.23.122.118 as root [2021-02-12 21:09:04,253] - [remote_util] [140099333379840] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:09:04,545] - [remote_util] [140099333379840] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:09:04,563] - [remote_util] [140099333379840] - INFO - 172.23.122.118 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 21:09:04,563] - [remote_util] [140099333379840] - INFO - Checking for process beam.smp on linux [2021-02-12 21:09:04,600] - [remote_util] [140099333379840] - INFO - Starting couchbase server [2021-02-12 21:09:04,689] - [remote_util] [140099333379840] - INFO - Running systemd command on this server [2021-02-12 21:09:04,689] - [remote_util] [140099333379840] - INFO - running command.raw on 172.23.122.118: systemctl start couchbase-server.service [2021-02-12 21:09:04,711] - [remote_util] [140099333379840] - INFO - command executed successfully with root [2021-02-12 21:09:04,711] - [remote_util] [140099333379840] - INFO - 172.23.122.118:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-12 21:09:09,716] - [remote_util] [140099333379840] - INFO - running command.raw on 172.23.122.118: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-12 21:09:09,742] - [remote_util] [140099333379840] - INFO - command executed successfully with root [2021-02-12 21:09:09,742] - [remote_util] [140099333379840] - INFO - Couchbase server status: [' Process: 70597 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-12 21:09:09,742] - [remote_util] [140099333379840] - INFO - Checking for process beam.smp on linux [2021-02-12 21:09:09,809] - [remote_util] [140099333379840] - INFO - process beam.smp is running on 172.23.122.118: with pid 70741 [2021-02-12 21:09:09,809] - [basetestcase] [140099333379840] - INFO - Couchbase started [2021-02-12 21:09:09,810] - [remote_util] [140099333379840] - INFO - SSH Connecting to 172.23.122.116 with username:root, attempt#1 of 5 [2021-02-12 21:09:09,913] - [remote_util] [140099333379840] - INFO - SSH Connected to 172.23.122.116 as root [2021-02-12 21:09:10,185] - [remote_util] [140099333379840] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:09:10,483] - [remote_util] [140099333379840] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:09:10,491] - [remote_util] [140099333379840] - INFO - SSH Connecting to 172.23.122.116 with username:root, attempt#1 of 5 [2021-02-12 21:09:10,590] - [remote_util] [140099333379840] - INFO - SSH Connected to 172.23.122.116 as root [2021-02-12 21:09:10,861] - [remote_util] [140099333379840] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:09:11,158] - [remote_util] [140099333379840] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:09:11,218] - [remote_util] [140099333379840] - INFO - 172.23.122.116 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 21:09:11,266] - [remote_util] [140099333379840] - INFO - Running systemd command on this server [2021-02-12 21:09:11,266] - [remote_util] [140099333379840] - INFO - running command.raw on 172.23.122.116: systemctl stop couchbase-server.service [2021-02-12 21:09:13,017] - [remote_util] [140099333379840] - INFO - command executed successfully with root [2021-02-12 21:09:13,018] - [basetestcase] [140099333379840] - INFO - Couchbase stopped [2021-02-12 21:09:13,018] - [basetestcase] [140099333379840] - INFO - sleep for 5 secs. ... [2021-02-12 21:09:18,020] - [remote_util] [140099333379840] - INFO - running command.raw on 172.23.122.116: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-12 21:09:18,034] - [remote_util] [140099333379840] - INFO - command executed successfully with root [2021-02-12 21:09:18,034] - [remote_util] [140099333379840] - INFO - running command.raw on 172.23.122.116: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-12 21:09:18,087] - [remote_util] [140099333379840] - INFO - command executed successfully with root [2021-02-12 21:09:18,089] - [remote_util] [140099333379840] - INFO - SSH Connecting to 172.23.122.116 with username:root, attempt#1 of 5 [2021-02-12 21:09:18,187] - [remote_util] [140099333379840] - INFO - SSH Connected to 172.23.122.116 as root [2021-02-12 21:09:18,458] - [remote_util] [140099333379840] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:09:18,746] - [remote_util] [140099333379840] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:09:18,804] - [remote_util] [140099333379840] - INFO - 172.23.122.116 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 21:09:18,804] - [remote_util] [140099333379840] - INFO - Checking for process beam.smp on linux [2021-02-12 21:09:18,840] - [remote_util] [140099333379840] - INFO - Starting couchbase server [2021-02-12 21:09:18,931] - [remote_util] [140099333379840] - INFO - Running systemd command on this server [2021-02-12 21:09:18,931] - [remote_util] [140099333379840] - INFO - running command.raw on 172.23.122.116: systemctl start couchbase-server.service [2021-02-12 21:09:18,952] - [remote_util] [140099333379840] - INFO - command executed successfully with root [2021-02-12 21:09:18,952] - [remote_util] [140099333379840] - INFO - 172.23.122.116:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-12 21:09:23,957] - [remote_util] [140099333379840] - INFO - running command.raw on 172.23.122.116: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-12 21:09:23,981] - [remote_util] [140099333379840] - INFO - command executed successfully with root [2021-02-12 21:09:23,981] - [remote_util] [140099333379840] - INFO - Couchbase server status: [' Process: 35280 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-12 21:09:23,981] - [remote_util] [140099333379840] - INFO - Checking for process beam.smp on linux [2021-02-12 21:09:24,049] - [remote_util] [140099333379840] - INFO - process beam.smp is running on 172.23.122.116: with pid 35425 [2021-02-12 21:09:24,049] - [basetestcase] [140099333379840] - INFO - Couchbase started [2021-02-12 21:09:24,050] - [basetestcase] [140099333379840] - INFO - sleep for 10 secs. ... [2021-02-12 21:09:34,589] - [task] [140099350165248] - INFO - server: ip:172.23.123.61 port:8091 ssh_username:root, nodes/self [2021-02-12 21:09:34,592] - [task] [140099350165248] - INFO - {'uptime': '110', 'memoryTotal': 4201672704, 'memoryFree': 3525197824, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.123.61:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4454-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.123.61', 'ip': '172.23.123.61', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} [2021-02-12 21:09:34,593] - [rest_client] [140099350165248] - ERROR - GET http://172.23.123.61:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-02-12 21:09:34,594] - [rest_client] [140099350165248] - INFO - pools/default params : memoryQuota=2147 [2021-02-12 21:09:34,599] - [rest_client] [140099350165248] - INFO - --> init_node_services(Administrator,password,172.23.123.61,8091,['kv']) [2021-02-12 21:09:34,599] - [rest_client] [140099350165248] - INFO - /node/controller/setupServices params on 172.23.123.61: 8091:hostname=172.23.123.61&user=Administrator&password=password&services=kv [2021-02-12 21:09:34,626] - [rest_client] [140099350165248] - INFO - --> in init_cluster...Administrator,password,8091 [2021-02-12 21:09:34,626] - [rest_client] [140099350165248] - INFO - settings/web params on 172.23.123.61:8091:port=8091&username=Administrator&password=password [2021-02-12 21:09:34,681] - [rest_client] [140099350165248] - INFO - --> status:True [2021-02-12 21:09:34,683] - [remote_util] [140099350165248] - INFO - SSH Connecting to 172.23.123.61 with username:root, attempt#1 of 5 [2021-02-12 21:09:34,782] - [remote_util] [140099350165248] - INFO - SSH Connected to 172.23.123.61 as root [2021-02-12 21:09:35,045] - [remote_util] [140099350165248] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:09:35,348] - [remote_util] [140099350165248] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:09:35,356] - [remote_util] [140099350165248] - INFO - running command.raw on 172.23.123.61: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-02-12 21:09:35,412] - [remote_util] [140099350165248] - INFO - command executed successfully with root [2021-02-12 21:09:35,412] - [remote_util] [140099350165248] - INFO - ['ok'] [2021-02-12 21:09:35,414] - [rest_client] [140099350165248] - INFO - /diag/eval status on 172.23.123.61:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-12 21:09:35,416] - [rest_client] [140099350165248] - INFO - /diag/eval status on 172.23.123.61:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-12 21:09:35,420] - [rest_client] [140099350165248] - INFO - settings/indexes params : storageMode=plasma [2021-02-12 21:09:35,437] - [task] [140099350165248] - INFO - server: ip:172.23.123.62 port:8091 ssh_username:root, nodes/self [2021-02-12 21:09:35,440] - [task] [140099350165248] - INFO - {'uptime': '94', 'memoryTotal': 4201676800, 'memoryFree': 3484684288, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.123.62:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4454-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.123.62', 'ip': '172.23.123.62', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} [2021-02-12 21:09:35,442] - [rest_client] [140099350165248] - ERROR - GET http://172.23.123.62:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-02-12 21:09:35,442] - [rest_client] [140099350165248] - INFO - pools/default params : memoryQuota=2147 [2021-02-12 21:09:35,447] - [rest_client] [140099350165248] - INFO - --> in init_cluster...Administrator,password,8091 [2021-02-12 21:09:35,447] - [rest_client] [140099350165248] - INFO - settings/web params on 172.23.123.62:8091:port=8091&username=Administrator&password=password [2021-02-12 21:09:35,503] - [rest_client] [140099350165248] - INFO - --> status:True [2021-02-12 21:09:35,504] - [remote_util] [140099350165248] - INFO - SSH Connecting to 172.23.123.62 with username:root, attempt#1 of 5 [2021-02-12 21:09:35,611] - [remote_util] [140099350165248] - INFO - SSH Connected to 172.23.123.62 as root [2021-02-12 21:09:35,886] - [remote_util] [140099350165248] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:09:36,183] - [remote_util] [140099350165248] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:09:36,190] - [remote_util] [140099350165248] - INFO - running command.raw on 172.23.123.62: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-02-12 21:09:36,245] - [remote_util] [140099350165248] - INFO - command executed successfully with root [2021-02-12 21:09:36,245] - [remote_util] [140099350165248] - INFO - ['ok'] [2021-02-12 21:09:36,247] - [rest_client] [140099350165248] - INFO - /diag/eval status on 172.23.123.62:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-12 21:09:36,249] - [rest_client] [140099350165248] - INFO - /diag/eval status on 172.23.123.62:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-12 21:09:36,253] - [rest_client] [140099350165248] - INFO - settings/indexes params : storageMode=plasma [2021-02-12 21:09:36,274] - [task] [140099350165248] - INFO - server: ip:172.23.122.90 port:8091 ssh_username:root, nodes/self [2021-02-12 21:09:36,277] - [task] [140099350165248] - INFO - {'uptime': '84', 'memoryTotal': 4201684992, 'memoryFree': 3561562112, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.122.90:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4454-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.122.90', 'ip': '172.23.122.90', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} [2021-02-12 21:09:36,279] - [rest_client] [140099350165248] - ERROR - GET http://172.23.122.90:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-02-12 21:09:36,279] - [rest_client] [140099350165248] - INFO - pools/default params : memoryQuota=2147 [2021-02-12 21:09:36,284] - [rest_client] [140099350165248] - INFO - --> in init_cluster...Administrator,password,8091 [2021-02-12 21:09:36,285] - [rest_client] [140099350165248] - INFO - settings/web params on 172.23.122.90:8091:port=8091&username=Administrator&password=password [2021-02-12 21:09:36,341] - [rest_client] [140099350165248] - INFO - --> status:True [2021-02-12 21:09:36,342] - [remote_util] [140099350165248] - INFO - SSH Connecting to 172.23.122.90 with username:root, attempt#1 of 5 [2021-02-12 21:09:36,443] - [remote_util] [140099350165248] - INFO - SSH Connected to 172.23.122.90 as root [2021-02-12 21:09:36,731] - [remote_util] [140099350165248] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:09:37,026] - [remote_util] [140099350165248] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:09:37,034] - [remote_util] [140099350165248] - INFO - running command.raw on 172.23.122.90: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-02-12 21:09:37,091] - [remote_util] [140099350165248] - INFO - command executed successfully with root [2021-02-12 21:09:37,091] - [remote_util] [140099350165248] - INFO - ['ok'] [2021-02-12 21:09:37,093] - [rest_client] [140099350165248] - INFO - /diag/eval status on 172.23.122.90:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-12 21:09:37,094] - [rest_client] [140099350165248] - INFO - /diag/eval status on 172.23.122.90:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-12 21:09:37,098] - [rest_client] [140099350165248] - INFO - settings/indexes params : storageMode=plasma [2021-02-12 21:09:37,115] - [task] [140099350165248] - INFO - server: ip:172.23.122.117 port:8091 ssh_username:root, nodes/self [2021-02-12 21:09:37,118] - [task] [140099350165248] - INFO - {'uptime': '69', 'memoryTotal': 4201684992, 'memoryFree': 3536113664, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.122.117:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4454-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.122.117', 'ip': '172.23.122.117', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} [2021-02-12 21:09:37,120] - [rest_client] [140099350165248] - ERROR - GET http://172.23.122.117:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-02-12 21:09:37,120] - [rest_client] [140099350165248] - INFO - pools/default params : memoryQuota=2147 [2021-02-12 21:09:37,125] - [rest_client] [140099350165248] - INFO - --> in init_cluster...Administrator,password,8091 [2021-02-12 21:09:37,125] - [rest_client] [140099350165248] - INFO - settings/web params on 172.23.122.117:8091:port=8091&username=Administrator&password=password [2021-02-12 21:09:37,181] - [rest_client] [140099350165248] - INFO - --> status:True [2021-02-12 21:09:37,183] - [remote_util] [140099350165248] - INFO - SSH Connecting to 172.23.122.117 with username:root, attempt#1 of 5 [2021-02-12 21:09:37,283] - [remote_util] [140099350165248] - INFO - SSH Connected to 172.23.122.117 as root [2021-02-12 21:09:37,537] - [remote_util] [140099350165248] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:09:37,823] - [remote_util] [140099350165248] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:09:37,830] - [remote_util] [140099350165248] - INFO - running command.raw on 172.23.122.117: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-02-12 21:09:37,886] - [remote_util] [140099350165248] - INFO - command executed successfully with root [2021-02-12 21:09:37,886] - [remote_util] [140099350165248] - INFO - ['ok'] [2021-02-12 21:09:37,887] - [rest_client] [140099350165248] - INFO - /diag/eval status on 172.23.122.117:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-12 21:09:37,890] - [rest_client] [140099350165248] - INFO - /diag/eval status on 172.23.122.117:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-12 21:09:37,895] - [rest_client] [140099350165248] - INFO - settings/indexes params : storageMode=plasma [2021-02-12 21:09:37,912] - [task] [140099350165248] - INFO - server: ip:172.23.122.97 port:8091 ssh_username:root, nodes/self [2021-02-12 21:09:37,914] - [task] [140099350165248] - INFO - {'uptime': '54', 'memoryTotal': 4201684992, 'memoryFree': 3554549760, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.122.97:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4454-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.122.97', 'ip': '172.23.122.97', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} [2021-02-12 21:09:37,916] - [rest_client] [140099350165248] - ERROR - GET http://172.23.122.97:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-02-12 21:09:37,916] - [rest_client] [140099350165248] - INFO - pools/default params : memoryQuota=2147 [2021-02-12 21:09:37,921] - [rest_client] [140099350165248] - INFO - --> in init_cluster...Administrator,password,8091 [2021-02-12 21:09:37,922] - [rest_client] [140099350165248] - INFO - settings/web params on 172.23.122.97:8091:port=8091&username=Administrator&password=password [2021-02-12 21:09:37,976] - [rest_client] [140099350165248] - INFO - --> status:True [2021-02-12 21:09:37,977] - [remote_util] [140099350165248] - INFO - SSH Connecting to 172.23.122.97 with username:root, attempt#1 of 5 [2021-02-12 21:09:38,076] - [remote_util] [140099350165248] - INFO - SSH Connected to 172.23.122.97 as root [2021-02-12 21:09:38,333] - [remote_util] [140099350165248] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:09:38,679] - [remote_util] [140099350165248] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:09:38,687] - [remote_util] [140099350165248] - INFO - running command.raw on 172.23.122.97: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-02-12 21:09:38,744] - [remote_util] [140099350165248] - INFO - command executed successfully with root [2021-02-12 21:09:38,744] - [remote_util] [140099350165248] - INFO - ['ok'] [2021-02-12 21:09:38,746] - [rest_client] [140099350165248] - INFO - /diag/eval status on 172.23.122.97:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-12 21:09:38,747] - [rest_client] [140099350165248] - INFO - /diag/eval status on 172.23.122.97:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-12 21:09:38,751] - [rest_client] [140099350165248] - INFO - settings/indexes params : storageMode=plasma [2021-02-12 21:09:38,769] - [task] [140099350165248] - INFO - server: ip:172.23.122.82 port:8091 ssh_username:root, nodes/self [2021-02-12 21:09:38,771] - [task] [140099350165248] - INFO - {'uptime': '44', 'memoryTotal': 4201684992, 'memoryFree': 3584798720, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.122.82:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4454-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.122.82', 'ip': '172.23.122.82', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} [2021-02-12 21:09:38,773] - [rest_client] [140099350165248] - ERROR - GET http://172.23.122.82:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-02-12 21:09:38,773] - [rest_client] [140099350165248] - INFO - pools/default params : memoryQuota=2147 [2021-02-12 21:09:38,778] - [rest_client] [140099350165248] - INFO - --> in init_cluster...Administrator,password,8091 [2021-02-12 21:09:38,778] - [rest_client] [140099350165248] - INFO - settings/web params on 172.23.122.82:8091:port=8091&username=Administrator&password=password [2021-02-12 21:09:38,836] - [rest_client] [140099350165248] - INFO - --> status:True [2021-02-12 21:09:38,838] - [remote_util] [140099350165248] - INFO - SSH Connecting to 172.23.122.82 with username:root, attempt#1 of 5 [2021-02-12 21:09:38,942] - [remote_util] [140099350165248] - INFO - SSH Connected to 172.23.122.82 as root [2021-02-12 21:09:39,206] - [remote_util] [140099350165248] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:09:39,510] - [remote_util] [140099350165248] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:09:39,518] - [remote_util] [140099350165248] - INFO - running command.raw on 172.23.122.82: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-02-12 21:09:39,576] - [remote_util] [140099350165248] - INFO - command executed successfully with root [2021-02-12 21:09:39,576] - [remote_util] [140099350165248] - INFO - ['ok'] [2021-02-12 21:09:39,578] - [rest_client] [140099350165248] - INFO - /diag/eval status on 172.23.122.82:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-12 21:09:39,580] - [rest_client] [140099350165248] - INFO - /diag/eval status on 172.23.122.82:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-12 21:09:39,584] - [rest_client] [140099350165248] - INFO - settings/indexes params : storageMode=plasma [2021-02-12 21:09:39,606] - [task] [140099350165248] - INFO - server: ip:172.23.122.118 port:8091 ssh_username:root, nodes/self [2021-02-12 21:09:39,609] - [task] [140099350165248] - INFO - {'uptime': '29', 'memoryTotal': 4201689088, 'memoryFree': 3539525632, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.122.118:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4454-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@cb.local', 'ip': '172.23.122.118', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} [2021-02-12 21:09:39,611] - [rest_client] [140099350165248] - ERROR - GET http://172.23.122.118:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-02-12 21:09:39,611] - [rest_client] [140099350165248] - INFO - pools/default params : memoryQuota=2147 [2021-02-12 21:09:39,617] - [rest_client] [140099350165248] - INFO - --> in init_cluster...Administrator,password,8091 [2021-02-12 21:09:39,617] - [rest_client] [140099350165248] - INFO - settings/web params on 172.23.122.118:8091:port=8091&username=Administrator&password=password [2021-02-12 21:09:39,671] - [rest_client] [140099350165248] - INFO - --> status:True [2021-02-12 21:09:39,672] - [remote_util] [140099350165248] - INFO - SSH Connecting to 172.23.122.118 with username:root, attempt#1 of 5 [2021-02-12 21:09:39,774] - [remote_util] [140099350165248] - INFO - SSH Connected to 172.23.122.118 as root [2021-02-12 21:09:40,019] - [remote_util] [140099350165248] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:09:40,308] - [remote_util] [140099350165248] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:09:40,316] - [remote_util] [140099350165248] - INFO - running command.raw on 172.23.122.118: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-02-12 21:09:40,372] - [remote_util] [140099350165248] - INFO - command executed successfully with root [2021-02-12 21:09:40,372] - [remote_util] [140099350165248] - INFO - ['ok'] [2021-02-12 21:09:40,374] - [rest_client] [140099350165248] - INFO - /diag/eval status on 172.23.122.118:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-12 21:09:40,376] - [rest_client] [140099350165248] - INFO - /diag/eval status on 172.23.122.118:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-12 21:09:40,379] - [rest_client] [140099350165248] - INFO - settings/indexes params : storageMode=plasma [2021-02-12 21:09:40,399] - [task] [140099350165248] - INFO - server: ip:172.23.122.116 port:8091 ssh_username:root, nodes/self [2021-02-12 21:09:40,402] - [task] [140099350165248] - INFO - {'uptime': '14', 'memoryTotal': 4201689088, 'memoryFree': 3574726656, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.122.116:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4454-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@cb.local', 'ip': '172.23.122.116', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} [2021-02-12 21:09:40,403] - [rest_client] [140099350165248] - ERROR - GET http://172.23.122.116:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-02-12 21:09:40,404] - [rest_client] [140099350165248] - INFO - pools/default params : memoryQuota=2147 [2021-02-12 21:09:40,409] - [rest_client] [140099350165248] - INFO - --> in init_cluster...Administrator,password,8091 [2021-02-12 21:09:40,409] - [rest_client] [140099350165248] - INFO - settings/web params on 172.23.122.116:8091:port=8091&username=Administrator&password=password [2021-02-12 21:09:40,464] - [rest_client] [140099350165248] - INFO - --> status:True [2021-02-12 21:09:40,465] - [remote_util] [140099350165248] - INFO - SSH Connecting to 172.23.122.116 with username:root, attempt#1 of 5 [2021-02-12 21:09:40,569] - [remote_util] [140099350165248] - INFO - SSH Connected to 172.23.122.116 as root [2021-02-12 21:09:40,847] - [remote_util] [140099350165248] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:09:41,141] - [remote_util] [140099350165248] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:09:41,149] - [remote_util] [140099350165248] - INFO - running command.raw on 172.23.122.116: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-02-12 21:09:41,204] - [remote_util] [140099350165248] - INFO - command executed successfully with root [2021-02-12 21:09:41,204] - [remote_util] [140099350165248] - INFO - ['ok'] [2021-02-12 21:09:41,207] - [rest_client] [140099350165248] - INFO - /diag/eval status on 172.23.122.116:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-12 21:09:41,210] - [rest_client] [140099350165248] - INFO - /diag/eval status on 172.23.122.116:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-12 21:09:41,215] - [rest_client] [140099350165248] - INFO - settings/indexes params : storageMode=plasma [2021-02-12 21:09:41,235] - [basetestcase] [140099333379840] - INFO - **** add built-in 'cbadminbucket' user to node 172.23.123.61 **** [2021-02-12 21:09:41,245] - [rest_client] [140099333379840] - ERROR - DELETE http://172.23.123.61:8091/settings/rbac/users/local/cbadminbucket body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:password [2021-02-12 21:09:41,246] - [internal_user] [140099333379840] - INFO - Exception while deleting user. Exception is -b'"User was not found."' [2021-02-12 21:09:41,307] - [basetestcase] [140099333379840] - INFO - sleep for 5 secs. ... [2021-02-12 21:09:46,312] - [basetestcase] [140099333379840] - INFO - **** add 'admin' role to 'cbadminbucket' user **** [2021-02-12 21:09:46,323] - [basetestcase] [140099333379840] - INFO - done initializing cluster [2021-02-12 21:09:47,233] - [task] [140099350165248] - INFO - adding node 172.23.123.62:8091 to cluster [2021-02-12 21:09:47,234] - [rest_client] [140099350165248] - INFO - adding remote node @172.23.123.62:8091 to this cluster @172.23.123.61:8091 [2021-02-12 21:09:49,760] - [task] [140099350165248] - INFO - adding node 172.23.122.90:8091 to cluster [2021-02-12 21:09:49,760] - [rest_client] [140099350165248] - INFO - adding remote node @172.23.122.90:8091 to this cluster @172.23.123.61:8091 [2021-02-12 21:09:51,951] - [task] [140099350165248] - INFO - adding node 172.23.122.117:8091 to cluster [2021-02-12 21:09:51,952] - [rest_client] [140099350165248] - INFO - adding remote node @172.23.122.117:8091 to this cluster @172.23.123.61:8091 [2021-02-12 21:09:54,264] - [task] [140099350165248] - INFO - adding node 172.23.122.97:8091 to cluster [2021-02-12 21:09:54,264] - [rest_client] [140099350165248] - INFO - adding remote node @172.23.122.97:8091 to this cluster @172.23.123.61:8091 [2021-02-12 21:09:56,541] - [task] [140099350165248] - INFO - adding node 172.23.122.82:8091 to cluster [2021-02-12 21:09:56,541] - [rest_client] [140099350165248] - INFO - adding remote node @172.23.122.82:8091 to this cluster @172.23.123.61:8091 [2021-02-12 21:09:58,870] - [rest_client] [140099350165248] - INFO - rebalance params : {'knownNodes': 'ns_1@172.23.122.117,ns_1@172.23.122.82,ns_1@172.23.122.90,ns_1@172.23.122.97,ns_1@172.23.123.61,ns_1@172.23.123.62', 'ejectedNodes': '', 'user': 'Administrator', 'password': 'password'} [2021-02-12 21:09:58,900] - [rest_client] [140099350165248] - INFO - rebalance operation started [2021-02-12 21:09:58,902] - [rest_client] [140099350165248] - INFO - rebalance percentage : 0.00 % [2021-02-12 21:09:58,902] - [task] [140099350165248] - INFO - Rebalance - status: running, progress: 0.00% [2021-02-12 21:10:08,927] - [task] [140099350165248] - INFO - Rebalance - status: none, progress: 100.00% [2021-02-12 21:10:08,936] - [task] [140099350165248] - INFO - rebalancing was completed with progress: 100% in 10.035871267318726 sec [2021-02-12 21:10:08,938] - [remote_util] [140099333379840] - INFO - SSH Connecting to 172.23.123.61 with username:root, attempt#1 of 5 [2021-02-12 21:10:09,041] - [remote_util] [140099333379840] - INFO - SSH Connected to 172.23.123.61 as root [2021-02-12 21:10:09,310] - [remote_util] [140099333379840] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:10:09,611] - [remote_util] [140099333379840] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:10:09,619] - [remote_util] [140099333379840] - INFO - running command.raw on 172.23.123.61: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-02-12 21:10:09,674] - [remote_util] [140099333379840] - INFO - command executed successfully with root [2021-02-12 21:10:09,675] - [remote_util] [140099333379840] - INFO - ['ok'] [2021-02-12 21:10:09,675] - [basetestcase] [140099333379840] - INFO - Enabled diag/eval for non-local hosts from 172.23.123.61 [2021-02-12 21:10:09,675] - [basetestcase] [140099333379840] - INFO - ============== basetestcase setup was finished for test #4 test_kv_rebalance_in_when_existing_eventing_node_is_processing_mutations ============== [2021-02-12 21:10:09,684] - [remote_util] [140099333379840] - INFO - SSH Connecting to 172.23.123.61 with username:root, attempt#1 of 5 [2021-02-12 21:10:09,783] - [remote_util] [140099333379840] - INFO - SSH Connected to 172.23.123.61 as root [2021-02-12 21:10:10,053] - [remote_util] [140099333379840] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:10:10,348] - [remote_util] [140099333379840] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:10:10,350] - [remote_util] [140099333379840] - INFO - SSH Connecting to 172.23.123.62 with username:root, attempt#1 of 5 [2021-02-12 21:10:10,453] - [remote_util] [140099333379840] - INFO - SSH Connected to 172.23.123.62 as root [2021-02-12 21:10:10,738] - [remote_util] [140099333379840] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:10:11,039] - [remote_util] [140099333379840] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:10:11,041] - [remote_util] [140099333379840] - INFO - SSH Connecting to 172.23.122.90 with username:root, attempt#1 of 5 [2021-02-12 21:10:11,143] - [remote_util] [140099333379840] - INFO - SSH Connected to 172.23.122.90 as root [2021-02-12 21:10:11,436] - [remote_util] [140099333379840] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:10:11,733] - [remote_util] [140099333379840] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:10:11,735] - [remote_util] [140099333379840] - INFO - SSH Connecting to 172.23.122.117 with username:root, attempt#1 of 5 [2021-02-12 21:10:11,835] - [remote_util] [140099333379840] - INFO - SSH Connected to 172.23.122.117 as root [2021-02-12 21:10:12,081] - [remote_util] [140099333379840] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:10:12,375] - [remote_util] [140099333379840] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:10:12,376] - [remote_util] [140099333379840] - INFO - SSH Connecting to 172.23.122.97 with username:root, attempt#1 of 5 [2021-02-12 21:10:12,480] - [remote_util] [140099333379840] - INFO - SSH Connected to 172.23.122.97 as root [2021-02-12 21:10:12,744] - [remote_util] [140099333379840] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:10:13,046] - [remote_util] [140099333379840] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:10:13,049] - [remote_util] [140099333379840] - INFO - SSH Connecting to 172.23.122.82 with username:root, attempt#1 of 5 [2021-02-12 21:10:13,150] - [remote_util] [140099333379840] - INFO - SSH Connected to 172.23.122.82 as root [2021-02-12 21:10:13,421] - [remote_util] [140099333379840] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:10:13,718] - [remote_util] [140099333379840] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:10:13,719] - [remote_util] [140099333379840] - INFO - SSH Connecting to 172.23.122.118 with username:root, attempt#1 of 5 [2021-02-12 21:10:13,820] - [remote_util] [140099333379840] - INFO - SSH Connected to 172.23.122.118 as root [2021-02-12 21:10:14,064] - [remote_util] [140099333379840] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:10:14,361] - [remote_util] [140099333379840] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:10:14,363] - [remote_util] [140099333379840] - INFO - SSH Connecting to 172.23.122.116 with username:root, attempt#1 of 5 [2021-02-12 21:10:14,469] - [remote_util] [140099333379840] - INFO - SSH Connected to 172.23.122.116 as root [2021-02-12 21:10:14,739] - [remote_util] [140099333379840] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:10:15,036] - [remote_util] [140099333379840] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:10:27,937] - [basetestcase] [140099333379840] - INFO - ------- Cluster statistics ------- [2021-02-12 21:10:27,937] - [basetestcase] [140099333379840] - INFO - 172.23.122.117:8091 => {'services': ['eventing'], 'cpu_utilization': 5.762455963764469, 'mem_free': 3501404160, 'mem_total': 4201684992, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-02-12 21:10:27,937] - [basetestcase] [140099333379840] - INFO - 172.23.122.82:8091 => {'services': ['index', 'n1ql'], 'cpu_utilization': 2.157010283421119, 'mem_free': 3490103296, 'mem_total': 4201684992, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-02-12 21:10:27,937] - [basetestcase] [140099333379840] - INFO - 172.23.122.90:8091 => {'services': ['eventing'], 'cpu_utilization': 1.858362631843295, 'mem_free': 3530555392, 'mem_total': 4201684992, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-02-12 21:10:27,937] - [basetestcase] [140099333379840] - INFO - 172.23.122.97:8091 => {'services': ['eventing'], 'cpu_utilization': 1.584507042253521, 'mem_free': 3511947264, 'mem_total': 4201684992, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-02-12 21:10:27,938] - [basetestcase] [140099333379840] - INFO - 172.23.123.61:8091 => {'services': ['kv'], 'cpu_utilization': 3.250812703175794, 'mem_free': 3470352384, 'mem_total': 4201672704, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-02-12 21:10:27,938] - [basetestcase] [140099333379840] - INFO - 172.23.123.62:8091 => {'services': ['kv'], 'cpu_utilization': 11.99594217600812, 'mem_free': 3458527232, 'mem_total': 4201676800, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-02-12 21:10:27,938] - [basetestcase] [140099333379840] - INFO - --- End of cluster statistics --- [2021-02-12 21:10:27,939] - [remote_util] [140099333379840] - INFO - SSH Connecting to 172.23.123.61 with username:root, attempt#1 of 5 [2021-02-12 21:10:28,041] - [remote_util] [140099333379840] - INFO - SSH Connected to 172.23.123.61 as root [2021-02-12 21:10:28,307] - [remote_util] [140099333379840] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:10:28,602] - [remote_util] [140099333379840] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:10:31,290] - [basetestcase] [140099333379840] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root] [2021-02-12 21:10:31,296] - [basetestcase] [140099333379840] - INFO - list of eventing nodes in cluster: [ip:172.23.122.117 port:8091 ssh_username:root, ip:172.23.122.90 port:8091 ssh_username:root, ip:172.23.122.97 port:8091 ssh_username:root] [2021-02-12 21:10:31,303] - [rest_client] [140099333379840] - INFO - settings/indexes params : storageMode=plasma [2021-02-12 21:10:31,307] - [eventing_base] [140099333379840] - INFO - Setting the min possible memory quota so that adding mode nodes to the cluster wouldn't be a problem. [2021-02-12 21:10:31,307] - [rest_client] [140099333379840] - INFO - pools/default params : memoryQuota=330 [2021-02-12 21:10:31,314] - [rest_client] [140099333379840] - INFO - pools/default params : indexMemoryQuota=256 [2021-02-12 21:10:31,319] - [rest_client] [140099333379840] - INFO - pools/default params : eventingMemoryQuota=512 [2021-02-12 21:10:31,341] - [basetestcase] [140099333379840] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root] [2021-02-12 21:10:31,346] - [rest_client] [140099333379840] - INFO - pools/default params : memoryQuota=700 [2021-02-12 21:10:31,351] - [eventing_rebalance_collection] [140099333379840] - INFO - 200 [2021-02-12 21:10:31,993] - [rest_client] [140099350165248] - INFO - http://172.23.123.61:8091/pools/default/buckets with param: name=src_bucket&ramQuotaMB=200&authType=none&replicaNumber=0&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore [2021-02-12 21:10:32,021] - [rest_client] [140099350165248] - INFO - 0.03 seconds to create bucket src_bucket [2021-02-12 21:10:32,021] - [bucket_helper] [140099350165248] - INFO - waiting for memcached bucket : src_bucket in 172.23.123.61 to accept set ops [2021-02-12 21:10:32,659] - [data_helper] [140099350165248] - INFO - creating direct client 172.23.123.61:11210 src_bucket [2021-02-12 21:10:32,733] - [data_helper] [140099350165248] - INFO - creating direct client 172.23.123.62:11210 src_bucket [2021-02-12 21:10:32,944] - [data_helper] [140099350165248] - INFO - creating direct client 172.23.123.61:11210 src_bucket [2021-02-12 21:10:33,033] - [data_helper] [140099350165248] - INFO - creating direct client 172.23.123.62:11210 src_bucket [2021-02-12 21:10:33,102] - [task] [140099350165248] - WARNING - vbucket map not ready after try 0 [2021-02-12 21:10:33,103] - [bucket_helper] [140099350165248] - INFO - waiting for memcached bucket : src_bucket in 172.23.123.61 to accept set ops [2021-02-12 21:10:33,314] - [data_helper] [140099350165248] - INFO - creating direct client 172.23.123.61:11210 src_bucket [2021-02-12 21:10:33,401] - [data_helper] [140099350165248] - INFO - creating direct client 172.23.123.62:11210 src_bucket [2021-02-12 21:10:33,589] - [data_helper] [140099350165248] - INFO - creating direct client 172.23.123.61:11210 src_bucket [2021-02-12 21:10:33,639] - [data_helper] [140099350165248] - INFO - creating direct client 172.23.123.62:11210 src_bucket [2021-02-12 21:10:33,674] - [task] [140099350165248] - WARNING - vbucket map not ready after try 1 [2021-02-12 21:10:33,674] - [bucket_helper] [140099350165248] - INFO - waiting for memcached bucket : src_bucket in 172.23.123.61 to accept set ops [2021-02-12 21:10:33,820] - [data_helper] [140099350165248] - INFO - creating direct client 172.23.123.61:11210 src_bucket [2021-02-12 21:10:33,868] - [data_helper] [140099350165248] - INFO - creating direct client 172.23.123.62:11210 src_bucket [2021-02-12 21:10:34,026] - [data_helper] [140099350165248] - INFO - creating direct client 172.23.123.61:11210 src_bucket [2021-02-12 21:10:34,074] - [data_helper] [140099350165248] - INFO - creating direct client 172.23.123.62:11210 src_bucket [2021-02-12 21:10:34,110] - [task] [140099350165248] - WARNING - vbucket map not ready after try 2 [2021-02-12 21:10:34,110] - [bucket_helper] [140099350165248] - INFO - waiting for memcached bucket : src_bucket in 172.23.123.61 to accept set ops [2021-02-12 21:10:34,264] - [data_helper] [140099350165248] - INFO - creating direct client 172.23.123.61:11210 src_bucket [2021-02-12 21:10:34,309] - [data_helper] [140099350165248] - INFO - creating direct client 172.23.123.62:11210 src_bucket [2021-02-12 21:10:34,374] - [data_helper] [140099350165248] - INFO - creating direct client 172.23.123.61:11210 src_bucket [2021-02-12 21:10:34,519] - [data_helper] [140099350165248] - INFO - creating direct client 172.23.123.62:11210 src_bucket [2021-02-12 21:10:34,559] - [task] [140099350165248] - WARNING - vbucket map not ready after try 3 [2021-02-12 21:10:34,559] - [bucket_helper] [140099350165248] - INFO - waiting for memcached bucket : src_bucket in 172.23.123.61 to accept set ops [2021-02-12 21:10:34,622] - [data_helper] [140099350165248] - INFO - creating direct client 172.23.123.61:11210 src_bucket [2021-02-12 21:10:34,763] - [data_helper] [140099350165248] - INFO - creating direct client 172.23.123.62:11210 src_bucket [2021-02-12 21:10:34,827] - [data_helper] [140099350165248] - INFO - creating direct client 172.23.123.61:11210 src_bucket [2021-02-12 21:10:34,874] - [data_helper] [140099350165248] - INFO - creating direct client 172.23.123.62:11210 src_bucket [2021-02-12 21:10:35,005] - [task] [140099350165248] - WARNING - vbucket map not ready after try 4 [2021-02-12 21:10:35,005] - [bucket_helper] [140099350165248] - INFO - waiting for memcached bucket : src_bucket in 172.23.123.61 to accept set ops [2021-02-12 21:10:35,067] - [data_helper] [140099350165248] - INFO - creating direct client 172.23.123.61:11210 src_bucket [2021-02-12 21:10:35,113] - [data_helper] [140099350165248] - INFO - creating direct client 172.23.123.62:11210 src_bucket [2021-02-12 21:10:35,274] - [data_helper] [140099350165248] - INFO - creating direct client 172.23.123.61:11210 src_bucket [2021-02-12 21:10:35,321] - [data_helper] [140099350165248] - INFO - creating direct client 172.23.123.62:11210 src_bucket [2021-02-12 21:10:35,362] - [task] [140099350165248] - WARNING - vbucket map not ready after try 5 [2021-02-12 21:10:36,386] - [rest_client] [140099350165248] - INFO - http://172.23.123.61:8091/pools/default/buckets with param: name=dst_bucket&ramQuotaMB=200&authType=none&replicaNumber=0&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore [2021-02-12 21:10:36,415] - [rest_client] [140099350165248] - INFO - 0.03 seconds to create bucket dst_bucket [2021-02-12 21:10:36,415] - [bucket_helper] [140099350165248] - INFO - waiting for memcached bucket : dst_bucket in 172.23.123.61 to accept set ops [2021-02-12 21:10:37,061] - [data_helper] [140099350165248] - INFO - creating direct client 172.23.123.61:11210 dst_bucket [2021-02-12 21:10:37,135] - [data_helper] [140099350165248] - INFO - creating direct client 172.23.123.62:11210 dst_bucket [2021-02-12 21:10:37,379] - [data_helper] [140099350165248] - INFO - creating direct client 172.23.123.61:11210 dst_bucket [2021-02-12 21:10:37,452] - [data_helper] [140099350165248] - INFO - creating direct client 172.23.123.62:11210 dst_bucket [2021-02-12 21:10:37,516] - [task] [140099350165248] - WARNING - vbucket map not ready after try 0 [2021-02-12 21:10:37,516] - [bucket_helper] [140099350165248] - INFO - waiting for memcached bucket : dst_bucket in 172.23.123.61 to accept set ops [2021-02-12 21:10:37,727] - [data_helper] [140099350165248] - INFO - creating direct client 172.23.123.61:11210 dst_bucket [2021-02-12 21:10:37,804] - [data_helper] [140099350165248] - INFO - creating direct client 172.23.123.62:11210 dst_bucket [2021-02-12 21:10:37,925] - [data_helper] [140099350165248] - INFO - creating direct client 172.23.123.61:11210 dst_bucket [2021-02-12 21:10:38,092] - [data_helper] [140099350165248] - INFO - creating direct client 172.23.123.62:11210 dst_bucket [2021-02-12 21:10:38,157] - [task] [140099350165248] - WARNING - vbucket map not ready after try 1 [2021-02-12 21:10:38,157] - [bucket_helper] [140099350165248] - INFO - waiting for memcached bucket : dst_bucket in 172.23.123.61 to accept set ops [2021-02-12 21:10:38,350] - [data_helper] [140099350165248] - INFO - creating direct client 172.23.123.61:11210 dst_bucket [2021-02-12 21:10:38,426] - [data_helper] [140099350165248] - INFO - creating direct client 172.23.123.62:11210 dst_bucket [2021-02-12 21:10:38,538] - [data_helper] [140099350165248] - INFO - creating direct client 172.23.123.61:11210 dst_bucket [2021-02-12 21:10:38,705] - [data_helper] [140099350165248] - INFO - creating direct client 172.23.123.62:11210 dst_bucket [2021-02-12 21:10:38,769] - [task] [140099350165248] - WARNING - vbucket map not ready after try 2 [2021-02-12 21:10:38,769] - [bucket_helper] [140099350165248] - INFO - waiting for memcached bucket : dst_bucket in 172.23.123.61 to accept set ops [2021-02-12 21:10:38,885] - [data_helper] [140099350165248] - INFO - creating direct client 172.23.123.61:11210 dst_bucket [2021-02-12 21:10:38,961] - [data_helper] [140099350165248] - INFO - creating direct client 172.23.123.62:11210 dst_bucket [2021-02-12 21:10:39,186] - [data_helper] [140099350165248] - INFO - creating direct client 172.23.123.61:11210 dst_bucket [2021-02-12 21:10:39,261] - [data_helper] [140099350165248] - INFO - creating direct client 172.23.123.62:11210 dst_bucket [2021-02-12 21:10:39,327] - [task] [140099350165248] - WARNING - vbucket map not ready after try 3 [2021-02-12 21:10:39,327] - [bucket_helper] [140099350165248] - INFO - waiting for memcached bucket : dst_bucket in 172.23.123.61 to accept set ops [2021-02-12 21:10:39,531] - [data_helper] [140099350165248] - INFO - creating direct client 172.23.123.61:11210 dst_bucket [2021-02-12 21:10:39,607] - [data_helper] [140099350165248] - INFO - creating direct client 172.23.123.62:11210 dst_bucket [2021-02-12 21:10:39,825] - [data_helper] [140099350165248] - INFO - creating direct client 172.23.123.61:11210 dst_bucket [2021-02-12 21:10:39,901] - [data_helper] [140099350165248] - INFO - creating direct client 172.23.123.62:11210 dst_bucket [2021-02-12 21:10:39,967] - [task] [140099350165248] - WARNING - vbucket map not ready after try 4 [2021-02-12 21:10:39,967] - [bucket_helper] [140099350165248] - INFO - waiting for memcached bucket : dst_bucket in 172.23.123.61 to accept set ops [2021-02-12 21:10:40,163] - [data_helper] [140099350165248] - INFO - creating direct client 172.23.123.61:11210 dst_bucket [2021-02-12 21:10:40,235] - [data_helper] [140099350165248] - INFO - creating direct client 172.23.123.62:11210 dst_bucket [2021-02-12 21:10:40,354] - [data_helper] [140099350165248] - INFO - creating direct client 172.23.123.61:11210 dst_bucket [2021-02-12 21:10:40,509] - [data_helper] [140099350165248] - INFO - creating direct client 172.23.123.62:11210 dst_bucket [2021-02-12 21:10:40,574] - [task] [140099350165248] - WARNING - vbucket map not ready after try 5 [2021-02-12 21:10:41,600] - [rest_client] [140099350165248] - INFO - http://172.23.123.61:8091/pools/default/buckets with param: name=metadata&ramQuotaMB=200&authType=none&replicaNumber=0&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore [2021-02-12 21:10:41,629] - [rest_client] [140099350165248] - INFO - 0.03 seconds to create bucket metadata [2021-02-12 21:10:41,629] - [bucket_helper] [140099350165248] - INFO - waiting for memcached bucket : metadata in 172.23.123.61 to accept set ops [2021-02-12 21:10:42,263] - [data_helper] [140099350165248] - INFO - creating direct client 172.23.123.61:11210 metadata [2021-02-12 21:10:42,433] - [data_helper] [140099350165248] - INFO - creating direct client 172.23.123.62:11210 metadata [2021-02-12 21:10:42,588] - [data_helper] [140099350165248] - INFO - creating direct client 172.23.123.61:11210 metadata [2021-02-12 21:10:42,664] - [data_helper] [140099350165248] - INFO - creating direct client 172.23.123.62:11210 metadata [2021-02-12 21:10:42,821] - [task] [140099350165248] - WARNING - vbucket map not ready after try 0 [2021-02-12 21:10:42,821] - [bucket_helper] [140099350165248] - INFO - waiting for memcached bucket : metadata in 172.23.123.61 to accept set ops [2021-02-12 21:10:42,962] - [data_helper] [140099350165248] - INFO - creating direct client 172.23.123.61:11210 metadata [2021-02-12 21:10:43,044] - [data_helper] [140099350165248] - INFO - creating direct client 172.23.123.62:11210 metadata [2021-02-12 21:10:43,279] - [data_helper] [140099350165248] - INFO - creating direct client 172.23.123.61:11210 metadata [2021-02-12 21:10:43,359] - [data_helper] [140099350165248] - INFO - creating direct client 172.23.123.62:11210 metadata [2021-02-12 21:10:43,431] - [task] [140099350165248] - WARNING - vbucket map not ready after try 1 [2021-02-12 21:10:43,431] - [bucket_helper] [140099350165248] - INFO - waiting for memcached bucket : metadata in 172.23.123.61 to accept set ops [2021-02-12 21:10:43,633] - [data_helper] [140099350165248] - INFO - creating direct client 172.23.123.61:11210 metadata [2021-02-12 21:10:43,709] - [data_helper] [140099350165248] - INFO - creating direct client 172.23.123.62:11210 metadata [2021-02-12 21:10:43,822] - [data_helper] [140099350165248] - INFO - creating direct client 172.23.123.61:11210 metadata [2021-02-12 21:10:43,991] - [data_helper] [140099350165248] - INFO - creating direct client 172.23.123.62:11210 metadata [2021-02-12 21:10:44,059] - [task] [140099350165248] - WARNING - vbucket map not ready after try 2 [2021-02-12 21:10:44,059] - [bucket_helper] [140099350165248] - INFO - waiting for memcached bucket : metadata in 172.23.123.61 to accept set ops [2021-02-12 21:10:44,174] - [data_helper] [140099350165248] - INFO - creating direct client 172.23.123.61:11210 metadata [2021-02-12 21:10:44,338] - [data_helper] [140099350165248] - INFO - creating direct client 172.23.123.62:11210 metadata [2021-02-12 21:10:44,464] - [data_helper] [140099350165248] - INFO - creating direct client 172.23.123.61:11210 metadata [2021-02-12 21:10:44,540] - [data_helper] [140099350165248] - INFO - creating direct client 172.23.123.62:11210 metadata [2021-02-12 21:10:44,698] - [task] [140099350165248] - WARNING - vbucket map not ready after try 3 [2021-02-12 21:10:44,698] - [bucket_helper] [140099350165248] - INFO - waiting for memcached bucket : metadata in 172.23.123.61 to accept set ops [2021-02-12 21:10:44,816] - [data_helper] [140099350165248] - INFO - creating direct client 172.23.123.61:11210 metadata [2021-02-12 21:10:44,898] - [data_helper] [140099350165248] - INFO - creating direct client 172.23.123.62:11210 metadata [2021-02-12 21:10:45,110] - [data_helper] [140099350165248] - INFO - creating direct client 172.23.123.61:11210 metadata [2021-02-12 21:10:45,188] - [data_helper] [140099350165248] - INFO - creating direct client 172.23.123.62:11210 metadata [2021-02-12 21:10:45,261] - [task] [140099350165248] - WARNING - vbucket map not ready after try 4 [2021-02-12 21:10:45,262] - [bucket_helper] [140099350165248] - INFO - waiting for memcached bucket : metadata in 172.23.123.61 to accept set ops [2021-02-12 21:10:45,467] - [data_helper] [140099350165248] - INFO - creating direct client 172.23.123.61:11210 metadata [2021-02-12 21:10:45,550] - [data_helper] [140099350165248] - INFO - creating direct client 172.23.123.62:11210 metadata [2021-02-12 21:10:45,680] - [data_helper] [140099350165248] - INFO - creating direct client 172.23.123.61:11210 metadata [2021-02-12 21:10:45,852] - [data_helper] [140099350165248] - INFO - creating direct client 172.23.123.62:11210 metadata [2021-02-12 21:10:45,923] - [task] [140099350165248] - WARNING - vbucket map not ready after try 5 [2021-02-12 21:10:45,996] - [rest_client] [140099333379840] - INFO - http://172.23.122.117:8091/pools/default/buckets/dst_bucket/scopes with params: name=scope_0 [2021-02-12 21:10:45,996] - [rest_client] [140099333379840] - INFO - Scope created dst_bucket->scope_0 {'uid': '1'} [2021-02-12 21:10:46,022] - [rest_client] [140099333379840] - INFO - http://172.23.122.117:8091/pools/default/buckets/dst_bucket/scopes with params: name=scope_1 [2021-02-12 21:10:46,023] - [rest_client] [140099333379840] - INFO - Scope created dst_bucket->scope_1 {'uid': '2'} [2021-02-12 21:10:46,051] - [rest_client] [140099333379840] - INFO - http://172.23.122.117:8091/pools/default/buckets/dst_bucket/scopes with params: name=scope_2 [2021-02-12 21:10:46,052] - [rest_client] [140099333379840] - INFO - Scope created dst_bucket->scope_2 {'uid': '3'} [2021-02-12 21:10:46,080] - [rest_client] [140099333379840] - INFO - http://172.23.122.117:8091/pools/default/buckets/dst_bucket/scopes with params: name=scope_3 [2021-02-12 21:10:46,080] - [rest_client] [140099333379840] - INFO - Scope created dst_bucket->scope_3 {'uid': '4'} [2021-02-12 21:10:46,112] - [rest_client] [140099333379840] - INFO - http://172.23.122.117:8091/pools/default/buckets/dst_bucket/scopes with params: name=scope_4 [2021-02-12 21:10:46,112] - [rest_client] [140099333379840] - INFO - Scope created dst_bucket->scope_4 {'uid': '5'} [2021-02-12 21:10:46,138] - [rest_client] [140099333379840] - INFO - http://172.23.122.117:8091/pools/default/buckets/src_bucket/scopes with params: name=scope_0 [2021-02-12 21:10:46,138] - [rest_client] [140099333379840] - INFO - Scope created src_bucket->scope_0 {'uid': '1'} [2021-02-12 21:10:46,167] - [rest_client] [140099333379840] - INFO - http://172.23.122.117:8091/pools/default/buckets/src_bucket/scopes with params: name=scope_1 [2021-02-12 21:10:46,167] - [rest_client] [140099333379840] - INFO - Scope created src_bucket->scope_1 {'uid': '2'} [2021-02-12 21:10:46,197] - [rest_client] [140099333379840] - INFO - http://172.23.122.117:8091/pools/default/buckets/src_bucket/scopes with params: name=scope_2 [2021-02-12 21:10:46,197] - [rest_client] [140099333379840] - INFO - Scope created src_bucket->scope_2 {'uid': '3'} [2021-02-12 21:10:46,229] - [rest_client] [140099333379840] - INFO - http://172.23.122.117:8091/pools/default/buckets/src_bucket/scopes with params: name=scope_3 [2021-02-12 21:10:46,229] - [rest_client] [140099333379840] - INFO - Scope created src_bucket->scope_3 {'uid': '4'} [2021-02-12 21:10:46,257] - [rest_client] [140099333379840] - INFO - http://172.23.122.117:8091/pools/default/buckets/src_bucket/scopes with params: name=scope_4 [2021-02-12 21:10:46,257] - [rest_client] [140099333379840] - INFO - Scope created src_bucket->scope_4 {'uid': '5'} [2021-02-12 21:10:46,291] - [rest_client] [140099333379840] - INFO - http://172.23.122.117:8091/pools/default/buckets/dst_bucket/scopes/scope_1/collections with params: name=coll_0 [2021-02-12 21:10:46,291] - [rest_client] [140099333379840] - INFO - Collection created dst_bucket->scope_1->coll_0 manifest:{'uid': '6'} [2021-02-12 21:10:46,322] - [rest_client] [140099333379840] - INFO - http://172.23.122.117:8091/pools/default/buckets/dst_bucket/scopes/scope_1/collections with params: name=coll_1 [2021-02-12 21:10:46,322] - [rest_client] [140099333379840] - INFO - Collection created dst_bucket->scope_1->coll_1 manifest:{'uid': '7'} [2021-02-12 21:10:46,351] - [rest_client] [140099333379840] - INFO - http://172.23.122.117:8091/pools/default/buckets/dst_bucket/scopes/scope_1/collections with params: name=coll_2 [2021-02-12 21:10:46,351] - [rest_client] [140099333379840] - INFO - Collection created dst_bucket->scope_1->coll_2 manifest:{'uid': '8'} [2021-02-12 21:10:46,385] - [rest_client] [140099333379840] - INFO - http://172.23.122.117:8091/pools/default/buckets/dst_bucket/scopes/scope_1/collections with params: name=coll_3 [2021-02-12 21:10:46,385] - [rest_client] [140099333379840] - INFO - Collection created dst_bucket->scope_1->coll_3 manifest:{'uid': '9'} [2021-02-12 21:10:46,413] - [rest_client] [140099333379840] - INFO - http://172.23.122.117:8091/pools/default/buckets/dst_bucket/scopes/scope_1/collections with params: name=coll_4 [2021-02-12 21:10:46,413] - [rest_client] [140099333379840] - INFO - Collection created dst_bucket->scope_1->coll_4 manifest:{'uid': 'a'} [2021-02-12 21:10:46,446] - [rest_client] [140099333379840] - INFO - http://172.23.122.117:8091/pools/default/buckets/src_bucket/scopes/scope_1/collections with params: name=coll_0 [2021-02-12 21:10:46,447] - [rest_client] [140099333379840] - INFO - Collection created src_bucket->scope_1->coll_0 manifest:{'uid': '6'} [2021-02-12 21:10:46,476] - [rest_client] [140099333379840] - INFO - http://172.23.122.117:8091/pools/default/buckets/src_bucket/scopes/scope_1/collections with params: name=coll_1 [2021-02-12 21:10:46,476] - [rest_client] [140099333379840] - INFO - Collection created src_bucket->scope_1->coll_1 manifest:{'uid': '7'} [2021-02-12 21:10:46,508] - [rest_client] [140099333379840] - INFO - http://172.23.122.117:8091/pools/default/buckets/src_bucket/scopes/scope_1/collections with params: name=coll_2 [2021-02-12 21:10:46,509] - [rest_client] [140099333379840] - INFO - Collection created src_bucket->scope_1->coll_2 manifest:{'uid': '8'} [2021-02-12 21:10:46,544] - [rest_client] [140099333379840] - INFO - http://172.23.122.117:8091/pools/default/buckets/src_bucket/scopes/scope_1/collections with params: name=coll_3 [2021-02-12 21:10:46,544] - [rest_client] [140099333379840] - INFO - Collection created src_bucket->scope_1->coll_3 manifest:{'uid': '9'} [2021-02-12 21:10:46,574] - [rest_client] [140099333379840] - INFO - http://172.23.122.117:8091/pools/default/buckets/src_bucket/scopes/scope_1/collections with params: name=coll_4 [2021-02-12 21:10:46,574] - [rest_client] [140099333379840] - INFO - Collection created src_bucket->scope_1->coll_4 manifest:{'uid': 'a'} [2021-02-12 21:10:47,834] - [eventing_base] [140099333379840] - INFO - saving function bucket_op [2021-02-12 21:10:49,046] - [eventing_base] [140099333379840] - INFO - saving function timers [2021-02-12 21:10:50,305] - [eventing_base] [140099333379840] - INFO - saving function sbm [2021-02-12 21:10:51,509] - [eventing_base] [140099333379840] - INFO - saving function curl [2021-02-12 21:10:52,723] - [eventing_base] [140099333379840] - INFO - saving function n1ql [2021-02-12 21:10:53,948] - [basetestcase] [140099333379840] - INFO - sleep for 5 secs. Waiting for bucket_op to deployed... ... [2021-02-12 21:10:58,988] - [basetestcase] [140099333379840] - INFO - sleep for 5 secs. Waiting for bucket_op to deployed... ... [2021-02-12 21:11:04,035] - [basetestcase] [140099333379840] - INFO - sleep for 5 secs. Waiting for bucket_op to deployed... ... [2021-02-12 21:11:09,087] - [basetestcase] [140099333379840] - INFO - sleep for 5 secs. Waiting for bucket_op to deployed... ... [2021-02-12 21:11:15,402] - [basetestcase] [140099333379840] - INFO - sleep for 5 secs. Waiting for timers to deployed... ... [2021-02-12 21:11:20,444] - [basetestcase] [140099333379840] - INFO - sleep for 5 secs. Waiting for timers to deployed... ... [2021-02-12 21:11:25,502] - [basetestcase] [140099333379840] - INFO - sleep for 5 secs. Waiting for timers to deployed... ... [2021-02-12 21:11:30,541] - [basetestcase] [140099333379840] - INFO - sleep for 5 secs. Waiting for timers to deployed... ... [2021-02-12 21:11:36,887] - [basetestcase] [140099333379840] - INFO - sleep for 5 secs. Waiting for sbm to deployed... ... [2021-02-12 21:11:41,938] - [basetestcase] [140099333379840] - INFO - sleep for 5 secs. Waiting for sbm to deployed... ... [2021-02-12 21:11:46,990] - [basetestcase] [140099333379840] - INFO - sleep for 5 secs. Waiting for sbm to deployed... ... [2021-02-12 21:11:52,038] - [basetestcase] [140099333379840] - INFO - sleep for 5 secs. Waiting for sbm to deployed... ... [2021-02-12 21:11:58,279] - [basetestcase] [140099333379840] - INFO - sleep for 5 secs. Waiting for curl to deployed... ... [2021-02-12 21:12:03,323] - [basetestcase] [140099333379840] - INFO - sleep for 5 secs. Waiting for curl to deployed... ... [2021-02-12 21:12:08,374] - [basetestcase] [140099333379840] - INFO - sleep for 5 secs. Waiting for curl to deployed... ... [2021-02-12 21:12:13,414] - [basetestcase] [140099333379840] - INFO - sleep for 5 secs. Waiting for curl to deployed... ... [2021-02-12 21:12:19,688] - [basetestcase] [140099333379840] - INFO - sleep for 5 secs. Waiting for n1ql to deployed... ... [2021-02-12 21:12:24,729] - [basetestcase] [140099333379840] - INFO - sleep for 5 secs. Waiting for n1ql to deployed... ... [2021-02-12 21:12:29,775] - [basetestcase] [140099333379840] - INFO - sleep for 5 secs. Waiting for n1ql to deployed... ... [2021-02-12 21:12:34,816] - [basetestcase] [140099333379840] - INFO - sleep for 5 secs. Waiting for n1ql to deployed... ... [2021-02-12 21:12:40,038] - [task] [140099350165248] - INFO - java -jar java_sdk_client/collections/target/javaclient/javaclient.jar -i 172.23.123.61 -u Administrator -p password -b src_bucket -s _default -c _default -n 40320 -pc 100 -pu 0 -pd 0 -l uniform -dsn 1 -dpx doc_ -dt Person -de 0 -ds 500 -ac False -st 0 -en 40319 -o False -sd False [2021-02-12 21:12:42,870] - [rest_client] [140099333379840] - INFO - rebalance reached >100% in 3.005713939666748 seconds [2021-02-12 21:12:51,204] - [task] [140099350165248] - INFO - java -jar java_sdk_client/collections/target/javaclient/javaclient.jar -i 172.23.123.61 -u Administrator -p password -b src_bucket -s scope_1 -c coll_1 -n 40320 -pc 100 -pu 0 -pd 0 -l uniform -dsn 1 -dpx doc_ -dt Person -de 0 -ds 500 -ac False -st 0 -en 40319 -o False -sd False [2021-02-12 21:13:02,711] - [task] [140099350165248] - INFO - adding node 172.23.122.118:8091 to cluster [2021-02-12 21:13:02,711] - [rest_client] [140099350165248] - INFO - adding remote node @172.23.122.118:8091 to this cluster @172.23.123.61:8091 [2021-02-12 21:13:05,526] - [rest_client] [140099350165248] - INFO - rebalance params : {'knownNodes': 'ns_1@172.23.122.117,ns_1@172.23.122.118,ns_1@172.23.122.82,ns_1@172.23.122.90,ns_1@172.23.122.97,ns_1@172.23.123.61,ns_1@172.23.123.62', 'ejectedNodes': '', 'user': 'Administrator', 'password': 'password'} [2021-02-12 21:13:05,559] - [rest_client] [140099350165248] - INFO - rebalance operation started [2021-02-12 21:13:05,568] - [rest_client] [140099350165248] - INFO - rebalance percentage : 0.00 % [2021-02-12 21:13:05,569] - [task] [140099350165248] - INFO - Rebalance - status: running, progress: 0.00% [2021-02-12 21:13:15,600] - [rest_client] [140099350165248] - INFO - rebalance percentage : 16.00 % [2021-02-12 21:13:15,600] - [task] [140099350165248] - INFO - Rebalance - status: running, progress: 16.00% [2021-02-12 21:13:25,632] - [rest_client] [140099350165248] - INFO - rebalance percentage : 34.00 % [2021-02-12 21:13:25,633] - [task] [140099350165248] - INFO - Rebalance - status: running, progress: 34.00% [2021-02-12 21:13:35,672] - [rest_client] [140099350165248] - INFO - rebalance percentage : 50.00 % [2021-02-12 21:13:35,672] - [task] [140099350165248] - INFO - Rebalance - status: running, progress: 50.00% [2021-02-12 21:13:45,699] - [rest_client] [140099350165248] - INFO - rebalance percentage : 53.00 % [2021-02-12 21:13:45,699] - [task] [140099350165248] - INFO - Rebalance - status: running, progress: 53.00% [2021-02-12 21:13:55,729] - [rest_client] [140099350165248] - INFO - rebalance percentage : 69.00 % [2021-02-12 21:13:55,729] - [task] [140099350165248] - INFO - Rebalance - status: running, progress: 69.00% [2021-02-12 21:14:05,755] - [rest_client] [140099350165248] - INFO - rebalance percentage : 83.00 % [2021-02-12 21:14:05,756] - [task] [140099350165248] - INFO - Rebalance - status: running, progress: 83.00% [2021-02-12 21:14:15,783] - [rest_client] [140099350165248] - INFO - rebalance percentage : 92.00 % [2021-02-12 21:14:15,784] - [task] [140099350165248] - INFO - Rebalance - status: running, progress: 92.00% [2021-02-12 21:14:25,813] - [task] [140099350165248] - INFO - Rebalance - status: none, progress: 100.00% [2021-02-12 21:14:25,827] - [task] [140099350165248] - INFO - rebalancing was completed with progress: 100% in 80.26724863052368 sec [2021-02-12 21:14:25,835] - [basetestcase] [140099333379840] - INFO - list of eventing nodes in cluster: [ip:172.23.122.117 port:8091 ssh_username:root, ip:172.23.122.90 port:8091 ssh_username:root, ip:172.23.122.97 port:8091 ssh_username:root] [2021-02-12 21:14:25,841] - [tuq_helper] [140099333379840] - INFO - RUN QUERY create primary index on dst_bucket.scope_1.coll_0 [2021-02-12 21:14:25,845] - [rest_client] [140099333379840] - INFO - query params : statement=create+primary+index+on+dst_bucket.scope_1.coll_0 [2021-02-12 21:14:28,328] - [tuq_helper] [140099333379840] - INFO - TOTAL ELAPSED TIME: 2.481054786s [2021-02-12 21:14:28,347] - [basetestcase] [140099333379840] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root] [2021-02-12 21:14:28,352] - [rest_client] [140099333379840] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_0 [2021-02-12 21:14:28,367] - [eventing_base] [140099333379840] - INFO - Final docs count... Current : 40320 Expected : 40320 [2021-02-12 21:14:28,378] - [basetestcase] [140099333379840] - INFO - list of eventing nodes in cluster: [ip:172.23.122.117 port:8091 ssh_username:root, ip:172.23.122.90 port:8091 ssh_username:root, ip:172.23.122.97 port:8091 ssh_username:root] [2021-02-12 21:14:28,382] - [tuq_helper] [140099333379840] - INFO - RUN QUERY create primary index on dst_bucket.scope_1.coll_1 [2021-02-12 21:14:28,387] - [rest_client] [140099333379840] - INFO - query params : statement=create+primary+index+on+dst_bucket.scope_1.coll_1 [2021-02-12 21:14:31,446] - [tuq_helper] [140099333379840] - INFO - TOTAL ELAPSED TIME: 3.057367008s [2021-02-12 21:14:31,468] - [basetestcase] [140099333379840] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root] [2021-02-12 21:14:31,474] - [rest_client] [140099333379840] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_1 [2021-02-12 21:14:31,485] - [eventing_base] [140099333379840] - INFO - Final docs count... Current : 40320 Expected : 40320 [2021-02-12 21:14:31,494] - [basetestcase] [140099333379840] - INFO - list of eventing nodes in cluster: [ip:172.23.122.117 port:8091 ssh_username:root, ip:172.23.122.90 port:8091 ssh_username:root, ip:172.23.122.97 port:8091 ssh_username:root] [2021-02-12 21:14:31,499] - [tuq_helper] [140099333379840] - INFO - RUN QUERY create primary index on dst_bucket.scope_1.coll_3 [2021-02-12 21:14:31,504] - [rest_client] [140099333379840] - INFO - query params : statement=create+primary+index+on+dst_bucket.scope_1.coll_3 [2021-02-12 21:14:34,381] - [tuq_helper] [140099333379840] - INFO - TOTAL ELAPSED TIME: 2.875408802s [2021-02-12 21:14:34,400] - [basetestcase] [140099333379840] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root] [2021-02-12 21:14:34,407] - [rest_client] [140099333379840] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_3 [2021-02-12 21:14:34,420] - [eventing_base] [140099333379840] - INFO - Final docs count... Current : 40320 Expected : 40320 [2021-02-12 21:14:34,428] - [basetestcase] [140099333379840] - INFO - list of eventing nodes in cluster: [ip:172.23.122.117 port:8091 ssh_username:root, ip:172.23.122.90 port:8091 ssh_username:root, ip:172.23.122.97 port:8091 ssh_username:root] [2021-02-12 21:14:34,433] - [tuq_helper] [140099333379840] - INFO - RUN QUERY create primary index on dst_bucket.scope_1.coll_4 [2021-02-12 21:14:34,437] - [rest_client] [140099333379840] - INFO - query params : statement=create+primary+index+on+dst_bucket.scope_1.coll_4 [2021-02-12 21:14:37,268] - [tuq_helper] [140099333379840] - INFO - TOTAL ELAPSED TIME: 2.829600687s [2021-02-12 21:14:37,291] - [basetestcase] [140099333379840] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root] [2021-02-12 21:14:37,296] - [rest_client] [140099333379840] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-12 21:14:37,306] - [basetestcase] [140099333379840] - INFO - sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_4 to complete bucket operations... Current : 40318 Expected : 40320 ... [2021-02-12 21:15:07,357] - [basetestcase] [140099333379840] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root] [2021-02-12 21:15:07,364] - [rest_client] [140099333379840] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-12 21:15:07,382] - [basetestcase] [140099333379840] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root] [2021-02-12 21:15:07,388] - [rest_client] [140099333379840] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-12 21:15:07,391] - [basetestcase] [140099333379840] - INFO - sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_4 to complete bucket operations... Current : 40318 Expected : 40320 ... [2021-02-12 21:15:37,442] - [basetestcase] [140099333379840] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root] [2021-02-12 21:15:37,449] - [rest_client] [140099333379840] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-12 21:15:37,468] - [basetestcase] [140099333379840] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root] [2021-02-12 21:15:37,473] - [rest_client] [140099333379840] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-12 21:15:37,476] - [basetestcase] [140099333379840] - INFO - sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_4 to complete bucket operations... Current : 40318 Expected : 40320 ... [2021-02-12 21:16:07,526] - [basetestcase] [140099333379840] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root] [2021-02-12 21:16:07,532] - [rest_client] [140099333379840] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-12 21:16:07,551] - [basetestcase] [140099333379840] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root] [2021-02-12 21:16:07,556] - [rest_client] [140099333379840] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-12 21:16:07,559] - [basetestcase] [140099333379840] - INFO - sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_4 to complete bucket operations... Current : 40318 Expected : 40320 ... [2021-02-12 21:16:37,609] - [basetestcase] [140099333379840] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root] [2021-02-12 21:16:37,616] - [rest_client] [140099333379840] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-12 21:16:37,635] - [basetestcase] [140099333379840] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root] [2021-02-12 21:16:37,640] - [rest_client] [140099333379840] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-12 21:16:37,643] - [basetestcase] [140099333379840] - INFO - sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_4 to complete bucket operations... Current : 40318 Expected : 40320 ... [2021-02-12 21:17:07,674] - [basetestcase] [140099333379840] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root] [2021-02-12 21:17:07,680] - [rest_client] [140099333379840] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-12 21:17:07,698] - [basetestcase] [140099333379840] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root] [2021-02-12 21:17:07,703] - [rest_client] [140099333379840] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-12 21:17:07,706] - [basetestcase] [140099333379840] - INFO - sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_4 to complete bucket operations... Current : 40318 Expected : 40320 ... [2021-02-12 21:17:37,756] - [basetestcase] [140099333379840] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root] [2021-02-12 21:17:37,764] - [rest_client] [140099333379840] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-12 21:17:37,783] - [basetestcase] [140099333379840] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root] [2021-02-12 21:17:37,790] - [rest_client] [140099333379840] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-12 21:17:37,793] - [basetestcase] [140099333379840] - INFO - sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_4 to complete bucket operations... Current : 40318 Expected : 40320 ... [2021-02-12 21:18:07,829] - [basetestcase] [140099333379840] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root] [2021-02-12 21:18:07,836] - [rest_client] [140099333379840] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-12 21:18:07,857] - [basetestcase] [140099333379840] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root] [2021-02-12 21:18:07,862] - [rest_client] [140099333379840] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-12 21:18:07,866] - [basetestcase] [140099333379840] - INFO - sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_4 to complete bucket operations... Current : 40318 Expected : 40320 ... [2021-02-12 21:18:37,913] - [basetestcase] [140099333379840] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root] [2021-02-12 21:18:37,919] - [rest_client] [140099333379840] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-12 21:18:37,938] - [basetestcase] [140099333379840] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root] [2021-02-12 21:18:37,942] - [rest_client] [140099333379840] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-12 21:18:37,945] - [basetestcase] [140099333379840] - INFO - sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_4 to complete bucket operations... Current : 40318 Expected : 40320 ... [2021-02-12 21:19:08,001] - [basetestcase] [140099333379840] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root] [2021-02-12 21:19:08,008] - [rest_client] [140099333379840] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-12 21:19:08,029] - [basetestcase] [140099333379840] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root] [2021-02-12 21:19:08,034] - [rest_client] [140099333379840] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-12 21:19:08,037] - [basetestcase] [140099333379840] - INFO - sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_4 to complete bucket operations... Current : 40318 Expected : 40320 ... [2021-02-12 21:19:38,093] - [basetestcase] [140099333379840] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root] [2021-02-12 21:19:38,101] - [rest_client] [140099333379840] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-12 21:19:38,121] - [basetestcase] [140099333379840] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root] [2021-02-12 21:19:38,126] - [rest_client] [140099333379840] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-12 21:19:38,129] - [basetestcase] [140099333379840] - INFO - sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_4 to complete bucket operations... Current : 40318 Expected : 40320 ... [2021-02-12 21:20:08,187] - [basetestcase] [140099333379840] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root] [2021-02-12 21:20:08,194] - [rest_client] [140099333379840] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-12 21:20:08,210] - [eventing_base] [140099333379840] - INFO - Compacting bucket : dst_bucket [2021-02-12 21:20:08,219] - [rest_client] [140099333379840] - INFO - bucket compaction successful [2021-02-12 21:20:08,219] - [eventing_base] [140099333379840] - INFO - Compacting bucket : metadata [2021-02-12 21:20:08,224] - [rest_client] [140099333379840] - INFO - bucket compaction successful [2021-02-12 21:20:08,225] - [eventing_base] [140099333379840] - INFO - Compacting bucket : src_bucket [2021-02-12 21:20:08,230] - [rest_client] [140099333379840] - INFO - bucket compaction successful [2021-02-12 21:20:08,242] - [basetestcase] [140099333379840] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root] [2021-02-12 21:20:08,248] - [rest_client] [140099333379840] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-12 21:20:08,251] - [basetestcase] [140099333379840] - INFO - sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_4 to complete bucket operations... Current : 40318 Expected : 40320 ... [2021-02-12 21:20:38,304] - [basetestcase] [140099333379840] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root] [2021-02-12 21:20:38,311] - [rest_client] [140099333379840] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-12 21:20:38,331] - [basetestcase] [140099333379840] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root] [2021-02-12 21:20:38,336] - [rest_client] [140099333379840] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-12 21:20:38,339] - [basetestcase] [140099333379840] - INFO - sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_4 to complete bucket operations... Current : 40318 Expected : 40320 ... [2021-02-12 21:21:08,386] - [basetestcase] [140099333379840] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root] [2021-02-12 21:21:08,393] - [rest_client] [140099333379840] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-12 21:21:08,414] - [basetestcase] [140099333379840] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root] [2021-02-12 21:21:08,421] - [rest_client] [140099333379840] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-12 21:21:08,424] - [basetestcase] [140099333379840] - INFO - sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_4 to complete bucket operations... Current : 40318 Expected : 40320 ... [2021-02-12 21:21:38,460] - [basetestcase] [140099333379840] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root] [2021-02-12 21:21:38,466] - [rest_client] [140099333379840] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-12 21:21:38,488] - [basetestcase] [140099333379840] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root] [2021-02-12 21:21:38,493] - [rest_client] [140099333379840] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-12 21:21:38,499] - [basetestcase] [140099333379840] - INFO - sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_4 to complete bucket operations... Current : 40318 Expected : 40320 ... [2021-02-12 21:22:08,552] - [basetestcase] [140099333379840] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root] [2021-02-12 21:22:08,558] - [rest_client] [140099333379840] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-12 21:22:08,577] - [basetestcase] [140099333379840] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root] [2021-02-12 21:22:08,583] - [rest_client] [140099333379840] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-12 21:22:08,586] - [basetestcase] [140099333379840] - INFO - sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_4 to complete bucket operations... Current : 40318 Expected : 40320 ... [2021-02-12 21:22:38,638] - [basetestcase] [140099333379840] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root] [2021-02-12 21:22:38,645] - [rest_client] [140099333379840] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-12 21:22:38,664] - [basetestcase] [140099333379840] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root] [2021-02-12 21:22:38,669] - [rest_client] [140099333379840] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-12 21:22:38,672] - [basetestcase] [140099333379840] - INFO - sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_4 to complete bucket operations... Current : 40318 Expected : 40320 ... [2021-02-12 21:23:08,726] - [basetestcase] [140099333379840] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root] [2021-02-12 21:23:08,732] - [rest_client] [140099333379840] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-12 21:23:08,753] - [basetestcase] [140099333379840] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root] [2021-02-12 21:23:08,759] - [rest_client] [140099333379840] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-12 21:23:08,761] - [basetestcase] [140099333379840] - INFO - sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_4 to complete bucket operations... Current : 40318 Expected : 40320 ... [2021-02-12 21:23:38,815] - [basetestcase] [140099333379840] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root] [2021-02-12 21:23:38,823] - [rest_client] [140099333379840] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-12 21:23:38,844] - [basetestcase] [140099333379840] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root] [2021-02-12 21:23:38,850] - [rest_client] [140099333379840] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-12 21:23:38,852] - [basetestcase] [140099333379840] - INFO - sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_4 to complete bucket operations... Current : 40318 Expected : 40320 ... [2021-02-12 21:24:08,910] - [basetestcase] [140099333379840] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root] [2021-02-12 21:24:08,916] - [rest_client] [140099333379840] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-12 21:24:08,936] - [basetestcase] [140099333379840] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root] [2021-02-12 21:24:08,941] - [rest_client] [140099333379840] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-12 21:24:08,944] - [basetestcase] [140099333379840] - INFO - sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_4 to complete bucket operations... Current : 40318 Expected : 40320 ... [2021-02-12 21:24:38,998] - [basetestcase] [140099333379840] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root] [2021-02-12 21:24:39,005] - [rest_client] [140099333379840] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-12 21:24:39,025] - [basetestcase] [140099333379840] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root] [2021-02-12 21:24:39,029] - [rest_client] [140099333379840] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-12 21:24:39,039] - [basetestcase] [140099333379840] - INFO - list of eventing nodes in cluster: [ip:172.23.122.117 port:8091 ssh_username:root, ip:172.23.122.90 port:8091 ssh_username:root, ip:172.23.122.97 port:8091 ssh_username:root] [2021-02-12 21:24:39,098] - [eventing_base] [140099333379840] - INFO - Stats for Node 172.23.122.117 is [ { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 2, "agg_messages_sent_to_worker": 18612, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 66981, "dcp_mutation_sent_to_worker": 13499, "dcp_snapshot": 40448, "dcp_stream_req_counter": 684, "dcp_streamend": 342, "dcp_streamreq": 684, "dcp_system_event": 3420, "execution_stats": 815, "failure_stats": 815, "latency_stats": 815, "lcb_exception_stats": 815, "log_level": 1, "num_processed_events": 13499, "processed_events_size": 12082156, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13499, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13499, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 18608, "num_processed_events": 13499, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13499, "processed_events_size": 12082156, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "124232": "2021-02-13T05:24:39Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "124232": "2021-02-13T05:24:39Z" }, "v8worker_events_lost": 0 }, "function_id": 1271581675, "function_name": "bucket_op", "gocb_creds_request_counter": 67, "internal_vb_distribution_stats": { "worker_bucket_op_0": "[0-341]" }, "latency_percentile_stats": { "100": 22400, "50": 700, "80": 1000, "90": 1300, "95": 1600, "99": 2700 }, "lcb_creds_request_counter": 27060, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.90:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.97:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_bucket_op_0": "[0-341]" }, "172.23.122.90:8096": { "worker_bucket_op_0": "[342-682]" }, "172.23.122.97:8096": { "worker_bucket_op_0": "[683-1023]" } }, "worker_pids": { "worker_bucket_op_0": 124232 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 2, "agg_messages_sent_to_worker": 18501, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 66981, "dcp_mutation_sent_to_worker": 13499, "dcp_snapshot": 40421, "dcp_stream_req_counter": 684, "dcp_streamend": 342, "dcp_streamreq": 684, "dcp_system_event": 3420, "execution_stats": 793, "failure_stats": 793, "latency_stats": 793, "lcb_exception_stats": 793, "log_level": 1, "num_processed_events": 13499, "processed_events_size": 12082156, "thr_count": 1, "thr_map": 1, "timer_events": 13528, "timer_responses_received": 13499, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13499, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13499, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 18497, "num_processed_events": 13499, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13499, "processed_events_size": 12082156, "timer_callback_failure": 0, "timer_callback_success": 13528, "timer_cancel_counter": 0, "timer_create_counter": 13499, "timer_create_failure": 0, "timer_msg_counter": 13528, "timer_responses_sent": 0, "timestamp": { "124274": "2021-02-13T05:24:38Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "124274": "2021-02-13T05:24:38Z" }, "v8worker_events_lost": 0 }, "function_id": 506151983, "function_name": "timers", "gocb_creds_request_counter": 67, "internal_vb_distribution_stats": { "worker_timers_0": "[0-341]" }, "latency_percentile_stats": { "100": 24100, "50": 1900, "80": 2600, "90": 3200, "95": 3900, "99": 6600 }, "lcb_creds_request_counter": 27060, "lcb_exception_stats": {}, "metastore_stats": { "metastore_delete_err": 0, "metastore_deletes": 0, "metastore_not_found": 0, "metastore_scan": 0, "metastore_scan_due": 0, "metastore_scan_err": 0, "metastore_set": 0, "metastore_set_err": 0 }, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.90:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.97:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_timers_0": "[0-341]" }, "172.23.122.90:8096": { "worker_timers_0": "[342-682]" }, "172.23.122.97:8096": { "worker_timers_0": "[683-1023]" } }, "worker_pids": { "worker_timers_0": 124274 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "agg_messages_sent_to_worker": 18395, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 40830, "dcp_mutation_sent_to_worker": 13499, "dcp_mutation_suppressed_counter": 13832, "dcp_snapshot": 40335, "dcp_stream_req_counter": 685, "dcp_stream_req_err_counter": 1, "dcp_streamend": 342, "dcp_streamreq": 684, "dcp_system_event": 3420, "execution_stats": 772, "failure_stats": 772, "latency_stats": 772, "lcb_exception_stats": 772, "log_level": 1, "num_processed_events": 13499, "processed_events_size": 12086172, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13499, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13499, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 18391, "num_processed_events": 13499, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13499, "processed_events_size": 12086172, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "124359": "2021-02-13T05:24:38Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "124359": "2021-02-13T05:24:38Z" }, "v8worker_events_lost": 0 }, "function_id": 3702549362, "function_name": "sbm", "gocb_creds_request_counter": 67, "internal_vb_distribution_stats": { "worker_sbm_0": "[0-341]" }, "latency_percentile_stats": { "100": 16400, "50": 800, "80": 1200, "90": 1700, "95": 2300, "99": 4700 }, "lcb_creds_request_counter": 27060, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.90:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.97:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_sbm_0": "[0-341]" }, "172.23.122.90:8096": { "worker_sbm_0": "[342-682]" }, "172.23.122.97:8096": { "worker_sbm_0": "[683-1023]" } }, "worker_pids": { "worker_sbm_0": 124359 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 2, "agg_messages_sent_to_worker": 17602, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 40331, "dcp_mutation_sent_to_worker": 13499, "dcp_snapshot": 40057, "dcp_stream_req_counter": 344, "dcp_streamend": 1, "dcp_streamreq": 344, "dcp_system_event": 3420, "execution_stats": 750, "failure_stats": 750, "latency_stats": 750, "lcb_exception_stats": 750, "log_level": 1, "num_processed_events": 13499, "processed_events_size": 12082156, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 13499, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 13499, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13499, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13499, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 17598, "num_processed_events": 13499, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13499, "processed_events_size": 12082156, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "124401": "2021-02-13T05:24:38Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "124401": "2021-02-13T05:24:38Z" }, "v8worker_events_lost": 0 }, "function_id": 3687844484, "function_name": "curl", "gocb_creds_request_counter": 67, "internal_vb_distribution_stats": { "worker_curl_0": "[0-341]" }, "latency_percentile_stats": { "100": 534100, "50": 1500, "80": 2000, "90": 2400, "95": 2900, "99": 4800 }, "lcb_creds_request_counter": 27060, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.90:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.97:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_curl_0": "[0-341]" }, "172.23.122.90:8096": { "worker_curl_0": "[342-682]" }, "172.23.122.97:8096": { "worker_curl_0": "[683-1023]" } }, "worker_pids": { "worker_curl_0": 124401 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 3, "agg_messages_sent_to_worker": 17496, "agg_queue_memory": 0, "agg_queue_memory_cap": 20971520, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 40331, "dcp_mutation_sent_to_worker": 13499, "dcp_snapshot": 40049, "dcp_stream_req_counter": 343, "dcp_streamend": 1, "dcp_streamreq": 343, "dcp_system_event": 3420, "execution_stats": 729, "failure_stats": 729, "latency_stats": 729, "lcb_exception_stats": 729, "log_level": 1, "num_processed_events": 13499, "processed_events_size": 12082156, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13499, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13499, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 17492, "num_processed_events": 13499, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13499, "processed_events_size": 12082156, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "124483": "2021-02-13T05:24:38Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "124483": "2021-02-13T05:24:38Z" }, "v8worker_events_lost": 0 }, "function_id": 1890800864, "function_name": "n1ql", "gocb_creds_request_counter": 67, "internal_vb_distribution_stats": { "worker_n1ql_0": "[0-341]" }, "latency_percentile_stats": { "100": 96500, "50": 3200, "80": 4300, "90": 5300, "95": 6700, "99": 11200 }, "lcb_creds_request_counter": 27060, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.90:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.97:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_n1ql_0": "[0-341]" }, "172.23.122.90:8096": { "worker_n1ql_0": "[342-682]" }, "172.23.122.97:8096": { "worker_n1ql_0": "[683-1023]" } }, "worker_pids": { "worker_n1ql_0": 124483 } } ] [2021-02-12 21:24:39,150] - [eventing_base] [140099333379840] - INFO - Stats for Node 172.23.122.90 is [ { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 2, "agg_messages_sent_to_worker": 17872, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 54088, "dcp_mutation_sent_to_worker": 13432, "dcp_snapshot": 40569, "dcp_stream_req_counter": 1392, "dcp_stream_req_err_counter": 837, "dcp_streamend": 170, "dcp_streamreq": 555, "dcp_system_event": 3410, "execution_stats": 750, "failure_stats": 750, "latency_stats": 750, "lcb_exception_stats": 750, "log_level": 1, "num_processed_events": 13432, "processed_events_size": 12023960, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 13432, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 13432, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13432, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13432, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 17868, "num_processed_events": 13432, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13432, "processed_events_size": 12023960, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "47426": "2021-02-13T05:24:38Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "47426": "2021-02-13T05:24:38Z" }, "v8worker_events_lost": 0 }, "function_id": 3687844484, "function_name": "curl", "gocb_creds_request_counter": 58, "internal_vb_distribution_stats": { "worker_curl_0": "[342-682]" }, "latency_percentile_stats": { "100": 534700, "50": 1600, "80": 2000, "90": 2400, "95": 3000, "99": 5300 }, "lcb_creds_request_counter": 26931, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.90:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.97:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_curl_0": "[0-341]" }, "172.23.122.90:8096": { "worker_curl_0": "[342-682]" }, "172.23.122.97:8096": { "worker_curl_0": "[683-1023]" } }, "worker_pids": { "worker_curl_0": 47426 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 2, "agg_messages_sent_to_worker": 17766, "agg_queue_memory": 0, "agg_queue_memory_cap": 20971520, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 54088, "dcp_mutation_sent_to_worker": 13432, "dcp_snapshot": 40540, "dcp_stream_req_counter": 1021, "dcp_stream_req_err_counter": 453, "dcp_streamend": 170, "dcp_streamreq": 568, "dcp_system_event": 3410, "execution_stats": 729, "failure_stats": 729, "latency_stats": 729, "lcb_exception_stats": 729, "log_level": 1, "num_processed_events": 13432, "processed_events_size": 12023960, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13432, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13432, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 17762, "num_processed_events": 13432, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 1, "on_update_success": 13431, "processed_events_size": 12023960, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "47458": "2021-02-13T05:24:38Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 1, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "47458": "2021-02-13T05:24:38Z" }, "v8worker_events_lost": 0 }, "function_id": 1890800864, "function_name": "n1ql", "gocb_creds_request_counter": 58, "internal_vb_distribution_stats": { "worker_n1ql_0": "[342-682]" }, "latency_percentile_stats": { "100": 142000, "50": 3400, "80": 4700, "90": 5800, "95": 7100, "99": 11800 }, "lcb_creds_request_counter": 26931, "lcb_exception_stats": { "12009": 1, "209": 1 }, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.90:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.97:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_n1ql_0": "[0-341]" }, "172.23.122.90:8096": { "worker_n1ql_0": "[342-682]" }, "172.23.122.97:8096": { "worker_n1ql_0": "[683-1023]" } }, "worker_pids": { "worker_n1ql_0": 47458 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 2, "agg_messages_sent_to_worker": 18600, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 67638, "dcp_mutation_sent_to_worker": 13432, "dcp_snapshot": 40678, "dcp_stream_req_counter": 822, "dcp_stream_req_err_counter": 6, "dcp_streamend": 399, "dcp_streamreq": 816, "dcp_system_event": 3410, "execution_stats": 815, "failure_stats": 815, "latency_stats": 815, "lcb_exception_stats": 815, "log_level": 1, "num_processed_events": 13432, "processed_events_size": 12023960, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13432, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13432, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 18596, "num_processed_events": 13432, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13432, "processed_events_size": 12023960, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "47216": "2021-02-13T05:24:39Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "47216": "2021-02-13T05:24:39Z" }, "v8worker_events_lost": 0 }, "function_id": 1271581675, "function_name": "bucket_op", "gocb_creds_request_counter": 58, "internal_vb_distribution_stats": { "worker_bucket_op_0": "[342-682]" }, "latency_percentile_stats": { "100": 26400, "50": 700, "80": 1000, "90": 1300, "95": 1700, "99": 3000 }, "lcb_creds_request_counter": 26931, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.90:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.97:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_bucket_op_0": "[0-341]" }, "172.23.122.90:8096": { "worker_bucket_op_0": "[342-682]" }, "172.23.122.97:8096": { "worker_bucket_op_0": "[683-1023]" } }, "worker_pids": { "worker_bucket_op_0": 47216 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 1, "agg_messages_sent_to_worker": 18485, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 67638, "dcp_mutation_sent_to_worker": 13432, "dcp_snapshot": 40655, "dcp_stream_req_counter": 834, "dcp_stream_req_err_counter": 6, "dcp_streamend": 395, "dcp_streamreq": 828, "dcp_system_event": 3410, "execution_stats": 793, "failure_stats": 793, "latency_stats": 793, "lcb_exception_stats": 793, "log_level": 1, "num_processed_events": 13432, "processed_events_size": 12023960, "thr_count": 1, "thr_map": 1, "timer_events": 13558, "timer_responses_received": 13432, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13432, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13432, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 18481, "num_processed_events": 13432, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13432, "processed_events_size": 12023960, "timer_callback_failure": 0, "timer_callback_success": 13558, "timer_cancel_counter": 0, "timer_create_counter": 13432, "timer_create_failure": 0, "timer_msg_counter": 13558, "timer_responses_sent": 0, "timestamp": { "47249": "2021-02-13T05:24:38Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "47249": "2021-02-13T05:24:38Z" }, "v8worker_events_lost": 0 }, "function_id": 506151983, "function_name": "timers", "gocb_creds_request_counter": 58, "internal_vb_distribution_stats": { "worker_timers_0": "[342-682]" }, "latency_percentile_stats": { "100": 26200, "50": 1900, "80": 2900, "90": 3700, "95": 4500, "99": 7200 }, "lcb_creds_request_counter": 26931, "lcb_exception_stats": {}, "metastore_stats": { "metastore_delete_err": 0, "metastore_deletes": 0, "metastore_not_found": 0, "metastore_scan": 0, "metastore_scan_due": 0, "metastore_scan_err": 0, "metastore_set": 0, "metastore_set_err": 0 }, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.90:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.97:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_timers_0": "[0-341]" }, "172.23.122.90:8096": { "worker_timers_0": "[342-682]" }, "172.23.122.97:8096": { "worker_timers_0": "[683-1023]" } }, "worker_pids": { "worker_timers_0": 47249 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "agg_messages_sent_to_worker": 18366, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 41156, "dcp_mutation_sent_to_worker": 13432, "dcp_mutation_suppressed_counter": 14292, "dcp_snapshot": 40593, "dcp_stream_req_counter": 848, "dcp_stream_req_err_counter": 6, "dcp_streamend": 382, "dcp_streamreq": 842, "dcp_system_event": 3410, "execution_stats": 772, "failure_stats": 772, "latency_stats": 772, "lcb_exception_stats": 772, "log_level": 1, "num_processed_events": 13432, "processed_events_size": 12029572, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13432, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13432, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 18362, "num_processed_events": 13432, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13432, "processed_events_size": 12029572, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "47363": "2021-02-13T05:24:39Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "47363": "2021-02-13T05:24:39Z" }, "v8worker_events_lost": 0 }, "function_id": 3702549362, "function_name": "sbm", "gocb_creds_request_counter": 58, "internal_vb_distribution_stats": { "worker_sbm_0": "[342-682]" }, "latency_percentile_stats": { "100": 17300, "50": 800, "80": 1300, "90": 1800, "95": 2300, "99": 4800 }, "lcb_creds_request_counter": 26931, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.90:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.97:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_sbm_0": "[0-341]" }, "172.23.122.90:8096": { "worker_sbm_0": "[342-682]" }, "172.23.122.97:8096": { "worker_sbm_0": "[683-1023]" } }, "worker_pids": { "worker_sbm_0": 47363 } } ] [2021-02-12 21:24:39,198] - [eventing_base] [140099333379840] - INFO - Stats for Node 172.23.122.97 is [ { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 3, "agg_messages_sent_to_worker": 18548, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 66981, "dcp_mutation_sent_to_worker": 13389, "dcp_snapshot": 40426, "dcp_stream_req_counter": 822, "dcp_stream_req_err_counter": 6, "dcp_streamend": 395, "dcp_streamreq": 816, "dcp_system_event": 3410, "execution_stats": 814, "failure_stats": 814, "latency_stats": 814, "lcb_exception_stats": 814, "log_level": 1, "num_processed_events": 13389, "processed_events_size": 11986976, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13389, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13389, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 18544, "num_processed_events": 13389, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13389, "processed_events_size": 11986976, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "26680": "2021-02-13T05:24:38Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "26680": "2021-02-13T05:24:38Z" }, "v8worker_events_lost": 0 }, "function_id": 1271581675, "function_name": "bucket_op", "gocb_creds_request_counter": 56, "internal_vb_distribution_stats": { "worker_bucket_op_0": "[683-1023]" }, "latency_percentile_stats": { "100": 36400, "50": 700, "80": 1100, "90": 1500, "95": 1900, "99": 3300 }, "lcb_creds_request_counter": 26852, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.90:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.97:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_bucket_op_0": "[0-341]" }, "172.23.122.90:8096": { "worker_bucket_op_0": "[342-682]" }, "172.23.122.97:8096": { "worker_bucket_op_0": "[683-1023]" } }, "worker_pids": { "worker_bucket_op_0": 26680 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 3, "agg_messages_sent_to_worker": 18438, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 66981, "dcp_mutation_sent_to_worker": 13389, "dcp_snapshot": 40394, "dcp_stream_req_counter": 833, "dcp_stream_req_err_counter": 7, "dcp_streamend": 391, "dcp_streamreq": 826, "dcp_system_event": 3410, "execution_stats": 793, "failure_stats": 793, "latency_stats": 793, "lcb_exception_stats": 793, "log_level": 1, "num_processed_events": 13389, "processed_events_size": 11986976, "thr_count": 1, "thr_map": 1, "timer_events": 13234, "timer_responses_received": 13389, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13389, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13389, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 18434, "num_processed_events": 13389, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13389, "processed_events_size": 11986976, "timer_callback_failure": 0, "timer_callback_success": 13234, "timer_cancel_counter": 0, "timer_create_counter": 13389, "timer_create_failure": 0, "timer_msg_counter": 13234, "timer_responses_sent": 0, "timestamp": { "26712": "2021-02-13T05:24:38Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "26712": "2021-02-13T05:24:38Z" }, "v8worker_events_lost": 0 }, "function_id": 506151983, "function_name": "timers", "gocb_creds_request_counter": 56, "internal_vb_distribution_stats": { "worker_timers_0": "[683-1023]" }, "latency_percentile_stats": { "100": 22000, "50": 2000, "80": 2900, "90": 3600, "95": 4500, "99": 7300 }, "lcb_creds_request_counter": 26852, "lcb_exception_stats": {}, "metastore_stats": { "metastore_delete_err": 0, "metastore_deletes": 0, "metastore_not_found": 0, "metastore_scan": 0, "metastore_scan_due": 0, "metastore_scan_err": 0, "metastore_set": 0, "metastore_set_err": 0 }, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.90:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.97:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_timers_0": "[0-341]" }, "172.23.122.90:8096": { "worker_timers_0": "[342-682]" }, "172.23.122.97:8096": { "worker_timers_0": "[683-1023]" } }, "worker_pids": { "worker_timers_0": 26712 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "agg_messages_sent_to_worker": 18322, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 40830, "dcp_mutation_sent_to_worker": 13389, "dcp_mutation_suppressed_counter": 14052, "dcp_snapshot": 40245, "dcp_stream_req_counter": 845, "dcp_stream_req_err_counter": 7, "dcp_streamend": 381, "dcp_streamreq": 838, "dcp_system_event": 3410, "execution_stats": 772, "failure_stats": 772, "latency_stats": 772, "lcb_exception_stats": 772, "log_level": 1, "num_processed_events": 13389, "processed_events_size": 11987720, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13389, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13389, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 18318, "num_processed_events": 13389, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13389, "processed_events_size": 11987720, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "26828": "2021-02-13T05:24:39Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "26828": "2021-02-13T05:24:39Z" }, "v8worker_events_lost": 0 }, "function_id": 3702549362, "function_name": "sbm", "gocb_creds_request_counter": 56, "internal_vb_distribution_stats": { "worker_sbm_0": "[683-1023]" }, "latency_percentile_stats": { "100": 21200, "50": 900, "80": 1400, "90": 1900, "95": 2500, "99": 5100 }, "lcb_creds_request_counter": 26852, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.90:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.97:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_sbm_0": "[0-341]" }, "172.23.122.90:8096": { "worker_sbm_0": "[342-682]" }, "172.23.122.97:8096": { "worker_sbm_0": "[683-1023]" } }, "worker_pids": { "worker_sbm_0": 26828 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 1, "agg_messages_sent_to_worker": 17831, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 53672, "dcp_mutation_sent_to_worker": 13389, "dcp_snapshot": 40104, "dcp_stream_req_counter": 1548, "dcp_stream_req_err_counter": 988, "dcp_streamend": 171, "dcp_streamreq": 560, "dcp_system_event": 3410, "execution_stats": 750, "failure_stats": 750, "latency_stats": 750, "lcb_exception_stats": 750, "log_level": 1, "num_processed_events": 13389, "processed_events_size": 11986976, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 13389, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 13389, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13389, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13389, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 17827, "num_processed_events": 13389, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13389, "processed_events_size": 11986976, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "26892": "2021-02-13T05:24:38Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "26892": "2021-02-13T05:24:38Z" }, "v8worker_events_lost": 0 }, "function_id": 3687844484, "function_name": "curl", "gocb_creds_request_counter": 56, "internal_vb_distribution_stats": { "worker_curl_0": "[683-1023]" }, "latency_percentile_stats": { "100": 534800, "50": 1600, "80": 2100, "90": 2500, "95": 2900, "99": 4700 }, "lcb_creds_request_counter": 26852, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.90:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.97:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_curl_0": "[0-341]" }, "172.23.122.90:8096": { "worker_curl_0": "[342-682]" }, "172.23.122.97:8096": { "worker_curl_0": "[683-1023]" } }, "worker_pids": { "worker_curl_0": 26892 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 1, "agg_messages_sent_to_worker": 17725, "agg_queue_memory": 0, "agg_queue_memory_cap": 20971520, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 53672, "dcp_mutation_sent_to_worker": 13389, "dcp_snapshot": 40129, "dcp_stream_req_counter": 1180, "dcp_stream_req_err_counter": 612, "dcp_streamend": 171, "dcp_streamreq": 568, "dcp_system_event": 3410, "execution_stats": 729, "failure_stats": 729, "latency_stats": 729, "lcb_exception_stats": 729, "log_level": 1, "num_processed_events": 13389, "processed_events_size": 11986976, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13389, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13389, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 17721, "num_processed_events": 13389, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 1, "on_update_success": 13388, "processed_events_size": 11986976, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "26924": "2021-02-13T05:24:38Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 1, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "26924": "2021-02-13T05:24:38Z" }, "v8worker_events_lost": 0 }, "function_id": 1890800864, "function_name": "n1ql", "gocb_creds_request_counter": 56, "internal_vb_distribution_stats": { "worker_n1ql_0": "[683-1023]" }, "latency_percentile_stats": { "100": 134200, "50": 3400, "80": 4700, "90": 5900, "95": 7400, "99": 12100 }, "lcb_creds_request_counter": 26852, "lcb_exception_stats": { "12009": 1, "209": 1 }, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.90:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.97:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_n1ql_0": "[0-341]" }, "172.23.122.90:8096": { "worker_n1ql_0": "[342-682]" }, "172.23.122.97:8096": { "worker_n1ql_0": "[683-1023]" } }, "worker_pids": { "worker_n1ql_0": 26924 } } ] [2021-02-12 21:24:43,659] - [eventing_base] [140099333379840] - INFO - Stats for Node 172.23.122.117 is [ { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 2, "agg_messages_sent_to_worker": 18612, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 66981, "dcp_mutation_sent_to_worker": 13499, "dcp_snapshot": 40448, "dcp_stream_req_counter": 684, "dcp_streamend": 342, "dcp_streamreq": 684, "dcp_system_event": 3420, "execution_stats": 815, "failure_stats": 815, "latency_stats": 815, "lcb_exception_stats": 815, "log_level": 1, "num_processed_events": 13499, "processed_events_size": 12082156, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13499, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13499, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 18608, "num_processed_events": 13499, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13499, "processed_events_size": 12082156, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "124232": "2021-02-13T05:24:39Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "124232": "2021-02-13T05:24:39Z" }, "v8worker_events_lost": 0 }, "function_id": 1271581675, "function_name": "bucket_op", "gocb_creds_request_counter": 67, "internal_vb_distribution_stats": { "worker_bucket_op_0": "[0-341]" }, "latency_percentile_stats": { "100": 22400, "50": 700, "80": 1000, "90": 1300, "95": 1600, "99": 2700 }, "lcb_creds_request_counter": 27060, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.90:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.97:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_bucket_op_0": "[0-341]" }, "172.23.122.90:8096": { "worker_bucket_op_0": "[342-682]" }, "172.23.122.97:8096": { "worker_bucket_op_0": "[683-1023]" } }, "worker_pids": { "worker_bucket_op_0": 124232 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 2, "agg_messages_sent_to_worker": 18501, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 66981, "dcp_mutation_sent_to_worker": 13499, "dcp_snapshot": 40421, "dcp_stream_req_counter": 684, "dcp_streamend": 342, "dcp_streamreq": 684, "dcp_system_event": 3420, "execution_stats": 793, "failure_stats": 793, "latency_stats": 793, "lcb_exception_stats": 793, "log_level": 1, "num_processed_events": 13499, "processed_events_size": 12082156, "thr_count": 1, "thr_map": 1, "timer_events": 13528, "timer_responses_received": 13499, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13499, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13499, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 18497, "num_processed_events": 13499, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13499, "processed_events_size": 12082156, "timer_callback_failure": 0, "timer_callback_success": 13528, "timer_cancel_counter": 0, "timer_create_counter": 13499, "timer_create_failure": 0, "timer_msg_counter": 13528, "timer_responses_sent": 0, "timestamp": { "124274": "2021-02-13T05:24:38Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "124274": "2021-02-13T05:24:38Z" }, "v8worker_events_lost": 0 }, "function_id": 506151983, "function_name": "timers", "gocb_creds_request_counter": 67, "internal_vb_distribution_stats": { "worker_timers_0": "[0-341]" }, "latency_percentile_stats": { "100": 24100, "50": 1900, "80": 2600, "90": 3200, "95": 3900, "99": 6600 }, "lcb_creds_request_counter": 27060, "lcb_exception_stats": {}, "metastore_stats": { "metastore_delete_err": 0, "metastore_deletes": 0, "metastore_not_found": 0, "metastore_scan": 0, "metastore_scan_due": 0, "metastore_scan_err": 0, "metastore_set": 0, "metastore_set_err": 0 }, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.90:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.97:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_timers_0": "[0-341]" }, "172.23.122.90:8096": { "worker_timers_0": "[342-682]" }, "172.23.122.97:8096": { "worker_timers_0": "[683-1023]" } }, "worker_pids": { "worker_timers_0": 124274 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "agg_messages_sent_to_worker": 18395, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 40830, "dcp_mutation_sent_to_worker": 13499, "dcp_mutation_suppressed_counter": 13832, "dcp_snapshot": 40335, "dcp_stream_req_counter": 685, "dcp_stream_req_err_counter": 1, "dcp_streamend": 342, "dcp_streamreq": 684, "dcp_system_event": 3420, "execution_stats": 772, "failure_stats": 772, "latency_stats": 772, "lcb_exception_stats": 772, "log_level": 1, "num_processed_events": 13499, "processed_events_size": 12086172, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13499, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13499, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 18391, "num_processed_events": 13499, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13499, "processed_events_size": 12086172, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "124359": "2021-02-13T05:24:38Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "124359": "2021-02-13T05:24:38Z" }, "v8worker_events_lost": 0 }, "function_id": 3702549362, "function_name": "sbm", "gocb_creds_request_counter": 67, "internal_vb_distribution_stats": { "worker_sbm_0": "[0-341]" }, "latency_percentile_stats": { "100": 16400, "50": 800, "80": 1200, "90": 1700, "95": 2300, "99": 4700 }, "lcb_creds_request_counter": 27060, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.90:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.97:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_sbm_0": "[0-341]" }, "172.23.122.90:8096": { "worker_sbm_0": "[342-682]" }, "172.23.122.97:8096": { "worker_sbm_0": "[683-1023]" } }, "worker_pids": { "worker_sbm_0": 124359 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 2, "agg_messages_sent_to_worker": 17602, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 40331, "dcp_mutation_sent_to_worker": 13499, "dcp_snapshot": 40057, "dcp_stream_req_counter": 344, "dcp_streamend": 1, "dcp_streamreq": 344, "dcp_system_event": 3420, "execution_stats": 750, "failure_stats": 750, "latency_stats": 750, "lcb_exception_stats": 750, "log_level": 1, "num_processed_events": 13499, "processed_events_size": 12082156, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 13499, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 13499, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13499, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13499, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 17598, "num_processed_events": 13499, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13499, "processed_events_size": 12082156, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "124401": "2021-02-13T05:24:38Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "124401": "2021-02-13T05:24:38Z" }, "v8worker_events_lost": 0 }, "function_id": 3687844484, "function_name": "curl", "gocb_creds_request_counter": 67, "internal_vb_distribution_stats": { "worker_curl_0": "[0-341]" }, "latency_percentile_stats": { "100": 534100, "50": 1500, "80": 2000, "90": 2400, "95": 2900, "99": 4800 }, "lcb_creds_request_counter": 27060, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.90:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.97:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_curl_0": "[0-341]" }, "172.23.122.90:8096": { "worker_curl_0": "[342-682]" }, "172.23.122.97:8096": { "worker_curl_0": "[683-1023]" } }, "worker_pids": { "worker_curl_0": 124401 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 3, "agg_messages_sent_to_worker": 17496, "agg_queue_memory": 0, "agg_queue_memory_cap": 20971520, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 40331, "dcp_mutation_sent_to_worker": 13499, "dcp_snapshot": 40049, "dcp_stream_req_counter": 343, "dcp_streamend": 1, "dcp_streamreq": 343, "dcp_system_event": 3420, "execution_stats": 729, "failure_stats": 729, "latency_stats": 729, "lcb_exception_stats": 729, "log_level": 1, "num_processed_events": 13499, "processed_events_size": 12082156, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13499, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13499, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 17492, "num_processed_events": 13499, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13499, "processed_events_size": 12082156, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "124483": "2021-02-13T05:24:38Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "124483": "2021-02-13T05:24:38Z" }, "v8worker_events_lost": 0 }, "function_id": 1890800864, "function_name": "n1ql", "gocb_creds_request_counter": 67, "internal_vb_distribution_stats": { "worker_n1ql_0": "[0-341]" }, "latency_percentile_stats": { "100": 96500, "50": 3200, "80": 4300, "90": 5300, "95": 6700, "99": 11200 }, "lcb_creds_request_counter": 27060, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.90:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.97:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_n1ql_0": "[0-341]" }, "172.23.122.90:8096": { "worker_n1ql_0": "[342-682]" }, "172.23.122.97:8096": { "worker_n1ql_0": "[683-1023]" } }, "worker_pids": { "worker_n1ql_0": 124483 } } ] [2021-02-12 21:24:49,406] - [eventing_base] [140099333379840] - INFO - Stats for Node 172.23.122.90 is [ { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 2, "agg_messages_sent_to_worker": 18625, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 67638, "dcp_mutation_sent_to_worker": 13432, "dcp_snapshot": 40678, "dcp_stream_req_counter": 822, "dcp_stream_req_err_counter": 6, "dcp_streamend": 399, "dcp_streamreq": 816, "dcp_system_event": 3410, "execution_stats": 820, "failure_stats": 820, "latency_stats": 820, "lcb_exception_stats": 820, "log_level": 1, "num_processed_events": 13432, "processed_events_size": 12023960, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13432, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13432, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 18621, "num_processed_events": 13432, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13432, "processed_events_size": 12023960, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "47216": "2021-02-13T05:24:44Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "47216": "2021-02-13T05:24:44Z" }, "v8worker_events_lost": 0 }, "function_id": 1271581675, "function_name": "bucket_op", "gocb_creds_request_counter": 58, "internal_vb_distribution_stats": { "worker_bucket_op_0": "[342-682]" }, "latency_percentile_stats": { "100": 26400, "50": 700, "80": 1000, "90": 1300, "95": 1700, "99": 3000 }, "lcb_creds_request_counter": 26931, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.90:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.97:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_bucket_op_0": "[0-341]" }, "172.23.122.90:8096": { "worker_bucket_op_0": "[342-682]" }, "172.23.122.97:8096": { "worker_bucket_op_0": "[683-1023]" } }, "worker_pids": { "worker_bucket_op_0": 47216 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 1, "agg_messages_sent_to_worker": 18510, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 67638, "dcp_mutation_sent_to_worker": 13432, "dcp_snapshot": 40655, "dcp_stream_req_counter": 834, "dcp_stream_req_err_counter": 6, "dcp_streamend": 395, "dcp_streamreq": 828, "dcp_system_event": 3410, "execution_stats": 798, "failure_stats": 798, "latency_stats": 798, "lcb_exception_stats": 798, "log_level": 1, "num_processed_events": 13432, "processed_events_size": 12023960, "thr_count": 1, "thr_map": 1, "timer_events": 13558, "timer_responses_received": 13432, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13432, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13432, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 18506, "num_processed_events": 13432, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13432, "processed_events_size": 12023960, "timer_callback_failure": 0, "timer_callback_success": 13558, "timer_cancel_counter": 0, "timer_create_counter": 13432, "timer_create_failure": 0, "timer_msg_counter": 13558, "timer_responses_sent": 0, "timestamp": { "47249": "2021-02-13T05:24:43Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "47249": "2021-02-13T05:24:43Z" }, "v8worker_events_lost": 0 }, "function_id": 506151983, "function_name": "timers", "gocb_creds_request_counter": 58, "internal_vb_distribution_stats": { "worker_timers_0": "[342-682]" }, "latency_percentile_stats": { "100": 26200, "50": 1900, "80": 2900, "90": 3700, "95": 4500, "99": 7200 }, "lcb_creds_request_counter": 26931, "lcb_exception_stats": {}, "metastore_stats": { "metastore_delete_err": 0, "metastore_deletes": 0, "metastore_not_found": 0, "metastore_scan": 0, "metastore_scan_due": 0, "metastore_scan_err": 0, "metastore_set": 0, "metastore_set_err": 0 }, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.90:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.97:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_timers_0": "[0-341]" }, "172.23.122.90:8096": { "worker_timers_0": "[342-682]" }, "172.23.122.97:8096": { "worker_timers_0": "[683-1023]" } }, "worker_pids": { "worker_timers_0": 47249 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "agg_messages_sent_to_worker": 18391, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 41156, "dcp_mutation_sent_to_worker": 13432, "dcp_mutation_suppressed_counter": 14292, "dcp_snapshot": 40593, "dcp_stream_req_counter": 848, "dcp_stream_req_err_counter": 6, "dcp_streamend": 382, "dcp_streamreq": 842, "dcp_system_event": 3410, "execution_stats": 777, "failure_stats": 777, "latency_stats": 777, "lcb_exception_stats": 777, "log_level": 1, "num_processed_events": 13432, "processed_events_size": 12029572, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13432, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13432, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 18387, "num_processed_events": 13432, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13432, "processed_events_size": 12029572, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "47363": "2021-02-13T05:24:44Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "47363": "2021-02-13T05:24:44Z" }, "v8worker_events_lost": 0 }, "function_id": 3702549362, "function_name": "sbm", "gocb_creds_request_counter": 58, "internal_vb_distribution_stats": { "worker_sbm_0": "[342-682]" }, "latency_percentile_stats": { "100": 17300, "50": 800, "80": 1300, "90": 1800, "95": 2300, "99": 4800 }, "lcb_creds_request_counter": 26931, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.90:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.97:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_sbm_0": "[0-341]" }, "172.23.122.90:8096": { "worker_sbm_0": "[342-682]" }, "172.23.122.97:8096": { "worker_sbm_0": "[683-1023]" } }, "worker_pids": { "worker_sbm_0": 47363 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 2, "agg_messages_sent_to_worker": 17902, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 54088, "dcp_mutation_sent_to_worker": 13432, "dcp_snapshot": 40569, "dcp_stream_req_counter": 1392, "dcp_stream_req_err_counter": 837, "dcp_streamend": 170, "dcp_streamreq": 555, "dcp_system_event": 3410, "execution_stats": 756, "failure_stats": 756, "latency_stats": 756, "lcb_exception_stats": 756, "log_level": 1, "num_processed_events": 13432, "processed_events_size": 12023960, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 13432, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 13432, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13432, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13432, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 17898, "num_processed_events": 13432, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13432, "processed_events_size": 12023960, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "47426": "2021-02-13T05:24:44Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "47426": "2021-02-13T05:24:44Z" }, "v8worker_events_lost": 0 }, "function_id": 3687844484, "function_name": "curl", "gocb_creds_request_counter": 58, "internal_vb_distribution_stats": { "worker_curl_0": "[342-682]" }, "latency_percentile_stats": { "100": 534700, "50": 1600, "80": 2000, "90": 2400, "95": 3000, "99": 5300 }, "lcb_creds_request_counter": 26931, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.90:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.97:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_curl_0": "[0-341]" }, "172.23.122.90:8096": { "worker_curl_0": "[342-682]" }, "172.23.122.97:8096": { "worker_curl_0": "[683-1023]" } }, "worker_pids": { "worker_curl_0": 47426 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 2, "agg_messages_sent_to_worker": 17791, "agg_queue_memory": 0, "agg_queue_memory_cap": 20971520, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 54088, "dcp_mutation_sent_to_worker": 13432, "dcp_snapshot": 40540, "dcp_stream_req_counter": 1021, "dcp_stream_req_err_counter": 453, "dcp_streamend": 170, "dcp_streamreq": 568, "dcp_system_event": 3410, "execution_stats": 734, "failure_stats": 734, "latency_stats": 734, "lcb_exception_stats": 734, "log_level": 1, "num_processed_events": 13432, "processed_events_size": 12023960, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13432, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13432, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 17787, "num_processed_events": 13432, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 1, "on_update_success": 13431, "processed_events_size": 12023960, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "47458": "2021-02-13T05:24:43Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 1, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "47458": "2021-02-13T05:24:43Z" }, "v8worker_events_lost": 0 }, "function_id": 1890800864, "function_name": "n1ql", "gocb_creds_request_counter": 58, "internal_vb_distribution_stats": { "worker_n1ql_0": "[342-682]" }, "latency_percentile_stats": { "100": 142000, "50": 3400, "80": 4700, "90": 5800, "95": 7100, "99": 11800 }, "lcb_creds_request_counter": 26931, "lcb_exception_stats": { "12009": 1, "209": 1 }, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.90:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.97:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_n1ql_0": "[0-341]" }, "172.23.122.90:8096": { "worker_n1ql_0": "[342-682]" }, "172.23.122.97:8096": { "worker_n1ql_0": "[683-1023]" } }, "worker_pids": { "worker_n1ql_0": 47458 } } ] [2021-02-12 21:24:55,147] - [eventing_base] [140099333379840] - INFO - Stats for Node 172.23.122.97 is [ { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 3, "agg_messages_sent_to_worker": 18608, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 66981, "dcp_mutation_sent_to_worker": 13389, "dcp_snapshot": 40426, "dcp_stream_req_counter": 822, "dcp_stream_req_err_counter": 6, "dcp_streamend": 395, "dcp_streamreq": 816, "dcp_system_event": 3410, "execution_stats": 826, "failure_stats": 826, "latency_stats": 826, "lcb_exception_stats": 826, "log_level": 1, "num_processed_events": 13389, "processed_events_size": 11986976, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13389, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13389, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 18604, "num_processed_events": 13389, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13389, "processed_events_size": 11986976, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "26680": "2021-02-13T05:24:50Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "26680": "2021-02-13T05:24:50Z" }, "v8worker_events_lost": 0 }, "function_id": 1271581675, "function_name": "bucket_op", "gocb_creds_request_counter": 56, "internal_vb_distribution_stats": { "worker_bucket_op_0": "[683-1023]" }, "latency_percentile_stats": { "100": 36400, "50": 700, "80": 1100, "90": 1500, "95": 1900, "99": 3300 }, "lcb_creds_request_counter": 26852, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.90:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.97:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_bucket_op_0": "[0-341]" }, "172.23.122.90:8096": { "worker_bucket_op_0": "[342-682]" }, "172.23.122.97:8096": { "worker_bucket_op_0": "[683-1023]" } }, "worker_pids": { "worker_bucket_op_0": 26680 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 3, "agg_messages_sent_to_worker": 18493, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 66981, "dcp_mutation_sent_to_worker": 13389, "dcp_snapshot": 40394, "dcp_stream_req_counter": 833, "dcp_stream_req_err_counter": 7, "dcp_streamend": 391, "dcp_streamreq": 826, "dcp_system_event": 3410, "execution_stats": 804, "failure_stats": 804, "latency_stats": 804, "lcb_exception_stats": 804, "log_level": 1, "num_processed_events": 13389, "processed_events_size": 11986976, "thr_count": 1, "thr_map": 1, "timer_events": 13234, "timer_responses_received": 13389, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13389, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13389, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 18489, "num_processed_events": 13389, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13389, "processed_events_size": 11986976, "timer_callback_failure": 0, "timer_callback_success": 13234, "timer_cancel_counter": 0, "timer_create_counter": 13389, "timer_create_failure": 0, "timer_msg_counter": 13234, "timer_responses_sent": 0, "timestamp": { "26712": "2021-02-13T05:24:49Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "26712": "2021-02-13T05:24:49Z" }, "v8worker_events_lost": 0 }, "function_id": 506151983, "function_name": "timers", "gocb_creds_request_counter": 56, "internal_vb_distribution_stats": { "worker_timers_0": "[683-1023]" }, "latency_percentile_stats": { "100": 22000, "50": 2000, "80": 2900, "90": 3600, "95": 4500, "99": 7300 }, "lcb_creds_request_counter": 26852, "lcb_exception_stats": {}, "metastore_stats": { "metastore_delete_err": 0, "metastore_deletes": 0, "metastore_not_found": 0, "metastore_scan": 0, "metastore_scan_due": 0, "metastore_scan_err": 0, "metastore_set": 0, "metastore_set_err": 0 }, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.90:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.97:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_timers_0": "[0-341]" }, "172.23.122.90:8096": { "worker_timers_0": "[342-682]" }, "172.23.122.97:8096": { "worker_timers_0": "[683-1023]" } }, "worker_pids": { "worker_timers_0": 26712 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "agg_messages_sent_to_worker": 18377, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 40830, "dcp_mutation_sent_to_worker": 13389, "dcp_mutation_suppressed_counter": 14052, "dcp_snapshot": 40245, "dcp_stream_req_counter": 845, "dcp_stream_req_err_counter": 7, "dcp_streamend": 381, "dcp_streamreq": 838, "dcp_system_event": 3410, "execution_stats": 783, "failure_stats": 783, "latency_stats": 783, "lcb_exception_stats": 783, "log_level": 1, "num_processed_events": 13389, "processed_events_size": 11987720, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13389, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13389, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 18373, "num_processed_events": 13389, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13389, "processed_events_size": 11987720, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "26828": "2021-02-13T05:24:50Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "26828": "2021-02-13T05:24:50Z" }, "v8worker_events_lost": 0 }, "function_id": 3702549362, "function_name": "sbm", "gocb_creds_request_counter": 56, "internal_vb_distribution_stats": { "worker_sbm_0": "[683-1023]" }, "latency_percentile_stats": { "100": 21200, "50": 900, "80": 1400, "90": 1900, "95": 2500, "99": 5100 }, "lcb_creds_request_counter": 26852, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.90:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.97:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_sbm_0": "[0-341]" }, "172.23.122.90:8096": { "worker_sbm_0": "[342-682]" }, "172.23.122.97:8096": { "worker_sbm_0": "[683-1023]" } }, "worker_pids": { "worker_sbm_0": 26828 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 1, "agg_messages_sent_to_worker": 17886, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 53672, "dcp_mutation_sent_to_worker": 13389, "dcp_snapshot": 40104, "dcp_stream_req_counter": 1548, "dcp_stream_req_err_counter": 988, "dcp_streamend": 171, "dcp_streamreq": 560, "dcp_system_event": 3410, "execution_stats": 761, "failure_stats": 761, "latency_stats": 761, "lcb_exception_stats": 761, "log_level": 1, "num_processed_events": 13389, "processed_events_size": 11986976, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 13389, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 13389, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13389, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13389, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 17882, "num_processed_events": 13389, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13389, "processed_events_size": 11986976, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "26892": "2021-02-13T05:24:49Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "26892": "2021-02-13T05:24:49Z" }, "v8worker_events_lost": 0 }, "function_id": 3687844484, "function_name": "curl", "gocb_creds_request_counter": 56, "internal_vb_distribution_stats": { "worker_curl_0": "[683-1023]" }, "latency_percentile_stats": { "100": 534800, "50": 1600, "80": 2100, "90": 2500, "95": 2900, "99": 4700 }, "lcb_creds_request_counter": 26852, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.90:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.97:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_curl_0": "[0-341]" }, "172.23.122.90:8096": { "worker_curl_0": "[342-682]" }, "172.23.122.97:8096": { "worker_curl_0": "[683-1023]" } }, "worker_pids": { "worker_curl_0": 26892 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 1, "agg_messages_sent_to_worker": 17780, "agg_queue_memory": 0, "agg_queue_memory_cap": 20971520, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 53672, "dcp_mutation_sent_to_worker": 13389, "dcp_snapshot": 40129, "dcp_stream_req_counter": 1180, "dcp_stream_req_err_counter": 612, "dcp_streamend": 171, "dcp_streamreq": 568, "dcp_system_event": 3410, "execution_stats": 740, "failure_stats": 740, "latency_stats": 740, "lcb_exception_stats": 740, "log_level": 1, "num_processed_events": 13389, "processed_events_size": 11986976, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13389, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13389, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 17776, "num_processed_events": 13389, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 1, "on_update_success": 13388, "processed_events_size": 11986976, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "26924": "2021-02-13T05:24:49Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 1, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "26924": "2021-02-13T05:24:49Z" }, "v8worker_events_lost": 0 }, "function_id": 1890800864, "function_name": "n1ql", "gocb_creds_request_counter": 56, "internal_vb_distribution_stats": { "worker_n1ql_0": "[683-1023]" }, "latency_percentile_stats": { "100": 134200, "50": 3400, "80": 4700, "90": 5900, "95": 7400, "99": 12100 }, "lcb_creds_request_counter": 26852, "lcb_exception_stats": { "12009": 1, "209": 1 }, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.90:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.97:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_n1ql_0": "[0-341]" }, "172.23.122.90:8096": { "worker_n1ql_0": "[342-682]" }, "172.23.122.97:8096": { "worker_n1ql_0": "[683-1023]" } }, "worker_pids": { "worker_n1ql_0": 26924 } } ] [2021-02-12 21:24:55,933] - [eventing_base] [140099333379840] - INFO - Execution stats update_success: 40320 update_failure: 0 delete_success: 0 delete_failure: 0 [2021-02-12 21:24:55,954] - [basetestcase] [140099333379840] - INFO - list of eventing nodes in cluster: [ip:172.23.122.117 port:8091 ssh_username:root, ip:172.23.122.90 port:8091 ssh_username:root, ip:172.23.122.97 port:8091 ssh_username:root] [2021-02-12 21:24:55,995] - [eventing_base] [140099333379840] - INFO - Stats for Node 172.23.122.117 is [ { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 2, "agg_messages_sent_to_worker": 18692, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 66981, "dcp_mutation_sent_to_worker": 13499, "dcp_snapshot": 40448, "dcp_stream_req_counter": 684, "dcp_streamend": 342, "dcp_streamreq": 684, "dcp_system_event": 3420, "execution_stats": 831, "failure_stats": 831, "latency_stats": 831, "lcb_exception_stats": 831, "log_level": 1, "num_processed_events": 13499, "processed_events_size": 12082156, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13499, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13499, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 18688, "num_processed_events": 13499, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13499, "processed_events_size": 12082156, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "124232": "2021-02-13T05:24:55Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "124232": "2021-02-13T05:24:55Z" }, "v8worker_events_lost": 0 }, "function_id": 1271581675, "function_name": "bucket_op", "gocb_creds_request_counter": 67, "internal_vb_distribution_stats": { "worker_bucket_op_0": "[0-341]" }, "latency_percentile_stats": { "100": 22400, "50": 700, "80": 1000, "90": 1300, "95": 1600, "99": 2700 }, "lcb_creds_request_counter": 27060, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.90:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.97:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_bucket_op_0": "[0-341]" }, "172.23.122.90:8096": { "worker_bucket_op_0": "[342-682]" }, "172.23.122.97:8096": { "worker_bucket_op_0": "[683-1023]" } }, "worker_pids": { "worker_bucket_op_0": 124232 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 2, "agg_messages_sent_to_worker": 18586, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 66981, "dcp_mutation_sent_to_worker": 13499, "dcp_snapshot": 40421, "dcp_stream_req_counter": 684, "dcp_streamend": 342, "dcp_streamreq": 684, "dcp_system_event": 3420, "execution_stats": 810, "failure_stats": 810, "latency_stats": 810, "lcb_exception_stats": 810, "log_level": 1, "num_processed_events": 13499, "processed_events_size": 12082156, "thr_count": 1, "thr_map": 1, "timer_events": 13528, "timer_responses_received": 13499, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13499, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13499, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 18582, "num_processed_events": 13499, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13499, "processed_events_size": 12082156, "timer_callback_failure": 0, "timer_callback_success": 13528, "timer_cancel_counter": 0, "timer_create_counter": 13499, "timer_create_failure": 0, "timer_msg_counter": 13528, "timer_responses_sent": 0, "timestamp": { "124274": "2021-02-13T05:24:55Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "124274": "2021-02-13T05:24:55Z" }, "v8worker_events_lost": 0 }, "function_id": 506151983, "function_name": "timers", "gocb_creds_request_counter": 67, "internal_vb_distribution_stats": { "worker_timers_0": "[0-341]" }, "latency_percentile_stats": { "100": 24100, "50": 1900, "80": 2600, "90": 3200, "95": 3900, "99": 6600 }, "lcb_creds_request_counter": 27060, "lcb_exception_stats": {}, "metastore_stats": { "metastore_delete_err": 0, "metastore_deletes": 0, "metastore_not_found": 0, "metastore_scan": 0, "metastore_scan_due": 0, "metastore_scan_err": 0, "metastore_set": 0, "metastore_set_err": 0 }, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.90:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.97:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_timers_0": "[0-341]" }, "172.23.122.90:8096": { "worker_timers_0": "[342-682]" }, "172.23.122.97:8096": { "worker_timers_0": "[683-1023]" } }, "worker_pids": { "worker_timers_0": 124274 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "agg_messages_sent_to_worker": 18480, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 40830, "dcp_mutation_sent_to_worker": 13499, "dcp_mutation_suppressed_counter": 13832, "dcp_snapshot": 40335, "dcp_stream_req_counter": 685, "dcp_stream_req_err_counter": 1, "dcp_streamend": 342, "dcp_streamreq": 684, "dcp_system_event": 3420, "execution_stats": 789, "failure_stats": 789, "latency_stats": 789, "lcb_exception_stats": 789, "log_level": 1, "num_processed_events": 13499, "processed_events_size": 12086172, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13499, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13499, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 18476, "num_processed_events": 13499, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13499, "processed_events_size": 12086172, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "124359": "2021-02-13T05:24:55Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "124359": "2021-02-13T05:24:55Z" }, "v8worker_events_lost": 0 }, "function_id": 3702549362, "function_name": "sbm", "gocb_creds_request_counter": 67, "internal_vb_distribution_stats": { "worker_sbm_0": "[0-341]" }, "latency_percentile_stats": { "100": 16400, "50": 800, "80": 1200, "90": 1700, "95": 2300, "99": 4700 }, "lcb_creds_request_counter": 27060, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.90:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.97:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_sbm_0": "[0-341]" }, "172.23.122.90:8096": { "worker_sbm_0": "[342-682]" }, "172.23.122.97:8096": { "worker_sbm_0": "[683-1023]" } }, "worker_pids": { "worker_sbm_0": 124359 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 2, "agg_messages_sent_to_worker": 17687, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 40331, "dcp_mutation_sent_to_worker": 13499, "dcp_snapshot": 40057, "dcp_stream_req_counter": 344, "dcp_streamend": 1, "dcp_streamreq": 344, "dcp_system_event": 3420, "execution_stats": 767, "failure_stats": 767, "latency_stats": 767, "lcb_exception_stats": 767, "log_level": 1, "num_processed_events": 13499, "processed_events_size": 12082156, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 13499, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 13499, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13499, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13499, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 17683, "num_processed_events": 13499, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13499, "processed_events_size": 12082156, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "124401": "2021-02-13T05:24:55Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "124401": "2021-02-13T05:24:55Z" }, "v8worker_events_lost": 0 }, "function_id": 3687844484, "function_name": "curl", "gocb_creds_request_counter": 67, "internal_vb_distribution_stats": { "worker_curl_0": "[0-341]" }, "latency_percentile_stats": { "100": 534100, "50": 1500, "80": 2000, "90": 2400, "95": 2900, "99": 4800 }, "lcb_creds_request_counter": 27060, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.90:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.97:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_curl_0": "[0-341]" }, "172.23.122.90:8096": { "worker_curl_0": "[342-682]" }, "172.23.122.97:8096": { "worker_curl_0": "[683-1023]" } }, "worker_pids": { "worker_curl_0": 124401 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 3, "agg_messages_sent_to_worker": 17581, "agg_queue_memory": 0, "agg_queue_memory_cap": 20971520, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 40331, "dcp_mutation_sent_to_worker": 13499, "dcp_snapshot": 40049, "dcp_stream_req_counter": 343, "dcp_streamend": 1, "dcp_streamreq": 343, "dcp_system_event": 3420, "execution_stats": 746, "failure_stats": 746, "latency_stats": 746, "lcb_exception_stats": 746, "log_level": 1, "num_processed_events": 13499, "processed_events_size": 12082156, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13499, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13499, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 17577, "num_processed_events": 13499, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13499, "processed_events_size": 12082156, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "124483": "2021-02-13T05:24:55Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "124483": "2021-02-13T05:24:55Z" }, "v8worker_events_lost": 0 }, "function_id": 1890800864, "function_name": "n1ql", "gocb_creds_request_counter": 67, "internal_vb_distribution_stats": { "worker_n1ql_0": "[0-341]" }, "latency_percentile_stats": { "100": 96500, "50": 3200, "80": 4300, "90": 5300, "95": 6700, "99": 11200 }, "lcb_creds_request_counter": 27060, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.90:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.97:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_n1ql_0": "[0-341]" }, "172.23.122.90:8096": { "worker_n1ql_0": "[342-682]" }, "172.23.122.97:8096": { "worker_n1ql_0": "[683-1023]" } }, "worker_pids": { "worker_n1ql_0": 124483 } } ] [2021-02-12 21:24:56,041] - [eventing_base] [140099333379840] - INFO - Stats for Node 172.23.122.90 is [ { "dcp_feed_boundary": "everything", "event_processing_stats": { "agg_messages_sent_to_worker": 18451, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 41156, "dcp_mutation_sent_to_worker": 13432, "dcp_mutation_suppressed_counter": 14292, "dcp_snapshot": 40593, "dcp_stream_req_counter": 848, "dcp_stream_req_err_counter": 6, "dcp_streamend": 382, "dcp_streamreq": 842, "dcp_system_event": 3410, "execution_stats": 789, "failure_stats": 789, "latency_stats": 789, "lcb_exception_stats": 789, "log_level": 1, "num_processed_events": 13432, "processed_events_size": 12029572, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13432, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13432, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 18447, "num_processed_events": 13432, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13432, "processed_events_size": 12029572, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "47363": "2021-02-13T05:24:56Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "47363": "2021-02-13T05:24:56Z" }, "v8worker_events_lost": 0 }, "function_id": 3702549362, "function_name": "sbm", "gocb_creds_request_counter": 58, "internal_vb_distribution_stats": { "worker_sbm_0": "[342-682]" }, "latency_percentile_stats": { "100": 17300, "50": 800, "80": 1300, "90": 1800, "95": 2300, "99": 4800 }, "lcb_creds_request_counter": 26931, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.90:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.97:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_sbm_0": "[0-341]" }, "172.23.122.90:8096": { "worker_sbm_0": "[342-682]" }, "172.23.122.97:8096": { "worker_sbm_0": "[683-1023]" } }, "worker_pids": { "worker_sbm_0": 47363 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 2, "agg_messages_sent_to_worker": 17957, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 54088, "dcp_mutation_sent_to_worker": 13432, "dcp_snapshot": 40569, "dcp_stream_req_counter": 1392, "dcp_stream_req_err_counter": 837, "dcp_streamend": 170, "dcp_streamreq": 555, "dcp_system_event": 3410, "execution_stats": 767, "failure_stats": 767, "latency_stats": 767, "lcb_exception_stats": 767, "log_level": 1, "num_processed_events": 13432, "processed_events_size": 12023960, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 13432, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 13432, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13432, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13432, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 17953, "num_processed_events": 13432, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13432, "processed_events_size": 12023960, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "47426": "2021-02-13T05:24:55Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "47426": "2021-02-13T05:24:55Z" }, "v8worker_events_lost": 0 }, "function_id": 3687844484, "function_name": "curl", "gocb_creds_request_counter": 58, "internal_vb_distribution_stats": { "worker_curl_0": "[342-682]" }, "latency_percentile_stats": { "100": 534700, "50": 1600, "80": 2000, "90": 2400, "95": 3000, "99": 5300 }, "lcb_creds_request_counter": 26931, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.90:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.97:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_curl_0": "[0-341]" }, "172.23.122.90:8096": { "worker_curl_0": "[342-682]" }, "172.23.122.97:8096": { "worker_curl_0": "[683-1023]" } }, "worker_pids": { "worker_curl_0": 47426 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 2, "agg_messages_sent_to_worker": 17851, "agg_queue_memory": 0, "agg_queue_memory_cap": 20971520, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 54088, "dcp_mutation_sent_to_worker": 13432, "dcp_snapshot": 40540, "dcp_stream_req_counter": 1021, "dcp_stream_req_err_counter": 453, "dcp_streamend": 170, "dcp_streamreq": 568, "dcp_system_event": 3410, "execution_stats": 746, "failure_stats": 746, "latency_stats": 746, "lcb_exception_stats": 746, "log_level": 1, "num_processed_events": 13432, "processed_events_size": 12023960, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13432, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13432, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 17847, "num_processed_events": 13432, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 1, "on_update_success": 13431, "processed_events_size": 12023960, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "47458": "2021-02-13T05:24:55Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 1, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "47458": "2021-02-13T05:24:55Z" }, "v8worker_events_lost": 0 }, "function_id": 1890800864, "function_name": "n1ql", "gocb_creds_request_counter": 58, "internal_vb_distribution_stats": { "worker_n1ql_0": "[342-682]" }, "latency_percentile_stats": { "100": 142000, "50": 3400, "80": 4700, "90": 5800, "95": 7100, "99": 11800 }, "lcb_creds_request_counter": 26931, "lcb_exception_stats": { "12009": 1, "209": 1 }, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.90:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.97:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_n1ql_0": "[0-341]" }, "172.23.122.90:8096": { "worker_n1ql_0": "[342-682]" }, "172.23.122.97:8096": { "worker_n1ql_0": "[683-1023]" } }, "worker_pids": { "worker_n1ql_0": 47458 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 2, "agg_messages_sent_to_worker": 18680, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 67638, "dcp_mutation_sent_to_worker": 13432, "dcp_snapshot": 40678, "dcp_stream_req_counter": 822, "dcp_stream_req_err_counter": 6, "dcp_streamend": 399, "dcp_streamreq": 816, "dcp_system_event": 3410, "execution_stats": 831, "failure_stats": 831, "latency_stats": 831, "lcb_exception_stats": 831, "log_level": 1, "num_processed_events": 13432, "processed_events_size": 12023960, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13432, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13432, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 18676, "num_processed_events": 13432, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13432, "processed_events_size": 12023960, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "47216": "2021-02-13T05:24:55Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "47216": "2021-02-13T05:24:55Z" }, "v8worker_events_lost": 0 }, "function_id": 1271581675, "function_name": "bucket_op", "gocb_creds_request_counter": 58, "internal_vb_distribution_stats": { "worker_bucket_op_0": "[342-682]" }, "latency_percentile_stats": { "100": 26400, "50": 700, "80": 1000, "90": 1300, "95": 1700, "99": 3000 }, "lcb_creds_request_counter": 26931, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.90:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.97:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_bucket_op_0": "[0-341]" }, "172.23.122.90:8096": { "worker_bucket_op_0": "[342-682]" }, "172.23.122.97:8096": { "worker_bucket_op_0": "[683-1023]" } }, "worker_pids": { "worker_bucket_op_0": 47216 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 1, "agg_messages_sent_to_worker": 18570, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 67638, "dcp_mutation_sent_to_worker": 13432, "dcp_snapshot": 40655, "dcp_stream_req_counter": 834, "dcp_stream_req_err_counter": 6, "dcp_streamend": 395, "dcp_streamreq": 828, "dcp_system_event": 3410, "execution_stats": 810, "failure_stats": 810, "latency_stats": 810, "lcb_exception_stats": 810, "log_level": 1, "num_processed_events": 13432, "processed_events_size": 12023960, "thr_count": 1, "thr_map": 1, "timer_events": 13558, "timer_responses_received": 13432, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13432, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13432, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 18566, "num_processed_events": 13432, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13432, "processed_events_size": 12023960, "timer_callback_failure": 0, "timer_callback_success": 13558, "timer_cancel_counter": 0, "timer_create_counter": 13432, "timer_create_failure": 0, "timer_msg_counter": 13558, "timer_responses_sent": 0, "timestamp": { "47249": "2021-02-13T05:24:55Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "47249": "2021-02-13T05:24:55Z" }, "v8worker_events_lost": 0 }, "function_id": 506151983, "function_name": "timers", "gocb_creds_request_counter": 58, "internal_vb_distribution_stats": { "worker_timers_0": "[342-682]" }, "latency_percentile_stats": { "100": 26200, "50": 1900, "80": 2900, "90": 3700, "95": 4500, "99": 7200 }, "lcb_creds_request_counter": 26931, "lcb_exception_stats": {}, "metastore_stats": { "metastore_delete_err": 0, "metastore_deletes": 0, "metastore_not_found": 0, "metastore_scan": 0, "metastore_scan_due": 0, "metastore_scan_err": 0, "metastore_set": 0, "metastore_set_err": 0 }, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.90:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.97:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_timers_0": "[0-341]" }, "172.23.122.90:8096": { "worker_timers_0": "[342-682]" }, "172.23.122.97:8096": { "worker_timers_0": "[683-1023]" } }, "worker_pids": { "worker_timers_0": 47249 } } ] [2021-02-12 21:24:56,085] - [eventing_base] [140099333379840] - INFO - Stats for Node 172.23.122.97 is [ { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 3, "agg_messages_sent_to_worker": 18523, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 66981, "dcp_mutation_sent_to_worker": 13389, "dcp_snapshot": 40394, "dcp_stream_req_counter": 833, "dcp_stream_req_err_counter": 7, "dcp_streamend": 391, "dcp_streamreq": 826, "dcp_system_event": 3410, "execution_stats": 810, "failure_stats": 810, "latency_stats": 810, "lcb_exception_stats": 810, "log_level": 1, "num_processed_events": 13389, "processed_events_size": 11986976, "thr_count": 1, "thr_map": 1, "timer_events": 13234, "timer_responses_received": 13389, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13389, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13389, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 18519, "num_processed_events": 13389, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13389, "processed_events_size": 11986976, "timer_callback_failure": 0, "timer_callback_success": 13234, "timer_cancel_counter": 0, "timer_create_counter": 13389, "timer_create_failure": 0, "timer_msg_counter": 13234, "timer_responses_sent": 0, "timestamp": { "26712": "2021-02-13T05:24:55Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "26712": "2021-02-13T05:24:55Z" }, "v8worker_events_lost": 0 }, "function_id": 506151983, "function_name": "timers", "gocb_creds_request_counter": 56, "internal_vb_distribution_stats": { "worker_timers_0": "[683-1023]" }, "latency_percentile_stats": { "100": 22000, "50": 2000, "80": 2900, "90": 3600, "95": 4500, "99": 7300 }, "lcb_creds_request_counter": 26852, "lcb_exception_stats": {}, "metastore_stats": { "metastore_delete_err": 0, "metastore_deletes": 0, "metastore_not_found": 0, "metastore_scan": 0, "metastore_scan_due": 0, "metastore_scan_err": 0, "metastore_set": 0, "metastore_set_err": 0 }, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.90:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.97:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_timers_0": "[0-341]" }, "172.23.122.90:8096": { "worker_timers_0": "[342-682]" }, "172.23.122.97:8096": { "worker_timers_0": "[683-1023]" } }, "worker_pids": { "worker_timers_0": 26712 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "agg_messages_sent_to_worker": 18407, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 40830, "dcp_mutation_sent_to_worker": 13389, "dcp_mutation_suppressed_counter": 14052, "dcp_snapshot": 40245, "dcp_stream_req_counter": 845, "dcp_stream_req_err_counter": 7, "dcp_streamend": 381, "dcp_streamreq": 838, "dcp_system_event": 3410, "execution_stats": 789, "failure_stats": 789, "latency_stats": 789, "lcb_exception_stats": 789, "log_level": 1, "num_processed_events": 13389, "processed_events_size": 11987720, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13389, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13389, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 18403, "num_processed_events": 13389, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13389, "processed_events_size": 11987720, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "26828": "2021-02-13T05:24:56Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "26828": "2021-02-13T05:24:56Z" }, "v8worker_events_lost": 0 }, "function_id": 3702549362, "function_name": "sbm", "gocb_creds_request_counter": 56, "internal_vb_distribution_stats": { "worker_sbm_0": "[683-1023]" }, "latency_percentile_stats": { "100": 21200, "50": 900, "80": 1400, "90": 1900, "95": 2500, "99": 5100 }, "lcb_creds_request_counter": 26852, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.90:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.97:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_sbm_0": "[0-341]" }, "172.23.122.90:8096": { "worker_sbm_0": "[342-682]" }, "172.23.122.97:8096": { "worker_sbm_0": "[683-1023]" } }, "worker_pids": { "worker_sbm_0": 26828 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 1, "agg_messages_sent_to_worker": 17916, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 53672, "dcp_mutation_sent_to_worker": 13389, "dcp_snapshot": 40104, "dcp_stream_req_counter": 1548, "dcp_stream_req_err_counter": 988, "dcp_streamend": 171, "dcp_streamreq": 560, "dcp_system_event": 3410, "execution_stats": 767, "failure_stats": 767, "latency_stats": 767, "lcb_exception_stats": 767, "log_level": 1, "num_processed_events": 13389, "processed_events_size": 11986976, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 13389, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 13389, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13389, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13389, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 17912, "num_processed_events": 13389, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13389, "processed_events_size": 11986976, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "26892": "2021-02-13T05:24:55Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "26892": "2021-02-13T05:24:55Z" }, "v8worker_events_lost": 0 }, "function_id": 3687844484, "function_name": "curl", "gocb_creds_request_counter": 56, "internal_vb_distribution_stats": { "worker_curl_0": "[683-1023]" }, "latency_percentile_stats": { "100": 534800, "50": 1600, "80": 2100, "90": 2500, "95": 2900, "99": 4700 }, "lcb_creds_request_counter": 26852, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.90:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.97:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_curl_0": "[0-341]" }, "172.23.122.90:8096": { "worker_curl_0": "[342-682]" }, "172.23.122.97:8096": { "worker_curl_0": "[683-1023]" } }, "worker_pids": { "worker_curl_0": 26892 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 1, "agg_messages_sent_to_worker": 17810, "agg_queue_memory": 0, "agg_queue_memory_cap": 20971520, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 53672, "dcp_mutation_sent_to_worker": 13389, "dcp_snapshot": 40129, "dcp_stream_req_counter": 1180, "dcp_stream_req_err_counter": 612, "dcp_streamend": 171, "dcp_streamreq": 568, "dcp_system_event": 3410, "execution_stats": 746, "failure_stats": 746, "latency_stats": 746, "lcb_exception_stats": 746, "log_level": 1, "num_processed_events": 13389, "processed_events_size": 11986976, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13389, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13389, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 17806, "num_processed_events": 13389, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 1, "on_update_success": 13388, "processed_events_size": 11986976, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "26924": "2021-02-13T05:24:55Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 1, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "26924": "2021-02-13T05:24:55Z" }, "v8worker_events_lost": 0 }, "function_id": 1890800864, "function_name": "n1ql", "gocb_creds_request_counter": 56, "internal_vb_distribution_stats": { "worker_n1ql_0": "[683-1023]" }, "latency_percentile_stats": { "100": 134200, "50": 3400, "80": 4700, "90": 5900, "95": 7400, "99": 12100 }, "lcb_creds_request_counter": 26852, "lcb_exception_stats": { "12009": 1, "209": 1 }, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.90:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.97:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_n1ql_0": "[0-341]" }, "172.23.122.90:8096": { "worker_n1ql_0": "[342-682]" }, "172.23.122.97:8096": { "worker_n1ql_0": "[683-1023]" } }, "worker_pids": { "worker_n1ql_0": 26924 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 3, "agg_messages_sent_to_worker": 18633, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 66981, "dcp_mutation_sent_to_worker": 13389, "dcp_snapshot": 40426, "dcp_stream_req_counter": 822, "dcp_stream_req_err_counter": 6, "dcp_streamend": 395, "dcp_streamreq": 816, "dcp_system_event": 3410, "execution_stats": 831, "failure_stats": 831, "latency_stats": 831, "lcb_exception_stats": 831, "log_level": 1, "num_processed_events": 13389, "processed_events_size": 11986976, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13389, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13389, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 18629, "num_processed_events": 13389, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13389, "processed_events_size": 11986976, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "26680": "2021-02-13T05:24:55Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "26680": "2021-02-13T05:24:55Z" }, "v8worker_events_lost": 0 }, "function_id": 1271581675, "function_name": "bucket_op", "gocb_creds_request_counter": 56, "internal_vb_distribution_stats": { "worker_bucket_op_0": "[683-1023]" }, "latency_percentile_stats": { "100": 36400, "50": 700, "80": 1100, "90": 1500, "95": 1900, "99": 3300 }, "lcb_creds_request_counter": 26852, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.90:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.97:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_bucket_op_0": "[0-341]" }, "172.23.122.90:8096": { "worker_bucket_op_0": "[342-682]" }, "172.23.122.97:8096": { "worker_bucket_op_0": "[683-1023]" } }, "worker_pids": { "worker_bucket_op_0": 26680 } } ] [2021-02-12 21:24:56,101] - [basetestcase] [140099333379840] - INFO - list of eventing nodes in cluster: [ip:172.23.122.117 port:8091 ssh_username:root, ip:172.23.122.90 port:8091 ssh_username:root, ip:172.23.122.97 port:8091 ssh_username:root] [2021-02-12 21:24:56,103] - [remote_util] [140099333379840] - INFO - SSH Connecting to 172.23.122.117 with username:root, attempt#1 of 5 [2021-02-12 21:24:56,202] - [remote_util] [140099333379840] - INFO - SSH Connected to 172.23.122.117 as root [2021-02-12 21:24:56,454] - [remote_util] [140099333379840] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:24:56,751] - [remote_util] [140099333379840] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:24:56,759] - [rest_client] [140099333379840] - INFO - /diag/eval status on 172.23.122.117:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). [2021-02-12 21:24:56,760] - [remote_util] [140099333379840] - INFO - running command.raw on 172.23.122.117: zgrep "panic" "/opt/couchbase/var/lib/couchbase/logs"/eventing.log* | wc -l [2021-02-12 21:24:56,830] - [remote_util] [140099333379840] - INFO - command executed successfully with root [2021-02-12 21:24:56,830] - [remote_util] [140099333379840] - INFO - running command.raw on 172.23.122.117: ls "/opt/couchbase/var/lib/couchbase/logs"/../crash/| wc -l [2021-02-12 21:24:56,884] - [remote_util] [140099333379840] - INFO - command executed successfully with root [2021-02-12 21:24:56,885] - [remote_util] [140099333379840] - INFO - SSH Connecting to 172.23.122.90 with username:root, attempt#1 of 5 [2021-02-12 21:24:56,993] - [remote_util] [140099333379840] - INFO - SSH Connected to 172.23.122.90 as root [2021-02-12 21:24:57,308] - [remote_util] [140099333379840] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:24:57,613] - [remote_util] [140099333379840] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:24:57,621] - [rest_client] [140099333379840] - INFO - /diag/eval status on 172.23.122.90:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). [2021-02-12 21:24:57,621] - [remote_util] [140099333379840] - INFO - running command.raw on 172.23.122.90: zgrep "panic" "/opt/couchbase/var/lib/couchbase/logs"/eventing.log* | wc -l [2021-02-12 21:24:57,900] - [remote_util] [140099333379840] - INFO - command executed successfully with root [2021-02-12 21:24:57,900] - [remote_util] [140099333379840] - INFO - running command.raw on 172.23.122.90: ls "/opt/couchbase/var/lib/couchbase/logs"/../crash/| wc -l [2021-02-12 21:24:57,954] - [remote_util] [140099333379840] - INFO - command executed successfully with root [2021-02-12 21:24:57,955] - [remote_util] [140099333379840] - INFO - SSH Connecting to 172.23.122.97 with username:root, attempt#1 of 5 [2021-02-12 21:24:58,065] - [remote_util] [140099333379840] - INFO - SSH Connected to 172.23.122.97 as root [2021-02-12 21:24:58,351] - [remote_util] [140099333379840] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:24:58,655] - [remote_util] [140099333379840] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:24:58,663] - [rest_client] [140099333379840] - INFO - /diag/eval status on 172.23.122.97:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). [2021-02-12 21:24:58,663] - [remote_util] [140099333379840] - INFO - running command.raw on 172.23.122.97: zgrep "panic" "/opt/couchbase/var/lib/couchbase/logs"/eventing.log* | wc -l [2021-02-12 21:24:58,929] - [remote_util] [140099333379840] - INFO - command executed successfully with root [2021-02-12 21:24:58,929] - [remote_util] [140099333379840] - INFO - running command.raw on 172.23.122.97: ls "/opt/couchbase/var/lib/couchbase/logs"/../crash/| wc -l [2021-02-12 21:24:58,982] - [remote_util] [140099333379840] - INFO - command executed successfully with root [2021-02-12 21:24:59,036] - [basetestcase] [140099333379840] - INFO - ------- Cluster statistics ------- [2021-02-12 21:24:59,036] - [basetestcase] [140099333379840] - INFO - 172.23.122.117:8091 => {'services': ['eventing'], 'cpu_utilization': 5.256410256410256, 'mem_free': 2955575296, 'mem_total': 4201684992, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-02-12 21:24:59,036] - [basetestcase] [140099333379840] - INFO - 172.23.122.118:8091 => {'services': ['kv'], 'cpu_utilization': 3.647028819178781, 'mem_free': 3022303232, 'mem_total': 4201689088, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-02-12 21:24:59,036] - [basetestcase] [140099333379840] - INFO - 172.23.122.82:8091 => {'services': ['index', 'n1ql'], 'cpu_utilization': 1.317122593718338, 'mem_free': 3224178688, 'mem_total': 4201684992, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-02-12 21:24:59,036] - [basetestcase] [140099333379840] - INFO - 172.23.122.90:8091 => {'services': ['eventing'], 'cpu_utilization': 9.44028888315708, 'mem_free': 2961907712, 'mem_total': 4201684992, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-02-12 21:24:59,036] - [basetestcase] [140099333379840] - INFO - 172.23.122.97:8091 => {'services': ['eventing'], 'cpu_utilization': 12.18879668049792, 'mem_free': 2965602304, 'mem_total': 4201684992, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-02-12 21:24:59,036] - [basetestcase] [140099333379840] - INFO - 172.23.123.61:8091 => {'services': ['kv'], 'cpu_utilization': 7.686434207177399, 'mem_free': 2926845952, 'mem_total': 4201672704, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-02-12 21:24:59,036] - [basetestcase] [140099333379840] - INFO - 172.23.123.62:8091 => {'services': ['kv'], 'cpu_utilization': 7.316449219749297, 'mem_free': 2945372160, 'mem_total': 4201676800, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-02-12 21:24:59,036] - [basetestcase] [140099333379840] - INFO - --- End of cluster statistics --- [2021-02-12 21:24:59,037] - [remote_util] [140099333379840] - INFO - SSH Connecting to 172.23.123.61 with username:root, attempt#1 of 5 [2021-02-12 21:24:59,139] - [remote_util] [140099333379840] - INFO - SSH Connected to 172.23.123.61 as root [2021-02-12 21:24:59,405] - [remote_util] [140099333379840] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:24:59,704] - [remote_util] [140099333379840] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:24:59,706] - [remote_util] [140099333379840] - INFO - SSH Connecting to 172.23.123.62 with username:root, attempt#1 of 5 [2021-02-12 21:24:59,821] - [remote_util] [140099333379840] - INFO - SSH Connected to 172.23.123.62 as root [2021-02-12 21:25:00,110] - [remote_util] [140099333379840] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:25:00,412] - [remote_util] [140099333379840] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:25:00,413] - [remote_util] [140099333379840] - INFO - SSH Connecting to 172.23.122.90 with username:root, attempt#1 of 5 [2021-02-12 21:25:00,520] - [remote_util] [140099333379840] - INFO - SSH Connected to 172.23.122.90 as root [2021-02-12 21:25:00,816] - [remote_util] [140099333379840] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:25:01,135] - [remote_util] [140099333379840] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:25:01,137] - [remote_util] [140099333379840] - INFO - SSH Connecting to 172.23.122.117 with username:root, attempt#1 of 5 [2021-02-12 21:25:01,235] - [remote_util] [140099333379840] - INFO - SSH Connected to 172.23.122.117 as root [2021-02-12 21:25:01,481] - [remote_util] [140099333379840] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:25:01,775] - [remote_util] [140099333379840] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:25:01,777] - [remote_util] [140099333379840] - INFO - SSH Connecting to 172.23.122.97 with username:root, attempt#1 of 5 [2021-02-12 21:25:01,881] - [remote_util] [140099333379840] - INFO - SSH Connected to 172.23.122.97 as root [2021-02-12 21:25:02,161] - [remote_util] [140099333379840] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:25:02,471] - [remote_util] [140099333379840] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:25:02,473] - [remote_util] [140099333379840] - INFO - SSH Connecting to 172.23.122.82 with username:root, attempt#1 of 5 [2021-02-12 21:25:02,574] - [remote_util] [140099333379840] - INFO - SSH Connected to 172.23.122.82 as root [2021-02-12 21:25:02,838] - [remote_util] [140099333379840] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:25:03,129] - [remote_util] [140099333379840] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:25:03,131] - [remote_util] [140099333379840] - INFO - SSH Connecting to 172.23.122.118 with username:root, attempt#1 of 5 [2021-02-12 21:25:03,232] - [remote_util] [140099333379840] - INFO - SSH Connected to 172.23.122.118 as root [2021-02-12 21:25:03,476] - [remote_util] [140099333379840] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:25:03,770] - [remote_util] [140099333379840] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:25:03,772] - [remote_util] [140099333379840] - INFO - SSH Connecting to 172.23.122.116 with username:root, attempt#1 of 5 [2021-02-12 21:25:03,871] - [remote_util] [140099333379840] - INFO - SSH Connected to 172.23.122.116 as root [2021-02-12 21:25:04,145] - [remote_util] [140099333379840] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:25:04,437] - [remote_util] [140099333379840] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:25:30,239] - [basetestcase] [140099333379840] - INFO - ============== basetestcase cleanup was started for test #4 test_kv_rebalance_in_when_existing_eventing_node_is_processing_mutations ============== [2021-02-12 21:25:30,418] - [bucket_helper] [140099333379840] - INFO - deleting existing buckets ['dst_bucket', 'metadata', 'src_bucket'] on 172.23.123.61 [2021-02-12 21:25:31,541] - [bucket_helper] [140099333379840] - INFO - waiting for bucket deletion to complete.... [2021-02-12 21:25:31,566] - [rest_client] [140099333379840] - INFO - node 172.23.123.61 existing buckets : ['metadata', 'src_bucket'] [2021-02-12 21:25:31,567] - [bucket_helper] [140099333379840] - INFO - deleted bucket : dst_bucket from 172.23.123.61 [2021-02-12 21:25:31,998] - [bucket_helper] [140099333379840] - INFO - waiting for bucket deletion to complete.... [2021-02-12 21:25:32,012] - [rest_client] [140099333379840] - INFO - node 172.23.123.61 existing buckets : ['src_bucket'] [2021-02-12 21:25:32,012] - [bucket_helper] [140099333379840] - INFO - deleted bucket : metadata from 172.23.123.61 [2021-02-12 21:25:33,138] - [bucket_helper] [140099333379840] - INFO - waiting for bucket deletion to complete.... [2021-02-12 21:25:33,140] - [rest_client] [140099333379840] - INFO - node 172.23.123.61 existing buckets : [] [2021-02-12 21:25:33,140] - [bucket_helper] [140099333379840] - INFO - deleted bucket : src_bucket from 172.23.123.61 [2021-02-12 21:25:33,149] - [bucket_helper] [140099333379840] - INFO - Could not find any buckets for node 172.23.123.62, nothing to delete [2021-02-12 21:25:33,157] - [bucket_helper] [140099333379840] - INFO - Could not find any buckets for node 172.23.122.90, nothing to delete [2021-02-12 21:25:33,163] - [bucket_helper] [140099333379840] - INFO - Could not find any buckets for node 172.23.122.117, nothing to delete [2021-02-12 21:25:33,172] - [bucket_helper] [140099333379840] - INFO - Could not find any buckets for node 172.23.122.97, nothing to delete [2021-02-12 21:25:33,179] - [bucket_helper] [140099333379840] - INFO - Could not find any buckets for node 172.23.122.82, nothing to delete [2021-02-12 21:25:33,185] - [bucket_helper] [140099333379840] - INFO - Could not find any buckets for node 172.23.122.118, nothing to delete [2021-02-12 21:25:33,190] - [bucket_helper] [140099333379840] - INFO - Could not find any buckets for node 172.23.122.116, nothing to delete [2021-02-12 21:25:33,197] - [rest_client] [140099333379840] - INFO - -->is_ns_server_running? [2021-02-12 21:25:33,215] - [cluster_helper] [140099333379840] - INFO - rebalancing all nodes in order to remove nodes [2021-02-12 21:25:33,217] - [rest_client] [140099333379840] - INFO - rebalance params : {'knownNodes': 'ns_1@172.23.122.117,ns_1@172.23.122.118,ns_1@172.23.122.82,ns_1@172.23.122.90,ns_1@172.23.122.97,ns_1@172.23.123.61,ns_1@172.23.123.62', 'ejectedNodes': 'ns_1@172.23.122.117,ns_1@172.23.122.118,ns_1@172.23.122.82,ns_1@172.23.122.90,ns_1@172.23.122.97,ns_1@172.23.123.62', 'user': 'Administrator', 'password': 'password'} [2021-02-12 21:25:33,244] - [rest_client] [140099333379840] - INFO - rebalance operation started [2021-02-12 21:25:33,247] - [rest_client] [140099333379840] - INFO - rebalance percentage : 0.00 % [2021-02-12 21:25:43,260] - [rest_client] [140099333379840] - INFO - rebalance percentage : 50.00 % [2021-02-12 21:25:53,274] - [rest_client] [140099333379840] - INFO - rebalance percentage : 92.00 % [2021-02-12 21:26:13,299] - [rest_client] [140099333379840] - INFO - rebalance progress took 40.06 seconds [2021-02-12 21:26:13,300] - [rest_client] [140099333379840] - INFO - sleep for 10 seconds after rebalance... [2021-02-12 21:26:23,344] - [cluster_helper] [140099333379840] - INFO - removed all the nodes from cluster associated with ip:172.23.123.61 port:8091 ssh_username:root ? [('ns_1@172.23.122.117', 8091), ('ns_1@172.23.122.118', 8091), ('ns_1@172.23.122.82', 8091), ('ns_1@172.23.122.90', 8091), ('ns_1@172.23.122.97', 8091), ('ns_1@172.23.123.62', 8091)] [2021-02-12 21:26:23,349] - [cluster_helper] [140099333379840] - INFO - waiting for ns_server @ 172.23.123.61:8091 [2021-02-12 21:26:23,350] - [rest_client] [140099333379840] - INFO - -->is_ns_server_running? [2021-02-12 21:26:23,355] - [cluster_helper] [140099333379840] - INFO - ns_server @ 172.23.123.61:8091 is running [2021-02-12 21:26:23,359] - [cluster_helper] [140099333379840] - INFO - waiting for ns_server @ 172.23.123.62:8091 [2021-02-12 21:26:23,359] - [rest_client] [140099333379840] - INFO - -->is_ns_server_running? [2021-02-12 21:26:23,362] - [cluster_helper] [140099333379840] - INFO - ns_server @ 172.23.123.62:8091 is running [2021-02-12 21:26:23,365] - [cluster_helper] [140099333379840] - INFO - waiting for ns_server @ 172.23.122.90:8091 [2021-02-12 21:26:23,365] - [rest_client] [140099333379840] - INFO - -->is_ns_server_running? [2021-02-12 21:26:23,368] - [cluster_helper] [140099333379840] - INFO - ns_server @ 172.23.122.90:8091 is running [2021-02-12 21:26:23,371] - [cluster_helper] [140099333379840] - INFO - waiting for ns_server @ 172.23.122.117:8091 [2021-02-12 21:26:23,371] - [rest_client] [140099333379840] - INFO - -->is_ns_server_running? [2021-02-12 21:26:23,374] - [cluster_helper] [140099333379840] - INFO - ns_server @ 172.23.122.117:8091 is running [2021-02-12 21:26:23,377] - [cluster_helper] [140099333379840] - INFO - waiting for ns_server @ 172.23.122.97:8091 [2021-02-12 21:26:23,377] - [rest_client] [140099333379840] - INFO - -->is_ns_server_running? [2021-02-12 21:26:23,380] - [cluster_helper] [140099333379840] - INFO - ns_server @ 172.23.122.97:8091 is running [2021-02-12 21:26:23,383] - [cluster_helper] [140099333379840] - INFO - waiting for ns_server @ 172.23.122.82:8091 [2021-02-12 21:26:23,383] - [rest_client] [140099333379840] - INFO - -->is_ns_server_running? [2021-02-12 21:26:23,385] - [cluster_helper] [140099333379840] - INFO - ns_server @ 172.23.122.82:8091 is running [2021-02-12 21:26:23,388] - [cluster_helper] [140099333379840] - INFO - waiting for ns_server @ 172.23.122.118:8091 [2021-02-12 21:26:23,389] - [rest_client] [140099333379840] - INFO - -->is_ns_server_running? [2021-02-12 21:26:23,391] - [cluster_helper] [140099333379840] - INFO - ns_server @ 172.23.122.118:8091 is running [2021-02-12 21:26:23,395] - [cluster_helper] [140099333379840] - INFO - waiting for ns_server @ 172.23.122.116:8091 [2021-02-12 21:26:23,395] - [rest_client] [140099333379840] - INFO - -->is_ns_server_running? [2021-02-12 21:26:23,398] - [cluster_helper] [140099333379840] - INFO - ns_server @ 172.23.122.116:8091 is running [2021-02-12 21:26:23,398] - [basetestcase] [140099333379840] - INFO - ============== basetestcase cleanup was finished for test #4 test_kv_rebalance_in_when_existing_eventing_node_is_processing_mutations ============== [2021-02-12 21:26:23,398] - [basetestcase] [140099333379840] - INFO - closing all ssh connections [2021-02-12 21:26:23,400] - [basetestcase] [140099333379840] - INFO - closing all memcached connections [2021-02-12 21:26:23,401] - [remote_util] [140099333379840] - INFO - SSH Connecting to 172.23.123.61 with username:root, attempt#1 of 5 [2021-02-12 21:26:23,506] - [remote_util] [140099333379840] - INFO - SSH Connected to 172.23.123.61 as root [2021-02-12 21:26:23,774] - [remote_util] [140099333379840] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:26:24,074] - [remote_util] [140099333379840] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:26:24,083] - [remote_util] [140099333379840] - INFO - SSH Connecting to 172.23.123.61 with username:root, attempt#1 of 5 [2021-02-12 21:26:24,182] - [remote_util] [140099333379840] - INFO - SSH Connected to 172.23.123.61 as root [2021-02-12 21:26:24,440] - [remote_util] [140099333379840] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:26:24,736] - [remote_util] [140099333379840] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:26:24,794] - [remote_util] [140099333379840] - INFO - 172.23.123.61 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 21:26:24,846] - [remote_util] [140099333379840] - INFO - Running systemd command on this server [2021-02-12 21:26:24,846] - [remote_util] [140099333379840] - INFO - running command.raw on 172.23.123.61: systemctl stop couchbase-server.service [2021-02-12 21:26:26,611] - [remote_util] [140099333379840] - INFO - command executed successfully with root [2021-02-12 21:26:26,611] - [basetestcase] [140099333379840] - INFO - Couchbase stopped [2021-02-12 21:26:26,612] - [basetestcase] [140099333379840] - INFO - sleep for 5 secs. ... [2021-02-12 21:26:31,617] - [remote_util] [140099333379840] - INFO - running command.raw on 172.23.123.61: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-12 21:26:31,633] - [remote_util] [140099333379840] - INFO - command executed successfully with root [2021-02-12 21:26:31,633] - [remote_util] [140099333379840] - INFO - running command.raw on 172.23.123.61: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-12 21:26:31,697] - [remote_util] [140099333379840] - INFO - command executed successfully with root [2021-02-12 21:26:31,699] - [remote_util] [140099333379840] - INFO - SSH Connecting to 172.23.123.61 with username:root, attempt#1 of 5 [2021-02-12 21:26:31,799] - [remote_util] [140099333379840] - INFO - SSH Connected to 172.23.123.61 as root [2021-02-12 21:26:32,076] - [remote_util] [140099333379840] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:26:32,373] - [remote_util] [140099333379840] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:26:32,431] - [remote_util] [140099333379840] - INFO - 172.23.123.61 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 21:26:32,432] - [remote_util] [140099333379840] - INFO - Checking for process beam.smp on linux [2021-02-12 21:26:32,470] - [remote_util] [140099333379840] - INFO - Starting couchbase server [2021-02-12 21:26:32,565] - [remote_util] [140099333379840] - INFO - Running systemd command on this server [2021-02-12 21:26:32,565] - [remote_util] [140099333379840] - INFO - running command.raw on 172.23.123.61: systemctl start couchbase-server.service [2021-02-12 21:26:32,587] - [remote_util] [140099333379840] - INFO - command executed successfully with root [2021-02-12 21:26:32,587] - [remote_util] [140099333379840] - INFO - 172.23.123.61:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-12 21:26:37,593] - [remote_util] [140099333379840] - INFO - running command.raw on 172.23.123.61: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-12 21:26:37,619] - [remote_util] [140099333379840] - INFO - command executed successfully with root [2021-02-12 21:26:37,619] - [remote_util] [140099333379840] - INFO - Couchbase server status: [' Process: 85882 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-12 21:26:37,619] - [remote_util] [140099333379840] - INFO - Checking for process beam.smp on linux [2021-02-12 21:26:37,686] - [remote_util] [140099333379840] - INFO - process beam.smp is running on 172.23.123.61: with pid 86026 [2021-02-12 21:26:37,686] - [basetestcase] [140099333379840] - INFO - Couchbase started [2021-02-12 21:26:37,687] - [remote_util] [140099333379840] - INFO - SSH Connecting to 172.23.123.62 with username:root, attempt#1 of 5 [2021-02-12 21:26:37,791] - [remote_util] [140099333379840] - INFO - SSH Connected to 172.23.123.62 as root [2021-02-12 21:26:38,065] - [remote_util] [140099333379840] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:26:38,362] - [remote_util] [140099333379840] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:26:38,370] - [remote_util] [140099333379840] - INFO - SSH Connecting to 172.23.123.62 with username:root, attempt#1 of 5 [2021-02-12 21:26:38,473] - [remote_util] [140099333379840] - INFO - SSH Connected to 172.23.123.62 as root [2021-02-12 21:26:38,752] - [remote_util] [140099333379840] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:26:39,061] - [remote_util] [140099333379840] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:26:39,080] - [remote_util] [140099333379840] - INFO - 172.23.123.62 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 21:26:39,135] - [remote_util] [140099333379840] - INFO - Running systemd command on this server [2021-02-12 21:26:39,135] - [remote_util] [140099333379840] - INFO - running command.raw on 172.23.123.62: systemctl stop couchbase-server.service [2021-02-12 21:26:40,905] - [remote_util] [140099333379840] - INFO - command executed successfully with root [2021-02-12 21:26:40,906] - [basetestcase] [140099333379840] - INFO - Couchbase stopped [2021-02-12 21:26:40,906] - [basetestcase] [140099333379840] - INFO - sleep for 5 secs. ... [2021-02-12 21:26:45,911] - [remote_util] [140099333379840] - INFO - running command.raw on 172.23.123.62: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-12 21:26:45,927] - [remote_util] [140099333379840] - INFO - command executed successfully with root [2021-02-12 21:26:45,927] - [remote_util] [140099333379840] - INFO - running command.raw on 172.23.123.62: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-12 21:26:45,982] - [remote_util] [140099333379840] - INFO - command executed successfully with root [2021-02-12 21:26:45,984] - [remote_util] [140099333379840] - INFO - SSH Connecting to 172.23.123.62 with username:root, attempt#1 of 5 [2021-02-12 21:26:46,085] - [remote_util] [140099333379840] - INFO - SSH Connected to 172.23.123.62 as root [2021-02-12 21:26:46,366] - [remote_util] [140099333379840] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:26:46,665] - [remote_util] [140099333379840] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:26:46,685] - [remote_util] [140099333379840] - INFO - 172.23.123.62 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 21:26:46,685] - [remote_util] [140099333379840] - INFO - Checking for process beam.smp on linux [2021-02-12 21:26:46,724] - [remote_util] [140099333379840] - INFO - Starting couchbase server [2021-02-12 21:26:46,822] - [remote_util] [140099333379840] - INFO - Running systemd command on this server [2021-02-12 21:26:46,822] - [remote_util] [140099333379840] - INFO - running command.raw on 172.23.123.62: systemctl start couchbase-server.service [2021-02-12 21:26:46,847] - [remote_util] [140099333379840] - INFO - command executed successfully with root [2021-02-12 21:26:46,847] - [remote_util] [140099333379840] - INFO - 172.23.123.62:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-12 21:26:51,852] - [remote_util] [140099333379840] - INFO - running command.raw on 172.23.123.62: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-12 21:26:51,878] - [remote_util] [140099333379840] - INFO - command executed successfully with root [2021-02-12 21:26:51,878] - [remote_util] [140099333379840] - INFO - Couchbase server status: [' Process: 99232 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-12 21:26:51,878] - [remote_util] [140099333379840] - INFO - Checking for process beam.smp on linux [2021-02-12 21:26:51,946] - [remote_util] [140099333379840] - INFO - process beam.smp is running on 172.23.123.62: with pid 99375 [2021-02-12 21:26:51,947] - [basetestcase] [140099333379840] - INFO - Couchbase started [2021-02-12 21:26:51,948] - [remote_util] [140099333379840] - INFO - SSH Connecting to 172.23.122.90 with username:root, attempt#1 of 5 [2021-02-12 21:26:52,055] - [remote_util] [140099333379840] - INFO - SSH Connected to 172.23.122.90 as root [2021-02-12 21:26:52,334] - [remote_util] [140099333379840] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:26:52,627] - [remote_util] [140099333379840] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:26:52,636] - [remote_util] [140099333379840] - INFO - SSH Connecting to 172.23.122.90 with username:root, attempt#1 of 5 [2021-02-12 21:26:52,734] - [remote_util] [140099333379840] - INFO - SSH Connected to 172.23.122.90 as root [2021-02-12 21:26:53,001] - [remote_util] [140099333379840] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:26:53,297] - [remote_util] [140099333379840] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:26:53,317] - [remote_util] [140099333379840] - INFO - 172.23.122.90 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 21:26:53,370] - [remote_util] [140099333379840] - INFO - Running systemd command on this server [2021-02-12 21:26:53,370] - [remote_util] [140099333379840] - INFO - running command.raw on 172.23.122.90: systemctl stop couchbase-server.service [2021-02-12 21:26:55,033] - [remote_util] [140099333379840] - INFO - command executed successfully with root [2021-02-12 21:26:55,033] - [basetestcase] [140099333379840] - INFO - Couchbase stopped [2021-02-12 21:26:55,033] - [basetestcase] [140099333379840] - INFO - sleep for 5 secs. ... [2021-02-12 21:27:00,039] - [remote_util] [140099333379840] - INFO - running command.raw on 172.23.122.90: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-12 21:27:00,062] - [remote_util] [140099333379840] - INFO - command executed successfully with root [2021-02-12 21:27:00,062] - [remote_util] [140099333379840] - INFO - running command.raw on 172.23.122.90: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-12 21:27:00,116] - [remote_util] [140099333379840] - INFO - command executed successfully with root [2021-02-12 21:27:00,118] - [remote_util] [140099333379840] - INFO - SSH Connecting to 172.23.122.90 with username:root, attempt#1 of 5 [2021-02-12 21:27:00,219] - [remote_util] [140099333379840] - INFO - SSH Connected to 172.23.122.90 as root [2021-02-12 21:27:00,498] - [remote_util] [140099333379840] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:27:00,790] - [remote_util] [140099333379840] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:27:00,808] - [remote_util] [140099333379840] - INFO - 172.23.122.90 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 21:27:00,808] - [remote_util] [140099333379840] - INFO - Checking for process beam.smp on linux [2021-02-12 21:27:00,846] - [remote_util] [140099333379840] - INFO - Starting couchbase server [2021-02-12 21:27:00,939] - [remote_util] [140099333379840] - INFO - Running systemd command on this server [2021-02-12 21:27:00,939] - [remote_util] [140099333379840] - INFO - running command.raw on 172.23.122.90: systemctl start couchbase-server.service [2021-02-12 21:27:00,963] - [remote_util] [140099333379840] - INFO - command executed successfully with root [2021-02-12 21:27:00,963] - [remote_util] [140099333379840] - INFO - 172.23.122.90:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-12 21:27:05,968] - [remote_util] [140099333379840] - INFO - running command.raw on 172.23.122.90: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-12 21:27:05,991] - [remote_util] [140099333379840] - INFO - command executed successfully with root [2021-02-12 21:27:05,991] - [remote_util] [140099333379840] - INFO - Couchbase server status: [' Process: 64491 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-12 21:27:05,992] - [remote_util] [140099333379840] - INFO - Checking for process beam.smp on linux [2021-02-12 21:27:06,059] - [remote_util] [140099333379840] - INFO - process beam.smp is running on 172.23.122.90: with pid 64635 [2021-02-12 21:27:06,059] - [basetestcase] [140099333379840] - INFO - Couchbase started [2021-02-12 21:27:06,060] - [remote_util] [140099333379840] - INFO - SSH Connecting to 172.23.122.117 with username:root, attempt#1 of 5 [2021-02-12 21:27:06,164] - [remote_util] [140099333379840] - INFO - SSH Connected to 172.23.122.117 as root [2021-02-12 21:27:06,412] - [remote_util] [140099333379840] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:27:06,702] - [remote_util] [140099333379840] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:27:06,709] - [remote_util] [140099333379840] - INFO - SSH Connecting to 172.23.122.117 with username:root, attempt#1 of 5 [2021-02-12 21:27:06,802] - [remote_util] [140099333379840] - INFO - SSH Connected to 172.23.122.117 as root [2021-02-12 21:27:07,044] - [remote_util] [140099333379840] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:27:07,335] - [remote_util] [140099333379840] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:27:07,354] - [remote_util] [140099333379840] - INFO - 172.23.122.117 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 21:27:07,404] - [remote_util] [140099333379840] - INFO - Running systemd command on this server [2021-02-12 21:27:07,404] - [remote_util] [140099333379840] - INFO - running command.raw on 172.23.122.117: systemctl stop couchbase-server.service [2021-02-12 21:27:09,185] - [remote_util] [140099333379840] - INFO - command executed successfully with root [2021-02-12 21:27:09,185] - [basetestcase] [140099333379840] - INFO - Couchbase stopped [2021-02-12 21:27:09,185] - [basetestcase] [140099333379840] - INFO - sleep for 5 secs. ... [2021-02-12 21:27:14,191] - [remote_util] [140099333379840] - INFO - running command.raw on 172.23.122.117: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-12 21:27:14,210] - [remote_util] [140099333379840] - INFO - command executed successfully with root [2021-02-12 21:27:14,210] - [remote_util] [140099333379840] - INFO - running command.raw on 172.23.122.117: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-12 21:27:14,263] - [remote_util] [140099333379840] - INFO - command executed successfully with root [2021-02-12 21:27:14,264] - [remote_util] [140099333379840] - INFO - SSH Connecting to 172.23.122.117 with username:root, attempt#1 of 5 [2021-02-12 21:27:14,363] - [remote_util] [140099333379840] - INFO - SSH Connected to 172.23.122.117 as root [2021-02-12 21:27:14,612] - [remote_util] [140099333379840] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:27:14,905] - [remote_util] [140099333379840] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:27:14,923] - [remote_util] [140099333379840] - INFO - 172.23.122.117 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 21:27:14,923] - [remote_util] [140099333379840] - INFO - Checking for process beam.smp on linux [2021-02-12 21:27:14,959] - [remote_util] [140099333379840] - INFO - Starting couchbase server [2021-02-12 21:27:15,050] - [remote_util] [140099333379840] - INFO - Running systemd command on this server [2021-02-12 21:27:15,051] - [remote_util] [140099333379840] - INFO - running command.raw on 172.23.122.117: systemctl start couchbase-server.service [2021-02-12 21:27:15,071] - [remote_util] [140099333379840] - INFO - command executed successfully with root [2021-02-12 21:27:15,072] - [remote_util] [140099333379840] - INFO - 172.23.122.117:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-12 21:27:20,076] - [remote_util] [140099333379840] - INFO - running command.raw on 172.23.122.117: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-12 21:27:20,101] - [remote_util] [140099333379840] - INFO - command executed successfully with root [2021-02-12 21:27:20,101] - [remote_util] [140099333379840] - INFO - Couchbase server status: [' Process: 10928 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-12 21:27:20,102] - [remote_util] [140099333379840] - INFO - Checking for process beam.smp on linux [2021-02-12 21:27:20,169] - [remote_util] [140099333379840] - INFO - process beam.smp is running on 172.23.122.117: with pid 11072 [2021-02-12 21:27:20,169] - [basetestcase] [140099333379840] - INFO - Couchbase started [2021-02-12 21:27:20,170] - [remote_util] [140099333379840] - INFO - SSH Connecting to 172.23.122.97 with username:root, attempt#1 of 5 [2021-02-12 21:27:20,275] - [remote_util] [140099333379840] - INFO - SSH Connected to 172.23.122.97 as root [2021-02-12 21:27:20,544] - [remote_util] [140099333379840] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:27:20,843] - [remote_util] [140099333379840] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:27:20,851] - [remote_util] [140099333379840] - INFO - SSH Connecting to 172.23.122.97 with username:root, attempt#1 of 5 [2021-02-12 21:27:20,949] - [remote_util] [140099333379840] - INFO - SSH Connected to 172.23.122.97 as root [2021-02-12 21:27:21,206] - [remote_util] [140099333379840] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:27:21,508] - [remote_util] [140099333379840] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:27:21,527] - [remote_util] [140099333379840] - INFO - 172.23.122.97 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 21:27:21,582] - [remote_util] [140099333379840] - INFO - Running systemd command on this server [2021-02-12 21:27:21,582] - [remote_util] [140099333379840] - INFO - running command.raw on 172.23.122.97: systemctl stop couchbase-server.service [2021-02-12 21:27:23,352] - [remote_util] [140099333379840] - INFO - command executed successfully with root [2021-02-12 21:27:23,352] - [basetestcase] [140099333379840] - INFO - Couchbase stopped [2021-02-12 21:27:23,352] - [basetestcase] [140099333379840] - INFO - sleep for 5 secs. ... [2021-02-12 21:27:28,355] - [remote_util] [140099333379840] - INFO - running command.raw on 172.23.122.97: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-12 21:27:28,378] - [remote_util] [140099333379840] - INFO - command executed successfully with root [2021-02-12 21:27:28,378] - [remote_util] [140099333379840] - INFO - running command.raw on 172.23.122.97: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-12 21:27:28,433] - [remote_util] [140099333379840] - INFO - command executed successfully with root [2021-02-12 21:27:28,435] - [remote_util] [140099333379840] - INFO - SSH Connecting to 172.23.122.97 with username:root, attempt#1 of 5 [2021-02-12 21:27:28,535] - [remote_util] [140099333379840] - INFO - SSH Connected to 172.23.122.97 as root [2021-02-12 21:27:28,813] - [remote_util] [140099333379840] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:27:29,117] - [remote_util] [140099333379840] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:27:29,138] - [remote_util] [140099333379840] - INFO - 172.23.122.97 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 21:27:29,138] - [remote_util] [140099333379840] - INFO - Checking for process beam.smp on linux [2021-02-12 21:27:29,178] - [remote_util] [140099333379840] - INFO - Starting couchbase server [2021-02-12 21:27:29,279] - [remote_util] [140099333379840] - INFO - Running systemd command on this server [2021-02-12 21:27:29,279] - [remote_util] [140099333379840] - INFO - running command.raw on 172.23.122.97: systemctl start couchbase-server.service [2021-02-12 21:27:29,302] - [remote_util] [140099333379840] - INFO - command executed successfully with root [2021-02-12 21:27:29,302] - [remote_util] [140099333379840] - INFO - 172.23.122.97:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-12 21:27:34,307] - [remote_util] [140099333379840] - INFO - running command.raw on 172.23.122.97: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-12 21:27:34,336] - [remote_util] [140099333379840] - INFO - command executed successfully with root [2021-02-12 21:27:34,336] - [remote_util] [140099333379840] - INFO - Couchbase server status: [' Process: 44031 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-12 21:27:34,336] - [remote_util] [140099333379840] - INFO - Checking for process beam.smp on linux [2021-02-12 21:27:34,403] - [remote_util] [140099333379840] - INFO - process beam.smp is running on 172.23.122.97: with pid 44176 [2021-02-12 21:27:34,403] - [basetestcase] [140099333379840] - INFO - Couchbase started [2021-02-12 21:27:34,405] - [remote_util] [140099333379840] - INFO - SSH Connecting to 172.23.122.82 with username:root, attempt#1 of 5 [2021-02-12 21:27:34,508] - [remote_util] [140099333379840] - INFO - SSH Connected to 172.23.122.82 as root [2021-02-12 21:27:34,771] - [remote_util] [140099333379840] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:27:35,061] - [remote_util] [140099333379840] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:27:35,069] - [remote_util] [140099333379840] - INFO - SSH Connecting to 172.23.122.82 with username:root, attempt#1 of 5 [2021-02-12 21:27:35,163] - [remote_util] [140099333379840] - INFO - SSH Connected to 172.23.122.82 as root [2021-02-12 21:27:35,416] - [remote_util] [140099333379840] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:27:35,703] - [remote_util] [140099333379840] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:27:35,761] - [remote_util] [140099333379840] - INFO - 172.23.122.82 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 21:27:35,812] - [remote_util] [140099333379840] - INFO - Running systemd command on this server [2021-02-12 21:27:35,812] - [remote_util] [140099333379840] - INFO - running command.raw on 172.23.122.82: systemctl stop couchbase-server.service [2021-02-12 21:27:37,521] - [remote_util] [140099333379840] - INFO - command executed successfully with root [2021-02-12 21:27:37,522] - [basetestcase] [140099333379840] - INFO - Couchbase stopped [2021-02-12 21:27:37,522] - [basetestcase] [140099333379840] - INFO - sleep for 5 secs. ... [2021-02-12 21:27:42,527] - [remote_util] [140099333379840] - INFO - running command.raw on 172.23.122.82: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-12 21:27:42,544] - [remote_util] [140099333379840] - INFO - command executed successfully with root [2021-02-12 21:27:42,545] - [remote_util] [140099333379840] - INFO - running command.raw on 172.23.122.82: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-12 21:27:42,598] - [remote_util] [140099333379840] - INFO - command executed successfully with root [2021-02-12 21:27:42,600] - [remote_util] [140099333379840] - INFO - SSH Connecting to 172.23.122.82 with username:root, attempt#1 of 5 [2021-02-12 21:27:42,699] - [remote_util] [140099333379840] - INFO - SSH Connected to 172.23.122.82 as root [2021-02-12 21:27:42,962] - [remote_util] [140099333379840] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:27:43,257] - [remote_util] [140099333379840] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:27:43,277] - [remote_util] [140099333379840] - INFO - 172.23.122.82 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 21:27:43,277] - [remote_util] [140099333379840] - INFO - Checking for process beam.smp on linux [2021-02-12 21:27:43,317] - [remote_util] [140099333379840] - INFO - Starting couchbase server [2021-02-12 21:27:43,409] - [remote_util] [140099333379840] - INFO - Running systemd command on this server [2021-02-12 21:27:43,409] - [remote_util] [140099333379840] - INFO - running command.raw on 172.23.122.82: systemctl start couchbase-server.service [2021-02-12 21:27:43,432] - [remote_util] [140099333379840] - INFO - command executed successfully with root [2021-02-12 21:27:43,432] - [remote_util] [140099333379840] - INFO - 172.23.122.82:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-12 21:27:48,433] - [remote_util] [140099333379840] - INFO - running command.raw on 172.23.122.82: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-12 21:27:48,461] - [remote_util] [140099333379840] - INFO - command executed successfully with root [2021-02-12 21:27:48,461] - [remote_util] [140099333379840] - INFO - Couchbase server status: [' Process: 35921 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-12 21:27:48,461] - [remote_util] [140099333379840] - INFO - Checking for process beam.smp on linux [2021-02-12 21:27:48,529] - [remote_util] [140099333379840] - INFO - process beam.smp is running on 172.23.122.82: with pid 36065 [2021-02-12 21:27:48,529] - [basetestcase] [140099333379840] - INFO - Couchbase started [2021-02-12 21:27:48,531] - [remote_util] [140099333379840] - INFO - SSH Connecting to 172.23.122.118 with username:root, attempt#1 of 5 [2021-02-12 21:27:48,630] - [remote_util] [140099333379840] - INFO - SSH Connected to 172.23.122.118 as root [2021-02-12 21:27:48,873] - [remote_util] [140099333379840] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:27:49,161] - [remote_util] [140099333379840] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:27:49,168] - [remote_util] [140099333379840] - INFO - SSH Connecting to 172.23.122.118 with username:root, attempt#1 of 5 [2021-02-12 21:27:49,262] - [remote_util] [140099333379840] - INFO - SSH Connected to 172.23.122.118 as root [2021-02-12 21:27:49,500] - [remote_util] [140099333379840] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:27:49,791] - [remote_util] [140099333379840] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:27:49,809] - [remote_util] [140099333379840] - INFO - 172.23.122.118 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 21:27:49,861] - [remote_util] [140099333379840] - INFO - Running systemd command on this server [2021-02-12 21:27:49,861] - [remote_util] [140099333379840] - INFO - running command.raw on 172.23.122.118: systemctl stop couchbase-server.service [2021-02-12 21:27:51,611] - [remote_util] [140099333379840] - INFO - command executed successfully with root [2021-02-12 21:27:51,611] - [basetestcase] [140099333379840] - INFO - Couchbase stopped [2021-02-12 21:27:51,611] - [basetestcase] [140099333379840] - INFO - sleep for 5 secs. ... [2021-02-12 21:27:56,616] - [remote_util] [140099333379840] - INFO - running command.raw on 172.23.122.118: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-12 21:27:56,632] - [remote_util] [140099333379840] - INFO - command executed successfully with root [2021-02-12 21:27:56,632] - [remote_util] [140099333379840] - INFO - running command.raw on 172.23.122.118: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-12 21:27:56,685] - [remote_util] [140099333379840] - INFO - command executed successfully with root [2021-02-12 21:27:56,686] - [remote_util] [140099333379840] - INFO - SSH Connecting to 172.23.122.118 with username:root, attempt#1 of 5 [2021-02-12 21:27:56,784] - [remote_util] [140099333379840] - INFO - SSH Connected to 172.23.122.118 as root [2021-02-12 21:27:57,024] - [remote_util] [140099333379840] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:27:57,315] - [remote_util] [140099333379840] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:27:57,333] - [remote_util] [140099333379840] - INFO - 172.23.122.118 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 21:27:57,334] - [remote_util] [140099333379840] - INFO - Checking for process beam.smp on linux [2021-02-12 21:27:57,369] - [remote_util] [140099333379840] - INFO - Starting couchbase server [2021-02-12 21:27:57,461] - [remote_util] [140099333379840] - INFO - Running systemd command on this server [2021-02-12 21:27:57,461] - [remote_util] [140099333379840] - INFO - running command.raw on 172.23.122.118: systemctl start couchbase-server.service [2021-02-12 21:27:57,484] - [remote_util] [140099333379840] - INFO - command executed successfully with root [2021-02-12 21:27:57,484] - [remote_util] [140099333379840] - INFO - 172.23.122.118:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-12 21:28:02,485] - [remote_util] [140099333379840] - INFO - running command.raw on 172.23.122.118: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-12 21:28:02,510] - [remote_util] [140099333379840] - INFO - command executed successfully with root [2021-02-12 21:28:02,511] - [remote_util] [140099333379840] - INFO - Couchbase server status: [' Process: 76698 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-12 21:28:02,511] - [remote_util] [140099333379840] - INFO - Checking for process beam.smp on linux [2021-02-12 21:28:02,578] - [remote_util] [140099333379840] - INFO - process beam.smp is running on 172.23.122.118: with pid 76843 [2021-02-12 21:28:02,578] - [basetestcase] [140099333379840] - INFO - Couchbase started [2021-02-12 21:28:02,580] - [remote_util] [140099333379840] - INFO - SSH Connecting to 172.23.122.116 with username:root, attempt#1 of 5 [2021-02-12 21:28:02,682] - [remote_util] [140099333379840] - INFO - SSH Connected to 172.23.122.116 as root [2021-02-12 21:28:02,951] - [remote_util] [140099333379840] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:28:03,246] - [remote_util] [140099333379840] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:28:03,255] - [remote_util] [140099333379840] - INFO - SSH Connecting to 172.23.122.116 with username:root, attempt#1 of 5 [2021-02-12 21:28:03,352] - [remote_util] [140099333379840] - INFO - SSH Connected to 172.23.122.116 as root [2021-02-12 21:28:03,618] - [remote_util] [140099333379840] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:28:03,910] - [remote_util] [140099333379840] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:28:03,928] - [remote_util] [140099333379840] - INFO - 172.23.122.116 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 21:28:03,979] - [remote_util] [140099333379840] - INFO - Running systemd command on this server [2021-02-12 21:28:03,979] - [remote_util] [140099333379840] - INFO - running command.raw on 172.23.122.116: systemctl stop couchbase-server.service [2021-02-12 21:28:06,176] - [remote_util] [140099333379840] - INFO - command executed successfully with root [2021-02-12 21:28:06,177] - [basetestcase] [140099333379840] - INFO - Couchbase stopped [2021-02-12 21:28:06,177] - [basetestcase] [140099333379840] - INFO - sleep for 5 secs. ... [2021-02-12 21:28:11,182] - [remote_util] [140099333379840] - INFO - running command.raw on 172.23.122.116: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-12 21:28:11,197] - [remote_util] [140099333379840] - INFO - command executed successfully with root [2021-02-12 21:28:11,197] - [remote_util] [140099333379840] - INFO - running command.raw on 172.23.122.116: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-12 21:28:11,250] - [remote_util] [140099333379840] - INFO - command executed successfully with root [2021-02-12 21:28:11,251] - [remote_util] [140099333379840] - INFO - SSH Connecting to 172.23.122.116 with username:root, attempt#1 of 5 [2021-02-12 21:28:11,350] - [remote_util] [140099333379840] - INFO - SSH Connected to 172.23.122.116 as root [2021-02-12 21:28:11,618] - [remote_util] [140099333379840] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:28:11,910] - [remote_util] [140099333379840] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:28:11,929] - [remote_util] [140099333379840] - INFO - 172.23.122.116 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 21:28:11,929] - [remote_util] [140099333379840] - INFO - Checking for process beam.smp on linux [2021-02-12 21:28:11,965] - [remote_util] [140099333379840] - INFO - Starting couchbase server [2021-02-12 21:28:12,057] - [remote_util] [140099333379840] - INFO - Running systemd command on this server [2021-02-12 21:28:12,057] - [remote_util] [140099333379840] - INFO - running command.raw on 172.23.122.116: systemctl start couchbase-server.service [2021-02-12 21:28:12,079] - [remote_util] [140099333379840] - INFO - command executed successfully with root [2021-02-12 21:28:12,079] - [remote_util] [140099333379840] - INFO - 172.23.122.116:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-12 21:28:17,084] - [remote_util] [140099333379840] - INFO - running command.raw on 172.23.122.116: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-12 21:28:17,108] - [remote_util] [140099333379840] - INFO - command executed successfully with root [2021-02-12 21:28:17,108] - [remote_util] [140099333379840] - INFO - Couchbase server status: [' Process: 40908 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-12 21:28:17,109] - [remote_util] [140099333379840] - INFO - Checking for process beam.smp on linux [2021-02-12 21:28:17,174] - [remote_util] [140099333379840] - INFO - process beam.smp is running on 172.23.122.116: with pid 41053 [2021-02-12 21:28:17,174] - [basetestcase] [140099333379840] - INFO - Couchbase started [2021-02-12 21:28:17,175] - [basetestcase] [140099333379840] - INFO - sleep for 10 secs. ... [2021-02-12 21:28:27,205] - [remote_util] [140099333379840] - INFO - SSH Connecting to 172.23.123.61 with username:root, attempt#1 of 5 [2021-02-12 21:28:27,207] - [remote_util] [140099567843072] - INFO - SSH Connecting to 172.23.123.62 with username:root, attempt#1 of 5 [2021-02-12 21:28:27,209] - [remote_util] [140099341772544] - INFO - SSH Connecting to 172.23.122.90 with username:root, attempt#1 of 5 [2021-02-12 21:28:27,213] - [remote_util] [140099316594432] - INFO - SSH Connecting to 172.23.122.117 with username:root, attempt#1 of 5 [2021-02-12 21:28:27,214] - [remote_util] [140098729400064] - INFO - SSH Connecting to 172.23.122.82 with username:root, attempt#1 of 5 [2021-02-12 21:28:27,216] - [remote_util] [140099308201728] - INFO - SSH Connecting to 172.23.122.97 with username:root, attempt#1 of 5 [2021-02-12 21:28:27,217] - [remote_util] [140098695829248] - INFO - SSH Connecting to 172.23.122.118 with username:root, attempt#1 of 5 [2021-02-12 21:28:27,220] - [remote_util] [140098185717504] - INFO - SSH Connecting to 172.23.122.116 with username:root, attempt#1 of 5 [2021-02-12 21:28:27,316] - [remote_util] [140099341772544] - INFO - SSH Connected to 172.23.122.90 as root [2021-02-12 21:28:27,319] - [remote_util] [140099316594432] - INFO - SSH Connected to 172.23.122.117 as root [2021-02-12 21:28:27,320] - [remote_util] [140098695829248] - INFO - SSH Connected to 172.23.122.118 as root [2021-02-12 21:28:27,321] - [remote_util] [140099333379840] - INFO - SSH Connected to 172.23.123.61 as root [2021-02-12 21:28:27,323] - [remote_util] [140098185717504] - INFO - SSH Connected to 172.23.122.116 as root [2021-02-12 21:28:27,324] - [remote_util] [140098729400064] - INFO - SSH Connected to 172.23.122.82 as root [2021-02-12 21:28:27,324] - [remote_util] [140099567843072] - INFO - SSH Connected to 172.23.123.62 as root [2021-02-12 21:28:27,330] - [remote_util] [140099308201728] - INFO - SSH Connected to 172.23.122.97 as root [2021-02-12 21:28:27,577] - [remote_util] [140098695829248] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:28:27,579] - [remote_util] [140099316594432] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:28:27,620] - [remote_util] [140099308201728] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:28:27,632] - [remote_util] [140098729400064] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:28:27,637] - [remote_util] [140098185717504] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:28:27,639] - [remote_util] [140099333379840] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:28:27,651] - [remote_util] [140099341772544] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:28:27,660] - [remote_util] [140099567843072] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:28:27,925] - [remote_util] [140099316594432] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:28:27,927] - [remote_util] [140099316594432] - INFO - running command.raw on 172.23.122.117: /opt/couchbase/bin/cbcollect_info 172.23.122.117-20210212-2128-diag.zip [2021-02-12 21:28:27,933] - [remote_util] [140098695829248] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:28:27,933] - [remote_util] [140098695829248] - INFO - running command.raw on 172.23.122.118: /opt/couchbase/bin/cbcollect_info 172.23.122.118-20210212-2128-diag.zip [2021-02-12 21:28:27,939] - [remote_util] [140098185717504] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:28:27,940] - [remote_util] [140098185717504] - INFO - running command.raw on 172.23.122.116: /opt/couchbase/bin/cbcollect_info 172.23.122.116-20210212-2128-diag.zip [2021-02-12 21:28:27,955] - [remote_util] [140099341772544] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:28:27,956] - [remote_util] [140099341772544] - INFO - running command.raw on 172.23.122.90: /opt/couchbase/bin/cbcollect_info 172.23.122.90-20210212-2128-diag.zip [2021-02-12 21:28:27,974] - [remote_util] [140099567843072] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:28:27,975] - [remote_util] [140099567843072] - INFO - running command.raw on 172.23.123.62: /opt/couchbase/bin/cbcollect_info 172.23.123.62-20210212-2128-diag.zip [2021-02-12 21:28:27,982] - [remote_util] [140098729400064] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:28:27,982] - [remote_util] [140098729400064] - INFO - running command.raw on 172.23.122.82: /opt/couchbase/bin/cbcollect_info 172.23.122.82-20210212-2128-diag.zip [2021-02-12 21:28:27,984] - [remote_util] [140099308201728] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:28:27,984] - [remote_util] [140099308201728] - INFO - running command.raw on 172.23.122.97: /opt/couchbase/bin/cbcollect_info 172.23.122.97-20210212-2128-diag.zip [2021-02-12 21:28:27,989] - [remote_util] [140099333379840] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:28:27,989] - [remote_util] [140099333379840] - INFO - running command.raw on 172.23.123.61: /opt/couchbase/bin/cbcollect_info 172.23.123.61-20210212-2128-diag.zip [2021-02-12 21:29:57,549] - [remote_util] [140098695829248] - INFO - command executed successfully with root [2021-02-12 21:29:57,891] - [remote_util] [140098695829248] - INFO - running command.raw on 172.23.122.118: rm -f /root/172.23.122.118-20210212-2128-diag.zip [2021-02-12 21:29:57,907] - [remote_util] [140098695829248] - INFO - command executed successfully with root [2021-02-12 21:29:59,228] - [remote_util] [140099316594432] - INFO - command executed successfully with root [2021-02-12 21:29:59,503] - [remote_util] [140099316594432] - INFO - running command.raw on 172.23.122.117: rm -f /root/172.23.122.117-20210212-2128-diag.zip [2021-02-12 21:29:59,519] - [remote_util] [140099316594432] - INFO - command executed successfully with root [2021-02-12 21:30:00,392] - [remote_util] [140098185717504] - INFO - command executed successfully with root [2021-02-12 21:30:00,609] - [remote_util] [140098185717504] - INFO - running command.raw on 172.23.122.116: rm -f /root/172.23.122.116-20210212-2128-diag.zip [2021-02-12 21:30:00,625] - [remote_util] [140098185717504] - INFO - command executed successfully with root [2021-02-12 21:30:03,933] - [remote_util] [140099333379840] - INFO - command executed successfully with root [2021-02-12 21:30:04,347] - [remote_util] [140099333379840] - INFO - running command.raw on 172.23.123.61: rm -f /root/172.23.123.61-20210212-2128-diag.zip [2021-02-12 21:30:04,366] - [remote_util] [140099333379840] - INFO - command executed successfully with root [2021-02-12 21:30:05,038] - [remote_util] [140099567843072] - INFO - command executed successfully with root [2021-02-12 21:30:05,570] - [remote_util] [140099567843072] - INFO - running command.raw on 172.23.123.62: rm -f /root/172.23.123.62-20210212-2128-diag.zip [2021-02-12 21:30:05,589] - [remote_util] [140099567843072] - INFO - command executed successfully with root [2021-02-12 21:30:08,734] - [remote_util] [140099308201728] - INFO - command executed successfully with root [2021-02-12 21:30:08,886] - [remote_util] [140098729400064] - INFO - command executed successfully with root [2021-02-12 21:30:08,932] - [remote_util] [140099341772544] - INFO - command executed successfully with root [2021-02-12 21:30:09,916] - [remote_util] [140099308201728] - INFO - running command.raw on 172.23.122.97: rm -f /root/172.23.122.97-20210212-2128-diag.zip [2021-02-12 21:30:09,974] - [remote_util] [140098729400064] - INFO - running command.raw on 172.23.122.82: rm -f /root/172.23.122.82-20210212-2128-diag.zip [2021-02-12 21:30:09,974] - [remote_util] [140099308201728] - INFO - command executed successfully with root [2021-02-12 21:30:10,031] - [remote_util] [140098729400064] - INFO - command executed successfully with root [2021-02-12 21:30:10,056] - [remote_util] [140099341772544] - INFO - running command.raw on 172.23.122.90: rm -f /root/172.23.122.90-20210212-2128-diag.zip [2021-02-12 21:30:10,075] - [remote_util] [140099341772544] - INFO - command executed successfully with root