[2021-02-12 21:54:42,791] - [eventing_base] [140099299809024] - INFO - Starting Test: test_kv_swap_rebalance_when_existing_eventing_node_is_processing_mutations [2021-02-12 21:54:42,793] - [remote_util] [140099299809024] - INFO - SSH Connecting to 172.23.123.61 with username:root, attempt#1 of 5 [2021-02-12 21:54:42,899] - [remote_util] [140099299809024] - INFO - SSH Connected to 172.23.123.61 as root [2021-02-12 21:54:43,189] - [remote_util] [140099299809024] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:54:43,489] - [remote_util] [140099299809024] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:54:43,493] - [rest_client] [140099299809024] - INFO - -->is_ns_server_running? [2021-02-12 21:54:43,501] - [rest_client] [140099299809024] - INFO - Node version in cluster 7.0.0-4454-enterprise [2021-02-12 21:54:43,505] - [rest_client] [140099299809024] - ERROR - GET http://172.23.123.61:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-02-12 21:54:43,506] - [rest_client] [140099299809024] - INFO - Node versions in cluster [] [2021-02-12 21:54:43,506] - [basetestcase] [140099299809024] - INFO - ============== basetestcase setup was started for test #6 test_kv_swap_rebalance_when_existing_eventing_node_is_processing_mutations============== [2021-02-12 21:54:43,511] - [rest_client] [140099299809024] - ERROR - GET http://172.23.123.61:8091/pools/default/ body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-02-12 21:54:43,526] - [bucket_helper] [140099299809024] - INFO - Could not find any buckets for node 172.23.123.61, nothing to delete [2021-02-12 21:54:43,536] - [bucket_helper] [140099299809024] - INFO - Could not find any buckets for node 172.23.123.62, nothing to delete [2021-02-12 21:54:43,545] - [bucket_helper] [140099299809024] - INFO - Could not find any buckets for node 172.23.122.90, nothing to delete [2021-02-12 21:54:43,555] - [bucket_helper] [140099299809024] - INFO - Could not find any buckets for node 172.23.122.117, nothing to delete [2021-02-12 21:54:43,566] - [bucket_helper] [140099299809024] - INFO - Could not find any buckets for node 172.23.122.97, nothing to delete [2021-02-12 21:54:43,575] - [bucket_helper] [140099299809024] - INFO - Could not find any buckets for node 172.23.122.82, nothing to delete [2021-02-12 21:54:43,585] - [bucket_helper] [140099299809024] - INFO - Could not find any buckets for node 172.23.122.118, nothing to delete [2021-02-12 21:54:43,596] - [bucket_helper] [140099299809024] - INFO - Could not find any buckets for node 172.23.122.116, nothing to delete [2021-02-12 21:54:43,600] - [rest_client] [140099299809024] - INFO - -->is_ns_server_running? [2021-02-12 21:54:43,610] - [cluster_helper] [140099299809024] - INFO - waiting for ns_server @ 172.23.123.61:8091 [2021-02-12 21:54:43,610] - [rest_client] [140099299809024] - INFO - -->is_ns_server_running? [2021-02-12 21:54:43,613] - [cluster_helper] [140099299809024] - INFO - ns_server @ 172.23.123.61:8091 is running [2021-02-12 21:54:43,617] - [cluster_helper] [140099299809024] - INFO - waiting for ns_server @ 172.23.123.62:8091 [2021-02-12 21:54:43,617] - [rest_client] [140099299809024] - INFO - -->is_ns_server_running? [2021-02-12 21:54:43,620] - [cluster_helper] [140099299809024] - INFO - ns_server @ 172.23.123.62:8091 is running [2021-02-12 21:54:43,623] - [cluster_helper] [140099299809024] - INFO - waiting for ns_server @ 172.23.122.90:8091 [2021-02-12 21:54:43,623] - [rest_client] [140099299809024] - INFO - -->is_ns_server_running? [2021-02-12 21:54:43,626] - [cluster_helper] [140099299809024] - INFO - ns_server @ 172.23.122.90:8091 is running [2021-02-12 21:54:43,629] - [cluster_helper] [140099299809024] - INFO - waiting for ns_server @ 172.23.122.117:8091 [2021-02-12 21:54:43,629] - [rest_client] [140099299809024] - INFO - -->is_ns_server_running? [2021-02-12 21:54:43,632] - [cluster_helper] [140099299809024] - INFO - ns_server @ 172.23.122.117:8091 is running [2021-02-12 21:54:43,635] - [cluster_helper] [140099299809024] - INFO - waiting for ns_server @ 172.23.122.97:8091 [2021-02-12 21:54:43,635] - [rest_client] [140099299809024] - INFO - -->is_ns_server_running? [2021-02-12 21:54:43,638] - [cluster_helper] [140099299809024] - INFO - ns_server @ 172.23.122.97:8091 is running [2021-02-12 21:54:43,641] - [cluster_helper] [140099299809024] - INFO - waiting for ns_server @ 172.23.122.82:8091 [2021-02-12 21:54:43,641] - [rest_client] [140099299809024] - INFO - -->is_ns_server_running? [2021-02-12 21:54:43,644] - [cluster_helper] [140099299809024] - INFO - ns_server @ 172.23.122.82:8091 is running [2021-02-12 21:54:43,647] - [cluster_helper] [140099299809024] - INFO - waiting for ns_server @ 172.23.122.118:8091 [2021-02-12 21:54:43,647] - [rest_client] [140099299809024] - INFO - -->is_ns_server_running? [2021-02-12 21:54:43,650] - [cluster_helper] [140099299809024] - INFO - ns_server @ 172.23.122.118:8091 is running [2021-02-12 21:54:43,653] - [cluster_helper] [140099299809024] - INFO - waiting for ns_server @ 172.23.122.116:8091 [2021-02-12 21:54:43,653] - [rest_client] [140099299809024] - INFO - -->is_ns_server_running? [2021-02-12 21:54:43,656] - [cluster_helper] [140099299809024] - INFO - ns_server @ 172.23.122.116:8091 is running [2021-02-12 21:54:43,656] - [basetestcase] [140099299809024] - INFO - Building docker image with java sdk client [2021-02-12 21:54:47,220] - [basetestcase] [140099299809024] - INFO - initializing cluster [2021-02-12 21:54:47,222] - [remote_util] [140099299809024] - INFO - SSH Connecting to 172.23.123.61 with username:root, attempt#1 of 5 [2021-02-12 21:54:47,327] - [remote_util] [140099299809024] - INFO - SSH Connected to 172.23.123.61 as root [2021-02-12 21:54:47,592] - [remote_util] [140099299809024] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:54:47,899] - [remote_util] [140099299809024] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:54:47,907] - [remote_util] [140099299809024] - INFO - SSH Connecting to 172.23.123.61 with username:root, attempt#1 of 5 [2021-02-12 21:54:48,010] - [remote_util] [140099299809024] - INFO - SSH Connected to 172.23.123.61 as root [2021-02-12 21:54:48,278] - [remote_util] [140099299809024] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:54:48,577] - [remote_util] [140099299809024] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:54:48,599] - [remote_util] [140099299809024] - INFO - 172.23.123.61 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 21:54:48,659] - [remote_util] [140099299809024] - INFO - Running systemd command on this server [2021-02-12 21:54:48,659] - [remote_util] [140099299809024] - INFO - running command.raw on 172.23.123.61: systemctl stop couchbase-server.service [2021-02-12 21:54:50,373] - [remote_util] [140099299809024] - INFO - command executed successfully with root [2021-02-12 21:54:50,373] - [basetestcase] [140099299809024] - INFO - Couchbase stopped [2021-02-12 21:54:50,374] - [basetestcase] [140099299809024] - INFO - sleep for 5 secs. ... [2021-02-12 21:54:55,379] - [remote_util] [140099299809024] - INFO - running command.raw on 172.23.123.61: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-12 21:54:55,395] - [remote_util] [140099299809024] - INFO - command executed successfully with root [2021-02-12 21:54:55,395] - [remote_util] [140099299809024] - INFO - running command.raw on 172.23.123.61: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-12 21:54:55,449] - [remote_util] [140099299809024] - INFO - command executed successfully with root [2021-02-12 21:54:55,451] - [remote_util] [140099299809024] - INFO - SSH Connecting to 172.23.123.61 with username:root, attempt#1 of 5 [2021-02-12 21:54:55,553] - [remote_util] [140099299809024] - INFO - SSH Connected to 172.23.123.61 as root [2021-02-12 21:54:55,836] - [remote_util] [140099299809024] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:54:56,141] - [remote_util] [140099299809024] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:54:56,161] - [remote_util] [140099299809024] - INFO - 172.23.123.61 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 21:54:56,161] - [remote_util] [140099299809024] - INFO - Checking for process beam.smp on linux [2021-02-12 21:54:56,202] - [remote_util] [140099299809024] - INFO - Starting couchbase server [2021-02-12 21:54:56,304] - [remote_util] [140099299809024] - INFO - Running systemd command on this server [2021-02-12 21:54:56,304] - [remote_util] [140099299809024] - INFO - running command.raw on 172.23.123.61: systemctl start couchbase-server.service [2021-02-12 21:54:56,331] - [remote_util] [140099299809024] - INFO - command executed successfully with root [2021-02-12 21:54:56,331] - [remote_util] [140099299809024] - INFO - 172.23.123.61:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-12 21:55:01,336] - [remote_util] [140099299809024] - INFO - running command.raw on 172.23.123.61: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-12 21:55:01,363] - [remote_util] [140099299809024] - INFO - command executed successfully with root [2021-02-12 21:55:01,363] - [remote_util] [140099299809024] - INFO - Couchbase server status: [' Process: 99239 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-12 21:55:01,363] - [remote_util] [140099299809024] - INFO - Checking for process beam.smp on linux [2021-02-12 21:55:01,440] - [remote_util] [140099299809024] - INFO - process beam.smp is running on 172.23.123.61: with pid 99381 [2021-02-12 21:55:01,440] - [basetestcase] [140099299809024] - INFO - Couchbase started [2021-02-12 21:55:01,441] - [remote_util] [140099299809024] - INFO - SSH Connecting to 172.23.123.62 with username:root, attempt#1 of 5 [2021-02-12 21:55:01,552] - [remote_util] [140099299809024] - INFO - SSH Connected to 172.23.123.62 as root [2021-02-12 21:55:01,859] - [remote_util] [140099299809024] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:55:02,168] - [remote_util] [140099299809024] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:55:02,176] - [remote_util] [140099299809024] - INFO - SSH Connecting to 172.23.123.62 with username:root, attempt#1 of 5 [2021-02-12 21:55:02,281] - [remote_util] [140099299809024] - INFO - SSH Connected to 172.23.123.62 as root [2021-02-12 21:55:02,575] - [remote_util] [140099299809024] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:55:02,885] - [remote_util] [140099299809024] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:55:02,948] - [remote_util] [140099299809024] - INFO - 172.23.123.62 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 21:55:03,014] - [remote_util] [140099299809024] - INFO - Running systemd command on this server [2021-02-12 21:55:03,014] - [remote_util] [140099299809024] - INFO - running command.raw on 172.23.123.62: systemctl stop couchbase-server.service [2021-02-12 21:55:04,775] - [remote_util] [140099299809024] - INFO - command executed successfully with root [2021-02-12 21:55:04,776] - [basetestcase] [140099299809024] - INFO - Couchbase stopped [2021-02-12 21:55:04,776] - [basetestcase] [140099299809024] - INFO - sleep for 5 secs. ... [2021-02-12 21:55:09,781] - [remote_util] [140099299809024] - INFO - running command.raw on 172.23.123.62: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-12 21:55:09,799] - [remote_util] [140099299809024] - INFO - command executed successfully with root [2021-02-12 21:55:09,799] - [remote_util] [140099299809024] - INFO - running command.raw on 172.23.123.62: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-12 21:55:09,854] - [remote_util] [140099299809024] - INFO - command executed successfully with root [2021-02-12 21:55:09,855] - [remote_util] [140099299809024] - INFO - SSH Connecting to 172.23.123.62 with username:root, attempt#1 of 5 [2021-02-12 21:55:09,960] - [remote_util] [140099299809024] - INFO - SSH Connected to 172.23.123.62 as root [2021-02-12 21:55:10,240] - [remote_util] [140099299809024] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:55:10,542] - [remote_util] [140099299809024] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:55:10,604] - [remote_util] [140099299809024] - INFO - 172.23.123.62 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 21:55:10,604] - [remote_util] [140099299809024] - INFO - Checking for process beam.smp on linux [2021-02-12 21:55:10,644] - [remote_util] [140099299809024] - INFO - Starting couchbase server [2021-02-12 21:55:10,742] - [remote_util] [140099299809024] - INFO - Running systemd command on this server [2021-02-12 21:55:10,742] - [remote_util] [140099299809024] - INFO - running command.raw on 172.23.123.62: systemctl start couchbase-server.service [2021-02-12 21:55:10,767] - [remote_util] [140099299809024] - INFO - command executed successfully with root [2021-02-12 21:55:10,767] - [remote_util] [140099299809024] - INFO - 172.23.123.62:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-12 21:55:15,772] - [remote_util] [140099299809024] - INFO - running command.raw on 172.23.123.62: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-12 21:55:15,802] - [remote_util] [140099299809024] - INFO - command executed successfully with root [2021-02-12 21:55:15,802] - [remote_util] [140099299809024] - INFO - Couchbase server status: [' Process: 111808 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-12 21:55:15,802] - [remote_util] [140099299809024] - INFO - Checking for process beam.smp on linux [2021-02-12 21:55:15,874] - [remote_util] [140099299809024] - INFO - process beam.smp is running on 172.23.123.62: with pid 111951 [2021-02-12 21:55:15,874] - [basetestcase] [140099299809024] - INFO - Couchbase started [2021-02-12 21:55:15,875] - [remote_util] [140099299809024] - INFO - SSH Connecting to 172.23.122.90 with username:root, attempt#1 of 5 [2021-02-12 21:55:15,981] - [remote_util] [140099299809024] - INFO - SSH Connected to 172.23.122.90 as root [2021-02-12 21:55:16,270] - [remote_util] [140099299809024] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:55:16,569] - [remote_util] [140099299809024] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:55:16,577] - [remote_util] [140099299809024] - INFO - SSH Connecting to 172.23.122.90 with username:root, attempt#1 of 5 [2021-02-12 21:55:16,679] - [remote_util] [140099299809024] - INFO - SSH Connected to 172.23.122.90 as root [2021-02-12 21:55:16,946] - [remote_util] [140099299809024] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:55:17,242] - [remote_util] [140099299809024] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:55:17,262] - [remote_util] [140099299809024] - INFO - 172.23.122.90 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 21:55:17,314] - [remote_util] [140099299809024] - INFO - Running systemd command on this server [2021-02-12 21:55:17,314] - [remote_util] [140099299809024] - INFO - running command.raw on 172.23.122.90: systemctl stop couchbase-server.service [2021-02-12 21:55:19,009] - [remote_util] [140099299809024] - INFO - command executed successfully with root [2021-02-12 21:55:19,009] - [basetestcase] [140099299809024] - INFO - Couchbase stopped [2021-02-12 21:55:19,009] - [basetestcase] [140099299809024] - INFO - sleep for 5 secs. ... [2021-02-12 21:55:24,014] - [remote_util] [140099299809024] - INFO - running command.raw on 172.23.122.90: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-12 21:55:24,028] - [remote_util] [140099299809024] - INFO - command executed successfully with root [2021-02-12 21:55:24,028] - [remote_util] [140099299809024] - INFO - running command.raw on 172.23.122.90: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-12 21:55:24,081] - [remote_util] [140099299809024] - INFO - command executed successfully with root [2021-02-12 21:55:24,082] - [remote_util] [140099299809024] - INFO - SSH Connecting to 172.23.122.90 with username:root, attempt#1 of 5 [2021-02-12 21:55:24,183] - [remote_util] [140099299809024] - INFO - SSH Connected to 172.23.122.90 as root [2021-02-12 21:55:24,462] - [remote_util] [140099299809024] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:55:24,759] - [remote_util] [140099299809024] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:55:24,778] - [remote_util] [140099299809024] - INFO - 172.23.122.90 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 21:55:24,778] - [remote_util] [140099299809024] - INFO - Checking for process beam.smp on linux [2021-02-12 21:55:24,817] - [remote_util] [140099299809024] - INFO - Starting couchbase server [2021-02-12 21:55:24,911] - [remote_util] [140099299809024] - INFO - Running systemd command on this server [2021-02-12 21:55:24,911] - [remote_util] [140099299809024] - INFO - running command.raw on 172.23.122.90: systemctl start couchbase-server.service [2021-02-12 21:55:24,934] - [remote_util] [140099299809024] - INFO - command executed successfully with root [2021-02-12 21:55:24,934] - [remote_util] [140099299809024] - INFO - 172.23.122.90:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-12 21:55:29,939] - [remote_util] [140099299809024] - INFO - running command.raw on 172.23.122.90: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-12 21:55:29,965] - [remote_util] [140099299809024] - INFO - command executed successfully with root [2021-02-12 21:55:29,965] - [remote_util] [140099299809024] - INFO - Couchbase server status: [' Process: 76934 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-12 21:55:29,965] - [remote_util] [140099299809024] - INFO - Checking for process beam.smp on linux [2021-02-12 21:55:30,034] - [remote_util] [140099299809024] - INFO - process beam.smp is running on 172.23.122.90: with pid 77078 [2021-02-12 21:55:30,034] - [basetestcase] [140099299809024] - INFO - Couchbase started [2021-02-12 21:55:30,036] - [remote_util] [140099299809024] - INFO - SSH Connecting to 172.23.122.117 with username:root, attempt#1 of 5 [2021-02-12 21:55:30,138] - [remote_util] [140099299809024] - INFO - SSH Connected to 172.23.122.117 as root [2021-02-12 21:55:30,389] - [remote_util] [140099299809024] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:55:30,680] - [remote_util] [140099299809024] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:55:30,689] - [remote_util] [140099299809024] - INFO - SSH Connecting to 172.23.122.117 with username:root, attempt#1 of 5 [2021-02-12 21:55:30,785] - [remote_util] [140099299809024] - INFO - SSH Connected to 172.23.122.117 as root [2021-02-12 21:55:31,031] - [remote_util] [140099299809024] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:55:31,320] - [remote_util] [140099299809024] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:55:31,338] - [remote_util] [140099299809024] - INFO - 172.23.122.117 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 21:55:31,388] - [remote_util] [140099299809024] - INFO - Running systemd command on this server [2021-02-12 21:55:31,389] - [remote_util] [140099299809024] - INFO - running command.raw on 172.23.122.117: systemctl stop couchbase-server.service [2021-02-12 21:55:33,187] - [remote_util] [140099299809024] - INFO - command executed successfully with root [2021-02-12 21:55:33,188] - [basetestcase] [140099299809024] - INFO - Couchbase stopped [2021-02-12 21:55:33,188] - [basetestcase] [140099299809024] - INFO - sleep for 5 secs. ... [2021-02-12 21:55:38,193] - [remote_util] [140099299809024] - INFO - running command.raw on 172.23.122.117: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-12 21:55:38,208] - [remote_util] [140099299809024] - INFO - command executed successfully with root [2021-02-12 21:55:38,208] - [remote_util] [140099299809024] - INFO - running command.raw on 172.23.122.117: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-12 21:55:38,261] - [remote_util] [140099299809024] - INFO - command executed successfully with root [2021-02-12 21:55:38,262] - [remote_util] [140099299809024] - INFO - SSH Connecting to 172.23.122.117 with username:root, attempt#1 of 5 [2021-02-12 21:55:38,359] - [remote_util] [140099299809024] - INFO - SSH Connected to 172.23.122.117 as root [2021-02-12 21:55:38,606] - [remote_util] [140099299809024] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:55:38,896] - [remote_util] [140099299809024] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:55:38,914] - [remote_util] [140099299809024] - INFO - 172.23.122.117 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 21:55:38,914] - [remote_util] [140099299809024] - INFO - Checking for process beam.smp on linux [2021-02-12 21:55:38,948] - [remote_util] [140099299809024] - INFO - Starting couchbase server [2021-02-12 21:55:39,039] - [remote_util] [140099299809024] - INFO - Running systemd command on this server [2021-02-12 21:55:39,039] - [remote_util] [140099299809024] - INFO - running command.raw on 172.23.122.117: systemctl start couchbase-server.service [2021-02-12 21:55:39,060] - [remote_util] [140099299809024] - INFO - command executed successfully with root [2021-02-12 21:55:39,060] - [remote_util] [140099299809024] - INFO - 172.23.122.117:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-12 21:55:44,066] - [remote_util] [140099299809024] - INFO - running command.raw on 172.23.122.117: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-12 21:55:44,090] - [remote_util] [140099299809024] - INFO - command executed successfully with root [2021-02-12 21:55:44,090] - [remote_util] [140099299809024] - INFO - Couchbase server status: [' Process: 37245 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-12 21:55:44,091] - [remote_util] [140099299809024] - INFO - Checking for process beam.smp on linux [2021-02-12 21:55:44,155] - [remote_util] [140099299809024] - INFO - process beam.smp is running on 172.23.122.117: with pid 37389 [2021-02-12 21:55:44,155] - [basetestcase] [140099299809024] - INFO - Couchbase started [2021-02-12 21:55:44,157] - [remote_util] [140099299809024] - INFO - SSH Connecting to 172.23.122.97 with username:root, attempt#1 of 5 [2021-02-12 21:55:44,258] - [remote_util] [140099299809024] - INFO - SSH Connected to 172.23.122.97 as root [2021-02-12 21:55:44,513] - [remote_util] [140099299809024] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:55:44,808] - [remote_util] [140099299809024] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:55:44,816] - [remote_util] [140099299809024] - INFO - SSH Connecting to 172.23.122.97 with username:root, attempt#1 of 5 [2021-02-12 21:55:44,911] - [remote_util] [140099299809024] - INFO - SSH Connected to 172.23.122.97 as root [2021-02-12 21:55:45,162] - [remote_util] [140099299809024] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:55:45,456] - [remote_util] [140099299809024] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:55:45,474] - [remote_util] [140099299809024] - INFO - 172.23.122.97 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 21:55:45,527] - [remote_util] [140099299809024] - INFO - Running systemd command on this server [2021-02-12 21:55:45,527] - [remote_util] [140099299809024] - INFO - running command.raw on 172.23.122.97: systemctl stop couchbase-server.service [2021-02-12 21:55:47,249] - [remote_util] [140099299809024] - INFO - command executed successfully with root [2021-02-12 21:55:47,250] - [basetestcase] [140099299809024] - INFO - Couchbase stopped [2021-02-12 21:55:47,250] - [basetestcase] [140099299809024] - INFO - sleep for 5 secs. ... [2021-02-12 21:55:52,255] - [remote_util] [140099299809024] - INFO - running command.raw on 172.23.122.97: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-12 21:55:52,272] - [remote_util] [140099299809024] - INFO - command executed successfully with root [2021-02-12 21:55:52,273] - [remote_util] [140099299809024] - INFO - running command.raw on 172.23.122.97: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-12 21:55:52,327] - [remote_util] [140099299809024] - INFO - command executed successfully with root [2021-02-12 21:55:52,329] - [remote_util] [140099299809024] - INFO - SSH Connecting to 172.23.122.97 with username:root, attempt#1 of 5 [2021-02-12 21:55:52,428] - [remote_util] [140099299809024] - INFO - SSH Connected to 172.23.122.97 as root [2021-02-12 21:55:52,688] - [remote_util] [140099299809024] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:55:52,984] - [remote_util] [140099299809024] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:55:53,003] - [remote_util] [140099299809024] - INFO - 172.23.122.97 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 21:55:53,003] - [remote_util] [140099299809024] - INFO - Checking for process beam.smp on linux [2021-02-12 21:55:53,038] - [remote_util] [140099299809024] - INFO - Starting couchbase server [2021-02-12 21:55:53,134] - [remote_util] [140099299809024] - INFO - Running systemd command on this server [2021-02-12 21:55:53,134] - [remote_util] [140099299809024] - INFO - running command.raw on 172.23.122.97: systemctl start couchbase-server.service [2021-02-12 21:55:53,158] - [remote_util] [140099299809024] - INFO - command executed successfully with root [2021-02-12 21:55:53,158] - [remote_util] [140099299809024] - INFO - 172.23.122.97:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-12 21:55:58,163] - [remote_util] [140099299809024] - INFO - running command.raw on 172.23.122.97: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-12 21:55:58,190] - [remote_util] [140099299809024] - INFO - command executed successfully with root [2021-02-12 21:55:58,190] - [remote_util] [140099299809024] - INFO - Couchbase server status: [' Process: 70442 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-12 21:55:58,190] - [remote_util] [140099299809024] - INFO - Checking for process beam.smp on linux [2021-02-12 21:55:58,268] - [remote_util] [140099299809024] - INFO - process beam.smp is running on 172.23.122.97: with pid 70588 [2021-02-12 21:55:58,268] - [basetestcase] [140099299809024] - INFO - Couchbase started [2021-02-12 21:55:58,269] - [remote_util] [140099299809024] - INFO - SSH Connecting to 172.23.122.82 with username:root, attempt#1 of 5 [2021-02-12 21:55:58,373] - [remote_util] [140099299809024] - INFO - SSH Connected to 172.23.122.82 as root [2021-02-12 21:55:58,648] - [remote_util] [140099299809024] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:55:58,945] - [remote_util] [140099299809024] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:55:58,953] - [remote_util] [140099299809024] - INFO - SSH Connecting to 172.23.122.82 with username:root, attempt#1 of 5 [2021-02-12 21:55:59,050] - [remote_util] [140099299809024] - INFO - SSH Connected to 172.23.122.82 as root [2021-02-12 21:55:59,305] - [remote_util] [140099299809024] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:55:59,594] - [remote_util] [140099299809024] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:55:59,651] - [remote_util] [140099299809024] - INFO - 172.23.122.82 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 21:55:59,699] - [remote_util] [140099299809024] - INFO - Running systemd command on this server [2021-02-12 21:55:59,699] - [remote_util] [140099299809024] - INFO - running command.raw on 172.23.122.82: systemctl stop couchbase-server.service [2021-02-12 21:56:01,286] - [remote_util] [140099299809024] - INFO - command executed successfully with root [2021-02-12 21:56:01,286] - [basetestcase] [140099299809024] - INFO - Couchbase stopped [2021-02-12 21:56:01,287] - [basetestcase] [140099299809024] - INFO - sleep for 5 secs. ... [2021-02-12 21:56:06,292] - [remote_util] [140099299809024] - INFO - running command.raw on 172.23.122.82: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-12 21:56:06,306] - [remote_util] [140099299809024] - INFO - command executed successfully with root [2021-02-12 21:56:06,306] - [remote_util] [140099299809024] - INFO - running command.raw on 172.23.122.82: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-12 21:56:06,359] - [remote_util] [140099299809024] - INFO - command executed successfully with root [2021-02-12 21:56:06,360] - [remote_util] [140099299809024] - INFO - SSH Connecting to 172.23.122.82 with username:root, attempt#1 of 5 [2021-02-12 21:56:06,460] - [remote_util] [140099299809024] - INFO - SSH Connected to 172.23.122.82 as root [2021-02-12 21:56:06,723] - [remote_util] [140099299809024] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:56:07,016] - [remote_util] [140099299809024] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:56:07,035] - [remote_util] [140099299809024] - INFO - 172.23.122.82 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 21:56:07,035] - [remote_util] [140099299809024] - INFO - Checking for process beam.smp on linux [2021-02-12 21:56:07,074] - [remote_util] [140099299809024] - INFO - Starting couchbase server [2021-02-12 21:56:07,167] - [remote_util] [140099299809024] - INFO - Running systemd command on this server [2021-02-12 21:56:07,167] - [remote_util] [140099299809024] - INFO - running command.raw on 172.23.122.82: systemctl start couchbase-server.service [2021-02-12 21:56:07,189] - [remote_util] [140099299809024] - INFO - command executed successfully with root [2021-02-12 21:56:07,189] - [remote_util] [140099299809024] - INFO - 172.23.122.82:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-12 21:56:12,194] - [remote_util] [140099299809024] - INFO - running command.raw on 172.23.122.82: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-12 21:56:12,225] - [remote_util] [140099299809024] - INFO - command executed successfully with root [2021-02-12 21:56:12,225] - [remote_util] [140099299809024] - INFO - Couchbase server status: [' Process: 62290 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-12 21:56:12,225] - [remote_util] [140099299809024] - INFO - Checking for process beam.smp on linux [2021-02-12 21:56:12,294] - [remote_util] [140099299809024] - INFO - process beam.smp is running on 172.23.122.82: with pid 62434 [2021-02-12 21:56:12,294] - [basetestcase] [140099299809024] - INFO - Couchbase started [2021-02-12 21:56:12,295] - [remote_util] [140099299809024] - INFO - SSH Connecting to 172.23.122.118 with username:root, attempt#1 of 5 [2021-02-12 21:56:12,396] - [remote_util] [140099299809024] - INFO - SSH Connected to 172.23.122.118 as root [2021-02-12 21:56:12,645] - [remote_util] [140099299809024] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:56:12,940] - [remote_util] [140099299809024] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:56:12,947] - [remote_util] [140099299809024] - INFO - SSH Connecting to 172.23.122.118 with username:root, attempt#1 of 5 [2021-02-12 21:56:13,042] - [remote_util] [140099299809024] - INFO - SSH Connected to 172.23.122.118 as root [2021-02-12 21:56:13,282] - [remote_util] [140099299809024] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:56:13,572] - [remote_util] [140099299809024] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:56:13,591] - [remote_util] [140099299809024] - INFO - 172.23.122.118 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 21:56:13,643] - [remote_util] [140099299809024] - INFO - Running systemd command on this server [2021-02-12 21:56:13,643] - [remote_util] [140099299809024] - INFO - running command.raw on 172.23.122.118: systemctl stop couchbase-server.service [2021-02-12 21:56:15,382] - [remote_util] [140099299809024] - INFO - command executed successfully with root [2021-02-12 21:56:15,382] - [basetestcase] [140099299809024] - INFO - Couchbase stopped [2021-02-12 21:56:15,383] - [basetestcase] [140099299809024] - INFO - sleep for 5 secs. ... [2021-02-12 21:56:20,388] - [remote_util] [140099299809024] - INFO - running command.raw on 172.23.122.118: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-12 21:56:20,403] - [remote_util] [140099299809024] - INFO - command executed successfully with root [2021-02-12 21:56:20,403] - [remote_util] [140099299809024] - INFO - running command.raw on 172.23.122.118: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-12 21:56:20,456] - [remote_util] [140099299809024] - INFO - command executed successfully with root [2021-02-12 21:56:20,458] - [remote_util] [140099299809024] - INFO - SSH Connecting to 172.23.122.118 with username:root, attempt#1 of 5 [2021-02-12 21:56:20,554] - [remote_util] [140099299809024] - INFO - SSH Connected to 172.23.122.118 as root [2021-02-12 21:56:20,794] - [remote_util] [140099299809024] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:56:21,088] - [remote_util] [140099299809024] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:56:21,146] - [remote_util] [140099299809024] - INFO - 172.23.122.118 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 21:56:21,147] - [remote_util] [140099299809024] - INFO - Checking for process beam.smp on linux [2021-02-12 21:56:21,182] - [remote_util] [140099299809024] - INFO - Starting couchbase server [2021-02-12 21:56:21,279] - [remote_util] [140099299809024] - INFO - Running systemd command on this server [2021-02-12 21:56:21,279] - [remote_util] [140099299809024] - INFO - running command.raw on 172.23.122.118: systemctl start couchbase-server.service [2021-02-12 21:56:21,302] - [remote_util] [140099299809024] - INFO - command executed successfully with root [2021-02-12 21:56:21,302] - [remote_util] [140099299809024] - INFO - 172.23.122.118:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-12 21:56:26,308] - [remote_util] [140099299809024] - INFO - running command.raw on 172.23.122.118: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-12 21:56:26,334] - [remote_util] [140099299809024] - INFO - command executed successfully with root [2021-02-12 21:56:26,334] - [remote_util] [140099299809024] - INFO - Couchbase server status: [' Process: 89245 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-12 21:56:26,334] - [remote_util] [140099299809024] - INFO - Checking for process beam.smp on linux [2021-02-12 21:56:26,398] - [remote_util] [140099299809024] - INFO - process beam.smp is running on 172.23.122.118: with pid 89389 [2021-02-12 21:56:26,399] - [basetestcase] [140099299809024] - INFO - Couchbase started [2021-02-12 21:56:26,400] - [remote_util] [140099299809024] - INFO - SSH Connecting to 172.23.122.116 with username:root, attempt#1 of 5 [2021-02-12 21:56:26,504] - [remote_util] [140099299809024] - INFO - SSH Connected to 172.23.122.116 as root [2021-02-12 21:56:26,781] - [remote_util] [140099299809024] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:56:27,076] - [remote_util] [140099299809024] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:56:27,084] - [remote_util] [140099299809024] - INFO - SSH Connecting to 172.23.122.116 with username:root, attempt#1 of 5 [2021-02-12 21:56:27,183] - [remote_util] [140099299809024] - INFO - SSH Connected to 172.23.122.116 as root [2021-02-12 21:56:27,446] - [remote_util] [140099299809024] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:56:27,741] - [remote_util] [140099299809024] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:56:27,760] - [remote_util] [140099299809024] - INFO - 172.23.122.116 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 21:56:27,813] - [remote_util] [140099299809024] - INFO - Running systemd command on this server [2021-02-12 21:56:27,813] - [remote_util] [140099299809024] - INFO - running command.raw on 172.23.122.116: systemctl stop couchbase-server.service [2021-02-12 21:56:29,735] - [remote_util] [140099299809024] - INFO - command executed successfully with root [2021-02-12 21:56:29,735] - [basetestcase] [140099299809024] - INFO - Couchbase stopped [2021-02-12 21:56:29,736] - [basetestcase] [140099299809024] - INFO - sleep for 5 secs. ... [2021-02-12 21:56:34,741] - [remote_util] [140099299809024] - INFO - running command.raw on 172.23.122.116: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-12 21:56:34,758] - [remote_util] [140099299809024] - INFO - command executed successfully with root [2021-02-12 21:56:34,758] - [remote_util] [140099299809024] - INFO - running command.raw on 172.23.122.116: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-12 21:56:34,812] - [remote_util] [140099299809024] - INFO - command executed successfully with root [2021-02-12 21:56:34,814] - [remote_util] [140099299809024] - INFO - SSH Connecting to 172.23.122.116 with username:root, attempt#1 of 5 [2021-02-12 21:56:34,912] - [remote_util] [140099299809024] - INFO - SSH Connected to 172.23.122.116 as root [2021-02-12 21:56:35,179] - [remote_util] [140099299809024] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:56:35,470] - [remote_util] [140099299809024] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:56:35,488] - [remote_util] [140099299809024] - INFO - 172.23.122.116 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 21:56:35,488] - [remote_util] [140099299809024] - INFO - Checking for process beam.smp on linux [2021-02-12 21:56:35,523] - [remote_util] [140099299809024] - INFO - Starting couchbase server [2021-02-12 21:56:35,615] - [remote_util] [140099299809024] - INFO - Running systemd command on this server [2021-02-12 21:56:35,615] - [remote_util] [140099299809024] - INFO - running command.raw on 172.23.122.116: systemctl start couchbase-server.service [2021-02-12 21:56:35,637] - [remote_util] [140099299809024] - INFO - command executed successfully with root [2021-02-12 21:56:35,637] - [remote_util] [140099299809024] - INFO - 172.23.122.116:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-12 21:56:40,642] - [remote_util] [140099299809024] - INFO - running command.raw on 172.23.122.116: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-12 21:56:40,665] - [remote_util] [140099299809024] - INFO - command executed successfully with root [2021-02-12 21:56:40,666] - [remote_util] [140099299809024] - INFO - Couchbase server status: [' Process: 52573 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-12 21:56:40,666] - [remote_util] [140099299809024] - INFO - Checking for process beam.smp on linux [2021-02-12 21:56:40,734] - [remote_util] [140099299809024] - INFO - process beam.smp is running on 172.23.122.116: with pid 52718 [2021-02-12 21:56:40,734] - [basetestcase] [140099299809024] - INFO - Couchbase started [2021-02-12 21:56:40,734] - [basetestcase] [140099299809024] - INFO - sleep for 10 secs. ... [2021-02-12 21:56:50,933] - [task] [140098209314560] - INFO - server: ip:172.23.123.61 port:8091 ssh_username:root, nodes/self [2021-02-12 21:56:50,936] - [task] [140098209314560] - INFO - {'uptime': '110', 'memoryTotal': 4201672704, 'memoryFree': 3507703808, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.123.61:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4454-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.123.61', 'ip': '172.23.123.61', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} [2021-02-12 21:56:50,938] - [rest_client] [140098209314560] - ERROR - GET http://172.23.123.61:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-02-12 21:56:50,938] - [rest_client] [140098209314560] - INFO - pools/default params : memoryQuota=2147 [2021-02-12 21:56:50,945] - [rest_client] [140098209314560] - INFO - --> init_node_services(Administrator,password,172.23.123.61,8091,['kv']) [2021-02-12 21:56:50,945] - [rest_client] [140098209314560] - INFO - /node/controller/setupServices params on 172.23.123.61: 8091:hostname=172.23.123.61&user=Administrator&password=password&services=kv [2021-02-12 21:56:50,973] - [rest_client] [140098209314560] - INFO - --> in init_cluster...Administrator,password,8091 [2021-02-12 21:56:50,973] - [rest_client] [140098209314560] - INFO - settings/web params on 172.23.123.61:8091:port=8091&username=Administrator&password=password [2021-02-12 21:56:51,031] - [rest_client] [140098209314560] - INFO - --> status:True [2021-02-12 21:56:51,032] - [remote_util] [140098209314560] - INFO - SSH Connecting to 172.23.123.61 with username:root, attempt#1 of 5 [2021-02-12 21:56:51,139] - [remote_util] [140098209314560] - INFO - SSH Connected to 172.23.123.61 as root [2021-02-12 21:56:51,430] - [remote_util] [140098209314560] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:56:51,738] - [remote_util] [140098209314560] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:56:51,746] - [remote_util] [140098209314560] - INFO - running command.raw on 172.23.123.61: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-02-12 21:56:51,804] - [remote_util] [140098209314560] - INFO - command executed successfully with root [2021-02-12 21:56:51,805] - [remote_util] [140098209314560] - INFO - ['ok'] [2021-02-12 21:56:51,806] - [rest_client] [140098209314560] - INFO - /diag/eval status on 172.23.123.61:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-12 21:56:51,808] - [rest_client] [140098209314560] - INFO - /diag/eval status on 172.23.123.61:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-12 21:56:51,812] - [rest_client] [140098209314560] - INFO - settings/indexes params : storageMode=plasma [2021-02-12 21:56:51,833] - [task] [140098209314560] - INFO - server: ip:172.23.123.62 port:8091 ssh_username:root, nodes/self [2021-02-12 21:56:51,836] - [task] [140098209314560] - INFO - {'uptime': '95', 'memoryTotal': 4201676800, 'memoryFree': 3480526848, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.123.62:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4454-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.123.62', 'ip': '172.23.123.62', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} [2021-02-12 21:56:51,837] - [rest_client] [140098209314560] - ERROR - GET http://172.23.123.62:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-02-12 21:56:51,838] - [rest_client] [140098209314560] - INFO - pools/default params : memoryQuota=2147 [2021-02-12 21:56:51,844] - [rest_client] [140098209314560] - INFO - --> in init_cluster...Administrator,password,8091 [2021-02-12 21:56:51,844] - [rest_client] [140098209314560] - INFO - settings/web params on 172.23.123.62:8091:port=8091&username=Administrator&password=password [2021-02-12 21:56:51,903] - [rest_client] [140098209314560] - INFO - --> status:True [2021-02-12 21:56:51,904] - [remote_util] [140098209314560] - INFO - SSH Connecting to 172.23.123.62 with username:root, attempt#1 of 5 [2021-02-12 21:56:52,013] - [remote_util] [140098209314560] - INFO - SSH Connected to 172.23.123.62 as root [2021-02-12 21:56:52,324] - [remote_util] [140098209314560] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:56:52,637] - [remote_util] [140098209314560] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:56:52,646] - [remote_util] [140098209314560] - INFO - running command.raw on 172.23.123.62: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-02-12 21:56:52,704] - [remote_util] [140098209314560] - INFO - command executed successfully with root [2021-02-12 21:56:52,704] - [remote_util] [140098209314560] - INFO - ['ok'] [2021-02-12 21:56:52,707] - [rest_client] [140098209314560] - INFO - /diag/eval status on 172.23.123.62:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-12 21:56:52,718] - [rest_client] [140098209314560] - INFO - /diag/eval status on 172.23.123.62:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-12 21:56:52,723] - [rest_client] [140098209314560] - INFO - settings/indexes params : storageMode=plasma [2021-02-12 21:56:52,743] - [task] [140098209314560] - INFO - server: ip:172.23.122.90 port:8091 ssh_username:root, nodes/self [2021-02-12 21:56:52,746] - [task] [140098209314560] - INFO - {'uptime': '84', 'memoryTotal': 4201684992, 'memoryFree': 3566641152, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.122.90:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4454-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.122.90', 'ip': '172.23.122.90', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} [2021-02-12 21:56:52,748] - [rest_client] [140098209314560] - ERROR - GET http://172.23.122.90:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-02-12 21:56:52,749] - [rest_client] [140098209314560] - INFO - pools/default params : memoryQuota=2147 [2021-02-12 21:56:52,755] - [rest_client] [140098209314560] - INFO - --> in init_cluster...Administrator,password,8091 [2021-02-12 21:56:52,755] - [rest_client] [140098209314560] - INFO - settings/web params on 172.23.122.90:8091:port=8091&username=Administrator&password=password [2021-02-12 21:56:52,810] - [rest_client] [140098209314560] - INFO - --> status:True [2021-02-12 21:56:52,812] - [remote_util] [140098209314560] - INFO - SSH Connecting to 172.23.122.90 with username:root, attempt#1 of 5 [2021-02-12 21:56:52,917] - [remote_util] [140098209314560] - INFO - SSH Connected to 172.23.122.90 as root [2021-02-12 21:56:53,196] - [remote_util] [140098209314560] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:56:53,491] - [remote_util] [140098209314560] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:56:53,498] - [remote_util] [140098209314560] - INFO - running command.raw on 172.23.122.90: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-02-12 21:56:53,555] - [remote_util] [140098209314560] - INFO - command executed successfully with root [2021-02-12 21:56:53,555] - [remote_util] [140098209314560] - INFO - ['ok'] [2021-02-12 21:56:53,556] - [rest_client] [140098209314560] - INFO - /diag/eval status on 172.23.122.90:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-12 21:56:53,558] - [rest_client] [140098209314560] - INFO - /diag/eval status on 172.23.122.90:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-12 21:56:53,562] - [rest_client] [140098209314560] - INFO - settings/indexes params : storageMode=plasma [2021-02-12 21:56:53,581] - [task] [140098209314560] - INFO - server: ip:172.23.122.117 port:8091 ssh_username:root, nodes/self [2021-02-12 21:56:53,584] - [task] [140098209314560] - INFO - {'uptime': '69', 'memoryTotal': 4201684992, 'memoryFree': 3533094912, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.122.117:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4454-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.122.117', 'ip': '172.23.122.117', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} [2021-02-12 21:56:53,586] - [rest_client] [140098209314560] - ERROR - GET http://172.23.122.117:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-02-12 21:56:53,586] - [rest_client] [140098209314560] - INFO - pools/default params : memoryQuota=2147 [2021-02-12 21:56:53,591] - [rest_client] [140098209314560] - INFO - --> in init_cluster...Administrator,password,8091 [2021-02-12 21:56:53,591] - [rest_client] [140098209314560] - INFO - settings/web params on 172.23.122.117:8091:port=8091&username=Administrator&password=password [2021-02-12 21:56:53,647] - [rest_client] [140098209314560] - INFO - --> status:True [2021-02-12 21:56:53,648] - [remote_util] [140098209314560] - INFO - SSH Connecting to 172.23.122.117 with username:root, attempt#1 of 5 [2021-02-12 21:56:53,753] - [remote_util] [140098209314560] - INFO - SSH Connected to 172.23.122.117 as root [2021-02-12 21:56:54,008] - [remote_util] [140098209314560] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:56:54,301] - [remote_util] [140098209314560] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:56:54,309] - [remote_util] [140098209314560] - INFO - running command.raw on 172.23.122.117: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-02-12 21:56:54,365] - [remote_util] [140098209314560] - INFO - command executed successfully with root [2021-02-12 21:56:54,365] - [remote_util] [140098209314560] - INFO - ['ok'] [2021-02-12 21:56:54,367] - [rest_client] [140098209314560] - INFO - /diag/eval status on 172.23.122.117:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-12 21:56:54,369] - [rest_client] [140098209314560] - INFO - /diag/eval status on 172.23.122.117:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-12 21:56:54,373] - [rest_client] [140098209314560] - INFO - settings/indexes params : storageMode=plasma [2021-02-12 21:56:54,392] - [task] [140098209314560] - INFO - server: ip:172.23.122.97 port:8091 ssh_username:root, nodes/self [2021-02-12 21:56:54,395] - [task] [140098209314560] - INFO - {'uptime': '55', 'memoryTotal': 4201684992, 'memoryFree': 3553415168, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.122.97:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4454-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.122.97', 'ip': '172.23.122.97', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} [2021-02-12 21:56:54,397] - [rest_client] [140098209314560] - ERROR - GET http://172.23.122.97:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-02-12 21:56:54,397] - [rest_client] [140098209314560] - INFO - pools/default params : memoryQuota=2147 [2021-02-12 21:56:54,403] - [rest_client] [140098209314560] - INFO - --> in init_cluster...Administrator,password,8091 [2021-02-12 21:56:54,403] - [rest_client] [140098209314560] - INFO - settings/web params on 172.23.122.97:8091:port=8091&username=Administrator&password=password [2021-02-12 21:56:54,460] - [rest_client] [140098209314560] - INFO - --> status:True [2021-02-12 21:56:54,461] - [remote_util] [140098209314560] - INFO - SSH Connecting to 172.23.122.97 with username:root, attempt#1 of 5 [2021-02-12 21:56:54,565] - [remote_util] [140098209314560] - INFO - SSH Connected to 172.23.122.97 as root [2021-02-12 21:56:54,835] - [remote_util] [140098209314560] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:56:55,133] - [remote_util] [140098209314560] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:56:55,142] - [remote_util] [140098209314560] - INFO - running command.raw on 172.23.122.97: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-02-12 21:56:55,199] - [remote_util] [140098209314560] - INFO - command executed successfully with root [2021-02-12 21:56:55,199] - [remote_util] [140098209314560] - INFO - ['ok'] [2021-02-12 21:56:55,202] - [rest_client] [140098209314560] - INFO - /diag/eval status on 172.23.122.97:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-12 21:56:55,204] - [rest_client] [140098209314560] - INFO - /diag/eval status on 172.23.122.97:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-12 21:56:55,208] - [rest_client] [140098209314560] - INFO - settings/indexes params : storageMode=plasma [2021-02-12 21:56:55,229] - [task] [140098209314560] - INFO - server: ip:172.23.122.82 port:8091 ssh_username:root, nodes/self [2021-02-12 21:56:55,232] - [task] [140098209314560] - INFO - {'uptime': '44', 'memoryTotal': 4201684992, 'memoryFree': 3582205952, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.122.82:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4454-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.122.82', 'ip': '172.23.122.82', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} [2021-02-12 21:56:55,234] - [rest_client] [140098209314560] - ERROR - GET http://172.23.122.82:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-02-12 21:56:55,234] - [rest_client] [140098209314560] - INFO - pools/default params : memoryQuota=2147 [2021-02-12 21:56:55,240] - [rest_client] [140098209314560] - INFO - --> in init_cluster...Administrator,password,8091 [2021-02-12 21:56:55,240] - [rest_client] [140098209314560] - INFO - settings/web params on 172.23.122.82:8091:port=8091&username=Administrator&password=password [2021-02-12 21:56:55,295] - [rest_client] [140098209314560] - INFO - --> status:True [2021-02-12 21:56:55,296] - [remote_util] [140098209314560] - INFO - SSH Connecting to 172.23.122.82 with username:root, attempt#1 of 5 [2021-02-12 21:56:55,401] - [remote_util] [140098209314560] - INFO - SSH Connected to 172.23.122.82 as root [2021-02-12 21:56:55,674] - [remote_util] [140098209314560] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:56:55,978] - [remote_util] [140098209314560] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:56:55,986] - [remote_util] [140098209314560] - INFO - running command.raw on 172.23.122.82: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-02-12 21:56:56,043] - [remote_util] [140098209314560] - INFO - command executed successfully with root [2021-02-12 21:56:56,043] - [remote_util] [140098209314560] - INFO - ['ok'] [2021-02-12 21:56:56,047] - [rest_client] [140098209314560] - INFO - /diag/eval status on 172.23.122.82:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-12 21:56:56,049] - [rest_client] [140098209314560] - INFO - /diag/eval status on 172.23.122.82:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-12 21:56:56,053] - [rest_client] [140098209314560] - INFO - settings/indexes params : storageMode=plasma [2021-02-12 21:56:56,071] - [task] [140098209314560] - INFO - server: ip:172.23.122.118 port:8091 ssh_username:root, nodes/self [2021-02-12 21:56:56,074] - [task] [140098209314560] - INFO - {'uptime': '29', 'memoryTotal': 4201689088, 'memoryFree': 3551313920, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.122.118:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4454-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.122.118', 'ip': '172.23.122.118', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} [2021-02-12 21:56:56,075] - [rest_client] [140098209314560] - ERROR - GET http://172.23.122.118:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-02-12 21:56:56,076] - [rest_client] [140098209314560] - INFO - pools/default params : memoryQuota=2147 [2021-02-12 21:56:56,081] - [rest_client] [140098209314560] - INFO - --> in init_cluster...Administrator,password,8091 [2021-02-12 21:56:56,081] - [rest_client] [140098209314560] - INFO - settings/web params on 172.23.122.118:8091:port=8091&username=Administrator&password=password [2021-02-12 21:56:56,136] - [rest_client] [140098209314560] - INFO - --> status:True [2021-02-12 21:56:56,137] - [remote_util] [140098209314560] - INFO - SSH Connecting to 172.23.122.118 with username:root, attempt#1 of 5 [2021-02-12 21:56:56,241] - [remote_util] [140098209314560] - INFO - SSH Connected to 172.23.122.118 as root [2021-02-12 21:56:56,485] - [remote_util] [140098209314560] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:56:56,778] - [remote_util] [140098209314560] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:56:56,786] - [remote_util] [140098209314560] - INFO - running command.raw on 172.23.122.118: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-02-12 21:56:56,842] - [remote_util] [140098209314560] - INFO - command executed successfully with root [2021-02-12 21:56:56,842] - [remote_util] [140098209314560] - INFO - ['ok'] [2021-02-12 21:56:56,844] - [rest_client] [140098209314560] - INFO - /diag/eval status on 172.23.122.118:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-12 21:56:56,847] - [rest_client] [140098209314560] - INFO - /diag/eval status on 172.23.122.118:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-12 21:56:56,851] - [rest_client] [140098209314560] - INFO - settings/indexes params : storageMode=plasma [2021-02-12 21:56:56,869] - [task] [140098209314560] - INFO - server: ip:172.23.122.116 port:8091 ssh_username:root, nodes/self [2021-02-12 21:56:56,872] - [task] [140098209314560] - INFO - {'uptime': '14', 'memoryTotal': 4201689088, 'memoryFree': 3569831936, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.122.116:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4454-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@cb.local', 'ip': '172.23.122.116', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} [2021-02-12 21:56:56,873] - [rest_client] [140098209314560] - ERROR - GET http://172.23.122.116:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-02-12 21:56:56,874] - [rest_client] [140098209314560] - INFO - pools/default params : memoryQuota=2147 [2021-02-12 21:56:56,879] - [rest_client] [140098209314560] - INFO - --> in init_cluster...Administrator,password,8091 [2021-02-12 21:56:56,879] - [rest_client] [140098209314560] - INFO - settings/web params on 172.23.122.116:8091:port=8091&username=Administrator&password=password [2021-02-12 21:56:56,934] - [rest_client] [140098209314560] - INFO - --> status:True [2021-02-12 21:56:56,935] - [remote_util] [140098209314560] - INFO - SSH Connecting to 172.23.122.116 with username:root, attempt#1 of 5 [2021-02-12 21:56:57,033] - [remote_util] [140098209314560] - INFO - SSH Connected to 172.23.122.116 as root [2021-02-12 21:56:57,309] - [remote_util] [140098209314560] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:56:57,603] - [remote_util] [140098209314560] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:56:57,611] - [remote_util] [140098209314560] - INFO - running command.raw on 172.23.122.116: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-02-12 21:56:57,665] - [remote_util] [140098209314560] - INFO - command executed successfully with root [2021-02-12 21:56:57,665] - [remote_util] [140098209314560] - INFO - ['ok'] [2021-02-12 21:56:57,667] - [rest_client] [140098209314560] - INFO - /diag/eval status on 172.23.122.116:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-12 21:56:57,669] - [rest_client] [140098209314560] - INFO - /diag/eval status on 172.23.122.116:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-12 21:56:57,674] - [rest_client] [140098209314560] - INFO - settings/indexes params : storageMode=plasma [2021-02-12 21:56:57,693] - [basetestcase] [140099299809024] - INFO - **** add built-in 'cbadminbucket' user to node 172.23.123.61 **** [2021-02-12 21:56:57,704] - [rest_client] [140099299809024] - ERROR - DELETE http://172.23.123.61:8091/settings/rbac/users/local/cbadminbucket body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:password [2021-02-12 21:56:57,704] - [internal_user] [140099299809024] - INFO - Exception while deleting user. Exception is -b'"User was not found."' [2021-02-12 21:56:57,765] - [basetestcase] [140099299809024] - INFO - sleep for 5 secs. ... [2021-02-12 21:57:02,771] - [basetestcase] [140099299809024] - INFO - **** add 'admin' role to 'cbadminbucket' user **** [2021-02-12 21:57:02,782] - [basetestcase] [140099299809024] - INFO - done initializing cluster [2021-02-12 21:57:03,691] - [task] [140098209314560] - INFO - adding node 172.23.123.62:8091 to cluster [2021-02-12 21:57:03,691] - [rest_client] [140098209314560] - INFO - adding remote node @172.23.123.62:8091 to this cluster @172.23.123.61:8091 [2021-02-12 21:57:06,096] - [task] [140098209314560] - INFO - adding node 172.23.122.90:8091 to cluster [2021-02-12 21:57:06,096] - [rest_client] [140098209314560] - INFO - adding remote node @172.23.122.90:8091 to this cluster @172.23.123.61:8091 [2021-02-12 21:57:08,386] - [task] [140098209314560] - INFO - adding node 172.23.122.117:8091 to cluster [2021-02-12 21:57:08,386] - [rest_client] [140098209314560] - INFO - adding remote node @172.23.122.117:8091 to this cluster @172.23.123.61:8091 [2021-02-12 21:57:10,666] - [task] [140098209314560] - INFO - adding node 172.23.122.97:8091 to cluster [2021-02-12 21:57:10,666] - [rest_client] [140098209314560] - INFO - adding remote node @172.23.122.97:8091 to this cluster @172.23.123.61:8091 [2021-02-12 21:57:13,120] - [task] [140098209314560] - INFO - adding node 172.23.122.82:8091 to cluster [2021-02-12 21:57:13,120] - [rest_client] [140098209314560] - INFO - adding remote node @172.23.122.82:8091 to this cluster @172.23.123.61:8091 [2021-02-12 21:57:15,494] - [rest_client] [140098209314560] - INFO - rebalance params : {'knownNodes': 'ns_1@172.23.122.117,ns_1@172.23.122.82,ns_1@172.23.122.90,ns_1@172.23.122.97,ns_1@172.23.123.61,ns_1@172.23.123.62', 'ejectedNodes': '', 'user': 'Administrator', 'password': 'password'} [2021-02-12 21:57:15,527] - [rest_client] [140098209314560] - INFO - rebalance operation started [2021-02-12 21:57:15,530] - [rest_client] [140098209314560] - INFO - rebalance percentage : 0.00 % [2021-02-12 21:57:15,530] - [task] [140098209314560] - INFO - Rebalance - status: running, progress: 0.00% [2021-02-12 21:57:25,551] - [task] [140098209314560] - INFO - Rebalance - status: none, progress: 100.00% [2021-02-12 21:57:25,560] - [task] [140098209314560] - INFO - rebalancing was completed with progress: 100% in 10.032175064086914 sec [2021-02-12 21:57:25,561] - [remote_util] [140099299809024] - INFO - SSH Connecting to 172.23.123.61 with username:root, attempt#1 of 5 [2021-02-12 21:57:25,665] - [remote_util] [140099299809024] - INFO - SSH Connected to 172.23.123.61 as root [2021-02-12 21:57:25,959] - [remote_util] [140099299809024] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:57:26,264] - [remote_util] [140099299809024] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:57:26,272] - [remote_util] [140099299809024] - INFO - running command.raw on 172.23.123.61: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-02-12 21:57:26,330] - [remote_util] [140099299809024] - INFO - command executed successfully with root [2021-02-12 21:57:26,330] - [remote_util] [140099299809024] - INFO - ['ok'] [2021-02-12 21:57:26,330] - [basetestcase] [140099299809024] - INFO - Enabled diag/eval for non-local hosts from 172.23.123.61 [2021-02-12 21:57:26,330] - [basetestcase] [140099299809024] - INFO - ============== basetestcase setup was finished for test #6 test_kv_swap_rebalance_when_existing_eventing_node_is_processing_mutations ============== [2021-02-12 21:57:26,338] - [remote_util] [140099299809024] - INFO - SSH Connecting to 172.23.123.61 with username:root, attempt#1 of 5 [2021-02-12 21:57:26,442] - [remote_util] [140099299809024] - INFO - SSH Connected to 172.23.123.61 as root [2021-02-12 21:57:26,705] - [remote_util] [140099299809024] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:57:27,001] - [remote_util] [140099299809024] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:57:27,003] - [remote_util] [140099299809024] - INFO - SSH Connecting to 172.23.123.62 with username:root, attempt#1 of 5 [2021-02-12 21:57:27,106] - [remote_util] [140099299809024] - INFO - SSH Connected to 172.23.123.62 as root [2021-02-12 21:57:27,390] - [remote_util] [140099299809024] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:57:27,697] - [remote_util] [140099299809024] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:57:27,699] - [remote_util] [140099299809024] - INFO - SSH Connecting to 172.23.122.90 with username:root, attempt#1 of 5 [2021-02-12 21:57:27,800] - [remote_util] [140099299809024] - INFO - SSH Connected to 172.23.122.90 as root [2021-02-12 21:57:28,081] - [remote_util] [140099299809024] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:57:28,376] - [remote_util] [140099299809024] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:57:28,378] - [remote_util] [140099299809024] - INFO - SSH Connecting to 172.23.122.117 with username:root, attempt#1 of 5 [2021-02-12 21:57:28,476] - [remote_util] [140099299809024] - INFO - SSH Connected to 172.23.122.117 as root [2021-02-12 21:57:28,733] - [remote_util] [140099299809024] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:57:29,025] - [remote_util] [140099299809024] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:57:29,027] - [remote_util] [140099299809024] - INFO - SSH Connecting to 172.23.122.97 with username:root, attempt#1 of 5 [2021-02-12 21:57:29,131] - [remote_util] [140099299809024] - INFO - SSH Connected to 172.23.122.97 as root [2021-02-12 21:57:29,392] - [remote_util] [140099299809024] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:57:29,693] - [remote_util] [140099299809024] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:57:29,696] - [remote_util] [140099299809024] - INFO - SSH Connecting to 172.23.122.82 with username:root, attempt#1 of 5 [2021-02-12 21:57:29,806] - [remote_util] [140099299809024] - INFO - SSH Connected to 172.23.122.82 as root [2021-02-12 21:57:30,083] - [remote_util] [140099299809024] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:57:30,385] - [remote_util] [140099299809024] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:57:30,387] - [remote_util] [140099299809024] - INFO - SSH Connecting to 172.23.122.118 with username:root, attempt#1 of 5 [2021-02-12 21:57:30,483] - [remote_util] [140099299809024] - INFO - SSH Connected to 172.23.122.118 as root [2021-02-12 21:57:30,727] - [remote_util] [140099299809024] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:57:31,023] - [remote_util] [140099299809024] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:57:31,025] - [remote_util] [140099299809024] - INFO - SSH Connecting to 172.23.122.116 with username:root, attempt#1 of 5 [2021-02-12 21:57:31,127] - [remote_util] [140099299809024] - INFO - SSH Connected to 172.23.122.116 as root [2021-02-12 21:57:31,392] - [remote_util] [140099299809024] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:57:31,682] - [remote_util] [140099299809024] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:58:04,530] - [basetestcase] [140099299809024] - INFO - ------- Cluster statistics ------- [2021-02-12 21:58:04,530] - [basetestcase] [140099299809024] - INFO - 172.23.122.117:8091 => {'services': ['eventing'], 'cpu_utilization': 10.33350176856998, 'mem_free': 3499151360, 'mem_total': 4201684992, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-02-12 21:58:04,530] - [basetestcase] [140099299809024] - INFO - 172.23.122.82:8091 => {'services': ['index', 'n1ql'], 'cpu_utilization': 0.9268537074148296, 'mem_free': 3500056576, 'mem_total': 4201684992, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-02-12 21:58:04,530] - [basetestcase] [140099299809024] - INFO - 172.23.122.90:8091 => {'services': ['eventing'], 'cpu_utilization': 0.8032128514056225, 'mem_free': 3536982016, 'mem_total': 4201684992, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-02-12 21:58:04,530] - [basetestcase] [140099299809024] - INFO - 172.23.122.97:8091 => {'services': ['eventing'], 'cpu_utilization': 23.12644862219933, 'mem_free': 3519442944, 'mem_total': 4201684992, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-02-12 21:58:04,530] - [basetestcase] [140099299809024] - INFO - 172.23.123.61:8091 => {'services': ['kv'], 'cpu_utilization': 0.907258064516129, 'mem_free': 3457597440, 'mem_total': 4201672704, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-02-12 21:58:04,530] - [basetestcase] [140099299809024] - INFO - 172.23.123.62:8091 => {'services': ['kv'], 'cpu_utilization': 0.8536279186542807, 'mem_free': 3452829696, 'mem_total': 4201676800, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-02-12 21:58:04,530] - [basetestcase] [140099299809024] - INFO - --- End of cluster statistics --- [2021-02-12 21:58:04,532] - [remote_util] [140099299809024] - INFO - SSH Connecting to 172.23.123.61 with username:root, attempt#1 of 5 [2021-02-12 21:58:04,634] - [remote_util] [140099299809024] - INFO - SSH Connected to 172.23.123.61 as root [2021-02-12 21:58:04,896] - [remote_util] [140099299809024] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:58:05,191] - [remote_util] [140099299809024] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:58:07,987] - [basetestcase] [140099299809024] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root] [2021-02-12 21:58:07,995] - [basetestcase] [140099299809024] - INFO - list of eventing nodes in cluster: [ip:172.23.122.117 port:8091 ssh_username:root, ip:172.23.122.90 port:8091 ssh_username:root, ip:172.23.122.97 port:8091 ssh_username:root] [2021-02-12 21:58:08,000] - [rest_client] [140099299809024] - INFO - settings/indexes params : storageMode=plasma [2021-02-12 21:58:08,007] - [eventing_base] [140099299809024] - INFO - Setting the min possible memory quota so that adding mode nodes to the cluster wouldn't be a problem. [2021-02-12 21:58:08,007] - [rest_client] [140099299809024] - INFO - pools/default params : memoryQuota=330 [2021-02-12 21:58:08,014] - [rest_client] [140099299809024] - INFO - pools/default params : indexMemoryQuota=256 [2021-02-12 21:58:08,023] - [rest_client] [140099299809024] - INFO - pools/default params : eventingMemoryQuota=512 [2021-02-12 21:58:08,041] - [basetestcase] [140099299809024] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root] [2021-02-12 21:58:08,046] - [rest_client] [140099299809024] - INFO - pools/default params : memoryQuota=700 [2021-02-12 21:58:08,051] - [eventing_rebalance_collection] [140099299809024] - INFO - 200 [2021-02-12 21:58:08,563] - [rest_client] [140098209314560] - INFO - http://172.23.123.61:8091/pools/default/buckets with param: name=src_bucket&ramQuotaMB=200&authType=none&replicaNumber=0&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore [2021-02-12 21:58:08,592] - [rest_client] [140098209314560] - INFO - 0.03 seconds to create bucket src_bucket [2021-02-12 21:58:08,592] - [bucket_helper] [140098209314560] - INFO - waiting for memcached bucket : src_bucket in 172.23.123.61 to accept set ops [2021-02-12 21:58:09,230] - [data_helper] [140098209314560] - INFO - creating direct client 172.23.123.61:11210 src_bucket [2021-02-12 21:58:09,305] - [data_helper] [140098209314560] - INFO - creating direct client 172.23.123.62:11210 src_bucket [2021-02-12 21:58:09,418] - [data_helper] [140098209314560] - INFO - creating direct client 172.23.123.61:11210 src_bucket [2021-02-12 21:58:09,660] - [data_helper] [140098209314560] - INFO - creating direct client 172.23.123.62:11210 src_bucket [2021-02-12 21:58:09,722] - [task] [140098209314560] - WARNING - vbucket map not ready after try 0 [2021-02-12 21:58:09,722] - [bucket_helper] [140098209314560] - INFO - waiting for memcached bucket : src_bucket in 172.23.123.61 to accept set ops [2021-02-12 21:58:09,831] - [data_helper] [140098209314560] - INFO - creating direct client 172.23.123.61:11210 src_bucket [2021-02-12 21:58:09,914] - [data_helper] [140098209314560] - INFO - creating direct client 172.23.123.62:11210 src_bucket [2021-02-12 21:58:10,032] - [data_helper] [140098209314560] - INFO - creating direct client 172.23.123.61:11210 src_bucket [2021-02-12 21:58:10,104] - [data_helper] [140098209314560] - INFO - creating direct client 172.23.123.62:11210 src_bucket [2021-02-12 21:58:10,327] - [task] [140098209314560] - WARNING - vbucket map not ready after try 1 [2021-02-12 21:58:10,327] - [bucket_helper] [140098209314560] - INFO - waiting for memcached bucket : src_bucket in 172.23.123.61 to accept set ops [2021-02-12 21:58:10,433] - [data_helper] [140098209314560] - INFO - creating direct client 172.23.123.61:11210 src_bucket [2021-02-12 21:58:10,504] - [data_helper] [140098209314560] - INFO - creating direct client 172.23.123.62:11210 src_bucket [2021-02-12 21:58:10,636] - [data_helper] [140098209314560] - INFO - creating direct client 172.23.123.61:11210 src_bucket [2021-02-12 21:58:10,707] - [data_helper] [140098209314560] - INFO - creating direct client 172.23.123.62:11210 src_bucket [2021-02-12 21:58:10,780] - [task] [140098209314560] - WARNING - vbucket map not ready after try 2 [2021-02-12 21:58:10,780] - [bucket_helper] [140098209314560] - INFO - waiting for memcached bucket : src_bucket in 172.23.123.61 to accept set ops [2021-02-12 21:58:11,055] - [data_helper] [140098209314560] - INFO - creating direct client 172.23.123.61:11210 src_bucket [2021-02-12 21:58:11,130] - [data_helper] [140098209314560] - INFO - creating direct client 172.23.123.62:11210 src_bucket [2021-02-12 21:58:11,246] - [data_helper] [140098209314560] - INFO - creating direct client 172.23.123.61:11210 src_bucket [2021-02-12 21:58:11,322] - [data_helper] [140098209314560] - INFO - creating direct client 172.23.123.62:11210 src_bucket [2021-02-12 21:58:11,547] - [task] [140098209314560] - WARNING - vbucket map not ready after try 3 [2021-02-12 21:58:11,547] - [bucket_helper] [140098209314560] - INFO - waiting for memcached bucket : src_bucket in 172.23.123.61 to accept set ops [2021-02-12 21:58:11,663] - [data_helper] [140098209314560] - INFO - creating direct client 172.23.123.61:11210 src_bucket [2021-02-12 21:58:11,736] - [data_helper] [140098209314560] - INFO - creating direct client 172.23.123.62:11210 src_bucket [2021-02-12 21:58:11,857] - [data_helper] [140098209314560] - INFO - creating direct client 172.23.123.61:11210 src_bucket [2021-02-12 21:58:11,934] - [data_helper] [140098209314560] - INFO - creating direct client 172.23.123.62:11210 src_bucket [2021-02-12 21:58:11,999] - [task] [140098209314560] - WARNING - vbucket map not ready after try 4 [2021-02-12 21:58:11,999] - [bucket_helper] [140098209314560] - INFO - waiting for memcached bucket : src_bucket in 172.23.123.61 to accept set ops [2021-02-12 21:58:12,292] - [data_helper] [140098209314560] - INFO - creating direct client 172.23.123.61:11210 src_bucket [2021-02-12 21:58:12,366] - [data_helper] [140098209314560] - INFO - creating direct client 172.23.123.62:11210 src_bucket [2021-02-12 21:58:12,484] - [data_helper] [140098209314560] - INFO - creating direct client 172.23.123.61:11210 src_bucket [2021-02-12 21:58:12,556] - [data_helper] [140098209314560] - INFO - creating direct client 172.23.123.62:11210 src_bucket [2021-02-12 21:58:12,622] - [task] [140098209314560] - WARNING - vbucket map not ready after try 5 [2021-02-12 21:58:13,647] - [rest_client] [140098209314560] - INFO - http://172.23.123.61:8091/pools/default/buckets with param: name=dst_bucket&ramQuotaMB=200&authType=none&replicaNumber=0&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore [2021-02-12 21:58:13,676] - [rest_client] [140098209314560] - INFO - 0.03 seconds to create bucket dst_bucket [2021-02-12 21:58:13,677] - [bucket_helper] [140098209314560] - INFO - waiting for memcached bucket : dst_bucket in 172.23.123.61 to accept set ops [2021-02-12 21:58:14,315] - [data_helper] [140098209314560] - INFO - creating direct client 172.23.123.61:11210 dst_bucket [2021-02-12 21:58:14,396] - [data_helper] [140098209314560] - INFO - creating direct client 172.23.123.62:11210 dst_bucket [2021-02-12 21:58:14,521] - [data_helper] [140098209314560] - INFO - creating direct client 172.23.123.61:11210 dst_bucket [2021-02-12 21:58:14,617] - [data_helper] [140098209314560] - INFO - creating direct client 172.23.123.62:11210 dst_bucket [2021-02-12 21:58:14,855] - [task] [140098209314560] - WARNING - vbucket map not ready after try 0 [2021-02-12 21:58:14,855] - [bucket_helper] [140098209314560] - INFO - waiting for memcached bucket : dst_bucket in 172.23.123.61 to accept set ops [2021-02-12 21:58:14,972] - [data_helper] [140098209314560] - INFO - creating direct client 172.23.123.61:11210 dst_bucket [2021-02-12 21:58:15,058] - [data_helper] [140098209314560] - INFO - creating direct client 172.23.123.62:11210 dst_bucket [2021-02-12 21:58:15,187] - [data_helper] [140098209314560] - INFO - creating direct client 172.23.123.61:11210 dst_bucket [2021-02-12 21:58:15,266] - [data_helper] [140098209314560] - INFO - creating direct client 172.23.123.62:11210 dst_bucket [2021-02-12 21:58:15,508] - [task] [140098209314560] - WARNING - vbucket map not ready after try 1 [2021-02-12 21:58:15,508] - [bucket_helper] [140098209314560] - INFO - waiting for memcached bucket : dst_bucket in 172.23.123.61 to accept set ops [2021-02-12 21:58:15,628] - [data_helper] [140098209314560] - INFO - creating direct client 172.23.123.61:11210 dst_bucket [2021-02-12 21:58:15,705] - [data_helper] [140098209314560] - INFO - creating direct client 172.23.123.62:11210 dst_bucket [2021-02-12 21:58:15,820] - [data_helper] [140098209314560] - INFO - creating direct client 172.23.123.61:11210 dst_bucket [2021-02-12 21:58:15,897] - [data_helper] [140098209314560] - INFO - creating direct client 172.23.123.62:11210 dst_bucket [2021-02-12 21:58:15,965] - [task] [140098209314560] - WARNING - vbucket map not ready after try 2 [2021-02-12 21:58:15,965] - [bucket_helper] [140098209314560] - INFO - waiting for memcached bucket : dst_bucket in 172.23.123.61 to accept set ops [2021-02-12 21:58:16,258] - [data_helper] [140098209314560] - INFO - creating direct client 172.23.123.61:11210 dst_bucket [2021-02-12 21:58:16,342] - [data_helper] [140098209314560] - INFO - creating direct client 172.23.123.62:11210 dst_bucket [2021-02-12 21:58:16,461] - [data_helper] [140098209314560] - INFO - creating direct client 172.23.123.61:11210 dst_bucket [2021-02-12 21:58:16,546] - [data_helper] [140098209314560] - INFO - creating direct client 172.23.123.62:11210 dst_bucket [2021-02-12 21:58:16,621] - [task] [140098209314560] - WARNING - vbucket map not ready after try 3 [2021-02-12 21:58:16,621] - [bucket_helper] [140098209314560] - INFO - waiting for memcached bucket : dst_bucket in 172.23.123.61 to accept set ops [2021-02-12 21:58:16,919] - [data_helper] [140098209314560] - INFO - creating direct client 172.23.123.61:11210 dst_bucket [2021-02-12 21:58:16,999] - [data_helper] [140098209314560] - INFO - creating direct client 172.23.123.62:11210 dst_bucket [2021-02-12 21:58:17,115] - [data_helper] [140098209314560] - INFO - creating direct client 172.23.123.61:11210 dst_bucket [2021-02-12 21:58:17,192] - [data_helper] [140098209314560] - INFO - creating direct client 172.23.123.62:11210 dst_bucket [2021-02-12 21:58:17,259] - [task] [140098209314560] - WARNING - vbucket map not ready after try 4 [2021-02-12 21:58:17,260] - [bucket_helper] [140098209314560] - INFO - waiting for memcached bucket : dst_bucket in 172.23.123.61 to accept set ops [2021-02-12 21:58:17,547] - [data_helper] [140098209314560] - INFO - creating direct client 172.23.123.61:11210 dst_bucket [2021-02-12 21:58:17,625] - [data_helper] [140098209314560] - INFO - creating direct client 172.23.123.62:11210 dst_bucket [2021-02-12 21:58:17,743] - [data_helper] [140098209314560] - INFO - creating direct client 172.23.123.61:11210 dst_bucket [2021-02-12 21:58:17,820] - [data_helper] [140098209314560] - INFO - creating direct client 172.23.123.62:11210 dst_bucket [2021-02-12 21:58:17,892] - [task] [140098209314560] - WARNING - vbucket map not ready after try 5 [2021-02-12 21:58:18,917] - [rest_client] [140098209314560] - INFO - http://172.23.123.61:8091/pools/default/buckets with param: name=metadata&ramQuotaMB=200&authType=none&replicaNumber=0&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore [2021-02-12 21:58:18,946] - [rest_client] [140098209314560] - INFO - 0.03 seconds to create bucket metadata [2021-02-12 21:58:18,947] - [bucket_helper] [140098209314560] - INFO - waiting for memcached bucket : metadata in 172.23.123.61 to accept set ops [2021-02-12 21:58:19,584] - [data_helper] [140098209314560] - INFO - creating direct client 172.23.123.61:11210 metadata [2021-02-12 21:58:19,876] - [data_helper] [140098209314560] - INFO - creating direct client 172.23.123.62:11210 metadata [2021-02-12 21:58:20,002] - [data_helper] [140098209314560] - INFO - creating direct client 172.23.123.61:11210 metadata [2021-02-12 21:58:20,081] - [data_helper] [140098209314560] - INFO - creating direct client 172.23.123.62:11210 metadata [2021-02-12 21:58:20,147] - [task] [140098209314560] - WARNING - vbucket map not ready after try 0 [2021-02-12 21:58:20,147] - [bucket_helper] [140098209314560] - INFO - waiting for memcached bucket : metadata in 172.23.123.61 to accept set ops [2021-02-12 21:58:20,259] - [data_helper] [140098209314560] - INFO - creating direct client 172.23.123.61:11210 metadata [2021-02-12 21:58:20,520] - [data_helper] [140098209314560] - INFO - creating direct client 172.23.123.62:11210 metadata [2021-02-12 21:58:20,651] - [data_helper] [140098209314560] - INFO - creating direct client 172.23.123.61:11210 metadata [2021-02-12 21:58:20,731] - [data_helper] [140098209314560] - INFO - creating direct client 172.23.123.62:11210 metadata [2021-02-12 21:58:20,806] - [task] [140098209314560] - WARNING - vbucket map not ready after try 1 [2021-02-12 21:58:20,806] - [bucket_helper] [140098209314560] - INFO - waiting for memcached bucket : metadata in 172.23.123.61 to accept set ops [2021-02-12 21:58:20,935] - [data_helper] [140098209314560] - INFO - creating direct client 172.23.123.61:11210 metadata [2021-02-12 21:58:21,178] - [data_helper] [140098209314560] - INFO - creating direct client 172.23.123.62:11210 metadata [2021-02-12 21:58:21,309] - [data_helper] [140098209314560] - INFO - creating direct client 172.23.123.61:11210 metadata [2021-02-12 21:58:21,394] - [data_helper] [140098209314560] - INFO - creating direct client 172.23.123.62:11210 metadata [2021-02-12 21:58:21,464] - [task] [140098209314560] - WARNING - vbucket map not ready after try 2 [2021-02-12 21:58:21,465] - [bucket_helper] [140098209314560] - INFO - waiting for memcached bucket : metadata in 172.23.123.61 to accept set ops [2021-02-12 21:58:21,584] - [data_helper] [140098209314560] - INFO - creating direct client 172.23.123.61:11210 metadata [2021-02-12 21:58:21,672] - [data_helper] [140098209314560] - INFO - creating direct client 172.23.123.62:11210 metadata [2021-02-12 21:58:21,967] - [data_helper] [140098209314560] - INFO - creating direct client 172.23.123.61:11210 metadata [2021-02-12 21:58:22,056] - [data_helper] [140098209314560] - INFO - creating direct client 172.23.123.62:11210 metadata [2021-02-12 21:58:22,128] - [task] [140098209314560] - WARNING - vbucket map not ready after try 3 [2021-02-12 21:58:22,128] - [bucket_helper] [140098209314560] - INFO - waiting for memcached bucket : metadata in 172.23.123.61 to accept set ops [2021-02-12 21:58:22,250] - [data_helper] [140098209314560] - INFO - creating direct client 172.23.123.61:11210 metadata [2021-02-12 21:58:22,338] - [data_helper] [140098209314560] - INFO - creating direct client 172.23.123.62:11210 metadata [2021-02-12 21:58:22,636] - [data_helper] [140098209314560] - INFO - creating direct client 172.23.123.61:11210 metadata [2021-02-12 21:58:22,726] - [data_helper] [140098209314560] - INFO - creating direct client 172.23.123.62:11210 metadata [2021-02-12 21:58:22,798] - [task] [140098209314560] - WARNING - vbucket map not ready after try 4 [2021-02-12 21:58:22,798] - [bucket_helper] [140098209314560] - INFO - waiting for memcached bucket : metadata in 172.23.123.61 to accept set ops [2021-02-12 21:58:22,933] - [data_helper] [140098209314560] - INFO - creating direct client 172.23.123.61:11210 metadata [2021-02-12 21:58:23,024] - [data_helper] [140098209314560] - INFO - creating direct client 172.23.123.62:11210 metadata [2021-02-12 21:58:23,325] - [data_helper] [140098209314560] - INFO - creating direct client 172.23.123.61:11210 metadata [2021-02-12 21:58:23,415] - [data_helper] [140098209314560] - INFO - creating direct client 172.23.123.62:11210 metadata [2021-02-12 21:58:23,493] - [task] [140098209314560] - WARNING - vbucket map not ready after try 5 [2021-02-12 21:58:23,581] - [rest_client] [140099299809024] - INFO - http://172.23.122.117:8091/pools/default/buckets/dst_bucket/scopes with params: name=scope_0 [2021-02-12 21:58:23,581] - [rest_client] [140099299809024] - INFO - Scope created dst_bucket->scope_0 {'uid': '1'} [2021-02-12 21:58:23,608] - [rest_client] [140099299809024] - INFO - http://172.23.122.117:8091/pools/default/buckets/dst_bucket/scopes with params: name=scope_1 [2021-02-12 21:58:23,609] - [rest_client] [140099299809024] - INFO - Scope created dst_bucket->scope_1 {'uid': '2'} [2021-02-12 21:58:23,640] - [rest_client] [140099299809024] - INFO - http://172.23.122.117:8091/pools/default/buckets/dst_bucket/scopes with params: name=scope_2 [2021-02-12 21:58:23,640] - [rest_client] [140099299809024] - INFO - Scope created dst_bucket->scope_2 {'uid': '3'} [2021-02-12 21:58:23,669] - [rest_client] [140099299809024] - INFO - http://172.23.122.117:8091/pools/default/buckets/dst_bucket/scopes with params: name=scope_3 [2021-02-12 21:58:23,669] - [rest_client] [140099299809024] - INFO - Scope created dst_bucket->scope_3 {'uid': '4'} [2021-02-12 21:58:23,700] - [rest_client] [140099299809024] - INFO - http://172.23.122.117:8091/pools/default/buckets/dst_bucket/scopes with params: name=scope_4 [2021-02-12 21:58:23,700] - [rest_client] [140099299809024] - INFO - Scope created dst_bucket->scope_4 {'uid': '5'} [2021-02-12 21:58:23,728] - [rest_client] [140099299809024] - INFO - http://172.23.122.117:8091/pools/default/buckets/src_bucket/scopes with params: name=scope_0 [2021-02-12 21:58:23,728] - [rest_client] [140099299809024] - INFO - Scope created src_bucket->scope_0 {'uid': '1'} [2021-02-12 21:58:23,760] - [rest_client] [140099299809024] - INFO - http://172.23.122.117:8091/pools/default/buckets/src_bucket/scopes with params: name=scope_1 [2021-02-12 21:58:23,760] - [rest_client] [140099299809024] - INFO - Scope created src_bucket->scope_1 {'uid': '2'} [2021-02-12 21:58:23,788] - [rest_client] [140099299809024] - INFO - http://172.23.122.117:8091/pools/default/buckets/src_bucket/scopes with params: name=scope_2 [2021-02-12 21:58:23,789] - [rest_client] [140099299809024] - INFO - Scope created src_bucket->scope_2 {'uid': '3'} [2021-02-12 21:58:23,820] - [rest_client] [140099299809024] - INFO - http://172.23.122.117:8091/pools/default/buckets/src_bucket/scopes with params: name=scope_3 [2021-02-12 21:58:23,820] - [rest_client] [140099299809024] - INFO - Scope created src_bucket->scope_3 {'uid': '4'} [2021-02-12 21:58:23,848] - [rest_client] [140099299809024] - INFO - http://172.23.122.117:8091/pools/default/buckets/src_bucket/scopes with params: name=scope_4 [2021-02-12 21:58:23,848] - [rest_client] [140099299809024] - INFO - Scope created src_bucket->scope_4 {'uid': '5'} [2021-02-12 21:58:23,879] - [rest_client] [140099299809024] - INFO - http://172.23.122.117:8091/pools/default/buckets/dst_bucket/scopes/scope_1/collections with params: name=coll_0 [2021-02-12 21:58:23,879] - [rest_client] [140099299809024] - INFO - Collection created dst_bucket->scope_1->coll_0 manifest:{'uid': '6'} [2021-02-12 21:58:23,909] - [rest_client] [140099299809024] - INFO - http://172.23.122.117:8091/pools/default/buckets/dst_bucket/scopes/scope_1/collections with params: name=coll_1 [2021-02-12 21:58:23,909] - [rest_client] [140099299809024] - INFO - Collection created dst_bucket->scope_1->coll_1 manifest:{'uid': '7'} [2021-02-12 21:58:23,937] - [rest_client] [140099299809024] - INFO - http://172.23.122.117:8091/pools/default/buckets/dst_bucket/scopes/scope_1/collections with params: name=coll_2 [2021-02-12 21:58:23,937] - [rest_client] [140099299809024] - INFO - Collection created dst_bucket->scope_1->coll_2 manifest:{'uid': '8'} [2021-02-12 21:58:23,967] - [rest_client] [140099299809024] - INFO - http://172.23.122.117:8091/pools/default/buckets/dst_bucket/scopes/scope_1/collections with params: name=coll_3 [2021-02-12 21:58:23,967] - [rest_client] [140099299809024] - INFO - Collection created dst_bucket->scope_1->coll_3 manifest:{'uid': '9'} [2021-02-12 21:58:24,001] - [rest_client] [140099299809024] - INFO - http://172.23.122.117:8091/pools/default/buckets/dst_bucket/scopes/scope_1/collections with params: name=coll_4 [2021-02-12 21:58:24,001] - [rest_client] [140099299809024] - INFO - Collection created dst_bucket->scope_1->coll_4 manifest:{'uid': 'a'} [2021-02-12 21:58:24,026] - [rest_client] [140099299809024] - INFO - http://172.23.122.117:8091/pools/default/buckets/src_bucket/scopes/scope_1/collections with params: name=coll_0 [2021-02-12 21:58:24,026] - [rest_client] [140099299809024] - INFO - Collection created src_bucket->scope_1->coll_0 manifest:{'uid': '6'} [2021-02-12 21:58:24,069] - [rest_client] [140099299809024] - INFO - http://172.23.122.117:8091/pools/default/buckets/src_bucket/scopes/scope_1/collections with params: name=coll_1 [2021-02-12 21:58:24,069] - [rest_client] [140099299809024] - INFO - Collection created src_bucket->scope_1->coll_1 manifest:{'uid': '7'} [2021-02-12 21:58:24,098] - [rest_client] [140099299809024] - INFO - http://172.23.122.117:8091/pools/default/buckets/src_bucket/scopes/scope_1/collections with params: name=coll_2 [2021-02-12 21:58:24,099] - [rest_client] [140099299809024] - INFO - Collection created src_bucket->scope_1->coll_2 manifest:{'uid': '8'} [2021-02-12 21:58:24,130] - [rest_client] [140099299809024] - INFO - http://172.23.122.117:8091/pools/default/buckets/src_bucket/scopes/scope_1/collections with params: name=coll_3 [2021-02-12 21:58:24,130] - [rest_client] [140099299809024] - INFO - Collection created src_bucket->scope_1->coll_3 manifest:{'uid': '9'} [2021-02-12 21:58:24,157] - [rest_client] [140099299809024] - INFO - http://172.23.122.117:8091/pools/default/buckets/src_bucket/scopes/scope_1/collections with params: name=coll_4 [2021-02-12 21:58:24,157] - [rest_client] [140099299809024] - INFO - Collection created src_bucket->scope_1->coll_4 manifest:{'uid': 'a'} [2021-02-12 21:58:25,482] - [eventing_base] [140099299809024] - INFO - saving function bucket_op [2021-02-12 21:58:26,652] - [eventing_base] [140099299809024] - INFO - saving function timers [2021-02-12 21:58:27,907] - [eventing_base] [140099299809024] - INFO - saving function sbm [2021-02-12 21:58:29,122] - [eventing_base] [140099299809024] - INFO - saving function curl [2021-02-12 21:58:30,352] - [eventing_base] [140099299809024] - INFO - saving function n1ql [2021-02-12 21:58:31,614] - [basetestcase] [140099299809024] - INFO - sleep for 5 secs. Waiting for bucket_op to deployed... ... [2021-02-12 21:58:36,656] - [basetestcase] [140099299809024] - INFO - sleep for 5 secs. Waiting for bucket_op to deployed... ... [2021-02-12 21:58:41,716] - [basetestcase] [140099299809024] - INFO - sleep for 5 secs. Waiting for bucket_op to deployed... ... [2021-02-12 21:58:46,760] - [basetestcase] [140099299809024] - INFO - sleep for 5 secs. Waiting for bucket_op to deployed... ... [2021-02-12 21:58:53,066] - [basetestcase] [140099299809024] - INFO - sleep for 5 secs. Waiting for timers to deployed... ... [2021-02-12 21:58:58,110] - [basetestcase] [140099299809024] - INFO - sleep for 5 secs. Waiting for timers to deployed... ... [2021-02-12 21:59:03,159] - [basetestcase] [140099299809024] - INFO - sleep for 5 secs. Waiting for timers to deployed... ... [2021-02-12 21:59:08,201] - [basetestcase] [140099299809024] - INFO - sleep for 5 secs. Waiting for timers to deployed... ... [2021-02-12 21:59:14,581] - [basetestcase] [140099299809024] - INFO - sleep for 5 secs. Waiting for sbm to deployed... ... [2021-02-12 21:59:19,622] - [basetestcase] [140099299809024] - INFO - sleep for 5 secs. Waiting for sbm to deployed... ... [2021-02-12 21:59:24,672] - [basetestcase] [140099299809024] - INFO - sleep for 5 secs. Waiting for sbm to deployed... ... [2021-02-12 21:59:29,713] - [basetestcase] [140099299809024] - INFO - sleep for 5 secs. Waiting for sbm to deployed... ... [2021-02-12 21:59:35,951] - [basetestcase] [140099299809024] - INFO - sleep for 5 secs. Waiting for curl to deployed... ... [2021-02-12 21:59:40,994] - [basetestcase] [140099299809024] - INFO - sleep for 5 secs. Waiting for curl to deployed... ... [2021-02-12 21:59:46,055] - [basetestcase] [140099299809024] - INFO - sleep for 5 secs. Waiting for curl to deployed... ... [2021-02-12 21:59:51,098] - [basetestcase] [140099299809024] - INFO - sleep for 5 secs. Waiting for curl to deployed... ... [2021-02-12 21:59:57,358] - [basetestcase] [140099299809024] - INFO - sleep for 5 secs. Waiting for n1ql to deployed... ... [2021-02-12 22:00:02,406] - [basetestcase] [140099299809024] - INFO - sleep for 5 secs. Waiting for n1ql to deployed... ... [2021-02-12 22:00:07,460] - [basetestcase] [140099299809024] - INFO - sleep for 5 secs. Waiting for n1ql to deployed... ... [2021-02-12 22:00:12,513] - [basetestcase] [140099299809024] - INFO - sleep for 5 secs. Waiting for n1ql to deployed... ... [2021-02-12 22:00:17,611] - [task] [140098209314560] - INFO - java -jar java_sdk_client/collections/target/javaclient/javaclient.jar -i 172.23.123.61 -u Administrator -p password -b src_bucket -s _default -c _default -n 40320 -pc 100 -pu 0 -pd 0 -l uniform -dsn 1 -dpx doc_ -dt Person -de 0 -ds 500 -ac False -st 0 -en 40319 -o False -sd False [2021-02-12 22:00:20,567] - [rest_client] [140099299809024] - INFO - rebalance reached >100% in 3.0047757625579834 seconds [2021-02-12 22:00:28,386] - [task] [140098209314560] - INFO - java -jar java_sdk_client/collections/target/javaclient/javaclient.jar -i 172.23.123.61 -u Administrator -p password -b src_bucket -s scope_1 -c coll_1 -n 40320 -pc 100 -pu 0 -pd 0 -l uniform -dsn 1 -dpx doc_ -dt Person -de 0 -ds 500 -ac False -st 0 -en 40319 -o False -sd False [2021-02-12 22:00:39,224] - [rest_client] [140098209314560] - INFO - Node versions in cluster ['7.0.0-4454-enterprise', '7.0.0-4454-enterprise', '7.0.0-4454-enterprise', '7.0.0-4454-enterprise', '7.0.0-4454-enterprise', '7.0.0-4454-enterprise'] [2021-02-12 22:00:39,640] - [task] [140098209314560] - INFO - This is swap rebalance and we will monitor vbuckets shuffling [2021-02-12 22:00:39,640] - [task] [140098209314560] - INFO - adding node 172.23.122.118:8091 to cluster [2021-02-12 22:00:39,641] - [rest_client] [140098209314560] - INFO - adding remote node @172.23.122.118:8091 to this cluster @172.23.123.61:8091 [2021-02-12 22:00:42,118] - [rest_client] [140098209314560] - INFO - rebalance params : {'knownNodes': 'ns_1@172.23.122.117,ns_1@172.23.122.118,ns_1@172.23.122.82,ns_1@172.23.122.90,ns_1@172.23.122.97,ns_1@172.23.123.61,ns_1@172.23.123.62', 'ejectedNodes': 'ns_1@172.23.123.62', 'user': 'Administrator', 'password': 'password'} [2021-02-12 22:00:42,158] - [rest_client] [140098209314560] - INFO - rebalance operation started [2021-02-12 22:00:42,158] - [task] [140098209314560] - INFO - This is swap rebalance and we will monitor vbuckets shuffling [2021-02-12 22:00:42,594] - [rest_client] [140098209314560] - INFO - rebalance percentage : 0.00 % [2021-02-12 22:00:42,594] - [task] [140098209314560] - INFO - Rebalance - status: running, progress: 0.00% [2021-02-12 22:00:52,622] - [task] [140098209314560] - INFO - This is swap rebalance and we will monitor vbuckets shuffling [2021-02-12 22:00:53,015] - [rest_client] [140098209314560] - INFO - rebalance percentage : 9.00 % [2021-02-12 22:00:53,016] - [task] [140098209314560] - INFO - Rebalance - status: running, progress: 9.00% [2021-02-12 22:01:03,043] - [task] [140098209314560] - INFO - This is swap rebalance and we will monitor vbuckets shuffling [2021-02-12 22:01:03,535] - [rest_client] [140098209314560] - INFO - rebalance percentage : 14.00 % [2021-02-12 22:01:03,535] - [task] [140098209314560] - INFO - Rebalance - status: running, progress: 14.00% [2021-02-12 22:01:13,559] - [task] [140098209314560] - INFO - This is swap rebalance and we will monitor vbuckets shuffling [2021-02-12 22:01:14,143] - [rest_client] [140098209314560] - INFO - rebalance percentage : 23.00 % [2021-02-12 22:01:14,143] - [task] [140098209314560] - INFO - Rebalance - status: running, progress: 23.00% [2021-02-12 22:01:24,168] - [task] [140098209314560] - INFO - This is swap rebalance and we will monitor vbuckets shuffling [2021-02-12 22:01:24,587] - [rest_client] [140098209314560] - INFO - rebalance percentage : 32.00 % [2021-02-12 22:01:24,588] - [task] [140098209314560] - INFO - Rebalance - status: running, progress: 32.00% [2021-02-12 22:01:34,613] - [task] [140098209314560] - INFO - This is swap rebalance and we will monitor vbuckets shuffling [2021-02-12 22:01:35,044] - [rest_client] [140098209314560] - INFO - rebalance percentage : 38.00 % [2021-02-12 22:01:35,044] - [task] [140098209314560] - INFO - Rebalance - status: running, progress: 38.00% [2021-02-12 22:01:45,066] - [task] [140098209314560] - INFO - This is swap rebalance and we will monitor vbuckets shuffling [2021-02-12 22:01:45,434] - [rest_client] [140098209314560] - INFO - rebalance percentage : 50.00 % [2021-02-12 22:01:45,434] - [task] [140098209314560] - INFO - Rebalance - status: running, progress: 50.00% [2021-02-12 22:01:55,456] - [task] [140098209314560] - INFO - This is swap rebalance and we will monitor vbuckets shuffling [2021-02-12 22:01:55,843] - [rest_client] [140098209314560] - INFO - rebalance percentage : 51.00 % [2021-02-12 22:01:55,843] - [task] [140098209314560] - INFO - Rebalance - status: running, progress: 51.00% [2021-02-12 22:02:05,864] - [task] [140098209314560] - INFO - This is swap rebalance and we will monitor vbuckets shuffling [2021-02-12 22:02:06,238] - [rest_client] [140098209314560] - INFO - rebalance percentage : 58.00 % [2021-02-12 22:02:06,238] - [task] [140098209314560] - INFO - Rebalance - status: running, progress: 58.00% [2021-02-12 22:02:16,261] - [task] [140098209314560] - INFO - This is swap rebalance and we will monitor vbuckets shuffling [2021-02-12 22:02:16,634] - [rest_client] [140098209314560] - INFO - rebalance percentage : 67.00 % [2021-02-12 22:02:16,634] - [task] [140098209314560] - INFO - Rebalance - status: running, progress: 67.00% [2021-02-12 22:02:26,658] - [task] [140098209314560] - INFO - This is swap rebalance and we will monitor vbuckets shuffling [2021-02-12 22:02:27,078] - [rest_client] [140098209314560] - INFO - rebalance percentage : 74.00 % [2021-02-12 22:02:27,078] - [task] [140098209314560] - INFO - Rebalance - status: running, progress: 74.00% [2021-02-12 22:02:37,101] - [task] [140098209314560] - INFO - This is swap rebalance and we will monitor vbuckets shuffling [2021-02-12 22:02:37,517] - [rest_client] [140098209314560] - INFO - rebalance percentage : 80.00 % [2021-02-12 22:02:37,518] - [task] [140098209314560] - INFO - Rebalance - status: running, progress: 80.00% [2021-02-12 22:02:47,541] - [task] [140098209314560] - INFO - This is swap rebalance and we will monitor vbuckets shuffling [2021-02-12 22:02:47,945] - [rest_client] [140098209314560] - INFO - rebalance percentage : 84.00 % [2021-02-12 22:02:47,945] - [task] [140098209314560] - INFO - Rebalance - status: running, progress: 84.00% [2021-02-12 22:02:57,967] - [task] [140098209314560] - INFO - This is swap rebalance and we will monitor vbuckets shuffling [2021-02-12 22:02:58,368] - [rest_client] [140098209314560] - INFO - rebalance percentage : 87.00 % [2021-02-12 22:02:58,368] - [task] [140098209314560] - INFO - Rebalance - status: running, progress: 87.00% [2021-02-12 22:03:08,393] - [task] [140098209314560] - INFO - This is swap rebalance and we will monitor vbuckets shuffling [2021-02-12 22:03:08,783] - [rest_client] [140098209314560] - INFO - rebalance percentage : 90.00 % [2021-02-12 22:03:08,784] - [task] [140098209314560] - INFO - Rebalance - status: running, progress: 90.00% [2021-02-12 22:03:18,808] - [task] [140098209314560] - INFO - This is swap rebalance and we will monitor vbuckets shuffling [2021-02-12 22:03:19,177] - [rest_client] [140098209314560] - INFO - rebalance percentage : 91.00 % [2021-02-12 22:03:19,177] - [task] [140098209314560] - INFO - Rebalance - status: running, progress: 91.00% [2021-02-12 22:03:29,198] - [task] [140098209314560] - INFO - This is swap rebalance and we will monitor vbuckets shuffling [2021-02-12 22:03:29,576] - [rest_client] [140098209314560] - INFO - rebalance percentage : 92.00 % [2021-02-12 22:03:29,577] - [task] [140098209314560] - INFO - Rebalance - status: running, progress: 92.00% [2021-02-12 22:03:39,598] - [task] [140098209314560] - INFO - This is swap rebalance and we will monitor vbuckets shuffling [2021-02-12 22:03:39,980] - [task] [140098209314560] - INFO - Rebalance - status: none, progress: 100.00% [2021-02-12 22:03:39,999] - [task] [140098209314560] - INFO - rebalancing was completed with progress: 100% in 177.84117650985718 sec [2021-02-12 22:03:40,007] - [basetestcase] [140099299809024] - INFO - list of eventing nodes in cluster: [ip:172.23.122.117 port:8091 ssh_username:root, ip:172.23.122.90 port:8091 ssh_username:root, ip:172.23.122.97 port:8091 ssh_username:root] [2021-02-12 22:03:40,012] - [tuq_helper] [140099299809024] - INFO - RUN QUERY create primary index on dst_bucket.scope_1.coll_0 [2021-02-12 22:03:40,017] - [rest_client] [140099299809024] - INFO - query params : statement=create+primary+index+on+dst_bucket.scope_1.coll_0 [2021-02-12 22:03:42,416] - [tuq_helper] [140099299809024] - INFO - TOTAL ELAPSED TIME: 2.397557076s [2021-02-12 22:03:42,433] - [basetestcase] [140099299809024] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root] [2021-02-12 22:03:42,441] - [rest_client] [140099299809024] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_0 [2021-02-12 22:03:42,453] - [eventing_base] [140099299809024] - INFO - Final docs count... Current : 40320 Expected : 40320 [2021-02-12 22:03:42,462] - [basetestcase] [140099299809024] - INFO - list of eventing nodes in cluster: [ip:172.23.122.117 port:8091 ssh_username:root, ip:172.23.122.90 port:8091 ssh_username:root, ip:172.23.122.97 port:8091 ssh_username:root] [2021-02-12 22:03:42,466] - [tuq_helper] [140099299809024] - INFO - RUN QUERY create primary index on dst_bucket.scope_1.coll_1 [2021-02-12 22:03:42,470] - [rest_client] [140099299809024] - INFO - query params : statement=create+primary+index+on+dst_bucket.scope_1.coll_1 [2021-02-12 22:03:45,507] - [tuq_helper] [140099299809024] - INFO - TOTAL ELAPSED TIME: 3.035272531s [2021-02-12 22:03:45,527] - [basetestcase] [140099299809024] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root] [2021-02-12 22:03:45,533] - [rest_client] [140099299809024] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_1 [2021-02-12 22:03:45,546] - [eventing_base] [140099299809024] - INFO - Final docs count... Current : 40320 Expected : 40320 [2021-02-12 22:03:45,554] - [basetestcase] [140099299809024] - INFO - list of eventing nodes in cluster: [ip:172.23.122.117 port:8091 ssh_username:root, ip:172.23.122.90 port:8091 ssh_username:root, ip:172.23.122.97 port:8091 ssh_username:root] [2021-02-12 22:03:45,560] - [tuq_helper] [140099299809024] - INFO - RUN QUERY create primary index on dst_bucket.scope_1.coll_3 [2021-02-12 22:03:45,565] - [rest_client] [140099299809024] - INFO - query params : statement=create+primary+index+on+dst_bucket.scope_1.coll_3 [2021-02-12 22:03:48,343] - [tuq_helper] [140099299809024] - INFO - TOTAL ELAPSED TIME: 2.776080347s [2021-02-12 22:03:48,361] - [basetestcase] [140099299809024] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root] [2021-02-12 22:03:48,366] - [rest_client] [140099299809024] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_3 [2021-02-12 22:03:48,375] - [eventing_base] [140099299809024] - INFO - Final docs count... Current : 40320 Expected : 40320 [2021-02-12 22:03:48,383] - [basetestcase] [140099299809024] - INFO - list of eventing nodes in cluster: [ip:172.23.122.117 port:8091 ssh_username:root, ip:172.23.122.90 port:8091 ssh_username:root, ip:172.23.122.97 port:8091 ssh_username:root] [2021-02-12 22:03:48,388] - [tuq_helper] [140099299809024] - INFO - RUN QUERY create primary index on dst_bucket.scope_1.coll_4 [2021-02-12 22:03:48,393] - [rest_client] [140099299809024] - INFO - query params : statement=create+primary+index+on+dst_bucket.scope_1.coll_4 [2021-02-12 22:03:51,171] - [tuq_helper] [140099299809024] - INFO - TOTAL ELAPSED TIME: 2.776647413s [2021-02-12 22:03:51,196] - [basetestcase] [140099299809024] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root] [2021-02-12 22:03:51,202] - [rest_client] [140099299809024] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-12 22:03:51,210] - [basetestcase] [140099299809024] - INFO - sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_4 to complete bucket operations... Current : 40319 Expected : 40320 ... [2021-02-12 22:04:21,233] - [basetestcase] [140099299809024] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root] [2021-02-12 22:04:21,239] - [rest_client] [140099299809024] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-12 22:04:21,255] - [basetestcase] [140099299809024] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root] [2021-02-12 22:04:21,260] - [rest_client] [140099299809024] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-12 22:04:21,263] - [basetestcase] [140099299809024] - INFO - sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_4 to complete bucket operations... Current : 40319 Expected : 40320 ... [2021-02-12 22:04:51,297] - [basetestcase] [140099299809024] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root] [2021-02-12 22:04:51,304] - [rest_client] [140099299809024] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-12 22:04:51,320] - [basetestcase] [140099299809024] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root] [2021-02-12 22:04:51,325] - [rest_client] [140099299809024] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-12 22:04:51,328] - [basetestcase] [140099299809024] - INFO - sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_4 to complete bucket operations... Current : 40319 Expected : 40320 ... [2021-02-12 22:05:21,360] - [basetestcase] [140099299809024] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root] [2021-02-12 22:05:21,367] - [rest_client] [140099299809024] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-12 22:05:21,384] - [basetestcase] [140099299809024] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root] [2021-02-12 22:05:21,391] - [rest_client] [140099299809024] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-12 22:05:21,394] - [basetestcase] [140099299809024] - INFO - sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_4 to complete bucket operations... Current : 40319 Expected : 40320 ... [2021-02-12 22:05:51,443] - [basetestcase] [140099299809024] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root] [2021-02-12 22:05:51,448] - [rest_client] [140099299809024] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-12 22:05:51,466] - [basetestcase] [140099299809024] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root] [2021-02-12 22:05:51,472] - [rest_client] [140099299809024] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-12 22:05:51,475] - [basetestcase] [140099299809024] - INFO - sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_4 to complete bucket operations... Current : 40319 Expected : 40320 ... [2021-02-12 22:06:21,526] - [basetestcase] [140099299809024] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root] [2021-02-12 22:06:21,532] - [rest_client] [140099299809024] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-12 22:06:21,551] - [basetestcase] [140099299809024] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root] [2021-02-12 22:06:21,557] - [rest_client] [140099299809024] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-12 22:06:21,560] - [basetestcase] [140099299809024] - INFO - sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_4 to complete bucket operations... Current : 40319 Expected : 40320 ... [2021-02-12 22:06:51,609] - [basetestcase] [140099299809024] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root] [2021-02-12 22:06:51,615] - [rest_client] [140099299809024] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-12 22:06:51,631] - [basetestcase] [140099299809024] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root] [2021-02-12 22:06:51,637] - [rest_client] [140099299809024] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-12 22:06:51,640] - [basetestcase] [140099299809024] - INFO - sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_4 to complete bucket operations... Current : 40319 Expected : 40320 ... [2021-02-12 22:07:21,690] - [basetestcase] [140099299809024] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root] [2021-02-12 22:07:21,696] - [rest_client] [140099299809024] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-12 22:07:21,712] - [basetestcase] [140099299809024] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root] [2021-02-12 22:07:21,717] - [rest_client] [140099299809024] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-12 22:07:21,721] - [basetestcase] [140099299809024] - INFO - sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_4 to complete bucket operations... Current : 40319 Expected : 40320 ... [2021-02-12 22:07:51,751] - [basetestcase] [140099299809024] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root] [2021-02-12 22:07:51,757] - [rest_client] [140099299809024] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-12 22:07:51,775] - [basetestcase] [140099299809024] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root] [2021-02-12 22:07:51,779] - [rest_client] [140099299809024] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-12 22:07:51,782] - [basetestcase] [140099299809024] - INFO - sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_4 to complete bucket operations... Current : 40319 Expected : 40320 ... [2021-02-12 22:08:21,832] - [basetestcase] [140099299809024] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root] [2021-02-12 22:08:21,838] - [rest_client] [140099299809024] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-12 22:08:21,854] - [basetestcase] [140099299809024] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root] [2021-02-12 22:08:21,860] - [rest_client] [140099299809024] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-12 22:08:21,863] - [basetestcase] [140099299809024] - INFO - sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_4 to complete bucket operations... Current : 40319 Expected : 40320 ... [2021-02-12 22:08:51,913] - [basetestcase] [140099299809024] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root] [2021-02-12 22:08:51,921] - [rest_client] [140099299809024] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-12 22:08:51,937] - [basetestcase] [140099299809024] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root] [2021-02-12 22:08:51,942] - [rest_client] [140099299809024] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-12 22:08:51,945] - [basetestcase] [140099299809024] - INFO - sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_4 to complete bucket operations... Current : 40319 Expected : 40320 ... [2021-02-12 22:09:21,996] - [basetestcase] [140099299809024] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root] [2021-02-12 22:09:22,002] - [rest_client] [140099299809024] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-12 22:09:22,016] - [eventing_base] [140099299809024] - INFO - Compacting bucket : dst_bucket [2021-02-12 22:09:22,027] - [rest_client] [140099299809024] - INFO - bucket compaction successful [2021-02-12 22:09:22,027] - [eventing_base] [140099299809024] - INFO - Compacting bucket : metadata [2021-02-12 22:09:22,037] - [rest_client] [140099299809024] - INFO - bucket compaction successful [2021-02-12 22:09:22,037] - [eventing_base] [140099299809024] - INFO - Compacting bucket : src_bucket [2021-02-12 22:09:22,044] - [rest_client] [140099299809024] - INFO - bucket compaction successful [2021-02-12 22:09:22,052] - [basetestcase] [140099299809024] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root] [2021-02-12 22:09:22,057] - [rest_client] [140099299809024] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-12 22:09:22,060] - [basetestcase] [140099299809024] - INFO - sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_4 to complete bucket operations... Current : 40319 Expected : 40320 ... [2021-02-12 22:09:52,108] - [basetestcase] [140099299809024] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root] [2021-02-12 22:09:52,115] - [rest_client] [140099299809024] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-12 22:09:52,132] - [basetestcase] [140099299809024] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root] [2021-02-12 22:09:52,138] - [rest_client] [140099299809024] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-12 22:09:52,140] - [basetestcase] [140099299809024] - INFO - sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_4 to complete bucket operations... Current : 40319 Expected : 40320 ... [2021-02-12 22:10:22,190] - [basetestcase] [140099299809024] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root] [2021-02-12 22:10:22,196] - [rest_client] [140099299809024] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-12 22:10:22,213] - [basetestcase] [140099299809024] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root] [2021-02-12 22:10:22,218] - [rest_client] [140099299809024] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-12 22:10:22,221] - [basetestcase] [140099299809024] - INFO - sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_4 to complete bucket operations... Current : 40319 Expected : 40320 ... [2021-02-12 22:10:52,272] - [basetestcase] [140099299809024] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root] [2021-02-12 22:10:52,278] - [rest_client] [140099299809024] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-12 22:10:52,296] - [basetestcase] [140099299809024] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root] [2021-02-12 22:10:52,302] - [rest_client] [140099299809024] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-12 22:10:52,304] - [basetestcase] [140099299809024] - INFO - sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_4 to complete bucket operations... Current : 40319 Expected : 40320 ... [2021-02-12 22:11:22,355] - [basetestcase] [140099299809024] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root] [2021-02-12 22:11:22,361] - [rest_client] [140099299809024] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-12 22:11:22,380] - [basetestcase] [140099299809024] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root] [2021-02-12 22:11:22,384] - [rest_client] [140099299809024] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-12 22:11:22,388] - [basetestcase] [140099299809024] - INFO - sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_4 to complete bucket operations... Current : 40319 Expected : 40320 ... [2021-02-12 22:11:52,411] - [basetestcase] [140099299809024] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root] [2021-02-12 22:11:52,417] - [rest_client] [140099299809024] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-12 22:11:52,436] - [basetestcase] [140099299809024] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root] [2021-02-12 22:11:52,440] - [rest_client] [140099299809024] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-12 22:11:52,444] - [basetestcase] [140099299809024] - INFO - sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_4 to complete bucket operations... Current : 40319 Expected : 40320 ... [2021-02-12 22:12:22,483] - [basetestcase] [140099299809024] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root] [2021-02-12 22:12:22,489] - [rest_client] [140099299809024] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-12 22:12:22,505] - [basetestcase] [140099299809024] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root] [2021-02-12 22:12:22,510] - [rest_client] [140099299809024] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-12 22:12:22,513] - [basetestcase] [140099299809024] - INFO - sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_4 to complete bucket operations... Current : 40319 Expected : 40320 ... [2021-02-12 22:12:52,562] - [basetestcase] [140099299809024] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root] [2021-02-12 22:12:52,568] - [rest_client] [140099299809024] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-12 22:12:52,584] - [basetestcase] [140099299809024] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root] [2021-02-12 22:12:52,590] - [rest_client] [140099299809024] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-12 22:12:52,593] - [basetestcase] [140099299809024] - INFO - sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_4 to complete bucket operations... Current : 40319 Expected : 40320 ... [2021-02-12 22:13:22,643] - [basetestcase] [140099299809024] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root] [2021-02-12 22:13:22,649] - [rest_client] [140099299809024] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-12 22:13:22,665] - [basetestcase] [140099299809024] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root] [2021-02-12 22:13:22,670] - [rest_client] [140099299809024] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-12 22:13:22,673] - [basetestcase] [140099299809024] - INFO - sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_4 to complete bucket operations... Current : 40319 Expected : 40320 ... [2021-02-12 22:13:52,702] - [basetestcase] [140099299809024] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root] [2021-02-12 22:13:52,708] - [rest_client] [140099299809024] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-12 22:13:52,726] - [basetestcase] [140099299809024] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root] [2021-02-12 22:13:52,731] - [rest_client] [140099299809024] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-12 22:13:52,742] - [basetestcase] [140099299809024] - INFO - list of eventing nodes in cluster: [ip:172.23.122.117 port:8091 ssh_username:root, ip:172.23.122.90 port:8091 ssh_username:root, ip:172.23.122.97 port:8091 ssh_username:root] [2021-02-12 22:13:52,784] - [eventing_base] [140099299809024] - INFO - Stats for Node 172.23.122.117 is [ { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 1, "agg_messages_sent_to_worker": 17974, "agg_queue_memory": 0, "agg_queue_memory_cap": 20971520, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 40513, "dcp_mutation_sent_to_worker": 13499, "dcp_snapshot": 40337, "dcp_stream_req_counter": 342, "dcp_streamreq": 342, "dcp_system_event": 3420, "execution_stats": 825, "failure_stats": 825, "latency_stats": 825, "lcb_exception_stats": 825, "log_level": 1, "num_processed_events": 13499, "processed_events_size": 12082876, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13499, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13499, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 17970, "num_processed_events": 13499, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13499, "processed_events_size": 12082876, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "40365": "2021-02-13T06:13:52Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "40365": "2021-02-13T06:13:52Z" }, "v8worker_events_lost": 0 }, "function_id": 1986265310, "function_name": "n1ql", "gocb_creds_request_counter": 81, "internal_vb_distribution_stats": { "worker_n1ql_0": "[0-341]" }, "latency_percentile_stats": { "100": 57200, "50": 3300, "80": 4600, "90": 5700, "95": 7100, "99": 11400 }, "lcb_creds_request_counter": 27062, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.90:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.97:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_n1ql_0": "[0-341]" }, "172.23.122.90:8096": { "worker_n1ql_0": "[342-682]" }, "172.23.122.97:8096": { "worker_n1ql_0": "[683-1023]" } }, "worker_pids": { "worker_n1ql_0": 40365 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 2, "agg_messages_sent_to_worker": 18403, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 40513, "dcp_mutation_sent_to_worker": 13499, "dcp_snapshot": 40388, "dcp_stream_req_counter": 342, "dcp_streamreq": 342, "dcp_system_event": 3420, "execution_stats": 910, "failure_stats": 910, "latency_stats": 910, "lcb_exception_stats": 910, "log_level": 1, "num_processed_events": 13499, "processed_events_size": 12082876, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13499, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13499, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 18399, "num_processed_events": 13499, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13499, "processed_events_size": 12082876, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "40126": "2021-02-13T06:13:51Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "40126": "2021-02-13T06:13:51Z" }, "v8worker_events_lost": 0 }, "function_id": 2434364620, "function_name": "bucket_op", "gocb_creds_request_counter": 81, "internal_vb_distribution_stats": { "worker_bucket_op_0": "[0-341]" }, "latency_percentile_stats": { "100": 13900, "50": 700, "80": 1000, "90": 1300, "95": 1700, "99": 3000 }, "lcb_creds_request_counter": 27062, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.90:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.97:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_bucket_op_0": "[0-341]" }, "172.23.122.90:8096": { "worker_bucket_op_0": "[342-682]" }, "172.23.122.97:8096": { "worker_bucket_op_0": "[683-1023]" } }, "worker_pids": { "worker_bucket_op_0": 40126 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 2, "agg_messages_sent_to_worker": 18297, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 40513, "dcp_mutation_sent_to_worker": 13499, "dcp_snapshot": 40374, "dcp_stream_req_counter": 342, "dcp_streamreq": 342, "dcp_system_event": 3420, "execution_stats": 889, "failure_stats": 889, "latency_stats": 889, "lcb_exception_stats": 889, "log_level": 1, "num_processed_events": 13499, "processed_events_size": 12082876, "thr_count": 1, "thr_map": 1, "timer_events": 13528, "timer_responses_received": 13499, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13499, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13499, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 18293, "num_processed_events": 13499, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13499, "processed_events_size": 12082876, "timer_callback_failure": 0, "timer_callback_success": 13528, "timer_cancel_counter": 0, "timer_create_counter": 13499, "timer_create_failure": 0, "timer_msg_counter": 13528, "timer_responses_sent": 0, "timestamp": { "40196": "2021-02-13T06:13:52Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "40196": "2021-02-13T06:13:52Z" }, "v8worker_events_lost": 0 }, "function_id": 972602119, "function_name": "timers", "gocb_creds_request_counter": 81, "internal_vb_distribution_stats": { "worker_timers_0": "[0-341]" }, "latency_percentile_stats": { "100": 18800, "50": 2000, "80": 2800, "90": 3400, "95": 4200, "99": 7000 }, "lcb_creds_request_counter": 27062, "lcb_exception_stats": {}, "metastore_stats": { "metastore_delete_err": 0, "metastore_deletes": 0, "metastore_not_found": 0, "metastore_scan": 0, "metastore_scan_due": 0, "metastore_scan_err": 0, "metastore_set": 0, "metastore_set_err": 0 }, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.90:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.97:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_timers_0": "[0-341]" }, "172.23.122.90:8096": { "worker_timers_0": "[342-682]" }, "172.23.122.97:8096": { "worker_timers_0": "[683-1023]" } }, "worker_pids": { "worker_timers_0": 40196 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 1, "agg_messages_sent_to_worker": 18191, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 40513, "dcp_mutation_sent_to_worker": 13499, "dcp_mutation_suppressed_counter": 13515, "dcp_snapshot": 40350, "dcp_stream_req_counter": 342, "dcp_streamreq": 342, "dcp_system_event": 3420, "execution_stats": 868, "failure_stats": 868, "latency_stats": 868, "lcb_exception_stats": 868, "log_level": 1, "num_processed_events": 13499, "processed_events_size": 12083724, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13499, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13499, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 18187, "num_processed_events": 13499, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13499, "processed_events_size": 12083724, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "40238": "2021-02-13T06:13:52Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "40238": "2021-02-13T06:13:52Z" }, "v8worker_events_lost": 0 }, "function_id": 4207347309, "function_name": "sbm", "gocb_creds_request_counter": 81, "internal_vb_distribution_stats": { "worker_sbm_0": "[0-341]" }, "latency_percentile_stats": { "100": 17100, "50": 800, "80": 1300, "90": 1700, "95": 2300, "99": 4000 }, "lcb_creds_request_counter": 27062, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.90:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.97:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_sbm_0": "[0-341]" }, "172.23.122.90:8096": { "worker_sbm_0": "[342-682]" }, "172.23.122.97:8096": { "worker_sbm_0": "[683-1023]" } }, "worker_pids": { "worker_sbm_0": 40238 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 2, "agg_messages_sent_to_worker": 18080, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 40513, "dcp_mutation_sent_to_worker": 13499, "dcp_snapshot": 40372, "dcp_stream_req_counter": 342, "dcp_streamreq": 342, "dcp_system_event": 3420, "execution_stats": 846, "failure_stats": 846, "latency_stats": 846, "lcb_exception_stats": 846, "log_level": 1, "num_processed_events": 13499, "processed_events_size": 12082876, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 13499, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 13499, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13499, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13499, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 18076, "num_processed_events": 13499, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13499, "processed_events_size": 12082876, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "40279": "2021-02-13T06:13:52Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "40279": "2021-02-13T06:13:52Z" }, "v8worker_events_lost": 0 }, "function_id": 1583657262, "function_name": "curl", "gocb_creds_request_counter": 81, "internal_vb_distribution_stats": { "worker_curl_0": "[0-341]" }, "latency_percentile_stats": { "100": 36500, "50": 1600, "80": 2100, "90": 2500, "95": 3000, "99": 4700 }, "lcb_creds_request_counter": 27062, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.90:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.97:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_curl_0": "[0-341]" }, "172.23.122.90:8096": { "worker_curl_0": "[342-682]" }, "172.23.122.97:8096": { "worker_curl_0": "[683-1023]" } }, "worker_pids": { "worker_curl_0": 40279 } } ] [2021-02-12 22:13:52,835] - [eventing_base] [140099299809024] - INFO - Stats for Node 172.23.122.90 is [ { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 3, "agg_messages_sent_to_worker": 18588, "agg_queue_memory": 0, "agg_queue_memory_cap": 20971520, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 67238, "dcp_mutation_sent_to_worker": 13432, "dcp_snapshot": 40560, "dcp_stream_req_counter": 11927, "dcp_stream_req_err_counter": 11108, "dcp_streamend": 341, "dcp_streamreq": 819, "dcp_system_event": 3410, "execution_stats": 825, "failure_stats": 825, "latency_stats": 825, "lcb_exception_stats": 825, "log_level": 1, "num_processed_events": 13432, "processed_events_size": 12029360, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13432, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13432, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 18584, "num_processed_events": 13432, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13432, "processed_events_size": 12029360, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "79926": "2021-02-13T06:13:52Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "79926": "2021-02-13T06:13:52Z" }, "v8worker_events_lost": 0 }, "function_id": 1986265310, "function_name": "n1ql", "gocb_creds_request_counter": 75, "internal_vb_distribution_stats": { "worker_n1ql_0": "[342-682]" }, "latency_percentile_stats": { "100": 106700, "50": 3400, "80": 4700, "90": 5900, "95": 7400, "99": 11900 }, "lcb_creds_request_counter": 26930, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.90:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.97:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_n1ql_0": "[0-341]" }, "172.23.122.90:8096": { "worker_n1ql_0": "[342-682]" }, "172.23.122.97:8096": { "worker_n1ql_0": "[683-1023]" } }, "worker_pids": { "worker_n1ql_0": 79926 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 3, "agg_messages_sent_to_worker": 19017, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 67238, "dcp_mutation_sent_to_worker": 13432, "dcp_snapshot": 40611, "dcp_stream_req_counter": 7383, "dcp_stream_req_err_counter": 6564, "dcp_streamend": 341, "dcp_streamreq": 819, "dcp_system_event": 3410, "execution_stats": 910, "failure_stats": 910, "latency_stats": 910, "lcb_exception_stats": 910, "log_level": 1, "num_processed_events": 13432, "processed_events_size": 12029360, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13432, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13432, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 19013, "num_processed_events": 13432, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13432, "processed_events_size": 12029360, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "79723": "2021-02-13T06:13:51Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "79723": "2021-02-13T06:13:51Z" }, "v8worker_events_lost": 0 }, "function_id": 2434364620, "function_name": "bucket_op", "gocb_creds_request_counter": 75, "internal_vb_distribution_stats": { "worker_bucket_op_0": "[342-682]" }, "latency_percentile_stats": { "100": 16100, "50": 700, "80": 1100, "90": 1400, "95": 1800, "99": 3000 }, "lcb_creds_request_counter": 26930, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.90:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.97:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_bucket_op_0": "[0-341]" }, "172.23.122.90:8096": { "worker_bucket_op_0": "[342-682]" }, "172.23.122.97:8096": { "worker_bucket_op_0": "[683-1023]" } }, "worker_pids": { "worker_bucket_op_0": 79723 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 2, "agg_messages_sent_to_worker": 18911, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 67238, "dcp_mutation_sent_to_worker": 13432, "dcp_snapshot": 40563, "dcp_stream_req_counter": 7034, "dcp_stream_req_err_counter": 6211, "dcp_streamend": 341, "dcp_streamreq": 823, "dcp_system_event": 3410, "execution_stats": 889, "failure_stats": 889, "latency_stats": 889, "lcb_exception_stats": 889, "log_level": 1, "num_processed_events": 13432, "processed_events_size": 12029360, "thr_count": 1, "thr_map": 1, "timer_events": 13558, "timer_responses_received": 13432, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13432, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13432, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 18907, "num_processed_events": 13432, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13432, "processed_events_size": 12029360, "timer_callback_failure": 0, "timer_callback_success": 13558, "timer_cancel_counter": 0, "timer_create_counter": 13432, "timer_create_failure": 0, "timer_msg_counter": 13558, "timer_responses_sent": 0, "timestamp": { "79800": "2021-02-13T06:13:52Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "79800": "2021-02-13T06:13:52Z" }, "v8worker_events_lost": 0 }, "function_id": 972602119, "function_name": "timers", "gocb_creds_request_counter": 75, "internal_vb_distribution_stats": { "worker_timers_0": "[342-682]" }, "latency_percentile_stats": { "100": 17700, "50": 1900, "80": 2800, "90": 3400, "95": 4300, "99": 6800 }, "lcb_creds_request_counter": 26930, "lcb_exception_stats": {}, "metastore_stats": { "metastore_delete_err": 0, "metastore_deletes": 0, "metastore_not_found": 0, "metastore_scan": 0, "metastore_scan_due": 0, "metastore_scan_err": 0, "metastore_set": 0, "metastore_set_err": 0 }, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.90:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.97:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_timers_0": "[0-341]" }, "172.23.122.90:8096": { "worker_timers_0": "[342-682]" }, "172.23.122.97:8096": { "worker_timers_0": "[683-1023]" } }, "worker_pids": { "worker_timers_0": 79800 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 2, "agg_messages_sent_to_worker": 18805, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 41052, "dcp_mutation_sent_to_worker": 13432, "dcp_mutation_suppressed_counter": 14188, "dcp_snapshot": 40526, "dcp_stream_req_counter": 1642, "dcp_stream_req_err_counter": 796, "dcp_streamend": 341, "dcp_streamreq": 846, "dcp_system_event": 3410, "execution_stats": 868, "failure_stats": 868, "latency_stats": 868, "lcb_exception_stats": 868, "log_level": 1, "num_processed_events": 13432, "processed_events_size": 12024580, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13432, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13432, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 18801, "num_processed_events": 13432, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13432, "processed_events_size": 12024580, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "79832": "2021-02-13T06:13:52Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "79832": "2021-02-13T06:13:52Z" }, "v8worker_events_lost": 0 }, "function_id": 4207347309, "function_name": "sbm", "gocb_creds_request_counter": 75, "internal_vb_distribution_stats": { "worker_sbm_0": "[342-682]" }, "latency_percentile_stats": { "100": 15300, "50": 800, "80": 1400, "90": 1800, "95": 2500, "99": 4300 }, "lcb_creds_request_counter": 26930, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.90:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.97:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_sbm_0": "[0-341]" }, "172.23.122.90:8096": { "worker_sbm_0": "[342-682]" }, "172.23.122.97:8096": { "worker_sbm_0": "[683-1023]" } }, "worker_pids": { "worker_sbm_0": 79832 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 2, "agg_messages_sent_to_worker": 18694, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 67238, "dcp_mutation_sent_to_worker": 13432, "dcp_snapshot": 40584, "dcp_stream_req_counter": 12369, "dcp_stream_req_err_counter": 11550, "dcp_streamend": 341, "dcp_streamreq": 819, "dcp_system_event": 3410, "execution_stats": 846, "failure_stats": 846, "latency_stats": 846, "lcb_exception_stats": 846, "log_level": 1, "num_processed_events": 13432, "processed_events_size": 12029360, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 13432, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 13432, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13432, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13432, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 18690, "num_processed_events": 13432, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13432, "processed_events_size": 12029360, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "79864": "2021-02-13T06:13:52Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "79864": "2021-02-13T06:13:52Z" }, "v8worker_events_lost": 0 }, "function_id": 1583657262, "function_name": "curl", "gocb_creds_request_counter": 75, "internal_vb_distribution_stats": { "worker_curl_0": "[342-682]" }, "latency_percentile_stats": { "100": 40700, "50": 1700, "80": 2200, "90": 2700, "95": 3300, "99": 5100 }, "lcb_creds_request_counter": 26930, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.90:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.97:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_curl_0": "[0-341]" }, "172.23.122.90:8096": { "worker_curl_0": "[342-682]" }, "172.23.122.97:8096": { "worker_curl_0": "[683-1023]" } }, "worker_pids": { "worker_curl_0": 79864 } } ] [2021-02-12 22:13:52,879] - [eventing_base] [140099299809024] - INFO - Stats for Node 172.23.122.97 is [ { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 2, "agg_messages_sent_to_worker": 18868, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 66835, "dcp_mutation_sent_to_worker": 13389, "dcp_snapshot": 40301, "dcp_stream_req_counter": 1220, "dcp_stream_req_err_counter": 208, "dcp_streamend": 341, "dcp_streamreq": 1012, "dcp_system_event": 3410, "execution_stats": 889, "failure_stats": 889, "latency_stats": 889, "lcb_exception_stats": 889, "log_level": 1, "num_processed_events": 13389, "processed_events_size": 11989232, "thr_count": 1, "thr_map": 1, "timer_events": 13234, "timer_responses_received": 13389, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13389, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13389, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 18864, "num_processed_events": 13389, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13389, "processed_events_size": 11989232, "timer_callback_failure": 0, "timer_callback_success": 13234, "timer_cancel_counter": 0, "timer_create_counter": 13389, "timer_create_failure": 0, "timer_msg_counter": 13234, "timer_responses_sent": 0, "timestamp": { "73476": "2021-02-13T06:13:52Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "73476": "2021-02-13T06:13:52Z" }, "v8worker_events_lost": 0 }, "function_id": 972602119, "function_name": "timers", "gocb_creds_request_counter": 77, "internal_vb_distribution_stats": { "worker_timers_0": "[683-1023]" }, "latency_percentile_stats": { "100": 23900, "50": 2000, "80": 2800, "90": 3500, "95": 4200, "99": 7000 }, "lcb_creds_request_counter": 26850, "lcb_exception_stats": {}, "metastore_stats": { "metastore_delete_err": 0, "metastore_deletes": 0, "metastore_not_found": 0, "metastore_scan": 0, "metastore_scan_due": 0, "metastore_scan_err": 0, "metastore_set": 0, "metastore_set_err": 0 }, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.90:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.97:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_timers_0": "[0-341]" }, "172.23.122.90:8096": { "worker_timers_0": "[342-682]" }, "172.23.122.97:8096": { "worker_timers_0": "[683-1023]" } }, "worker_pids": { "worker_timers_0": 73476 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 1, "agg_messages_sent_to_worker": 18773, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 40836, "dcp_mutation_sent_to_worker": 13389, "dcp_mutation_suppressed_counter": 14058, "dcp_snapshot": 40249, "dcp_stream_req_counter": 13149, "dcp_stream_req_err_counter": 12196, "dcp_streamend": 352, "dcp_streamreq": 953, "dcp_system_event": 3410, "execution_stats": 868, "failure_stats": 868, "latency_stats": 868, "lcb_exception_stats": 868, "log_level": 1, "num_processed_events": 13389, "processed_events_size": 11990912, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13389, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13389, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 18769, "num_processed_events": 13389, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13389, "processed_events_size": 11990912, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "73553": "2021-02-13T06:13:52Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "73553": "2021-02-13T06:13:52Z" }, "v8worker_events_lost": 0 }, "function_id": 4207347309, "function_name": "sbm", "gocb_creds_request_counter": 77, "internal_vb_distribution_stats": { "worker_sbm_0": "[683-1023]" }, "latency_percentile_stats": { "100": 13300, "50": 900, "80": 1400, "90": 1900, "95": 2500, "99": 4400 }, "lcb_creds_request_counter": 26850, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.90:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.97:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_sbm_0": "[0-341]" }, "172.23.122.90:8096": { "worker_sbm_0": "[342-682]" }, "172.23.122.97:8096": { "worker_sbm_0": "[683-1023]" } }, "worker_pids": { "worker_sbm_0": 73553 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 2, "agg_messages_sent_to_worker": 18662, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 66835, "dcp_mutation_sent_to_worker": 13389, "dcp_snapshot": 40403, "dcp_stream_req_counter": 12994, "dcp_stream_req_err_counter": 12095, "dcp_streamend": 352, "dcp_streamreq": 899, "dcp_system_event": 3410, "execution_stats": 846, "failure_stats": 846, "latency_stats": 846, "lcb_exception_stats": 846, "log_level": 1, "num_processed_events": 13389, "processed_events_size": 11989232, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 13389, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 13389, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13389, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13389, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 18658, "num_processed_events": 13389, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13389, "processed_events_size": 11989232, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "73585": "2021-02-13T06:13:52Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "73585": "2021-02-13T06:13:52Z" }, "v8worker_events_lost": 0 }, "function_id": 1583657262, "function_name": "curl", "gocb_creds_request_counter": 77, "internal_vb_distribution_stats": { "worker_curl_0": "[683-1023]" }, "latency_percentile_stats": { "100": 43900, "50": 1700, "80": 2200, "90": 2600, "95": 3000, "99": 4800 }, "lcb_creds_request_counter": 26850, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.90:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.97:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_curl_0": "[0-341]" }, "172.23.122.90:8096": { "worker_curl_0": "[342-682]" }, "172.23.122.97:8096": { "worker_curl_0": "[683-1023]" } }, "worker_pids": { "worker_curl_0": 73585 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 3, "agg_messages_sent_to_worker": 18556, "agg_queue_memory": 0, "agg_queue_memory_cap": 20971520, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 66835, "dcp_mutation_sent_to_worker": 13389, "dcp_snapshot": 40269, "dcp_stream_req_counter": 12432, "dcp_stream_req_err_counter": 11504, "dcp_streamend": 352, "dcp_streamreq": 928, "dcp_system_event": 3410, "execution_stats": 825, "failure_stats": 825, "latency_stats": 825, "lcb_exception_stats": 825, "log_level": 1, "num_processed_events": 13389, "processed_events_size": 11989232, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13389, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13389, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 18552, "num_processed_events": 13389, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 1, "on_update_success": 13388, "processed_events_size": 11989232, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "73647": "2021-02-13T06:13:52Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 1, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "73647": "2021-02-13T06:13:52Z" }, "v8worker_events_lost": 0 }, "function_id": 1986265310, "function_name": "n1ql", "gocb_creds_request_counter": 77, "internal_vb_distribution_stats": { "worker_n1ql_0": "[683-1023]" }, "latency_percentile_stats": { "100": 198100, "50": 3500, "80": 4800, "90": 6000, "95": 7500, "99": 12900 }, "lcb_creds_request_counter": 26850, "lcb_exception_stats": { "12009": 1, "209": 1 }, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.90:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.97:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_n1ql_0": "[0-341]" }, "172.23.122.90:8096": { "worker_n1ql_0": "[342-682]" }, "172.23.122.97:8096": { "worker_n1ql_0": "[683-1023]" } }, "worker_pids": { "worker_n1ql_0": 73647 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 1, "agg_messages_sent_to_worker": 18990, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 66835, "dcp_mutation_sent_to_worker": 13389, "dcp_snapshot": 40314, "dcp_stream_req_counter": 5346, "dcp_stream_req_err_counter": 4343, "dcp_streamend": 352, "dcp_streamreq": 1003, "dcp_system_event": 3410, "execution_stats": 911, "failure_stats": 911, "latency_stats": 911, "lcb_exception_stats": 911, "log_level": 1, "num_processed_events": 13389, "processed_events_size": 11989232, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13389, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13389, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 18986, "num_processed_events": 13389, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13389, "processed_events_size": 11989232, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "73445": "2021-02-13T06:13:52Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "73445": "2021-02-13T06:13:52Z" }, "v8worker_events_lost": 0 }, "function_id": 2434364620, "function_name": "bucket_op", "gocb_creds_request_counter": 77, "internal_vb_distribution_stats": { "worker_bucket_op_0": "[683-1023]" }, "latency_percentile_stats": { "100": 23900, "50": 700, "80": 1100, "90": 1500, "95": 1800, "99": 3100 }, "lcb_creds_request_counter": 26850, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.90:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.97:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_bucket_op_0": "[0-341]" }, "172.23.122.90:8096": { "worker_bucket_op_0": "[342-682]" }, "172.23.122.97:8096": { "worker_bucket_op_0": "[683-1023]" } }, "worker_pids": { "worker_bucket_op_0": 73445 } } ] [2021-02-12 22:13:57,200] - [eventing_base] [140099299809024] - INFO - Stats for Node 172.23.122.117 is [ { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 2, "agg_messages_sent_to_worker": 18297, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 40513, "dcp_mutation_sent_to_worker": 13499, "dcp_snapshot": 40374, "dcp_stream_req_counter": 342, "dcp_streamreq": 342, "dcp_system_event": 3420, "execution_stats": 889, "failure_stats": 889, "latency_stats": 889, "lcb_exception_stats": 889, "log_level": 1, "num_processed_events": 13499, "processed_events_size": 12082876, "thr_count": 1, "thr_map": 1, "timer_events": 13528, "timer_responses_received": 13499, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13499, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13499, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 18293, "num_processed_events": 13499, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13499, "processed_events_size": 12082876, "timer_callback_failure": 0, "timer_callback_success": 13528, "timer_cancel_counter": 0, "timer_create_counter": 13499, "timer_create_failure": 0, "timer_msg_counter": 13528, "timer_responses_sent": 0, "timestamp": { "40196": "2021-02-13T06:13:52Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "40196": "2021-02-13T06:13:52Z" }, "v8worker_events_lost": 0 }, "function_id": 972602119, "function_name": "timers", "gocb_creds_request_counter": 81, "internal_vb_distribution_stats": { "worker_timers_0": "[0-341]" }, "latency_percentile_stats": { "100": 18800, "50": 2000, "80": 2800, "90": 3400, "95": 4200, "99": 7000 }, "lcb_creds_request_counter": 27062, "lcb_exception_stats": {}, "metastore_stats": { "metastore_delete_err": 0, "metastore_deletes": 0, "metastore_not_found": 0, "metastore_scan": 0, "metastore_scan_due": 0, "metastore_scan_err": 0, "metastore_set": 0, "metastore_set_err": 0 }, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.90:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.97:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_timers_0": "[0-341]" }, "172.23.122.90:8096": { "worker_timers_0": "[342-682]" }, "172.23.122.97:8096": { "worker_timers_0": "[683-1023]" } }, "worker_pids": { "worker_timers_0": 40196 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 1, "agg_messages_sent_to_worker": 18191, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 40513, "dcp_mutation_sent_to_worker": 13499, "dcp_mutation_suppressed_counter": 13515, "dcp_snapshot": 40350, "dcp_stream_req_counter": 342, "dcp_streamreq": 342, "dcp_system_event": 3420, "execution_stats": 868, "failure_stats": 868, "latency_stats": 868, "lcb_exception_stats": 868, "log_level": 1, "num_processed_events": 13499, "processed_events_size": 12083724, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13499, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13499, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 18187, "num_processed_events": 13499, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13499, "processed_events_size": 12083724, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "40238": "2021-02-13T06:13:52Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "40238": "2021-02-13T06:13:52Z" }, "v8worker_events_lost": 0 }, "function_id": 4207347309, "function_name": "sbm", "gocb_creds_request_counter": 81, "internal_vb_distribution_stats": { "worker_sbm_0": "[0-341]" }, "latency_percentile_stats": { "100": 17100, "50": 800, "80": 1300, "90": 1700, "95": 2300, "99": 4000 }, "lcb_creds_request_counter": 27062, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.90:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.97:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_sbm_0": "[0-341]" }, "172.23.122.90:8096": { "worker_sbm_0": "[342-682]" }, "172.23.122.97:8096": { "worker_sbm_0": "[683-1023]" } }, "worker_pids": { "worker_sbm_0": 40238 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 2, "agg_messages_sent_to_worker": 18080, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 40513, "dcp_mutation_sent_to_worker": 13499, "dcp_snapshot": 40372, "dcp_stream_req_counter": 342, "dcp_streamreq": 342, "dcp_system_event": 3420, "execution_stats": 846, "failure_stats": 846, "latency_stats": 846, "lcb_exception_stats": 846, "log_level": 1, "num_processed_events": 13499, "processed_events_size": 12082876, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 13499, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 13499, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13499, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13499, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 18076, "num_processed_events": 13499, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13499, "processed_events_size": 12082876, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "40279": "2021-02-13T06:13:52Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "40279": "2021-02-13T06:13:52Z" }, "v8worker_events_lost": 0 }, "function_id": 1583657262, "function_name": "curl", "gocb_creds_request_counter": 81, "internal_vb_distribution_stats": { "worker_curl_0": "[0-341]" }, "latency_percentile_stats": { "100": 36500, "50": 1600, "80": 2100, "90": 2500, "95": 3000, "99": 4700 }, "lcb_creds_request_counter": 27062, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.90:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.97:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_curl_0": "[0-341]" }, "172.23.122.90:8096": { "worker_curl_0": "[342-682]" }, "172.23.122.97:8096": { "worker_curl_0": "[683-1023]" } }, "worker_pids": { "worker_curl_0": 40279 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 1, "agg_messages_sent_to_worker": 17974, "agg_queue_memory": 0, "agg_queue_memory_cap": 20971520, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 40513, "dcp_mutation_sent_to_worker": 13499, "dcp_snapshot": 40337, "dcp_stream_req_counter": 342, "dcp_streamreq": 342, "dcp_system_event": 3420, "execution_stats": 825, "failure_stats": 825, "latency_stats": 825, "lcb_exception_stats": 825, "log_level": 1, "num_processed_events": 13499, "processed_events_size": 12082876, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13499, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13499, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 17970, "num_processed_events": 13499, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13499, "processed_events_size": 12082876, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "40365": "2021-02-13T06:13:52Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "40365": "2021-02-13T06:13:52Z" }, "v8worker_events_lost": 0 }, "function_id": 1986265310, "function_name": "n1ql", "gocb_creds_request_counter": 81, "internal_vb_distribution_stats": { "worker_n1ql_0": "[0-341]" }, "latency_percentile_stats": { "100": 57200, "50": 3300, "80": 4600, "90": 5700, "95": 7100, "99": 11400 }, "lcb_creds_request_counter": 27062, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.90:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.97:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_n1ql_0": "[0-341]" }, "172.23.122.90:8096": { "worker_n1ql_0": "[342-682]" }, "172.23.122.97:8096": { "worker_n1ql_0": "[683-1023]" } }, "worker_pids": { "worker_n1ql_0": 40365 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 2, "agg_messages_sent_to_worker": 18408, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 40513, "dcp_mutation_sent_to_worker": 13499, "dcp_snapshot": 40388, "dcp_stream_req_counter": 342, "dcp_streamreq": 342, "dcp_system_event": 3420, "execution_stats": 911, "failure_stats": 911, "latency_stats": 911, "lcb_exception_stats": 911, "log_level": 1, "num_processed_events": 13499, "processed_events_size": 12082876, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13499, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13499, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 18404, "num_processed_events": 13499, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13499, "processed_events_size": 12082876, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "40126": "2021-02-13T06:13:52Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "40126": "2021-02-13T06:13:52Z" }, "v8worker_events_lost": 0 }, "function_id": 2434364620, "function_name": "bucket_op", "gocb_creds_request_counter": 81, "internal_vb_distribution_stats": { "worker_bucket_op_0": "[0-341]" }, "latency_percentile_stats": { "100": 13900, "50": 700, "80": 1000, "90": 1300, "95": 1700, "99": 3000 }, "lcb_creds_request_counter": 27062, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.90:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.97:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_bucket_op_0": "[0-341]" }, "172.23.122.90:8096": { "worker_bucket_op_0": "[342-682]" }, "172.23.122.97:8096": { "worker_bucket_op_0": "[683-1023]" } }, "worker_pids": { "worker_bucket_op_0": 40126 } } ] [2021-02-12 22:14:02,834] - [eventing_base] [140099299809024] - INFO - Stats for Node 172.23.122.90 is [ { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 3, "agg_messages_sent_to_worker": 19047, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 67238, "dcp_mutation_sent_to_worker": 13432, "dcp_snapshot": 40611, "dcp_stream_req_counter": 7383, "dcp_stream_req_err_counter": 6564, "dcp_streamend": 341, "dcp_streamreq": 819, "dcp_system_event": 3410, "execution_stats": 916, "failure_stats": 916, "latency_stats": 916, "lcb_exception_stats": 916, "log_level": 1, "num_processed_events": 13432, "processed_events_size": 12029360, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13432, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13432, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 19043, "num_processed_events": 13432, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13432, "processed_events_size": 12029360, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "79723": "2021-02-13T06:13:57Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "79723": "2021-02-13T06:13:57Z" }, "v8worker_events_lost": 0 }, "function_id": 2434364620, "function_name": "bucket_op", "gocb_creds_request_counter": 75, "internal_vb_distribution_stats": { "worker_bucket_op_0": "[342-682]" }, "latency_percentile_stats": { "100": 16100, "50": 700, "80": 1100, "90": 1400, "95": 1800, "99": 3000 }, "lcb_creds_request_counter": 26930, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.90:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.97:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_bucket_op_0": "[0-341]" }, "172.23.122.90:8096": { "worker_bucket_op_0": "[342-682]" }, "172.23.122.97:8096": { "worker_bucket_op_0": "[683-1023]" } }, "worker_pids": { "worker_bucket_op_0": 79723 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 2, "agg_messages_sent_to_worker": 18936, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 67238, "dcp_mutation_sent_to_worker": 13432, "dcp_snapshot": 40563, "dcp_stream_req_counter": 7034, "dcp_stream_req_err_counter": 6211, "dcp_streamend": 341, "dcp_streamreq": 823, "dcp_system_event": 3410, "execution_stats": 894, "failure_stats": 894, "latency_stats": 894, "lcb_exception_stats": 894, "log_level": 1, "num_processed_events": 13432, "processed_events_size": 12029360, "thr_count": 1, "thr_map": 1, "timer_events": 13558, "timer_responses_received": 13432, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13432, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13432, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 18932, "num_processed_events": 13432, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13432, "processed_events_size": 12029360, "timer_callback_failure": 0, "timer_callback_success": 13558, "timer_cancel_counter": 0, "timer_create_counter": 13432, "timer_create_failure": 0, "timer_msg_counter": 13558, "timer_responses_sent": 0, "timestamp": { "79800": "2021-02-13T06:13:57Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "79800": "2021-02-13T06:13:57Z" }, "v8worker_events_lost": 0 }, "function_id": 972602119, "function_name": "timers", "gocb_creds_request_counter": 75, "internal_vb_distribution_stats": { "worker_timers_0": "[342-682]" }, "latency_percentile_stats": { "100": 17700, "50": 1900, "80": 2800, "90": 3400, "95": 4300, "99": 6800 }, "lcb_creds_request_counter": 26930, "lcb_exception_stats": {}, "metastore_stats": { "metastore_delete_err": 0, "metastore_deletes": 0, "metastore_not_found": 0, "metastore_scan": 0, "metastore_scan_due": 0, "metastore_scan_err": 0, "metastore_set": 0, "metastore_set_err": 0 }, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.90:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.97:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_timers_0": "[0-341]" }, "172.23.122.90:8096": { "worker_timers_0": "[342-682]" }, "172.23.122.97:8096": { "worker_timers_0": "[683-1023]" } }, "worker_pids": { "worker_timers_0": 79800 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 2, "agg_messages_sent_to_worker": 18830, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 41052, "dcp_mutation_sent_to_worker": 13432, "dcp_mutation_suppressed_counter": 14188, "dcp_snapshot": 40526, "dcp_stream_req_counter": 1642, "dcp_stream_req_err_counter": 796, "dcp_streamend": 341, "dcp_streamreq": 846, "dcp_system_event": 3410, "execution_stats": 873, "failure_stats": 873, "latency_stats": 873, "lcb_exception_stats": 873, "log_level": 1, "num_processed_events": 13432, "processed_events_size": 12024580, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13432, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13432, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 18826, "num_processed_events": 13432, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13432, "processed_events_size": 12024580, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "79832": "2021-02-13T06:13:57Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "79832": "2021-02-13T06:13:57Z" }, "v8worker_events_lost": 0 }, "function_id": 4207347309, "function_name": "sbm", "gocb_creds_request_counter": 75, "internal_vb_distribution_stats": { "worker_sbm_0": "[342-682]" }, "latency_percentile_stats": { "100": 15300, "50": 800, "80": 1400, "90": 1800, "95": 2500, "99": 4300 }, "lcb_creds_request_counter": 26930, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.90:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.97:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_sbm_0": "[0-341]" }, "172.23.122.90:8096": { "worker_sbm_0": "[342-682]" }, "172.23.122.97:8096": { "worker_sbm_0": "[683-1023]" } }, "worker_pids": { "worker_sbm_0": 79832 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 2, "agg_messages_sent_to_worker": 18719, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 67238, "dcp_mutation_sent_to_worker": 13432, "dcp_snapshot": 40584, "dcp_stream_req_counter": 12369, "dcp_stream_req_err_counter": 11550, "dcp_streamend": 341, "dcp_streamreq": 819, "dcp_system_event": 3410, "execution_stats": 851, "failure_stats": 851, "latency_stats": 851, "lcb_exception_stats": 851, "log_level": 1, "num_processed_events": 13432, "processed_events_size": 12029360, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 13432, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 13432, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13432, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13432, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 18715, "num_processed_events": 13432, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13432, "processed_events_size": 12029360, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "79864": "2021-02-13T06:13:57Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "79864": "2021-02-13T06:13:57Z" }, "v8worker_events_lost": 0 }, "function_id": 1583657262, "function_name": "curl", "gocb_creds_request_counter": 75, "internal_vb_distribution_stats": { "worker_curl_0": "[342-682]" }, "latency_percentile_stats": { "100": 40700, "50": 1700, "80": 2200, "90": 2700, "95": 3300, "99": 5100 }, "lcb_creds_request_counter": 26930, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.90:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.97:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_curl_0": "[0-341]" }, "172.23.122.90:8096": { "worker_curl_0": "[342-682]" }, "172.23.122.97:8096": { "worker_curl_0": "[683-1023]" } }, "worker_pids": { "worker_curl_0": 79864 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 3, "agg_messages_sent_to_worker": 18613, "agg_queue_memory": 0, "agg_queue_memory_cap": 20971520, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 67238, "dcp_mutation_sent_to_worker": 13432, "dcp_snapshot": 40560, "dcp_stream_req_counter": 11927, "dcp_stream_req_err_counter": 11108, "dcp_streamend": 341, "dcp_streamreq": 819, "dcp_system_event": 3410, "execution_stats": 830, "failure_stats": 830, "latency_stats": 830, "lcb_exception_stats": 830, "log_level": 1, "num_processed_events": 13432, "processed_events_size": 12029360, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13432, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13432, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 18609, "num_processed_events": 13432, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13432, "processed_events_size": 12029360, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "79926": "2021-02-13T06:13:57Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "79926": "2021-02-13T06:13:57Z" }, "v8worker_events_lost": 0 }, "function_id": 1986265310, "function_name": "n1ql", "gocb_creds_request_counter": 75, "internal_vb_distribution_stats": { "worker_n1ql_0": "[342-682]" }, "latency_percentile_stats": { "100": 106700, "50": 3400, "80": 4700, "90": 5900, "95": 7400, "99": 11900 }, "lcb_creds_request_counter": 26930, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.90:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.97:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_n1ql_0": "[0-341]" }, "172.23.122.90:8096": { "worker_n1ql_0": "[342-682]" }, "172.23.122.97:8096": { "worker_n1ql_0": "[683-1023]" } }, "worker_pids": { "worker_n1ql_0": 79926 } } ] [2021-02-12 22:14:08,269] - [eventing_base] [140099299809024] - INFO - Stats for Node 172.23.122.97 is [ { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 1, "agg_messages_sent_to_worker": 19040, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 66835, "dcp_mutation_sent_to_worker": 13389, "dcp_snapshot": 40314, "dcp_stream_req_counter": 5346, "dcp_stream_req_err_counter": 4343, "dcp_streamend": 352, "dcp_streamreq": 1003, "dcp_system_event": 3410, "execution_stats": 921, "failure_stats": 921, "latency_stats": 921, "lcb_exception_stats": 921, "log_level": 1, "num_processed_events": 13389, "processed_events_size": 11989232, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13389, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13389, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 19036, "num_processed_events": 13389, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13389, "processed_events_size": 11989232, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "73445": "2021-02-13T06:14:02Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "73445": "2021-02-13T06:14:02Z" }, "v8worker_events_lost": 0 }, "function_id": 2434364620, "function_name": "bucket_op", "gocb_creds_request_counter": 77, "internal_vb_distribution_stats": { "worker_bucket_op_0": "[683-1023]" }, "latency_percentile_stats": { "100": 23900, "50": 700, "80": 1100, "90": 1500, "95": 1800, "99": 3100 }, "lcb_creds_request_counter": 26850, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.90:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.97:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_bucket_op_0": "[0-341]" }, "172.23.122.90:8096": { "worker_bucket_op_0": "[342-682]" }, "172.23.122.97:8096": { "worker_bucket_op_0": "[683-1023]" } }, "worker_pids": { "worker_bucket_op_0": 73445 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 2, "agg_messages_sent_to_worker": 18923, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 66835, "dcp_mutation_sent_to_worker": 13389, "dcp_snapshot": 40301, "dcp_stream_req_counter": 1220, "dcp_stream_req_err_counter": 208, "dcp_streamend": 341, "dcp_streamreq": 1012, "dcp_system_event": 3410, "execution_stats": 900, "failure_stats": 900, "latency_stats": 900, "lcb_exception_stats": 900, "log_level": 1, "num_processed_events": 13389, "processed_events_size": 11989232, "thr_count": 1, "thr_map": 1, "timer_events": 13234, "timer_responses_received": 13389, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13389, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13389, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 18919, "num_processed_events": 13389, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13389, "processed_events_size": 11989232, "timer_callback_failure": 0, "timer_callback_success": 13234, "timer_cancel_counter": 0, "timer_create_counter": 13389, "timer_create_failure": 0, "timer_msg_counter": 13234, "timer_responses_sent": 0, "timestamp": { "73476": "2021-02-13T06:14:03Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "73476": "2021-02-13T06:14:03Z" }, "v8worker_events_lost": 0 }, "function_id": 972602119, "function_name": "timers", "gocb_creds_request_counter": 77, "internal_vb_distribution_stats": { "worker_timers_0": "[683-1023]" }, "latency_percentile_stats": { "100": 23900, "50": 2000, "80": 2800, "90": 3500, "95": 4200, "99": 7000 }, "lcb_creds_request_counter": 26850, "lcb_exception_stats": {}, "metastore_stats": { "metastore_delete_err": 0, "metastore_deletes": 0, "metastore_not_found": 0, "metastore_scan": 0, "metastore_scan_due": 0, "metastore_scan_err": 0, "metastore_set": 0, "metastore_set_err": 0 }, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.90:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.97:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_timers_0": "[0-341]" }, "172.23.122.90:8096": { "worker_timers_0": "[342-682]" }, "172.23.122.97:8096": { "worker_timers_0": "[683-1023]" } }, "worker_pids": { "worker_timers_0": 73476 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 1, "agg_messages_sent_to_worker": 18823, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 40836, "dcp_mutation_sent_to_worker": 13389, "dcp_mutation_suppressed_counter": 14058, "dcp_snapshot": 40249, "dcp_stream_req_counter": 13149, "dcp_stream_req_err_counter": 12196, "dcp_streamend": 352, "dcp_streamreq": 953, "dcp_system_event": 3410, "execution_stats": 878, "failure_stats": 878, "latency_stats": 878, "lcb_exception_stats": 878, "log_level": 1, "num_processed_events": 13389, "processed_events_size": 11990912, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13389, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13389, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 18819, "num_processed_events": 13389, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13389, "processed_events_size": 11990912, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "73553": "2021-02-13T06:14:02Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "73553": "2021-02-13T06:14:02Z" }, "v8worker_events_lost": 0 }, "function_id": 4207347309, "function_name": "sbm", "gocb_creds_request_counter": 77, "internal_vb_distribution_stats": { "worker_sbm_0": "[683-1023]" }, "latency_percentile_stats": { "100": 13300, "50": 900, "80": 1400, "90": 1900, "95": 2500, "99": 4400 }, "lcb_creds_request_counter": 26850, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.90:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.97:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_sbm_0": "[0-341]" }, "172.23.122.90:8096": { "worker_sbm_0": "[342-682]" }, "172.23.122.97:8096": { "worker_sbm_0": "[683-1023]" } }, "worker_pids": { "worker_sbm_0": 73553 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 2, "agg_messages_sent_to_worker": 18717, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 66835, "dcp_mutation_sent_to_worker": 13389, "dcp_snapshot": 40403, "dcp_stream_req_counter": 12994, "dcp_stream_req_err_counter": 12095, "dcp_streamend": 352, "dcp_streamreq": 899, "dcp_system_event": 3410, "execution_stats": 857, "failure_stats": 857, "latency_stats": 857, "lcb_exception_stats": 857, "log_level": 1, "num_processed_events": 13389, "processed_events_size": 11989232, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 13389, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 13389, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13389, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13389, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 18713, "num_processed_events": 13389, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13389, "processed_events_size": 11989232, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "73585": "2021-02-13T06:14:03Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "73585": "2021-02-13T06:14:03Z" }, "v8worker_events_lost": 0 }, "function_id": 1583657262, "function_name": "curl", "gocb_creds_request_counter": 77, "internal_vb_distribution_stats": { "worker_curl_0": "[683-1023]" }, "latency_percentile_stats": { "100": 43900, "50": 1700, "80": 2200, "90": 2600, "95": 3000, "99": 4800 }, "lcb_creds_request_counter": 26850, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.90:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.97:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_curl_0": "[0-341]" }, "172.23.122.90:8096": { "worker_curl_0": "[342-682]" }, "172.23.122.97:8096": { "worker_curl_0": "[683-1023]" } }, "worker_pids": { "worker_curl_0": 73585 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 3, "agg_messages_sent_to_worker": 18611, "agg_queue_memory": 0, "agg_queue_memory_cap": 20971520, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 66835, "dcp_mutation_sent_to_worker": 13389, "dcp_snapshot": 40269, "dcp_stream_req_counter": 12432, "dcp_stream_req_err_counter": 11504, "dcp_streamend": 352, "dcp_streamreq": 928, "dcp_system_event": 3410, "execution_stats": 836, "failure_stats": 836, "latency_stats": 836, "lcb_exception_stats": 836, "log_level": 1, "num_processed_events": 13389, "processed_events_size": 11989232, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13389, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13389, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 18607, "num_processed_events": 13389, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 1, "on_update_success": 13388, "processed_events_size": 11989232, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "73647": "2021-02-13T06:14:03Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 1, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "73647": "2021-02-13T06:14:03Z" }, "v8worker_events_lost": 0 }, "function_id": 1986265310, "function_name": "n1ql", "gocb_creds_request_counter": 77, "internal_vb_distribution_stats": { "worker_n1ql_0": "[683-1023]" }, "latency_percentile_stats": { "100": 198100, "50": 3500, "80": 4800, "90": 6000, "95": 7500, "99": 12900 }, "lcb_creds_request_counter": 26850, "lcb_exception_stats": { "12009": 1, "209": 1 }, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.90:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.97:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_n1ql_0": "[0-341]" }, "172.23.122.90:8096": { "worker_n1ql_0": "[342-682]" }, "172.23.122.97:8096": { "worker_n1ql_0": "[683-1023]" } }, "worker_pids": { "worker_n1ql_0": 73647 } } ] [2021-02-12 22:14:09,058] - [eventing_base] [140099299809024] - INFO - Execution stats update_success: 40320 update_failure: 0 delete_success: 0 delete_failure: 0 [2021-02-12 22:14:09,079] - [basetestcase] [140099299809024] - INFO - list of eventing nodes in cluster: [ip:172.23.122.117 port:8091 ssh_username:root, ip:172.23.122.90 port:8091 ssh_username:root, ip:172.23.122.97 port:8091 ssh_username:root] [2021-02-12 22:14:09,123] - [eventing_base] [140099299809024] - INFO - Stats for Node 172.23.122.117 is [ { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 2, "agg_messages_sent_to_worker": 18488, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 40513, "dcp_mutation_sent_to_worker": 13499, "dcp_snapshot": 40388, "dcp_stream_req_counter": 342, "dcp_streamreq": 342, "dcp_system_event": 3420, "execution_stats": 927, "failure_stats": 927, "latency_stats": 927, "lcb_exception_stats": 927, "log_level": 1, "num_processed_events": 13499, "processed_events_size": 12082876, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13499, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13499, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 18484, "num_processed_events": 13499, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13499, "processed_events_size": 12082876, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "40126": "2021-02-13T06:14:08Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "40126": "2021-02-13T06:14:08Z" }, "v8worker_events_lost": 0 }, "function_id": 2434364620, "function_name": "bucket_op", "gocb_creds_request_counter": 81, "internal_vb_distribution_stats": { "worker_bucket_op_0": "[0-341]" }, "latency_percentile_stats": { "100": 13900, "50": 700, "80": 1000, "90": 1300, "95": 1700, "99": 3000 }, "lcb_creds_request_counter": 27062, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.90:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.97:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_bucket_op_0": "[0-341]" }, "172.23.122.90:8096": { "worker_bucket_op_0": "[342-682]" }, "172.23.122.97:8096": { "worker_bucket_op_0": "[683-1023]" } }, "worker_pids": { "worker_bucket_op_0": 40126 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 2, "agg_messages_sent_to_worker": 18377, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 40513, "dcp_mutation_sent_to_worker": 13499, "dcp_snapshot": 40374, "dcp_stream_req_counter": 342, "dcp_streamreq": 342, "dcp_system_event": 3420, "execution_stats": 905, "failure_stats": 905, "latency_stats": 905, "lcb_exception_stats": 905, "log_level": 1, "num_processed_events": 13499, "processed_events_size": 12082876, "thr_count": 1, "thr_map": 1, "timer_events": 13528, "timer_responses_received": 13499, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13499, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13499, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 18373, "num_processed_events": 13499, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13499, "processed_events_size": 12082876, "timer_callback_failure": 0, "timer_callback_success": 13528, "timer_cancel_counter": 0, "timer_create_counter": 13499, "timer_create_failure": 0, "timer_msg_counter": 13528, "timer_responses_sent": 0, "timestamp": { "40196": "2021-02-13T06:14:08Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "40196": "2021-02-13T06:14:08Z" }, "v8worker_events_lost": 0 }, "function_id": 972602119, "function_name": "timers", "gocb_creds_request_counter": 81, "internal_vb_distribution_stats": { "worker_timers_0": "[0-341]" }, "latency_percentile_stats": { "100": 18800, "50": 2000, "80": 2800, "90": 3400, "95": 4200, "99": 7000 }, "lcb_creds_request_counter": 27062, "lcb_exception_stats": {}, "metastore_stats": { "metastore_delete_err": 0, "metastore_deletes": 0, "metastore_not_found": 0, "metastore_scan": 0, "metastore_scan_due": 0, "metastore_scan_err": 0, "metastore_set": 0, "metastore_set_err": 0 }, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.90:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.97:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_timers_0": "[0-341]" }, "172.23.122.90:8096": { "worker_timers_0": "[342-682]" }, "172.23.122.97:8096": { "worker_timers_0": "[683-1023]" } }, "worker_pids": { "worker_timers_0": 40196 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 1, "agg_messages_sent_to_worker": 18271, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 40513, "dcp_mutation_sent_to_worker": 13499, "dcp_mutation_suppressed_counter": 13515, "dcp_snapshot": 40350, "dcp_stream_req_counter": 342, "dcp_streamreq": 342, "dcp_system_event": 3420, "execution_stats": 884, "failure_stats": 884, "latency_stats": 884, "lcb_exception_stats": 884, "log_level": 1, "num_processed_events": 13499, "processed_events_size": 12083724, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13499, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13499, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 18267, "num_processed_events": 13499, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13499, "processed_events_size": 12083724, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "40238": "2021-02-13T06:14:08Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "40238": "2021-02-13T06:14:08Z" }, "v8worker_events_lost": 0 }, "function_id": 4207347309, "function_name": "sbm", "gocb_creds_request_counter": 81, "internal_vb_distribution_stats": { "worker_sbm_0": "[0-341]" }, "latency_percentile_stats": { "100": 17100, "50": 800, "80": 1300, "90": 1700, "95": 2300, "99": 4000 }, "lcb_creds_request_counter": 27062, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.90:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.97:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_sbm_0": "[0-341]" }, "172.23.122.90:8096": { "worker_sbm_0": "[342-682]" }, "172.23.122.97:8096": { "worker_sbm_0": "[683-1023]" } }, "worker_pids": { "worker_sbm_0": 40238 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 2, "agg_messages_sent_to_worker": 18165, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 40513, "dcp_mutation_sent_to_worker": 13499, "dcp_snapshot": 40372, "dcp_stream_req_counter": 342, "dcp_streamreq": 342, "dcp_system_event": 3420, "execution_stats": 863, "failure_stats": 863, "latency_stats": 863, "lcb_exception_stats": 863, "log_level": 1, "num_processed_events": 13499, "processed_events_size": 12082876, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 13499, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 13499, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13499, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13499, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 18161, "num_processed_events": 13499, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13499, "processed_events_size": 12082876, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "40279": "2021-02-13T06:14:09Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "40279": "2021-02-13T06:14:09Z" }, "v8worker_events_lost": 0 }, "function_id": 1583657262, "function_name": "curl", "gocb_creds_request_counter": 81, "internal_vb_distribution_stats": { "worker_curl_0": "[0-341]" }, "latency_percentile_stats": { "100": 36500, "50": 1600, "80": 2100, "90": 2500, "95": 3000, "99": 4700 }, "lcb_creds_request_counter": 27062, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.90:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.97:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_curl_0": "[0-341]" }, "172.23.122.90:8096": { "worker_curl_0": "[342-682]" }, "172.23.122.97:8096": { "worker_curl_0": "[683-1023]" } }, "worker_pids": { "worker_curl_0": 40279 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 1, "agg_messages_sent_to_worker": 18054, "agg_queue_memory": 0, "agg_queue_memory_cap": 20971520, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 40513, "dcp_mutation_sent_to_worker": 13499, "dcp_snapshot": 40337, "dcp_stream_req_counter": 342, "dcp_streamreq": 342, "dcp_system_event": 3420, "execution_stats": 841, "failure_stats": 841, "latency_stats": 841, "lcb_exception_stats": 841, "log_level": 1, "num_processed_events": 13499, "processed_events_size": 12082876, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13499, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13499, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 18050, "num_processed_events": 13499, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13499, "processed_events_size": 12082876, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "40365": "2021-02-13T06:14:08Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "40365": "2021-02-13T06:14:08Z" }, "v8worker_events_lost": 0 }, "function_id": 1986265310, "function_name": "n1ql", "gocb_creds_request_counter": 81, "internal_vb_distribution_stats": { "worker_n1ql_0": "[0-341]" }, "latency_percentile_stats": { "100": 57200, "50": 3300, "80": 4600, "90": 5700, "95": 7100, "99": 11400 }, "lcb_creds_request_counter": 27062, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.90:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.97:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_n1ql_0": "[0-341]" }, "172.23.122.90:8096": { "worker_n1ql_0": "[342-682]" }, "172.23.122.97:8096": { "worker_n1ql_0": "[683-1023]" } }, "worker_pids": { "worker_n1ql_0": 40365 } } ] [2021-02-12 22:14:09,165] - [eventing_base] [140099299809024] - INFO - Stats for Node 172.23.122.90 is [ { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 3, "agg_messages_sent_to_worker": 19102, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 67238, "dcp_mutation_sent_to_worker": 13432, "dcp_snapshot": 40611, "dcp_stream_req_counter": 7383, "dcp_stream_req_err_counter": 6564, "dcp_streamend": 341, "dcp_streamreq": 819, "dcp_system_event": 3410, "execution_stats": 927, "failure_stats": 927, "latency_stats": 927, "lcb_exception_stats": 927, "log_level": 1, "num_processed_events": 13432, "processed_events_size": 12029360, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13432, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13432, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 19098, "num_processed_events": 13432, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13432, "processed_events_size": 12029360, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "79723": "2021-02-13T06:14:08Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "79723": "2021-02-13T06:14:08Z" }, "v8worker_events_lost": 0 }, "function_id": 2434364620, "function_name": "bucket_op", "gocb_creds_request_counter": 75, "internal_vb_distribution_stats": { "worker_bucket_op_0": "[342-682]" }, "latency_percentile_stats": { "100": 16100, "50": 700, "80": 1100, "90": 1400, "95": 1800, "99": 3000 }, "lcb_creds_request_counter": 26930, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.90:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.97:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_bucket_op_0": "[0-341]" }, "172.23.122.90:8096": { "worker_bucket_op_0": "[342-682]" }, "172.23.122.97:8096": { "worker_bucket_op_0": "[683-1023]" } }, "worker_pids": { "worker_bucket_op_0": 79723 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 2, "agg_messages_sent_to_worker": 18991, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 67238, "dcp_mutation_sent_to_worker": 13432, "dcp_snapshot": 40563, "dcp_stream_req_counter": 7034, "dcp_stream_req_err_counter": 6211, "dcp_streamend": 341, "dcp_streamreq": 823, "dcp_system_event": 3410, "execution_stats": 905, "failure_stats": 905, "latency_stats": 905, "lcb_exception_stats": 905, "log_level": 1, "num_processed_events": 13432, "processed_events_size": 12029360, "thr_count": 1, "thr_map": 1, "timer_events": 13558, "timer_responses_received": 13432, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13432, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13432, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 18987, "num_processed_events": 13432, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13432, "processed_events_size": 12029360, "timer_callback_failure": 0, "timer_callback_success": 13558, "timer_cancel_counter": 0, "timer_create_counter": 13432, "timer_create_failure": 0, "timer_msg_counter": 13558, "timer_responses_sent": 0, "timestamp": { "79800": "2021-02-13T06:14:08Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "79800": "2021-02-13T06:14:08Z" }, "v8worker_events_lost": 0 }, "function_id": 972602119, "function_name": "timers", "gocb_creds_request_counter": 75, "internal_vb_distribution_stats": { "worker_timers_0": "[342-682]" }, "latency_percentile_stats": { "100": 17700, "50": 1900, "80": 2800, "90": 3400, "95": 4300, "99": 6800 }, "lcb_creds_request_counter": 26930, "lcb_exception_stats": {}, "metastore_stats": { "metastore_delete_err": 0, "metastore_deletes": 0, "metastore_not_found": 0, "metastore_scan": 0, "metastore_scan_due": 0, "metastore_scan_err": 0, "metastore_set": 0, "metastore_set_err": 0 }, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.90:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.97:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_timers_0": "[0-341]" }, "172.23.122.90:8096": { "worker_timers_0": "[342-682]" }, "172.23.122.97:8096": { "worker_timers_0": "[683-1023]" } }, "worker_pids": { "worker_timers_0": 79800 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 2, "agg_messages_sent_to_worker": 18885, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 41052, "dcp_mutation_sent_to_worker": 13432, "dcp_mutation_suppressed_counter": 14188, "dcp_snapshot": 40526, "dcp_stream_req_counter": 1642, "dcp_stream_req_err_counter": 796, "dcp_streamend": 341, "dcp_streamreq": 846, "dcp_system_event": 3410, "execution_stats": 884, "failure_stats": 884, "latency_stats": 884, "lcb_exception_stats": 884, "log_level": 1, "num_processed_events": 13432, "processed_events_size": 12024580, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13432, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13432, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 18881, "num_processed_events": 13432, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13432, "processed_events_size": 12024580, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "79832": "2021-02-13T06:14:08Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "79832": "2021-02-13T06:14:08Z" }, "v8worker_events_lost": 0 }, "function_id": 4207347309, "function_name": "sbm", "gocb_creds_request_counter": 75, "internal_vb_distribution_stats": { "worker_sbm_0": "[342-682]" }, "latency_percentile_stats": { "100": 15300, "50": 800, "80": 1400, "90": 1800, "95": 2500, "99": 4300 }, "lcb_creds_request_counter": 26930, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.90:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.97:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_sbm_0": "[0-341]" }, "172.23.122.90:8096": { "worker_sbm_0": "[342-682]" }, "172.23.122.97:8096": { "worker_sbm_0": "[683-1023]" } }, "worker_pids": { "worker_sbm_0": 79832 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 2, "agg_messages_sent_to_worker": 18779, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 67238, "dcp_mutation_sent_to_worker": 13432, "dcp_snapshot": 40584, "dcp_stream_req_counter": 12369, "dcp_stream_req_err_counter": 11550, "dcp_streamend": 341, "dcp_streamreq": 819, "dcp_system_event": 3410, "execution_stats": 863, "failure_stats": 863, "latency_stats": 863, "lcb_exception_stats": 863, "log_level": 1, "num_processed_events": 13432, "processed_events_size": 12029360, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 13432, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 13432, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13432, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13432, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 18775, "num_processed_events": 13432, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13432, "processed_events_size": 12029360, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "79864": "2021-02-13T06:14:09Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "79864": "2021-02-13T06:14:09Z" }, "v8worker_events_lost": 0 }, "function_id": 1583657262, "function_name": "curl", "gocb_creds_request_counter": 75, "internal_vb_distribution_stats": { "worker_curl_0": "[342-682]" }, "latency_percentile_stats": { "100": 40700, "50": 1700, "80": 2200, "90": 2700, "95": 3300, "99": 5100 }, "lcb_creds_request_counter": 26930, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.90:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.97:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_curl_0": "[0-341]" }, "172.23.122.90:8096": { "worker_curl_0": "[342-682]" }, "172.23.122.97:8096": { "worker_curl_0": "[683-1023]" } }, "worker_pids": { "worker_curl_0": 79864 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 3, "agg_messages_sent_to_worker": 18668, "agg_queue_memory": 0, "agg_queue_memory_cap": 20971520, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 67238, "dcp_mutation_sent_to_worker": 13432, "dcp_snapshot": 40560, "dcp_stream_req_counter": 11927, "dcp_stream_req_err_counter": 11108, "dcp_streamend": 341, "dcp_streamreq": 819, "dcp_system_event": 3410, "execution_stats": 841, "failure_stats": 841, "latency_stats": 841, "lcb_exception_stats": 841, "log_level": 1, "num_processed_events": 13432, "processed_events_size": 12029360, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13432, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13432, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 18664, "num_processed_events": 13432, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13432, "processed_events_size": 12029360, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "79926": "2021-02-13T06:14:08Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "79926": "2021-02-13T06:14:08Z" }, "v8worker_events_lost": 0 }, "function_id": 1986265310, "function_name": "n1ql", "gocb_creds_request_counter": 75, "internal_vb_distribution_stats": { "worker_n1ql_0": "[342-682]" }, "latency_percentile_stats": { "100": 106700, "50": 3400, "80": 4700, "90": 5900, "95": 7400, "99": 11900 }, "lcb_creds_request_counter": 26930, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.90:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.97:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_n1ql_0": "[0-341]" }, "172.23.122.90:8096": { "worker_n1ql_0": "[342-682]" }, "172.23.122.97:8096": { "worker_n1ql_0": "[683-1023]" } }, "worker_pids": { "worker_n1ql_0": 79926 } } ] [2021-02-12 22:14:09,210] - [eventing_base] [140099299809024] - INFO - Stats for Node 172.23.122.97 is [ { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 1, "agg_messages_sent_to_worker": 19070, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 66835, "dcp_mutation_sent_to_worker": 13389, "dcp_snapshot": 40314, "dcp_stream_req_counter": 5346, "dcp_stream_req_err_counter": 4343, "dcp_streamend": 352, "dcp_streamreq": 1003, "dcp_system_event": 3410, "execution_stats": 927, "failure_stats": 927, "latency_stats": 927, "lcb_exception_stats": 927, "log_level": 1, "num_processed_events": 13389, "processed_events_size": 11989232, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13389, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13389, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 19066, "num_processed_events": 13389, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13389, "processed_events_size": 11989232, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "73445": "2021-02-13T06:14:08Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "73445": "2021-02-13T06:14:08Z" }, "v8worker_events_lost": 0 }, "function_id": 2434364620, "function_name": "bucket_op", "gocb_creds_request_counter": 77, "internal_vb_distribution_stats": { "worker_bucket_op_0": "[683-1023]" }, "latency_percentile_stats": { "100": 23900, "50": 700, "80": 1100, "90": 1500, "95": 1800, "99": 3100 }, "lcb_creds_request_counter": 26850, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.90:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.97:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_bucket_op_0": "[0-341]" }, "172.23.122.90:8096": { "worker_bucket_op_0": "[342-682]" }, "172.23.122.97:8096": { "worker_bucket_op_0": "[683-1023]" } }, "worker_pids": { "worker_bucket_op_0": 73445 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 2, "agg_messages_sent_to_worker": 18953, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 66835, "dcp_mutation_sent_to_worker": 13389, "dcp_snapshot": 40301, "dcp_stream_req_counter": 1220, "dcp_stream_req_err_counter": 208, "dcp_streamend": 341, "dcp_streamreq": 1012, "dcp_system_event": 3410, "execution_stats": 906, "failure_stats": 906, "latency_stats": 906, "lcb_exception_stats": 906, "log_level": 1, "num_processed_events": 13389, "processed_events_size": 11989232, "thr_count": 1, "thr_map": 1, "timer_events": 13234, "timer_responses_received": 13389, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13389, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13389, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 18949, "num_processed_events": 13389, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13389, "processed_events_size": 11989232, "timer_callback_failure": 0, "timer_callback_success": 13234, "timer_cancel_counter": 0, "timer_create_counter": 13389, "timer_create_failure": 0, "timer_msg_counter": 13234, "timer_responses_sent": 0, "timestamp": { "73476": "2021-02-13T06:14:09Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "73476": "2021-02-13T06:14:09Z" }, "v8worker_events_lost": 0 }, "function_id": 972602119, "function_name": "timers", "gocb_creds_request_counter": 77, "internal_vb_distribution_stats": { "worker_timers_0": "[683-1023]" }, "latency_percentile_stats": { "100": 23900, "50": 2000, "80": 2800, "90": 3500, "95": 4200, "99": 7000 }, "lcb_creds_request_counter": 26850, "lcb_exception_stats": {}, "metastore_stats": { "metastore_delete_err": 0, "metastore_deletes": 0, "metastore_not_found": 0, "metastore_scan": 0, "metastore_scan_due": 0, "metastore_scan_err": 0, "metastore_set": 0, "metastore_set_err": 0 }, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.90:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.97:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_timers_0": "[0-341]" }, "172.23.122.90:8096": { "worker_timers_0": "[342-682]" }, "172.23.122.97:8096": { "worker_timers_0": "[683-1023]" } }, "worker_pids": { "worker_timers_0": 73476 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 1, "agg_messages_sent_to_worker": 18853, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 40836, "dcp_mutation_sent_to_worker": 13389, "dcp_mutation_suppressed_counter": 14058, "dcp_snapshot": 40249, "dcp_stream_req_counter": 13149, "dcp_stream_req_err_counter": 12196, "dcp_streamend": 352, "dcp_streamreq": 953, "dcp_system_event": 3410, "execution_stats": 884, "failure_stats": 884, "latency_stats": 884, "lcb_exception_stats": 884, "log_level": 1, "num_processed_events": 13389, "processed_events_size": 11990912, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13389, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13389, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 18849, "num_processed_events": 13389, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13389, "processed_events_size": 11990912, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "73553": "2021-02-13T06:14:08Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "73553": "2021-02-13T06:14:08Z" }, "v8worker_events_lost": 0 }, "function_id": 4207347309, "function_name": "sbm", "gocb_creds_request_counter": 77, "internal_vb_distribution_stats": { "worker_sbm_0": "[683-1023]" }, "latency_percentile_stats": { "100": 13300, "50": 900, "80": 1400, "90": 1900, "95": 2500, "99": 4400 }, "lcb_creds_request_counter": 26850, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.90:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.97:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_sbm_0": "[0-341]" }, "172.23.122.90:8096": { "worker_sbm_0": "[342-682]" }, "172.23.122.97:8096": { "worker_sbm_0": "[683-1023]" } }, "worker_pids": { "worker_sbm_0": 73553 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 2, "agg_messages_sent_to_worker": 18747, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 66835, "dcp_mutation_sent_to_worker": 13389, "dcp_snapshot": 40403, "dcp_stream_req_counter": 12994, "dcp_stream_req_err_counter": 12095, "dcp_streamend": 352, "dcp_streamreq": 899, "dcp_system_event": 3410, "execution_stats": 863, "failure_stats": 863, "latency_stats": 863, "lcb_exception_stats": 863, "log_level": 1, "num_processed_events": 13389, "processed_events_size": 11989232, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 13389, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 13389, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13389, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13389, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 18743, "num_processed_events": 13389, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13389, "processed_events_size": 11989232, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "73585": "2021-02-13T06:14:09Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "73585": "2021-02-13T06:14:09Z" }, "v8worker_events_lost": 0 }, "function_id": 1583657262, "function_name": "curl", "gocb_creds_request_counter": 77, "internal_vb_distribution_stats": { "worker_curl_0": "[683-1023]" }, "latency_percentile_stats": { "100": 43900, "50": 1700, "80": 2200, "90": 2600, "95": 3000, "99": 4800 }, "lcb_creds_request_counter": 26850, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.90:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.97:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_curl_0": "[0-341]" }, "172.23.122.90:8096": { "worker_curl_0": "[342-682]" }, "172.23.122.97:8096": { "worker_curl_0": "[683-1023]" } }, "worker_pids": { "worker_curl_0": 73585 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 3, "agg_messages_sent_to_worker": 18636, "agg_queue_memory": 0, "agg_queue_memory_cap": 20971520, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 66835, "dcp_mutation_sent_to_worker": 13389, "dcp_snapshot": 40269, "dcp_stream_req_counter": 12432, "dcp_stream_req_err_counter": 11504, "dcp_streamend": 352, "dcp_streamreq": 928, "dcp_system_event": 3410, "execution_stats": 841, "failure_stats": 841, "latency_stats": 841, "lcb_exception_stats": 841, "log_level": 1, "num_processed_events": 13389, "processed_events_size": 11989232, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13389, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13389, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 18632, "num_processed_events": 13389, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 1, "on_update_success": 13388, "processed_events_size": 11989232, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "73647": "2021-02-13T06:14:08Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 1, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "73647": "2021-02-13T06:14:08Z" }, "v8worker_events_lost": 0 }, "function_id": 1986265310, "function_name": "n1ql", "gocb_creds_request_counter": 77, "internal_vb_distribution_stats": { "worker_n1ql_0": "[683-1023]" }, "latency_percentile_stats": { "100": 198100, "50": 3500, "80": 4800, "90": 6000, "95": 7500, "99": 12900 }, "lcb_creds_request_counter": 26850, "lcb_exception_stats": { "12009": 1, "209": 1 }, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.90:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.97:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_n1ql_0": "[0-341]" }, "172.23.122.90:8096": { "worker_n1ql_0": "[342-682]" }, "172.23.122.97:8096": { "worker_n1ql_0": "[683-1023]" } }, "worker_pids": { "worker_n1ql_0": 73647 } } ] [2021-02-12 22:14:09,234] - [basetestcase] [140099299809024] - INFO - list of eventing nodes in cluster: [ip:172.23.122.117 port:8091 ssh_username:root, ip:172.23.122.90 port:8091 ssh_username:root, ip:172.23.122.97 port:8091 ssh_username:root] [2021-02-12 22:14:09,236] - [remote_util] [140099299809024] - INFO - SSH Connecting to 172.23.122.117 with username:root, attempt#1 of 5 [2021-02-12 22:14:09,344] - [remote_util] [140099299809024] - INFO - SSH Connected to 172.23.122.117 as root [2021-02-12 22:14:09,611] - [remote_util] [140099299809024] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 22:14:09,910] - [remote_util] [140099299809024] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 22:14:09,917] - [rest_client] [140099299809024] - INFO - /diag/eval status on 172.23.122.117:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). [2021-02-12 22:14:09,917] - [remote_util] [140099299809024] - INFO - running command.raw on 172.23.122.117: zgrep "panic" "/opt/couchbase/var/lib/couchbase/logs"/eventing.log* | wc -l [2021-02-12 22:14:10,010] - [remote_util] [140099299809024] - INFO - command executed successfully with root [2021-02-12 22:14:10,010] - [remote_util] [140099299809024] - INFO - running command.raw on 172.23.122.117: ls "/opt/couchbase/var/lib/couchbase/logs"/../crash/| wc -l [2021-02-12 22:14:10,063] - [remote_util] [140099299809024] - INFO - command executed successfully with root [2021-02-12 22:14:10,064] - [remote_util] [140099299809024] - INFO - SSH Connecting to 172.23.122.90 with username:root, attempt#1 of 5 [2021-02-12 22:14:10,166] - [remote_util] [140099299809024] - INFO - SSH Connected to 172.23.122.90 as root [2021-02-12 22:14:10,471] - [remote_util] [140099299809024] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 22:14:10,769] - [remote_util] [140099299809024] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 22:14:10,776] - [rest_client] [140099299809024] - INFO - /diag/eval status on 172.23.122.90:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). [2021-02-12 22:14:10,777] - [remote_util] [140099299809024] - INFO - running command.raw on 172.23.122.90: zgrep "panic" "/opt/couchbase/var/lib/couchbase/logs"/eventing.log* | wc -l [2021-02-12 22:14:11,619] - [remote_util] [140099299809024] - INFO - command executed successfully with root [2021-02-12 22:14:11,620] - [remote_util] [140099299809024] - INFO - running command.raw on 172.23.122.90: ls "/opt/couchbase/var/lib/couchbase/logs"/../crash/| wc -l [2021-02-12 22:14:11,673] - [remote_util] [140099299809024] - INFO - command executed successfully with root [2021-02-12 22:14:11,675] - [remote_util] [140099299809024] - INFO - SSH Connecting to 172.23.122.97 with username:root, attempt#1 of 5 [2021-02-12 22:14:11,776] - [remote_util] [140099299809024] - INFO - SSH Connected to 172.23.122.97 as root [2021-02-12 22:14:12,039] - [remote_util] [140099299809024] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 22:14:12,345] - [remote_util] [140099299809024] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 22:14:12,354] - [rest_client] [140099299809024] - INFO - /diag/eval status on 172.23.122.97:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). [2021-02-12 22:14:12,354] - [remote_util] [140099299809024] - INFO - running command.raw on 172.23.122.97: zgrep "panic" "/opt/couchbase/var/lib/couchbase/logs"/eventing.log* | wc -l [2021-02-12 22:14:14,019] - [remote_util] [140099299809024] - INFO - command executed successfully with root [2021-02-12 22:14:14,020] - [remote_util] [140099299809024] - INFO - running command.raw on 172.23.122.97: ls "/opt/couchbase/var/lib/couchbase/logs"/../crash/| wc -l [2021-02-12 22:14:14,074] - [remote_util] [140099299809024] - INFO - command executed successfully with root [2021-02-12 22:14:14,124] - [basetestcase] [140099299809024] - INFO - ------- Cluster statistics ------- [2021-02-12 22:14:14,124] - [basetestcase] [140099299809024] - INFO - 172.23.122.117:8091 => {'services': ['eventing'], 'cpu_utilization': 11.18608876200363, 'mem_free': 2976538624, 'mem_total': 4201684992, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-02-12 22:14:14,124] - [basetestcase] [140099299809024] - INFO - 172.23.122.118:8091 => {'services': ['kv'], 'cpu_utilization': 5.195473251028806, 'mem_free': 2952859648, 'mem_total': 4201689088, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-02-12 22:14:14,125] - [basetestcase] [140099299809024] - INFO - 172.23.122.82:8091 => {'services': ['index', 'n1ql'], 'cpu_utilization': 1.467240070832279, 'mem_free': 3215114240, 'mem_total': 4201684992, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-02-12 22:14:14,125] - [basetestcase] [140099299809024] - INFO - 172.23.122.90:8091 => {'services': ['eventing'], 'cpu_utilization': 12.33480176211454, 'mem_free': 2976083968, 'mem_total': 4201684992, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-02-12 22:14:14,125] - [basetestcase] [140099299809024] - INFO - 172.23.122.97:8091 => {'services': ['eventing'], 'cpu_utilization': 11.31831131831132, 'mem_free': 3092893696, 'mem_total': 4201684992, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-02-12 22:14:14,125] - [basetestcase] [140099299809024] - INFO - 172.23.123.61:8091 => {'services': ['kv'], 'cpu_utilization': 4.577373211963589, 'mem_free': 2821836800, 'mem_total': 4201672704, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-02-12 22:14:14,125] - [basetestcase] [140099299809024] - INFO - --- End of cluster statistics --- [2021-02-12 22:14:14,126] - [remote_util] [140099299809024] - INFO - SSH Connecting to 172.23.123.61 with username:root, attempt#1 of 5 [2021-02-12 22:14:14,233] - [remote_util] [140099299809024] - INFO - SSH Connected to 172.23.123.61 as root [2021-02-12 22:14:14,501] - [remote_util] [140099299809024] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 22:14:14,804] - [remote_util] [140099299809024] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 22:14:14,806] - [remote_util] [140099299809024] - INFO - SSH Connecting to 172.23.123.62 with username:root, attempt#1 of 5 [2021-02-12 22:14:14,912] - [remote_util] [140099299809024] - INFO - SSH Connected to 172.23.123.62 as root [2021-02-12 22:14:15,184] - [remote_util] [140099299809024] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 22:14:15,480] - [remote_util] [140099299809024] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 22:14:15,482] - [remote_util] [140099299809024] - INFO - SSH Connecting to 172.23.122.90 with username:root, attempt#1 of 5 [2021-02-12 22:14:15,581] - [remote_util] [140099299809024] - INFO - SSH Connected to 172.23.122.90 as root [2021-02-12 22:14:15,855] - [remote_util] [140099299809024] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 22:14:16,144] - [remote_util] [140099299809024] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 22:14:16,147] - [remote_util] [140099299809024] - INFO - SSH Connecting to 172.23.122.117 with username:root, attempt#1 of 5 [2021-02-12 22:14:16,243] - [remote_util] [140099299809024] - INFO - SSH Connected to 172.23.122.117 as root [2021-02-12 22:14:16,496] - [remote_util] [140099299809024] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 22:14:16,789] - [remote_util] [140099299809024] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 22:14:16,792] - [remote_util] [140099299809024] - INFO - SSH Connecting to 172.23.122.97 with username:root, attempt#1 of 5 [2021-02-12 22:14:16,892] - [remote_util] [140099299809024] - INFO - SSH Connected to 172.23.122.97 as root [2021-02-12 22:14:17,157] - [remote_util] [140099299809024] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 22:14:17,460] - [remote_util] [140099299809024] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 22:14:17,462] - [remote_util] [140099299809024] - INFO - SSH Connecting to 172.23.122.82 with username:root, attempt#1 of 5 [2021-02-12 22:14:17,562] - [remote_util] [140099299809024] - INFO - SSH Connected to 172.23.122.82 as root [2021-02-12 22:14:17,828] - [remote_util] [140099299809024] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 22:14:18,119] - [remote_util] [140099299809024] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 22:14:18,121] - [remote_util] [140099299809024] - INFO - SSH Connecting to 172.23.122.118 with username:root, attempt#1 of 5 [2021-02-12 22:14:18,226] - [remote_util] [140099299809024] - INFO - SSH Connected to 172.23.122.118 as root [2021-02-12 22:14:18,478] - [remote_util] [140099299809024] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 22:14:18,773] - [remote_util] [140099299809024] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 22:14:18,775] - [remote_util] [140099299809024] - INFO - SSH Connecting to 172.23.122.116 with username:root, attempt#1 of 5 [2021-02-12 22:14:18,880] - [remote_util] [140099299809024] - INFO - SSH Connected to 172.23.122.116 as root [2021-02-12 22:14:19,167] - [remote_util] [140099299809024] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 22:14:19,469] - [remote_util] [140099299809024] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 22:15:16,551] - [basetestcase] [140099299809024] - INFO - ============== basetestcase cleanup was started for test #6 test_kv_swap_rebalance_when_existing_eventing_node_is_processing_mutations ============== [2021-02-12 22:15:16,610] - [bucket_helper] [140099299809024] - INFO - deleting existing buckets ['dst_bucket', 'metadata', 'src_bucket'] on 172.23.123.61 [2021-02-12 22:15:17,763] - [bucket_helper] [140099299809024] - INFO - waiting for bucket deletion to complete.... [2021-02-12 22:15:17,982] - [rest_client] [140099299809024] - INFO - node 172.23.123.61 existing buckets : ['metadata', 'src_bucket'] [2021-02-12 22:15:17,983] - [bucket_helper] [140099299809024] - INFO - deleted bucket : dst_bucket from 172.23.123.61 [2021-02-12 22:15:18,422] - [bucket_helper] [140099299809024] - INFO - waiting for bucket deletion to complete.... [2021-02-12 22:15:18,432] - [rest_client] [140099299809024] - INFO - node 172.23.123.61 existing buckets : ['src_bucket'] [2021-02-12 22:15:18,433] - [bucket_helper] [140099299809024] - INFO - deleted bucket : metadata from 172.23.123.61 [2021-02-12 22:15:19,478] - [bucket_helper] [140099299809024] - INFO - waiting for bucket deletion to complete.... [2021-02-12 22:15:19,481] - [rest_client] [140099299809024] - INFO - node 172.23.123.61 existing buckets : [] [2021-02-12 22:15:19,481] - [bucket_helper] [140099299809024] - INFO - deleted bucket : src_bucket from 172.23.123.61 [2021-02-12 22:15:19,488] - [bucket_helper] [140099299809024] - INFO - Could not find any buckets for node 172.23.123.62, nothing to delete [2021-02-12 22:15:19,496] - [bucket_helper] [140099299809024] - INFO - Could not find any buckets for node 172.23.122.90, nothing to delete [2021-02-12 22:15:19,503] - [bucket_helper] [140099299809024] - INFO - Could not find any buckets for node 172.23.122.117, nothing to delete [2021-02-12 22:15:19,510] - [bucket_helper] [140099299809024] - INFO - Could not find any buckets for node 172.23.122.97, nothing to delete [2021-02-12 22:15:19,517] - [bucket_helper] [140099299809024] - INFO - Could not find any buckets for node 172.23.122.82, nothing to delete [2021-02-12 22:15:19,525] - [bucket_helper] [140099299809024] - INFO - Could not find any buckets for node 172.23.122.118, nothing to delete [2021-02-12 22:15:19,530] - [bucket_helper] [140099299809024] - INFO - Could not find any buckets for node 172.23.122.116, nothing to delete [2021-02-12 22:15:19,535] - [rest_client] [140099299809024] - INFO - -->is_ns_server_running? [2021-02-12 22:15:19,549] - [cluster_helper] [140099299809024] - INFO - rebalancing all nodes in order to remove nodes [2021-02-12 22:15:19,551] - [rest_client] [140099299809024] - INFO - rebalance params : {'knownNodes': 'ns_1@172.23.122.117,ns_1@172.23.122.118,ns_1@172.23.122.82,ns_1@172.23.122.90,ns_1@172.23.122.97,ns_1@172.23.123.61', 'ejectedNodes': 'ns_1@172.23.122.117,ns_1@172.23.122.118,ns_1@172.23.122.82,ns_1@172.23.122.90,ns_1@172.23.122.97', 'user': 'Administrator', 'password': 'password'} [2021-02-12 22:15:19,578] - [rest_client] [140099299809024] - INFO - rebalance operation started [2021-02-12 22:15:19,581] - [rest_client] [140099299809024] - INFO - rebalance percentage : 0.00 % [2021-02-12 22:15:29,588] - [rest_client] [140099299809024] - INFO - rebalance percentage : 41.00 % [2021-02-12 22:15:39,601] - [rest_client] [140099299809024] - INFO - rebalance percentage : 91.00 % [2021-02-12 22:15:59,627] - [rest_client] [140099299809024] - INFO - rebalance progress took 40.05 seconds [2021-02-12 22:15:59,627] - [rest_client] [140099299809024] - INFO - sleep for 10 seconds after rebalance... [2021-02-12 22:16:09,667] - [cluster_helper] [140099299809024] - INFO - removed all the nodes from cluster associated with ip:172.23.123.61 port:8091 ssh_username:root ? [('ns_1@172.23.122.117', 8091), ('ns_1@172.23.122.118', 8091), ('ns_1@172.23.122.82', 8091), ('ns_1@172.23.122.90', 8091), ('ns_1@172.23.122.97', 8091)] [2021-02-12 22:16:09,672] - [cluster_helper] [140099299809024] - INFO - waiting for ns_server @ 172.23.123.61:8091 [2021-02-12 22:16:09,672] - [rest_client] [140099299809024] - INFO - -->is_ns_server_running? [2021-02-12 22:16:09,676] - [cluster_helper] [140099299809024] - INFO - ns_server @ 172.23.123.61:8091 is running [2021-02-12 22:16:09,680] - [cluster_helper] [140099299809024] - INFO - waiting for ns_server @ 172.23.123.62:8091 [2021-02-12 22:16:09,680] - [rest_client] [140099299809024] - INFO - -->is_ns_server_running? [2021-02-12 22:16:09,683] - [cluster_helper] [140099299809024] - INFO - ns_server @ 172.23.123.62:8091 is running [2021-02-12 22:16:09,686] - [cluster_helper] [140099299809024] - INFO - waiting for ns_server @ 172.23.122.90:8091 [2021-02-12 22:16:09,686] - [rest_client] [140099299809024] - INFO - -->is_ns_server_running? [2021-02-12 22:16:09,688] - [cluster_helper] [140099299809024] - INFO - ns_server @ 172.23.122.90:8091 is running [2021-02-12 22:16:09,691] - [cluster_helper] [140099299809024] - INFO - waiting for ns_server @ 172.23.122.117:8091 [2021-02-12 22:16:09,691] - [rest_client] [140099299809024] - INFO - -->is_ns_server_running? [2021-02-12 22:16:09,694] - [cluster_helper] [140099299809024] - INFO - ns_server @ 172.23.122.117:8091 is running [2021-02-12 22:16:09,697] - [cluster_helper] [140099299809024] - INFO - waiting for ns_server @ 172.23.122.97:8091 [2021-02-12 22:16:09,697] - [rest_client] [140099299809024] - INFO - -->is_ns_server_running? [2021-02-12 22:16:09,700] - [cluster_helper] [140099299809024] - INFO - ns_server @ 172.23.122.97:8091 is running [2021-02-12 22:16:09,703] - [cluster_helper] [140099299809024] - INFO - waiting for ns_server @ 172.23.122.82:8091 [2021-02-12 22:16:09,703] - [rest_client] [140099299809024] - INFO - -->is_ns_server_running? [2021-02-12 22:16:09,705] - [cluster_helper] [140099299809024] - INFO - ns_server @ 172.23.122.82:8091 is running [2021-02-12 22:16:09,708] - [cluster_helper] [140099299809024] - INFO - waiting for ns_server @ 172.23.122.118:8091 [2021-02-12 22:16:09,709] - [rest_client] [140099299809024] - INFO - -->is_ns_server_running? [2021-02-12 22:16:09,711] - [cluster_helper] [140099299809024] - INFO - ns_server @ 172.23.122.118:8091 is running [2021-02-12 22:16:09,715] - [cluster_helper] [140099299809024] - INFO - waiting for ns_server @ 172.23.122.116:8091 [2021-02-12 22:16:09,715] - [rest_client] [140099299809024] - INFO - -->is_ns_server_running? [2021-02-12 22:16:09,717] - [cluster_helper] [140099299809024] - INFO - ns_server @ 172.23.122.116:8091 is running [2021-02-12 22:16:09,717] - [basetestcase] [140099299809024] - INFO - ============== basetestcase cleanup was finished for test #6 test_kv_swap_rebalance_when_existing_eventing_node_is_processing_mutations ============== [2021-02-12 22:16:09,717] - [basetestcase] [140099299809024] - INFO - closing all ssh connections [2021-02-12 22:16:09,719] - [basetestcase] [140099299809024] - INFO - closing all memcached connections [2021-02-12 22:16:09,720] - [remote_util] [140099299809024] - INFO - SSH Connecting to 172.23.123.61 with username:root, attempt#1 of 5 [2021-02-12 22:16:09,829] - [remote_util] [140099299809024] - INFO - SSH Connected to 172.23.123.61 as root [2021-02-12 22:16:10,122] - [remote_util] [140099299809024] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 22:16:10,433] - [remote_util] [140099299809024] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 22:16:10,443] - [remote_util] [140099299809024] - INFO - SSH Connecting to 172.23.123.61 with username:root, attempt#1 of 5 [2021-02-12 22:16:10,549] - [remote_util] [140099299809024] - INFO - SSH Connected to 172.23.123.61 as root [2021-02-12 22:16:10,848] - [remote_util] [140099299809024] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 22:16:11,165] - [remote_util] [140099299809024] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 22:16:11,225] - [remote_util] [140099299809024] - INFO - 172.23.123.61 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 22:16:11,285] - [remote_util] [140099299809024] - INFO - Running systemd command on this server [2021-02-12 22:16:11,285] - [remote_util] [140099299809024] - INFO - running command.raw on 172.23.123.61: systemctl stop couchbase-server.service [2021-02-12 22:16:13,139] - [remote_util] [140099299809024] - INFO - command executed successfully with root [2021-02-12 22:16:13,139] - [basetestcase] [140099299809024] - INFO - Couchbase stopped [2021-02-12 22:16:13,139] - [basetestcase] [140099299809024] - INFO - sleep for 5 secs. ... [2021-02-12 22:16:18,145] - [remote_util] [140099299809024] - INFO - running command.raw on 172.23.123.61: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-12 22:16:18,165] - [remote_util] [140099299809024] - INFO - command executed successfully with root [2021-02-12 22:16:18,165] - [remote_util] [140099299809024] - INFO - running command.raw on 172.23.123.61: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-12 22:16:18,228] - [remote_util] [140099299809024] - INFO - command executed successfully with root [2021-02-12 22:16:18,230] - [remote_util] [140099299809024] - INFO - SSH Connecting to 172.23.123.61 with username:root, attempt#1 of 5 [2021-02-12 22:16:18,331] - [remote_util] [140099299809024] - INFO - SSH Connected to 172.23.123.61 as root [2021-02-12 22:16:18,616] - [remote_util] [140099299809024] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 22:16:18,931] - [remote_util] [140099299809024] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 22:16:18,952] - [remote_util] [140099299809024] - INFO - 172.23.123.61 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 22:16:18,952] - [remote_util] [140099299809024] - INFO - Checking for process beam.smp on linux [2021-02-12 22:16:18,992] - [remote_util] [140099299809024] - INFO - Starting couchbase server [2021-02-12 22:16:19,099] - [remote_util] [140099299809024] - INFO - Running systemd command on this server [2021-02-12 22:16:19,099] - [remote_util] [140099299809024] - INFO - running command.raw on 172.23.123.61: systemctl start couchbase-server.service [2021-02-12 22:16:19,127] - [remote_util] [140099299809024] - INFO - command executed successfully with root [2021-02-12 22:16:19,127] - [remote_util] [140099299809024] - INFO - 172.23.123.61:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-12 22:16:24,130] - [remote_util] [140099299809024] - INFO - running command.raw on 172.23.123.61: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-12 22:16:24,159] - [remote_util] [140099299809024] - INFO - command executed successfully with root [2021-02-12 22:16:24,159] - [remote_util] [140099299809024] - INFO - Couchbase server status: [' Process: 107179 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-12 22:16:24,159] - [remote_util] [140099299809024] - INFO - Checking for process beam.smp on linux [2021-02-12 22:16:24,233] - [remote_util] [140099299809024] - INFO - process beam.smp is running on 172.23.123.61: with pid 107353 [2021-02-12 22:16:24,233] - [basetestcase] [140099299809024] - INFO - Couchbase started [2021-02-12 22:16:24,235] - [remote_util] [140099299809024] - INFO - SSH Connecting to 172.23.123.62 with username:root, attempt#1 of 5 [2021-02-12 22:16:24,350] - [remote_util] [140099299809024] - INFO - SSH Connected to 172.23.123.62 as root [2021-02-12 22:16:24,680] - [remote_util] [140099299809024] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 22:16:24,997] - [remote_util] [140099299809024] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 22:16:25,005] - [remote_util] [140099299809024] - INFO - SSH Connecting to 172.23.123.62 with username:root, attempt#1 of 5 [2021-02-12 22:16:25,109] - [remote_util] [140099299809024] - INFO - SSH Connected to 172.23.123.62 as root [2021-02-12 22:16:25,413] - [remote_util] [140099299809024] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 22:16:25,727] - [remote_util] [140099299809024] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 22:16:25,755] - [remote_util] [140099299809024] - INFO - 172.23.123.62 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 22:16:25,825] - [remote_util] [140099299809024] - INFO - Running systemd command on this server [2021-02-12 22:16:25,825] - [remote_util] [140099299809024] - INFO - running command.raw on 172.23.123.62: systemctl stop couchbase-server.service [2021-02-12 22:16:27,540] - [remote_util] [140099299809024] - INFO - command executed successfully with root [2021-02-12 22:16:27,540] - [basetestcase] [140099299809024] - INFO - Couchbase stopped [2021-02-12 22:16:27,540] - [basetestcase] [140099299809024] - INFO - sleep for 5 secs. ... [2021-02-12 22:16:32,546] - [remote_util] [140099299809024] - INFO - running command.raw on 172.23.123.62: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-12 22:16:32,568] - [remote_util] [140099299809024] - INFO - command executed successfully with root [2021-02-12 22:16:32,568] - [remote_util] [140099299809024] - INFO - running command.raw on 172.23.123.62: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-12 22:16:32,622] - [remote_util] [140099299809024] - INFO - command executed successfully with root [2021-02-12 22:16:32,623] - [remote_util] [140099299809024] - INFO - SSH Connecting to 172.23.123.62 with username:root, attempt#1 of 5 [2021-02-12 22:16:32,723] - [remote_util] [140099299809024] - INFO - SSH Connected to 172.23.123.62 as root [2021-02-12 22:16:33,003] - [remote_util] [140099299809024] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 22:16:33,308] - [remote_util] [140099299809024] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 22:16:33,329] - [remote_util] [140099299809024] - INFO - 172.23.123.62 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 22:16:33,329] - [remote_util] [140099299809024] - INFO - Checking for process beam.smp on linux [2021-02-12 22:16:33,373] - [remote_util] [140099299809024] - INFO - Starting couchbase server [2021-02-12 22:16:33,468] - [remote_util] [140099299809024] - INFO - Running systemd command on this server [2021-02-12 22:16:33,468] - [remote_util] [140099299809024] - INFO - running command.raw on 172.23.123.62: systemctl start couchbase-server.service [2021-02-12 22:16:33,492] - [remote_util] [140099299809024] - INFO - command executed successfully with root [2021-02-12 22:16:33,492] - [remote_util] [140099299809024] - INFO - 172.23.123.62:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-12 22:16:38,497] - [remote_util] [140099299809024] - INFO - running command.raw on 172.23.123.62: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-12 22:16:38,525] - [remote_util] [140099299809024] - INFO - command executed successfully with root [2021-02-12 22:16:38,525] - [remote_util] [140099299809024] - INFO - Couchbase server status: [' Process: 118977 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-12 22:16:38,525] - [remote_util] [140099299809024] - INFO - Checking for process beam.smp on linux [2021-02-12 22:16:38,592] - [remote_util] [140099299809024] - INFO - process beam.smp is running on 172.23.123.62: with pid 119118 [2021-02-12 22:16:38,593] - [basetestcase] [140099299809024] - INFO - Couchbase started [2021-02-12 22:16:38,594] - [remote_util] [140099299809024] - INFO - SSH Connecting to 172.23.122.90 with username:root, attempt#1 of 5 [2021-02-12 22:16:38,697] - [remote_util] [140099299809024] - INFO - SSH Connected to 172.23.122.90 as root [2021-02-12 22:16:38,984] - [remote_util] [140099299809024] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 22:16:39,283] - [remote_util] [140099299809024] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 22:16:39,291] - [remote_util] [140099299809024] - INFO - SSH Connecting to 172.23.122.90 with username:root, attempt#1 of 5 [2021-02-12 22:16:39,389] - [remote_util] [140099299809024] - INFO - SSH Connected to 172.23.122.90 as root [2021-02-12 22:16:39,663] - [remote_util] [140099299809024] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 22:16:39,958] - [remote_util] [140099299809024] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 22:16:39,976] - [remote_util] [140099299809024] - INFO - 172.23.122.90 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 22:16:40,028] - [remote_util] [140099299809024] - INFO - Running systemd command on this server [2021-02-12 22:16:40,028] - [remote_util] [140099299809024] - INFO - running command.raw on 172.23.122.90: systemctl stop couchbase-server.service [2021-02-12 22:16:41,833] - [remote_util] [140099299809024] - INFO - command executed successfully with root [2021-02-12 22:16:41,834] - [basetestcase] [140099299809024] - INFO - Couchbase stopped [2021-02-12 22:16:41,834] - [basetestcase] [140099299809024] - INFO - sleep for 5 secs. ... [2021-02-12 22:16:46,839] - [remote_util] [140099299809024] - INFO - running command.raw on 172.23.122.90: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-12 22:16:46,859] - [remote_util] [140099299809024] - INFO - command executed successfully with root [2021-02-12 22:16:46,859] - [remote_util] [140099299809024] - INFO - running command.raw on 172.23.122.90: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-12 22:16:46,912] - [remote_util] [140099299809024] - INFO - command executed successfully with root [2021-02-12 22:16:46,914] - [remote_util] [140099299809024] - INFO - SSH Connecting to 172.23.122.90 with username:root, attempt#1 of 5 [2021-02-12 22:16:47,014] - [remote_util] [140099299809024] - INFO - SSH Connected to 172.23.122.90 as root [2021-02-12 22:16:47,293] - [remote_util] [140099299809024] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 22:16:47,588] - [remote_util] [140099299809024] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 22:16:47,645] - [remote_util] [140099299809024] - INFO - 172.23.122.90 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 22:16:47,645] - [remote_util] [140099299809024] - INFO - Checking for process beam.smp on linux [2021-02-12 22:16:47,682] - [remote_util] [140099299809024] - INFO - Starting couchbase server [2021-02-12 22:16:47,773] - [remote_util] [140099299809024] - INFO - Running systemd command on this server [2021-02-12 22:16:47,774] - [remote_util] [140099299809024] - INFO - running command.raw on 172.23.122.90: systemctl start couchbase-server.service [2021-02-12 22:16:47,795] - [remote_util] [140099299809024] - INFO - command executed successfully with root [2021-02-12 22:16:47,795] - [remote_util] [140099299809024] - INFO - 172.23.122.90:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-12 22:16:52,800] - [remote_util] [140099299809024] - INFO - running command.raw on 172.23.122.90: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-12 22:16:52,828] - [remote_util] [140099299809024] - INFO - command executed successfully with root [2021-02-12 22:16:52,828] - [remote_util] [140099299809024] - INFO - Couchbase server status: [' Process: 97454 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-12 22:16:52,828] - [remote_util] [140099299809024] - INFO - Checking for process beam.smp on linux [2021-02-12 22:16:52,895] - [remote_util] [140099299809024] - INFO - process beam.smp is running on 172.23.122.90: with pid 97598 [2021-02-12 22:16:52,895] - [basetestcase] [140099299809024] - INFO - Couchbase started [2021-02-12 22:16:52,897] - [remote_util] [140099299809024] - INFO - SSH Connecting to 172.23.122.117 with username:root, attempt#1 of 5 [2021-02-12 22:16:52,998] - [remote_util] [140099299809024] - INFO - SSH Connected to 172.23.122.117 as root [2021-02-12 22:16:53,250] - [remote_util] [140099299809024] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 22:16:53,548] - [remote_util] [140099299809024] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 22:16:53,557] - [remote_util] [140099299809024] - INFO - SSH Connecting to 172.23.122.117 with username:root, attempt#1 of 5 [2021-02-12 22:16:53,656] - [remote_util] [140099299809024] - INFO - SSH Connected to 172.23.122.117 as root [2021-02-12 22:16:53,910] - [remote_util] [140099299809024] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 22:16:54,206] - [remote_util] [140099299809024] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 22:16:54,266] - [remote_util] [140099299809024] - INFO - 172.23.122.117 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 22:16:54,321] - [remote_util] [140099299809024] - INFO - Running systemd command on this server [2021-02-12 22:16:54,321] - [remote_util] [140099299809024] - INFO - running command.raw on 172.23.122.117: systemctl stop couchbase-server.service [2021-02-12 22:16:56,024] - [remote_util] [140099299809024] - INFO - command executed successfully with root [2021-02-12 22:16:56,024] - [basetestcase] [140099299809024] - INFO - Couchbase stopped [2021-02-12 22:16:56,025] - [basetestcase] [140099299809024] - INFO - sleep for 5 secs. ... [2021-02-12 22:17:01,030] - [remote_util] [140099299809024] - INFO - running command.raw on 172.23.122.117: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-12 22:17:01,051] - [remote_util] [140099299809024] - INFO - command executed successfully with root [2021-02-12 22:17:01,051] - [remote_util] [140099299809024] - INFO - running command.raw on 172.23.122.117: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-12 22:17:01,105] - [remote_util] [140099299809024] - INFO - command executed successfully with root [2021-02-12 22:17:01,107] - [remote_util] [140099299809024] - INFO - SSH Connecting to 172.23.122.117 with username:root, attempt#1 of 5 [2021-02-12 22:17:01,208] - [remote_util] [140099299809024] - INFO - SSH Connected to 172.23.122.117 as root [2021-02-12 22:17:01,473] - [remote_util] [140099299809024] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 22:17:01,774] - [remote_util] [140099299809024] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 22:17:01,794] - [remote_util] [140099299809024] - INFO - 172.23.122.117 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 22:17:01,794] - [remote_util] [140099299809024] - INFO - Checking for process beam.smp on linux [2021-02-12 22:17:01,830] - [remote_util] [140099299809024] - INFO - Starting couchbase server [2021-02-12 22:17:01,923] - [remote_util] [140099299809024] - INFO - Running systemd command on this server [2021-02-12 22:17:01,923] - [remote_util] [140099299809024] - INFO - running command.raw on 172.23.122.117: systemctl start couchbase-server.service [2021-02-12 22:17:01,946] - [remote_util] [140099299809024] - INFO - command executed successfully with root [2021-02-12 22:17:01,946] - [remote_util] [140099299809024] - INFO - 172.23.122.117:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-12 22:17:06,952] - [remote_util] [140099299809024] - INFO - running command.raw on 172.23.122.117: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-12 22:17:06,975] - [remote_util] [140099299809024] - INFO - command executed successfully with root [2021-02-12 22:17:06,975] - [remote_util] [140099299809024] - INFO - Couchbase server status: [' Process: 57858 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-12 22:17:06,975] - [remote_util] [140099299809024] - INFO - Checking for process beam.smp on linux [2021-02-12 22:17:07,039] - [remote_util] [140099299809024] - INFO - process beam.smp is running on 172.23.122.117: with pid 58001 [2021-02-12 22:17:07,039] - [basetestcase] [140099299809024] - INFO - Couchbase started [2021-02-12 22:17:07,040] - [remote_util] [140099299809024] - INFO - SSH Connecting to 172.23.122.97 with username:root, attempt#1 of 5 [2021-02-12 22:17:07,144] - [remote_util] [140099299809024] - INFO - SSH Connected to 172.23.122.97 as root [2021-02-12 22:17:07,410] - [remote_util] [140099299809024] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 22:17:07,703] - [remote_util] [140099299809024] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 22:17:07,710] - [remote_util] [140099299809024] - INFO - SSH Connecting to 172.23.122.97 with username:root, attempt#1 of 5 [2021-02-12 22:17:07,809] - [remote_util] [140099299809024] - INFO - SSH Connected to 172.23.122.97 as root [2021-02-12 22:17:08,062] - [remote_util] [140099299809024] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 22:17:08,358] - [remote_util] [140099299809024] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 22:17:08,378] - [remote_util] [140099299809024] - INFO - 172.23.122.97 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 22:17:08,433] - [remote_util] [140099299809024] - INFO - Running systemd command on this server [2021-02-12 22:17:08,433] - [remote_util] [140099299809024] - INFO - running command.raw on 172.23.122.97: systemctl stop couchbase-server.service [2021-02-12 22:17:10,064] - [remote_util] [140099299809024] - INFO - command executed successfully with root [2021-02-12 22:17:10,065] - [basetestcase] [140099299809024] - INFO - Couchbase stopped [2021-02-12 22:17:10,065] - [basetestcase] [140099299809024] - INFO - sleep for 5 secs. ... [2021-02-12 22:17:15,070] - [remote_util] [140099299809024] - INFO - running command.raw on 172.23.122.97: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-12 22:17:15,090] - [remote_util] [140099299809024] - INFO - command executed successfully with root [2021-02-12 22:17:15,090] - [remote_util] [140099299809024] - INFO - running command.raw on 172.23.122.97: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-12 22:17:15,143] - [remote_util] [140099299809024] - INFO - command executed successfully with root [2021-02-12 22:17:15,144] - [remote_util] [140099299809024] - INFO - SSH Connecting to 172.23.122.97 with username:root, attempt#1 of 5 [2021-02-12 22:17:15,243] - [remote_util] [140099299809024] - INFO - SSH Connected to 172.23.122.97 as root [2021-02-12 22:17:15,503] - [remote_util] [140099299809024] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 22:17:15,801] - [remote_util] [140099299809024] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 22:17:15,821] - [remote_util] [140099299809024] - INFO - 172.23.122.97 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 22:17:15,821] - [remote_util] [140099299809024] - INFO - Checking for process beam.smp on linux [2021-02-12 22:17:15,857] - [remote_util] [140099299809024] - INFO - Starting couchbase server [2021-02-12 22:17:15,951] - [remote_util] [140099299809024] - INFO - Running systemd command on this server [2021-02-12 22:17:15,951] - [remote_util] [140099299809024] - INFO - running command.raw on 172.23.122.97: systemctl start couchbase-server.service [2021-02-12 22:17:15,974] - [remote_util] [140099299809024] - INFO - command executed successfully with root [2021-02-12 22:17:15,974] - [remote_util] [140099299809024] - INFO - 172.23.122.97:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-12 22:17:20,980] - [remote_util] [140099299809024] - INFO - running command.raw on 172.23.122.97: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-12 22:17:21,008] - [remote_util] [140099299809024] - INFO - command executed successfully with root [2021-02-12 22:17:21,008] - [remote_util] [140099299809024] - INFO - Couchbase server status: [' Process: 91510 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-12 22:17:21,008] - [remote_util] [140099299809024] - INFO - Checking for process beam.smp on linux [2021-02-12 22:17:21,076] - [remote_util] [140099299809024] - INFO - process beam.smp is running on 172.23.122.97: with pid 91654 [2021-02-12 22:17:21,076] - [basetestcase] [140099299809024] - INFO - Couchbase started [2021-02-12 22:17:21,078] - [remote_util] [140099299809024] - INFO - SSH Connecting to 172.23.122.82 with username:root, attempt#1 of 5 [2021-02-12 22:17:21,186] - [remote_util] [140099299809024] - INFO - SSH Connected to 172.23.122.82 as root [2021-02-12 22:17:21,461] - [remote_util] [140099299809024] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 22:17:21,757] - [remote_util] [140099299809024] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 22:17:21,766] - [remote_util] [140099299809024] - INFO - SSH Connecting to 172.23.122.82 with username:root, attempt#1 of 5 [2021-02-12 22:17:21,861] - [remote_util] [140099299809024] - INFO - SSH Connected to 172.23.122.82 as root [2021-02-12 22:17:22,125] - [remote_util] [140099299809024] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 22:17:22,428] - [remote_util] [140099299809024] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 22:17:22,448] - [remote_util] [140099299809024] - INFO - 172.23.122.82 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 22:17:22,503] - [remote_util] [140099299809024] - INFO - Running systemd command on this server [2021-02-12 22:17:22,503] - [remote_util] [140099299809024] - INFO - running command.raw on 172.23.122.82: systemctl stop couchbase-server.service [2021-02-12 22:17:24,273] - [remote_util] [140099299809024] - INFO - command executed successfully with root [2021-02-12 22:17:24,273] - [basetestcase] [140099299809024] - INFO - Couchbase stopped [2021-02-12 22:17:24,274] - [basetestcase] [140099299809024] - INFO - sleep for 5 secs. ... [2021-02-12 22:17:29,275] - [remote_util] [140099299809024] - INFO - running command.raw on 172.23.122.82: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-12 22:17:29,293] - [remote_util] [140099299809024] - INFO - command executed successfully with root [2021-02-12 22:17:29,293] - [remote_util] [140099299809024] - INFO - running command.raw on 172.23.122.82: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-12 22:17:29,348] - [remote_util] [140099299809024] - INFO - command executed successfully with root [2021-02-12 22:17:29,350] - [remote_util] [140099299809024] - INFO - SSH Connecting to 172.23.122.82 with username:root, attempt#1 of 5 [2021-02-12 22:17:29,451] - [remote_util] [140099299809024] - INFO - SSH Connected to 172.23.122.82 as root [2021-02-12 22:17:29,726] - [remote_util] [140099299809024] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 22:17:30,025] - [remote_util] [140099299809024] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 22:17:30,045] - [remote_util] [140099299809024] - INFO - 172.23.122.82 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 22:17:30,045] - [remote_util] [140099299809024] - INFO - Checking for process beam.smp on linux [2021-02-12 22:17:30,084] - [remote_util] [140099299809024] - INFO - Starting couchbase server [2021-02-12 22:17:30,182] - [remote_util] [140099299809024] - INFO - Running systemd command on this server [2021-02-12 22:17:30,183] - [remote_util] [140099299809024] - INFO - running command.raw on 172.23.122.82: systemctl start couchbase-server.service [2021-02-12 22:17:30,207] - [remote_util] [140099299809024] - INFO - command executed successfully with root [2021-02-12 22:17:30,207] - [remote_util] [140099299809024] - INFO - 172.23.122.82:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-12 22:17:35,213] - [remote_util] [140099299809024] - INFO - running command.raw on 172.23.122.82: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-12 22:17:35,243] - [remote_util] [140099299809024] - INFO - command executed successfully with root [2021-02-12 22:17:35,243] - [remote_util] [140099299809024] - INFO - Couchbase server status: [' Process: 69509 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-12 22:17:35,243] - [remote_util] [140099299809024] - INFO - Checking for process beam.smp on linux [2021-02-12 22:17:35,312] - [remote_util] [140099299809024] - INFO - process beam.smp is running on 172.23.122.82: with pid 69652 [2021-02-12 22:17:35,312] - [basetestcase] [140099299809024] - INFO - Couchbase started [2021-02-12 22:17:35,314] - [remote_util] [140099299809024] - INFO - SSH Connecting to 172.23.122.118 with username:root, attempt#1 of 5 [2021-02-12 22:17:35,419] - [remote_util] [140099299809024] - INFO - SSH Connected to 172.23.122.118 as root [2021-02-12 22:17:35,675] - [remote_util] [140099299809024] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 22:17:35,970] - [remote_util] [140099299809024] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 22:17:35,978] - [remote_util] [140099299809024] - INFO - SSH Connecting to 172.23.122.118 with username:root, attempt#1 of 5 [2021-02-12 22:17:36,076] - [remote_util] [140099299809024] - INFO - SSH Connected to 172.23.122.118 as root [2021-02-12 22:17:36,324] - [remote_util] [140099299809024] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 22:17:36,614] - [remote_util] [140099299809024] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 22:17:36,631] - [remote_util] [140099299809024] - INFO - 172.23.122.118 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 22:17:36,680] - [remote_util] [140099299809024] - INFO - Running systemd command on this server [2021-02-12 22:17:36,680] - [remote_util] [140099299809024] - INFO - running command.raw on 172.23.122.118: systemctl stop couchbase-server.service [2021-02-12 22:17:38,483] - [remote_util] [140099299809024] - INFO - command executed successfully with root [2021-02-12 22:17:38,483] - [basetestcase] [140099299809024] - INFO - Couchbase stopped [2021-02-12 22:17:38,483] - [basetestcase] [140099299809024] - INFO - sleep for 5 secs. ... [2021-02-12 22:17:43,484] - [remote_util] [140099299809024] - INFO - running command.raw on 172.23.122.118: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-12 22:17:43,499] - [remote_util] [140099299809024] - INFO - command executed successfully with root [2021-02-12 22:17:43,499] - [remote_util] [140099299809024] - INFO - running command.raw on 172.23.122.118: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-12 22:17:43,551] - [remote_util] [140099299809024] - INFO - command executed successfully with root [2021-02-12 22:17:43,553] - [remote_util] [140099299809024] - INFO - SSH Connecting to 172.23.122.118 with username:root, attempt#1 of 5 [2021-02-12 22:17:43,649] - [remote_util] [140099299809024] - INFO - SSH Connected to 172.23.122.118 as root [2021-02-12 22:17:43,900] - [remote_util] [140099299809024] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 22:17:44,200] - [remote_util] [140099299809024] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 22:17:44,258] - [remote_util] [140099299809024] - INFO - 172.23.122.118 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 22:17:44,258] - [remote_util] [140099299809024] - INFO - Checking for process beam.smp on linux [2021-02-12 22:17:44,293] - [remote_util] [140099299809024] - INFO - Starting couchbase server [2021-02-12 22:17:44,391] - [remote_util] [140099299809024] - INFO - Running systemd command on this server [2021-02-12 22:17:44,391] - [remote_util] [140099299809024] - INFO - running command.raw on 172.23.122.118: systemctl start couchbase-server.service [2021-02-12 22:17:44,414] - [remote_util] [140099299809024] - INFO - command executed successfully with root [2021-02-12 22:17:44,414] - [remote_util] [140099299809024] - INFO - 172.23.122.118:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-12 22:17:49,419] - [remote_util] [140099299809024] - INFO - running command.raw on 172.23.122.118: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-12 22:17:49,444] - [remote_util] [140099299809024] - INFO - command executed successfully with root [2021-02-12 22:17:49,444] - [remote_util] [140099299809024] - INFO - Couchbase server status: [' Process: 96169 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-12 22:17:49,445] - [remote_util] [140099299809024] - INFO - Checking for process beam.smp on linux [2021-02-12 22:17:49,514] - [remote_util] [140099299809024] - INFO - process beam.smp is running on 172.23.122.118: with pid 96344 [2021-02-12 22:17:49,514] - [basetestcase] [140099299809024] - INFO - Couchbase started [2021-02-12 22:17:49,516] - [remote_util] [140099299809024] - INFO - SSH Connecting to 172.23.122.116 with username:root, attempt#1 of 5 [2021-02-12 22:17:49,622] - [remote_util] [140099299809024] - INFO - SSH Connected to 172.23.122.116 as root [2021-02-12 22:17:49,902] - [remote_util] [140099299809024] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 22:17:50,206] - [remote_util] [140099299809024] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 22:17:50,215] - [remote_util] [140099299809024] - INFO - SSH Connecting to 172.23.122.116 with username:root, attempt#1 of 5 [2021-02-12 22:17:50,316] - [remote_util] [140099299809024] - INFO - SSH Connected to 172.23.122.116 as root [2021-02-12 22:17:50,582] - [remote_util] [140099299809024] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 22:17:50,884] - [remote_util] [140099299809024] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 22:17:50,945] - [remote_util] [140099299809024] - INFO - 172.23.122.116 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 22:17:51,002] - [remote_util] [140099299809024] - INFO - Running systemd command on this server [2021-02-12 22:17:51,002] - [remote_util] [140099299809024] - INFO - running command.raw on 172.23.122.116: systemctl stop couchbase-server.service [2021-02-12 22:17:52,874] - [remote_util] [140099299809024] - INFO - command executed successfully with root [2021-02-12 22:17:52,875] - [basetestcase] [140099299809024] - INFO - Couchbase stopped [2021-02-12 22:17:52,875] - [basetestcase] [140099299809024] - INFO - sleep for 5 secs. ... [2021-02-12 22:17:57,880] - [remote_util] [140099299809024] - INFO - running command.raw on 172.23.122.116: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-12 22:17:57,897] - [remote_util] [140099299809024] - INFO - command executed successfully with root [2021-02-12 22:17:57,897] - [remote_util] [140099299809024] - INFO - running command.raw on 172.23.122.116: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-12 22:17:57,951] - [remote_util] [140099299809024] - INFO - command executed successfully with root [2021-02-12 22:17:57,953] - [remote_util] [140099299809024] - INFO - SSH Connecting to 172.23.122.116 with username:root, attempt#1 of 5 [2021-02-12 22:17:58,059] - [remote_util] [140099299809024] - INFO - SSH Connected to 172.23.122.116 as root [2021-02-12 22:17:58,344] - [remote_util] [140099299809024] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 22:17:58,641] - [remote_util] [140099299809024] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 22:17:58,660] - [remote_util] [140099299809024] - INFO - 172.23.122.116 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 22:17:58,660] - [remote_util] [140099299809024] - INFO - Checking for process beam.smp on linux [2021-02-12 22:17:58,696] - [remote_util] [140099299809024] - INFO - Starting couchbase server [2021-02-12 22:17:58,790] - [remote_util] [140099299809024] - INFO - Running systemd command on this server [2021-02-12 22:17:58,790] - [remote_util] [140099299809024] - INFO - running command.raw on 172.23.122.116: systemctl start couchbase-server.service [2021-02-12 22:17:58,812] - [remote_util] [140099299809024] - INFO - command executed successfully with root [2021-02-12 22:17:58,812] - [remote_util] [140099299809024] - INFO - 172.23.122.116:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-12 22:18:03,817] - [remote_util] [140099299809024] - INFO - running command.raw on 172.23.122.116: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-12 22:18:03,844] - [remote_util] [140099299809024] - INFO - command executed successfully with root [2021-02-12 22:18:03,844] - [remote_util] [140099299809024] - INFO - Couchbase server status: [' Process: 58467 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-12 22:18:03,845] - [remote_util] [140099299809024] - INFO - Checking for process beam.smp on linux [2021-02-12 22:18:03,911] - [remote_util] [140099299809024] - INFO - process beam.smp is running on 172.23.122.116: with pid 58612 [2021-02-12 22:18:03,911] - [basetestcase] [140099299809024] - INFO - Couchbase started [2021-02-12 22:18:03,911] - [basetestcase] [140099299809024] - INFO - sleep for 10 secs. ... [2021-02-12 22:18:13,945] - [remote_util] [140099341772544] - INFO - SSH Connecting to 172.23.123.62 with username:root, attempt#1 of 5 [2021-02-12 22:18:13,947] - [remote_util] [140099333379840] - INFO - SSH Connecting to 172.23.122.90 with username:root, attempt#1 of 5 [2021-02-12 22:18:13,948] - [remote_util] [140099299809024] - INFO - SSH Connecting to 172.23.123.61 with username:root, attempt#1 of 5 [2021-02-12 22:18:13,950] - [remote_util] [140099316594432] - INFO - SSH Connecting to 172.23.122.97 with username:root, attempt#1 of 5 [2021-02-12 22:18:13,953] - [remote_util] [140099324987136] - INFO - SSH Connecting to 172.23.122.117 with username:root, attempt#1 of 5 [2021-02-12 22:18:13,955] - [remote_util] [140097183020800] - INFO - SSH Connecting to 172.23.122.116 with username:root, attempt#1 of 5 [2021-02-12 22:18:13,956] - [remote_util] [140098737792768] - INFO - SSH Connecting to 172.23.122.82 with username:root, attempt#1 of 5 [2021-02-12 22:18:13,958] - [remote_util] [140098704226048] - INFO - SSH Connecting to 172.23.122.118 with username:root, attempt#1 of 5 [2021-02-12 22:18:14,059] - [remote_util] [140099333379840] - INFO - SSH Connected to 172.23.122.90 as root [2021-02-12 22:18:14,060] - [remote_util] [140099341772544] - INFO - SSH Connected to 172.23.123.62 as root [2021-02-12 22:18:14,061] - [remote_util] [140099299809024] - INFO - SSH Connected to 172.23.123.61 as root [2021-02-12 22:18:14,063] - [remote_util] [140098704226048] - INFO - SSH Connected to 172.23.122.118 as root [2021-02-12 22:18:14,064] - [remote_util] [140097183020800] - INFO - SSH Connected to 172.23.122.116 as root [2021-02-12 22:18:14,065] - [remote_util] [140099324987136] - INFO - SSH Connected to 172.23.122.117 as root [2021-02-12 22:18:14,066] - [remote_util] [140099316594432] - INFO - SSH Connected to 172.23.122.97 as root [2021-02-12 22:18:14,067] - [remote_util] [140098737792768] - INFO - SSH Connected to 172.23.122.82 as root [2021-02-12 22:18:14,361] - [remote_util] [140098704226048] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 22:18:14,362] - [remote_util] [140099324987136] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 22:18:14,394] - [remote_util] [140099316594432] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 22:18:14,395] - [remote_util] [140099299809024] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 22:18:14,399] - [remote_util] [140098737792768] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 22:18:14,410] - [remote_util] [140099333379840] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 22:18:14,414] - [remote_util] [140099341772544] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 22:18:14,417] - [remote_util] [140097183020800] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 22:18:14,711] - [remote_util] [140099333379840] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 22:18:14,712] - [remote_util] [140099333379840] - INFO - running command.raw on 172.23.122.90: /opt/couchbase/bin/cbcollect_info 172.23.122.90-20210212-2218-diag.zip [2021-02-12 22:18:14,715] - [remote_util] [140099299809024] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 22:18:14,716] - [remote_util] [140099299809024] - INFO - running command.raw on 172.23.123.61: /opt/couchbase/bin/cbcollect_info 172.23.123.61-20210212-2218-diag.zip [2021-02-12 22:18:14,719] - [remote_util] [140098737792768] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 22:18:14,720] - [remote_util] [140098737792768] - INFO - running command.raw on 172.23.122.82: /opt/couchbase/bin/cbcollect_info 172.23.122.82-20210212-2218-diag.zip [2021-02-12 22:18:14,727] - [remote_util] [140098704226048] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 22:18:14,727] - [remote_util] [140098704226048] - INFO - running command.raw on 172.23.122.118: /opt/couchbase/bin/cbcollect_info 172.23.122.118-20210212-2218-diag.zip [2021-02-12 22:18:14,728] - [remote_util] [140099324987136] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 22:18:14,729] - [remote_util] [140099324987136] - INFO - running command.raw on 172.23.122.117: /opt/couchbase/bin/cbcollect_info 172.23.122.117-20210212-2218-diag.zip [2021-02-12 22:18:14,745] - [remote_util] [140099316594432] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 22:18:14,745] - [remote_util] [140099316594432] - INFO - running command.raw on 172.23.122.97: /opt/couchbase/bin/cbcollect_info 172.23.122.97-20210212-2218-diag.zip [2021-02-12 22:18:14,764] - [remote_util] [140097183020800] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 22:18:14,765] - [remote_util] [140097183020800] - INFO - running command.raw on 172.23.122.116: /opt/couchbase/bin/cbcollect_info 172.23.122.116-20210212-2218-diag.zip [2021-02-12 22:18:14,775] - [remote_util] [140099341772544] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 22:18:14,775] - [remote_util] [140099341772544] - INFO - running command.raw on 172.23.123.62: /opt/couchbase/bin/cbcollect_info 172.23.123.62-20210212-2218-diag.zip [2021-02-12 22:19:42,733] - [remote_util] [140097183020800] - INFO - command executed successfully with root [2021-02-12 22:19:42,972] - [remote_util] [140097183020800] - INFO - running command.raw on 172.23.122.116: rm -f /root/172.23.122.116-20210212-2218-diag.zip [2021-02-12 22:19:42,988] - [remote_util] [140097183020800] - INFO - command executed successfully with root [2021-02-12 22:19:48,135] - [remote_util] [140099324987136] - INFO - command executed successfully with root [2021-02-12 22:19:48,466] - [remote_util] [140099324987136] - INFO - running command.raw on 172.23.122.117: rm -f /root/172.23.122.117-20210212-2218-diag.zip [2021-02-12 22:19:48,483] - [remote_util] [140099324987136] - INFO - command executed successfully with root [2021-02-12 22:19:50,073] - [remote_util] [140098704226048] - INFO - command executed successfully with root [2021-02-12 22:19:50,497] - [remote_util] [140098704226048] - INFO - running command.raw on 172.23.122.118: rm -f /root/172.23.122.118-20210212-2218-diag.zip [2021-02-12 22:19:50,515] - [remote_util] [140098704226048] - INFO - command executed successfully with root [2021-02-12 22:19:55,938] - [remote_util] [140099341772544] - INFO - command executed successfully with root [2021-02-12 22:19:56,588] - [remote_util] [140099341772544] - INFO - running command.raw on 172.23.123.62: rm -f /root/172.23.123.62-20210212-2218-diag.zip [2021-02-12 22:19:56,612] - [remote_util] [140099341772544] - INFO - command executed successfully with root [2021-02-12 22:19:56,711] - [remote_util] [140098737792768] - INFO - command executed successfully with root [2021-02-12 22:19:57,201] - [remote_util] [140098737792768] - INFO - running command.raw on 172.23.122.82: rm -f /root/172.23.122.82-20210212-2218-diag.zip [2021-02-12 22:19:57,219] - [remote_util] [140098737792768] - INFO - command executed successfully with root [2021-02-12 22:19:58,096] - [remote_util] [140099299809024] - INFO - command executed successfully with root [2021-02-12 22:19:58,615] - [remote_util] [140099299809024] - INFO - running command.raw on 172.23.123.61: rm -f /root/172.23.123.61-20210212-2218-diag.zip [2021-02-12 22:19:58,636] - [remote_util] [140099299809024] - INFO - command executed successfully with root [2021-02-12 22:19:59,975] - [remote_util] [140099333379840] - INFO - command executed successfully with root [2021-02-12 22:20:00,611] - [remote_util] [140099333379840] - INFO - running command.raw on 172.23.122.90: rm -f /root/172.23.122.90-20210212-2218-diag.zip [2021-02-12 22:20:00,631] - [remote_util] [140099333379840] - INFO - command executed successfully with root [2021-02-12 22:20:01,964] - [remote_util] [140099316594432] - INFO - command executed successfully with root [2021-02-12 22:20:02,736] - [remote_util] [140099316594432] - INFO - running command.raw on 172.23.122.97: rm -f /root/172.23.122.97-20210212-2218-diag.zip [2021-02-12 22:20:02,759] - [remote_util] [140099316594432] - INFO - command executed successfully with root