[2021-02-12 23:03:51,979] - [eventing_base] [140099324987136] - INFO - Starting Test: test_eventing_rebalance_in_delete_recreate_collections [2021-02-12 23:03:51,980] - [remote_util] [140099324987136] - INFO - SSH Connecting to 172.23.123.61 with username:root, attempt#1 of 5 [2021-02-12 23:03:52,084] - [remote_util] [140099324987136] - INFO - SSH Connected to 172.23.123.61 as root [2021-02-12 23:03:52,353] - [remote_util] [140099324987136] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:03:52,654] - [remote_util] [140099324987136] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:03:52,659] - [rest_client] [140099324987136] - INFO - -->is_ns_server_running? [2021-02-12 23:03:52,666] - [rest_client] [140099324987136] - INFO - Node version in cluster 7.0.0-4454-enterprise [2021-02-12 23:03:52,670] - [rest_client] [140099324987136] - ERROR - GET http://172.23.123.61:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-02-12 23:03:52,670] - [rest_client] [140099324987136] - INFO - Node versions in cluster [] [2021-02-12 23:03:52,670] - [basetestcase] [140099324987136] - INFO - ============== basetestcase setup was started for test #9 test_eventing_rebalance_in_delete_recreate_collections============== [2021-02-12 23:03:52,674] - [rest_client] [140099324987136] - ERROR - GET http://172.23.123.61:8091/pools/default/ body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-02-12 23:03:52,687] - [bucket_helper] [140099324987136] - INFO - Could not find any buckets for node 172.23.123.61, nothing to delete [2021-02-12 23:03:52,697] - [bucket_helper] [140099324987136] - INFO - Could not find any buckets for node 172.23.123.62, nothing to delete [2021-02-12 23:03:52,707] - [bucket_helper] [140099324987136] - INFO - Could not find any buckets for node 172.23.122.90, nothing to delete [2021-02-12 23:03:52,717] - [bucket_helper] [140099324987136] - INFO - Could not find any buckets for node 172.23.122.117, nothing to delete [2021-02-12 23:03:52,727] - [bucket_helper] [140099324987136] - INFO - Could not find any buckets for node 172.23.122.97, nothing to delete [2021-02-12 23:03:52,737] - [bucket_helper] [140099324987136] - INFO - Could not find any buckets for node 172.23.122.82, nothing to delete [2021-02-12 23:03:52,747] - [bucket_helper] [140099324987136] - INFO - Could not find any buckets for node 172.23.122.118, nothing to delete [2021-02-12 23:03:52,757] - [bucket_helper] [140099324987136] - INFO - Could not find any buckets for node 172.23.122.116, nothing to delete [2021-02-12 23:03:52,760] - [rest_client] [140099324987136] - INFO - -->is_ns_server_running? [2021-02-12 23:03:52,772] - [cluster_helper] [140099324987136] - INFO - waiting for ns_server @ 172.23.123.61:8091 [2021-02-12 23:03:52,772] - [rest_client] [140099324987136] - INFO - -->is_ns_server_running? [2021-02-12 23:03:52,775] - [cluster_helper] [140099324987136] - INFO - ns_server @ 172.23.123.61:8091 is running [2021-02-12 23:03:52,778] - [cluster_helper] [140099324987136] - INFO - waiting for ns_server @ 172.23.123.62:8091 [2021-02-12 23:03:52,778] - [rest_client] [140099324987136] - INFO - -->is_ns_server_running? [2021-02-12 23:03:52,780] - [cluster_helper] [140099324987136] - INFO - ns_server @ 172.23.123.62:8091 is running [2021-02-12 23:03:52,784] - [cluster_helper] [140099324987136] - INFO - waiting for ns_server @ 172.23.122.90:8091 [2021-02-12 23:03:52,784] - [rest_client] [140099324987136] - INFO - -->is_ns_server_running? [2021-02-12 23:03:52,786] - [cluster_helper] [140099324987136] - INFO - ns_server @ 172.23.122.90:8091 is running [2021-02-12 23:03:52,789] - [cluster_helper] [140099324987136] - INFO - waiting for ns_server @ 172.23.122.117:8091 [2021-02-12 23:03:52,789] - [rest_client] [140099324987136] - INFO - -->is_ns_server_running? [2021-02-12 23:03:52,792] - [cluster_helper] [140099324987136] - INFO - ns_server @ 172.23.122.117:8091 is running [2021-02-12 23:03:52,795] - [cluster_helper] [140099324987136] - INFO - waiting for ns_server @ 172.23.122.97:8091 [2021-02-12 23:03:52,795] - [rest_client] [140099324987136] - INFO - -->is_ns_server_running? [2021-02-12 23:03:52,798] - [cluster_helper] [140099324987136] - INFO - ns_server @ 172.23.122.97:8091 is running [2021-02-12 23:03:52,801] - [cluster_helper] [140099324987136] - INFO - waiting for ns_server @ 172.23.122.82:8091 [2021-02-12 23:03:52,801] - [rest_client] [140099324987136] - INFO - -->is_ns_server_running? [2021-02-12 23:03:52,804] - [cluster_helper] [140099324987136] - INFO - ns_server @ 172.23.122.82:8091 is running [2021-02-12 23:03:52,806] - [cluster_helper] [140099324987136] - INFO - waiting for ns_server @ 172.23.122.118:8091 [2021-02-12 23:03:52,807] - [rest_client] [140099324987136] - INFO - -->is_ns_server_running? [2021-02-12 23:03:52,809] - [cluster_helper] [140099324987136] - INFO - ns_server @ 172.23.122.118:8091 is running [2021-02-12 23:03:52,812] - [cluster_helper] [140099324987136] - INFO - waiting for ns_server @ 172.23.122.116:8091 [2021-02-12 23:03:52,812] - [rest_client] [140099324987136] - INFO - -->is_ns_server_running? [2021-02-12 23:03:52,815] - [cluster_helper] [140099324987136] - INFO - ns_server @ 172.23.122.116:8091 is running [2021-02-12 23:03:52,815] - [basetestcase] [140099324987136] - INFO - Building docker image with java sdk client [2021-02-12 23:03:56,379] - [basetestcase] [140099324987136] - INFO - initializing cluster [2021-02-12 23:03:56,381] - [remote_util] [140099324987136] - INFO - SSH Connecting to 172.23.123.61 with username:root, attempt#1 of 5 [2021-02-12 23:03:56,487] - [remote_util] [140099324987136] - INFO - SSH Connected to 172.23.123.61 as root [2021-02-12 23:03:56,761] - [remote_util] [140099324987136] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:03:57,057] - [remote_util] [140099324987136] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:03:57,065] - [remote_util] [140099324987136] - INFO - SSH Connecting to 172.23.123.61 with username:root, attempt#1 of 5 [2021-02-12 23:03:57,161] - [remote_util] [140099324987136] - INFO - SSH Connected to 172.23.123.61 as root [2021-02-12 23:03:57,424] - [remote_util] [140099324987136] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:03:57,714] - [remote_util] [140099324987136] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:03:57,732] - [remote_util] [140099324987136] - INFO - 172.23.123.61 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 23:03:57,781] - [remote_util] [140099324987136] - INFO - Running systemd command on this server [2021-02-12 23:03:57,781] - [remote_util] [140099324987136] - INFO - running command.raw on 172.23.123.61: systemctl stop couchbase-server.service [2021-02-12 23:03:59,577] - [remote_util] [140099324987136] - INFO - command executed successfully with root [2021-02-12 23:03:59,577] - [basetestcase] [140099324987136] - INFO - Couchbase stopped [2021-02-12 23:03:59,577] - [basetestcase] [140099324987136] - INFO - sleep for 5 secs. ... [2021-02-12 23:04:04,583] - [remote_util] [140099324987136] - INFO - running command.raw on 172.23.123.61: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-12 23:04:04,599] - [remote_util] [140099324987136] - INFO - command executed successfully with root [2021-02-12 23:04:04,599] - [remote_util] [140099324987136] - INFO - running command.raw on 172.23.123.61: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-12 23:04:04,653] - [remote_util] [140099324987136] - INFO - command executed successfully with root [2021-02-12 23:04:04,654] - [remote_util] [140099324987136] - INFO - SSH Connecting to 172.23.123.61 with username:root, attempt#1 of 5 [2021-02-12 23:04:04,754] - [remote_util] [140099324987136] - INFO - SSH Connected to 172.23.123.61 as root [2021-02-12 23:04:05,017] - [remote_util] [140099324987136] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:04:05,312] - [remote_util] [140099324987136] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:04:05,371] - [remote_util] [140099324987136] - INFO - 172.23.123.61 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 23:04:05,371] - [remote_util] [140099324987136] - INFO - Checking for process beam.smp on linux [2021-02-12 23:04:05,408] - [remote_util] [140099324987136] - INFO - Starting couchbase server [2021-02-12 23:04:05,502] - [remote_util] [140099324987136] - INFO - Running systemd command on this server [2021-02-12 23:04:05,503] - [remote_util] [140099324987136] - INFO - running command.raw on 172.23.123.61: systemctl start couchbase-server.service [2021-02-12 23:04:05,525] - [remote_util] [140099324987136] - INFO - command executed successfully with root [2021-02-12 23:04:05,525] - [remote_util] [140099324987136] - INFO - 172.23.123.61:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-12 23:04:10,530] - [remote_util] [140099324987136] - INFO - running command.raw on 172.23.123.61: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-12 23:04:10,558] - [remote_util] [140099324987136] - INFO - command executed successfully with root [2021-02-12 23:04:10,559] - [remote_util] [140099324987136] - INFO - Couchbase server status: [' Process: 1722 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-12 23:04:10,559] - [remote_util] [140099324987136] - INFO - Checking for process beam.smp on linux [2021-02-12 23:04:10,625] - [remote_util] [140099324987136] - INFO - process beam.smp is running on 172.23.123.61: with pid 1865 [2021-02-12 23:04:10,626] - [basetestcase] [140099324987136] - INFO - Couchbase started [2021-02-12 23:04:10,627] - [remote_util] [140099324987136] - INFO - SSH Connecting to 172.23.123.62 with username:root, attempt#1 of 5 [2021-02-12 23:04:10,735] - [remote_util] [140099324987136] - INFO - SSH Connected to 172.23.123.62 as root [2021-02-12 23:04:11,026] - [remote_util] [140099324987136] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:04:11,330] - [remote_util] [140099324987136] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:04:11,338] - [remote_util] [140099324987136] - INFO - SSH Connecting to 172.23.123.62 with username:root, attempt#1 of 5 [2021-02-12 23:04:11,443] - [remote_util] [140099324987136] - INFO - SSH Connected to 172.23.123.62 as root [2021-02-12 23:04:11,726] - [remote_util] [140099324987136] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:04:12,032] - [remote_util] [140099324987136] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:04:12,091] - [remote_util] [140099324987136] - INFO - 172.23.123.62 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 23:04:12,147] - [remote_util] [140099324987136] - INFO - Running systemd command on this server [2021-02-12 23:04:12,147] - [remote_util] [140099324987136] - INFO - running command.raw on 172.23.123.62: systemctl stop couchbase-server.service [2021-02-12 23:04:13,700] - [remote_util] [140099324987136] - INFO - command executed successfully with root [2021-02-12 23:04:13,701] - [basetestcase] [140099324987136] - INFO - Couchbase stopped [2021-02-12 23:04:13,701] - [basetestcase] [140099324987136] - INFO - sleep for 5 secs. ... [2021-02-12 23:04:18,706] - [remote_util] [140099324987136] - INFO - running command.raw on 172.23.123.62: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-12 23:04:18,725] - [remote_util] [140099324987136] - INFO - command executed successfully with root [2021-02-12 23:04:18,725] - [remote_util] [140099324987136] - INFO - running command.raw on 172.23.123.62: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-12 23:04:18,777] - [remote_util] [140099324987136] - INFO - command executed successfully with root [2021-02-12 23:04:18,779] - [remote_util] [140099324987136] - INFO - SSH Connecting to 172.23.123.62 with username:root, attempt#1 of 5 [2021-02-12 23:04:18,882] - [remote_util] [140099324987136] - INFO - SSH Connected to 172.23.123.62 as root [2021-02-12 23:04:19,161] - [remote_util] [140099324987136] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:04:19,467] - [remote_util] [140099324987136] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:04:19,489] - [remote_util] [140099324987136] - INFO - 172.23.123.62 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 23:04:19,489] - [remote_util] [140099324987136] - INFO - Checking for process beam.smp on linux [2021-02-12 23:04:19,527] - [remote_util] [140099324987136] - INFO - Starting couchbase server [2021-02-12 23:04:19,624] - [remote_util] [140099324987136] - INFO - Running systemd command on this server [2021-02-12 23:04:19,624] - [remote_util] [140099324987136] - INFO - running command.raw on 172.23.123.62: systemctl start couchbase-server.service [2021-02-12 23:04:19,649] - [remote_util] [140099324987136] - INFO - command executed successfully with root [2021-02-12 23:04:19,649] - [remote_util] [140099324987136] - INFO - 172.23.123.62:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-12 23:04:24,654] - [remote_util] [140099324987136] - INFO - running command.raw on 172.23.123.62: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-12 23:04:24,682] - [remote_util] [140099324987136] - INFO - command executed successfully with root [2021-02-12 23:04:24,682] - [remote_util] [140099324987136] - INFO - Couchbase server status: [' Process: 22310 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-12 23:04:24,682] - [remote_util] [140099324987136] - INFO - Checking for process beam.smp on linux [2021-02-12 23:04:24,750] - [remote_util] [140099324987136] - INFO - process beam.smp is running on 172.23.123.62: with pid 22497 [2021-02-12 23:04:24,751] - [basetestcase] [140099324987136] - INFO - Couchbase started [2021-02-12 23:04:24,752] - [remote_util] [140099324987136] - INFO - SSH Connecting to 172.23.122.90 with username:root, attempt#1 of 5 [2021-02-12 23:04:24,856] - [remote_util] [140099324987136] - INFO - SSH Connected to 172.23.122.90 as root [2021-02-12 23:04:25,148] - [remote_util] [140099324987136] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:04:25,448] - [remote_util] [140099324987136] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:04:25,456] - [remote_util] [140099324987136] - INFO - SSH Connecting to 172.23.122.90 with username:root, attempt#1 of 5 [2021-02-12 23:04:25,555] - [remote_util] [140099324987136] - INFO - SSH Connected to 172.23.122.90 as root [2021-02-12 23:04:25,828] - [remote_util] [140099324987136] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:04:26,125] - [remote_util] [140099324987136] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:04:26,183] - [remote_util] [140099324987136] - INFO - 172.23.122.90 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 23:04:26,277] - [remote_util] [140099324987136] - INFO - Running systemd command on this server [2021-02-12 23:04:26,277] - [remote_util] [140099324987136] - INFO - running command.raw on 172.23.122.90: systemctl stop couchbase-server.service [2021-02-12 23:04:29,636] - [remote_util] [140099324987136] - INFO - command executed successfully with root [2021-02-12 23:04:29,637] - [basetestcase] [140099324987136] - INFO - Couchbase stopped [2021-02-12 23:04:29,637] - [basetestcase] [140099324987136] - INFO - sleep for 5 secs. ... [2021-02-12 23:04:34,642] - [remote_util] [140099324987136] - INFO - running command.raw on 172.23.122.90: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-12 23:04:34,658] - [remote_util] [140099324987136] - INFO - command executed successfully with root [2021-02-12 23:04:34,659] - [remote_util] [140099324987136] - INFO - running command.raw on 172.23.122.90: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-12 23:04:34,710] - [remote_util] [140099324987136] - INFO - command executed successfully with root [2021-02-12 23:04:34,712] - [remote_util] [140099324987136] - INFO - SSH Connecting to 172.23.122.90 with username:root, attempt#1 of 5 [2021-02-12 23:04:34,809] - [remote_util] [140099324987136] - INFO - SSH Connected to 172.23.122.90 as root [2021-02-12 23:04:35,086] - [remote_util] [140099324987136] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:04:35,384] - [remote_util] [140099324987136] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:04:35,442] - [remote_util] [140099324987136] - INFO - 172.23.122.90 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 23:04:35,443] - [remote_util] [140099324987136] - INFO - Checking for process beam.smp on linux [2021-02-12 23:04:35,480] - [remote_util] [140099324987136] - INFO - Starting couchbase server [2021-02-12 23:04:35,574] - [remote_util] [140099324987136] - INFO - Running systemd command on this server [2021-02-12 23:04:35,574] - [remote_util] [140099324987136] - INFO - running command.raw on 172.23.122.90: systemctl start couchbase-server.service [2021-02-12 23:04:35,595] - [remote_util] [140099324987136] - INFO - command executed successfully with root [2021-02-12 23:04:35,596] - [remote_util] [140099324987136] - INFO - 172.23.122.90:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-12 23:04:40,600] - [remote_util] [140099324987136] - INFO - running command.raw on 172.23.122.90: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-12 23:04:40,623] - [remote_util] [140099324987136] - INFO - command executed successfully with root [2021-02-12 23:04:40,623] - [remote_util] [140099324987136] - INFO - Couchbase server status: [' Process: 918 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-12 23:04:40,623] - [remote_util] [140099324987136] - INFO - Checking for process beam.smp on linux [2021-02-12 23:04:40,690] - [remote_util] [140099324987136] - INFO - process beam.smp is running on 172.23.122.90: with pid 1070 [2021-02-12 23:04:40,690] - [basetestcase] [140099324987136] - INFO - Couchbase started [2021-02-12 23:04:40,692] - [remote_util] [140099324987136] - INFO - SSH Connecting to 172.23.122.117 with username:root, attempt#1 of 5 [2021-02-12 23:04:40,793] - [remote_util] [140099324987136] - INFO - SSH Connected to 172.23.122.117 as root [2021-02-12 23:04:41,043] - [remote_util] [140099324987136] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:04:41,336] - [remote_util] [140099324987136] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:04:41,343] - [remote_util] [140099324987136] - INFO - SSH Connecting to 172.23.122.117 with username:root, attempt#1 of 5 [2021-02-12 23:04:41,438] - [remote_util] [140099324987136] - INFO - SSH Connected to 172.23.122.117 as root [2021-02-12 23:04:41,688] - [remote_util] [140099324987136] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:04:41,979] - [remote_util] [140099324987136] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:04:42,037] - [remote_util] [140099324987136] - INFO - 172.23.122.117 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 23:04:42,089] - [remote_util] [140099324987136] - INFO - Running systemd command on this server [2021-02-12 23:04:42,089] - [remote_util] [140099324987136] - INFO - running command.raw on 172.23.122.117: systemctl stop couchbase-server.service [2021-02-12 23:04:43,945] - [remote_util] [140099324987136] - INFO - command executed successfully with root [2021-02-12 23:04:43,945] - [basetestcase] [140099324987136] - INFO - Couchbase stopped [2021-02-12 23:04:43,945] - [basetestcase] [140099324987136] - INFO - sleep for 5 secs. ... [2021-02-12 23:04:48,950] - [remote_util] [140099324987136] - INFO - running command.raw on 172.23.122.117: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-12 23:04:48,966] - [remote_util] [140099324987136] - INFO - command executed successfully with root [2021-02-12 23:04:48,967] - [remote_util] [140099324987136] - INFO - running command.raw on 172.23.122.117: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-12 23:04:49,020] - [remote_util] [140099324987136] - INFO - command executed successfully with root [2021-02-12 23:04:49,022] - [remote_util] [140099324987136] - INFO - SSH Connecting to 172.23.122.117 with username:root, attempt#1 of 5 [2021-02-12 23:04:49,121] - [remote_util] [140099324987136] - INFO - SSH Connected to 172.23.122.117 as root [2021-02-12 23:04:49,377] - [remote_util] [140099324987136] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:04:49,673] - [remote_util] [140099324987136] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:04:49,732] - [remote_util] [140099324987136] - INFO - 172.23.122.117 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 23:04:49,732] - [remote_util] [140099324987136] - INFO - Checking for process beam.smp on linux [2021-02-12 23:04:49,767] - [remote_util] [140099324987136] - INFO - Starting couchbase server [2021-02-12 23:04:49,860] - [remote_util] [140099324987136] - INFO - Running systemd command on this server [2021-02-12 23:04:49,860] - [remote_util] [140099324987136] - INFO - running command.raw on 172.23.122.117: systemctl start couchbase-server.service [2021-02-12 23:04:49,883] - [remote_util] [140099324987136] - INFO - command executed successfully with root [2021-02-12 23:04:49,883] - [remote_util] [140099324987136] - INFO - 172.23.122.117:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-12 23:04:54,888] - [remote_util] [140099324987136] - INFO - running command.raw on 172.23.122.117: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-12 23:04:54,913] - [remote_util] [140099324987136] - INFO - command executed successfully with root [2021-02-12 23:04:54,913] - [remote_util] [140099324987136] - INFO - Couchbase server status: [' Process: 1702 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-12 23:04:54,913] - [remote_util] [140099324987136] - INFO - Checking for process beam.smp on linux [2021-02-12 23:04:54,981] - [remote_util] [140099324987136] - INFO - process beam.smp is running on 172.23.122.117: with pid 1846 [2021-02-12 23:04:54,981] - [basetestcase] [140099324987136] - INFO - Couchbase started [2021-02-12 23:04:54,982] - [remote_util] [140099324987136] - INFO - SSH Connecting to 172.23.122.97 with username:root, attempt#1 of 5 [2021-02-12 23:04:55,088] - [remote_util] [140099324987136] - INFO - SSH Connected to 172.23.122.97 as root [2021-02-12 23:04:55,353] - [remote_util] [140099324987136] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:04:55,653] - [remote_util] [140099324987136] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:04:55,661] - [remote_util] [140099324987136] - INFO - SSH Connecting to 172.23.122.97 with username:root, attempt#1 of 5 [2021-02-12 23:04:55,762] - [remote_util] [140099324987136] - INFO - SSH Connected to 172.23.122.97 as root [2021-02-12 23:04:56,020] - [remote_util] [140099324987136] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:04:56,320] - [remote_util] [140099324987136] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:04:56,339] - [remote_util] [140099324987136] - INFO - 172.23.122.97 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 23:04:56,393] - [remote_util] [140099324987136] - INFO - Running systemd command on this server [2021-02-12 23:04:56,393] - [remote_util] [140099324987136] - INFO - running command.raw on 172.23.122.97: systemctl stop couchbase-server.service [2021-02-12 23:04:58,028] - [remote_util] [140099324987136] - INFO - command executed successfully with root [2021-02-12 23:04:58,029] - [basetestcase] [140099324987136] - INFO - Couchbase stopped [2021-02-12 23:04:58,029] - [basetestcase] [140099324987136] - INFO - sleep for 5 secs. ... [2021-02-12 23:05:03,033] - [remote_util] [140099324987136] - INFO - running command.raw on 172.23.122.97: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-12 23:05:03,048] - [remote_util] [140099324987136] - INFO - command executed successfully with root [2021-02-12 23:05:03,049] - [remote_util] [140099324987136] - INFO - running command.raw on 172.23.122.97: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-12 23:05:03,103] - [remote_util] [140099324987136] - INFO - command executed successfully with root [2021-02-12 23:05:03,104] - [remote_util] [140099324987136] - INFO - SSH Connecting to 172.23.122.97 with username:root, attempt#1 of 5 [2021-02-12 23:05:03,207] - [remote_util] [140099324987136] - INFO - SSH Connected to 172.23.122.97 as root [2021-02-12 23:05:03,472] - [remote_util] [140099324987136] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:05:03,776] - [remote_util] [140099324987136] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:05:03,797] - [remote_util] [140099324987136] - INFO - 172.23.122.97 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 23:05:03,797] - [remote_util] [140099324987136] - INFO - Checking for process beam.smp on linux [2021-02-12 23:05:03,835] - [remote_util] [140099324987136] - INFO - Starting couchbase server [2021-02-12 23:05:03,934] - [remote_util] [140099324987136] - INFO - Running systemd command on this server [2021-02-12 23:05:03,934] - [remote_util] [140099324987136] - INFO - running command.raw on 172.23.122.97: systemctl start couchbase-server.service [2021-02-12 23:05:03,958] - [remote_util] [140099324987136] - INFO - command executed successfully with root [2021-02-12 23:05:03,958] - [remote_util] [140099324987136] - INFO - 172.23.122.97:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-12 23:05:08,961] - [remote_util] [140099324987136] - INFO - running command.raw on 172.23.122.97: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-12 23:05:08,987] - [remote_util] [140099324987136] - INFO - command executed successfully with root [2021-02-12 23:05:08,987] - [remote_util] [140099324987136] - INFO - Couchbase server status: [' Process: 113097 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-12 23:05:08,987] - [remote_util] [140099324987136] - INFO - Checking for process beam.smp on linux [2021-02-12 23:05:09,056] - [remote_util] [140099324987136] - INFO - process beam.smp is running on 172.23.122.97: with pid 113241 [2021-02-12 23:05:09,057] - [basetestcase] [140099324987136] - INFO - Couchbase started [2021-02-12 23:05:09,058] - [remote_util] [140099324987136] - INFO - SSH Connecting to 172.23.122.82 with username:root, attempt#1 of 5 [2021-02-12 23:05:09,164] - [remote_util] [140099324987136] - INFO - SSH Connected to 172.23.122.82 as root [2021-02-12 23:05:09,437] - [remote_util] [140099324987136] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:05:09,734] - [remote_util] [140099324987136] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:05:09,742] - [remote_util] [140099324987136] - INFO - SSH Connecting to 172.23.122.82 with username:root, attempt#1 of 5 [2021-02-12 23:05:09,841] - [remote_util] [140099324987136] - INFO - SSH Connected to 172.23.122.82 as root [2021-02-12 23:05:10,101] - [remote_util] [140099324987136] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:05:10,402] - [remote_util] [140099324987136] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:05:10,423] - [remote_util] [140099324987136] - INFO - 172.23.122.82 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 23:05:10,483] - [remote_util] [140099324987136] - INFO - Running systemd command on this server [2021-02-12 23:05:10,483] - [remote_util] [140099324987136] - INFO - running command.raw on 172.23.122.82: systemctl stop couchbase-server.service [2021-02-12 23:05:12,338] - [remote_util] [140099324987136] - INFO - command executed successfully with root [2021-02-12 23:05:12,339] - [basetestcase] [140099324987136] - INFO - Couchbase stopped [2021-02-12 23:05:12,339] - [basetestcase] [140099324987136] - INFO - sleep for 5 secs. ... [2021-02-12 23:05:17,344] - [remote_util] [140099324987136] - INFO - running command.raw on 172.23.122.82: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-12 23:05:17,360] - [remote_util] [140099324987136] - INFO - command executed successfully with root [2021-02-12 23:05:17,361] - [remote_util] [140099324987136] - INFO - running command.raw on 172.23.122.82: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-12 23:05:17,414] - [remote_util] [140099324987136] - INFO - command executed successfully with root [2021-02-12 23:05:17,416] - [remote_util] [140099324987136] - INFO - SSH Connecting to 172.23.122.82 with username:root, attempt#1 of 5 [2021-02-12 23:05:17,515] - [remote_util] [140099324987136] - INFO - SSH Connected to 172.23.122.82 as root [2021-02-12 23:05:17,781] - [remote_util] [140099324987136] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:05:18,077] - [remote_util] [140099324987136] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:05:18,096] - [remote_util] [140099324987136] - INFO - 172.23.122.82 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 23:05:18,096] - [remote_util] [140099324987136] - INFO - Checking for process beam.smp on linux [2021-02-12 23:05:18,135] - [remote_util] [140099324987136] - INFO - Starting couchbase server [2021-02-12 23:05:18,229] - [remote_util] [140099324987136] - INFO - Running systemd command on this server [2021-02-12 23:05:18,229] - [remote_util] [140099324987136] - INFO - running command.raw on 172.23.122.82: systemctl start couchbase-server.service [2021-02-12 23:05:18,253] - [remote_util] [140099324987136] - INFO - command executed successfully with root [2021-02-12 23:05:18,253] - [remote_util] [140099324987136] - INFO - 172.23.122.82:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-12 23:05:23,258] - [remote_util] [140099324987136] - INFO - running command.raw on 172.23.122.82: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-12 23:05:23,287] - [remote_util] [140099324987136] - INFO - command executed successfully with root [2021-02-12 23:05:23,287] - [remote_util] [140099324987136] - INFO - Couchbase server status: [' Process: 90277 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-12 23:05:23,287] - [remote_util] [140099324987136] - INFO - Checking for process beam.smp on linux [2021-02-12 23:05:23,356] - [remote_util] [140099324987136] - INFO - process beam.smp is running on 172.23.122.82: with pid 90466 [2021-02-12 23:05:23,356] - [basetestcase] [140099324987136] - INFO - Couchbase started [2021-02-12 23:05:23,358] - [remote_util] [140099324987136] - INFO - SSH Connecting to 172.23.122.118 with username:root, attempt#1 of 5 [2021-02-12 23:05:23,464] - [remote_util] [140099324987136] - INFO - SSH Connected to 172.23.122.118 as root [2021-02-12 23:05:23,725] - [remote_util] [140099324987136] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:05:24,034] - [remote_util] [140099324987136] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:05:24,044] - [remote_util] [140099324987136] - INFO - SSH Connecting to 172.23.122.118 with username:root, attempt#1 of 5 [2021-02-12 23:05:24,143] - [remote_util] [140099324987136] - INFO - SSH Connected to 172.23.122.118 as root [2021-02-12 23:05:24,389] - [remote_util] [140099324987136] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:05:24,689] - [remote_util] [140099324987136] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:05:24,747] - [remote_util] [140099324987136] - INFO - 172.23.122.118 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 23:05:24,800] - [remote_util] [140099324987136] - INFO - Running systemd command on this server [2021-02-12 23:05:24,800] - [remote_util] [140099324987136] - INFO - running command.raw on 172.23.122.118: systemctl stop couchbase-server.service [2021-02-12 23:05:26,587] - [remote_util] [140099324987136] - INFO - command executed successfully with root [2021-02-12 23:05:26,587] - [basetestcase] [140099324987136] - INFO - Couchbase stopped [2021-02-12 23:05:26,588] - [basetestcase] [140099324987136] - INFO - sleep for 5 secs. ... [2021-02-12 23:05:31,591] - [remote_util] [140099324987136] - INFO - running command.raw on 172.23.122.118: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-12 23:05:31,608] - [remote_util] [140099324987136] - INFO - command executed successfully with root [2021-02-12 23:05:31,608] - [remote_util] [140099324987136] - INFO - running command.raw on 172.23.122.118: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-12 23:05:31,661] - [remote_util] [140099324987136] - INFO - command executed successfully with root [2021-02-12 23:05:31,663] - [remote_util] [140099324987136] - INFO - SSH Connecting to 172.23.122.118 with username:root, attempt#1 of 5 [2021-02-12 23:05:31,760] - [remote_util] [140099324987136] - INFO - SSH Connected to 172.23.122.118 as root [2021-02-12 23:05:32,005] - [remote_util] [140099324987136] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:05:32,297] - [remote_util] [140099324987136] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:05:32,315] - [remote_util] [140099324987136] - INFO - 172.23.122.118 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 23:05:32,315] - [remote_util] [140099324987136] - INFO - Checking for process beam.smp on linux [2021-02-12 23:05:32,350] - [remote_util] [140099324987136] - INFO - Starting couchbase server [2021-02-12 23:05:32,443] - [remote_util] [140099324987136] - INFO - Running systemd command on this server [2021-02-12 23:05:32,443] - [remote_util] [140099324987136] - INFO - running command.raw on 172.23.122.118: systemctl start couchbase-server.service [2021-02-12 23:05:32,465] - [remote_util] [140099324987136] - INFO - command executed successfully with root [2021-02-12 23:05:32,465] - [remote_util] [140099324987136] - INFO - 172.23.122.118:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-12 23:05:37,469] - [remote_util] [140099324987136] - INFO - running command.raw on 172.23.122.118: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-12 23:05:37,494] - [remote_util] [140099324987136] - INFO - command executed successfully with root [2021-02-12 23:05:37,495] - [remote_util] [140099324987136] - INFO - Couchbase server status: [' Process: 116726 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-12 23:05:37,495] - [remote_util] [140099324987136] - INFO - Checking for process beam.smp on linux [2021-02-12 23:05:37,561] - [remote_util] [140099324987136] - INFO - process beam.smp is running on 172.23.122.118: with pid 116869 [2021-02-12 23:05:37,561] - [basetestcase] [140099324987136] - INFO - Couchbase started [2021-02-12 23:05:37,563] - [remote_util] [140099324987136] - INFO - SSH Connecting to 172.23.122.116 with username:root, attempt#1 of 5 [2021-02-12 23:05:37,669] - [remote_util] [140099324987136] - INFO - SSH Connected to 172.23.122.116 as root [2021-02-12 23:05:37,949] - [remote_util] [140099324987136] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:05:38,245] - [remote_util] [140099324987136] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:05:38,252] - [remote_util] [140099324987136] - INFO - SSH Connecting to 172.23.122.116 with username:root, attempt#1 of 5 [2021-02-12 23:05:38,350] - [remote_util] [140099324987136] - INFO - SSH Connected to 172.23.122.116 as root [2021-02-12 23:05:38,623] - [remote_util] [140099324987136] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:05:38,916] - [remote_util] [140099324987136] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:05:38,935] - [remote_util] [140099324987136] - INFO - 172.23.122.116 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 23:05:38,987] - [remote_util] [140099324987136] - INFO - Running systemd command on this server [2021-02-12 23:05:38,987] - [remote_util] [140099324987136] - INFO - running command.raw on 172.23.122.116: systemctl stop couchbase-server.service [2021-02-12 23:05:40,840] - [remote_util] [140099324987136] - INFO - command executed successfully with root [2021-02-12 23:05:40,841] - [basetestcase] [140099324987136] - INFO - Couchbase stopped [2021-02-12 23:05:40,841] - [basetestcase] [140099324987136] - INFO - sleep for 5 secs. ... [2021-02-12 23:05:45,846] - [remote_util] [140099324987136] - INFO - running command.raw on 172.23.122.116: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-12 23:05:45,862] - [remote_util] [140099324987136] - INFO - command executed successfully with root [2021-02-12 23:05:45,862] - [remote_util] [140099324987136] - INFO - running command.raw on 172.23.122.116: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-12 23:05:45,915] - [remote_util] [140099324987136] - INFO - command executed successfully with root [2021-02-12 23:05:45,917] - [remote_util] [140099324987136] - INFO - SSH Connecting to 172.23.122.116 with username:root, attempt#1 of 5 [2021-02-12 23:05:46,015] - [remote_util] [140099324987136] - INFO - SSH Connected to 172.23.122.116 as root [2021-02-12 23:05:46,292] - [remote_util] [140099324987136] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:05:46,589] - [remote_util] [140099324987136] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:05:46,608] - [remote_util] [140099324987136] - INFO - 172.23.122.116 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 23:05:46,609] - [remote_util] [140099324987136] - INFO - Checking for process beam.smp on linux [2021-02-12 23:05:46,645] - [remote_util] [140099324987136] - INFO - Starting couchbase server [2021-02-12 23:05:46,739] - [remote_util] [140099324987136] - INFO - Running systemd command on this server [2021-02-12 23:05:46,739] - [remote_util] [140099324987136] - INFO - running command.raw on 172.23.122.116: systemctl start couchbase-server.service [2021-02-12 23:05:46,761] - [remote_util] [140099324987136] - INFO - command executed successfully with root [2021-02-12 23:05:46,761] - [remote_util] [140099324987136] - INFO - 172.23.122.116:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-12 23:05:51,766] - [remote_util] [140099324987136] - INFO - running command.raw on 172.23.122.116: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-12 23:05:51,792] - [remote_util] [140099324987136] - INFO - command executed successfully with root [2021-02-12 23:05:51,792] - [remote_util] [140099324987136] - INFO - Couchbase server status: [' Process: 77026 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-12 23:05:51,792] - [remote_util] [140099324987136] - INFO - Checking for process beam.smp on linux [2021-02-12 23:05:51,857] - [remote_util] [140099324987136] - INFO - process beam.smp is running on 172.23.122.116: with pid 77171 [2021-02-12 23:05:51,858] - [basetestcase] [140099324987136] - INFO - Couchbase started [2021-02-12 23:05:51,858] - [basetestcase] [140099324987136] - INFO - sleep for 10 secs. ... [2021-02-12 23:06:02,108] - [task] [140097183020800] - INFO - server: ip:172.23.123.61 port:8091 ssh_username:root, nodes/self [2021-02-12 23:06:02,111] - [task] [140097183020800] - INFO - {'uptime': '109', 'memoryTotal': 4201672704, 'memoryFree': 3516211200, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.123.61:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4454-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.123.61', 'ip': '172.23.123.61', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} [2021-02-12 23:06:02,113] - [rest_client] [140097183020800] - ERROR - GET http://172.23.123.61:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-02-12 23:06:02,113] - [rest_client] [140097183020800] - INFO - pools/default params : memoryQuota=2147 [2021-02-12 23:06:02,119] - [rest_client] [140097183020800] - INFO - --> init_node_services(Administrator,password,172.23.123.61,8091,['kv']) [2021-02-12 23:06:02,119] - [rest_client] [140097183020800] - INFO - /node/controller/setupServices params on 172.23.123.61: 8091:hostname=172.23.123.61&user=Administrator&password=password&services=kv [2021-02-12 23:06:02,147] - [rest_client] [140097183020800] - INFO - --> in init_cluster...Administrator,password,8091 [2021-02-12 23:06:02,147] - [rest_client] [140097183020800] - INFO - settings/web params on 172.23.123.61:8091:port=8091&username=Administrator&password=password [2021-02-12 23:06:02,200] - [rest_client] [140097183020800] - INFO - --> status:True [2021-02-12 23:06:02,202] - [remote_util] [140097183020800] - INFO - SSH Connecting to 172.23.123.61 with username:root, attempt#1 of 5 [2021-02-12 23:06:02,305] - [remote_util] [140097183020800] - INFO - SSH Connected to 172.23.123.61 as root [2021-02-12 23:06:02,574] - [remote_util] [140097183020800] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:06:02,865] - [remote_util] [140097183020800] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:06:02,873] - [remote_util] [140097183020800] - INFO - running command.raw on 172.23.123.61: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-02-12 23:06:02,930] - [remote_util] [140097183020800] - INFO - command executed successfully with root [2021-02-12 23:06:02,930] - [remote_util] [140097183020800] - INFO - ['ok'] [2021-02-12 23:06:02,932] - [rest_client] [140097183020800] - INFO - /diag/eval status on 172.23.123.61:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-12 23:06:02,934] - [rest_client] [140097183020800] - INFO - /diag/eval status on 172.23.123.61:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-12 23:06:02,938] - [rest_client] [140097183020800] - INFO - settings/indexes params : storageMode=plasma [2021-02-12 23:06:02,955] - [task] [140097183020800] - INFO - server: ip:172.23.123.62 port:8091 ssh_username:root, nodes/self [2021-02-12 23:06:02,958] - [task] [140097183020800] - INFO - {'uptime': '100', 'memoryTotal': 4201676800, 'memoryFree': 3485650944, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.123.62:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4454-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.123.62', 'ip': '172.23.123.62', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} [2021-02-12 23:06:02,959] - [rest_client] [140097183020800] - ERROR - GET http://172.23.123.62:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-02-12 23:06:02,960] - [rest_client] [140097183020800] - INFO - pools/default params : memoryQuota=2147 [2021-02-12 23:06:02,966] - [rest_client] [140097183020800] - INFO - --> in init_cluster...Administrator,password,8091 [2021-02-12 23:06:02,966] - [rest_client] [140097183020800] - INFO - settings/web params on 172.23.123.62:8091:port=8091&username=Administrator&password=password [2021-02-12 23:06:03,024] - [rest_client] [140097183020800] - INFO - --> status:True [2021-02-12 23:06:03,025] - [remote_util] [140097183020800] - INFO - SSH Connecting to 172.23.123.62 with username:root, attempt#1 of 5 [2021-02-12 23:06:03,128] - [remote_util] [140097183020800] - INFO - SSH Connected to 172.23.123.62 as root [2021-02-12 23:06:03,403] - [remote_util] [140097183020800] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:06:03,701] - [remote_util] [140097183020800] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:06:03,710] - [remote_util] [140097183020800] - INFO - running command.raw on 172.23.123.62: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-02-12 23:06:03,766] - [remote_util] [140097183020800] - INFO - command executed successfully with root [2021-02-12 23:06:03,766] - [remote_util] [140097183020800] - INFO - ['ok'] [2021-02-12 23:06:03,768] - [rest_client] [140097183020800] - INFO - /diag/eval status on 172.23.123.62:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-12 23:06:03,770] - [rest_client] [140097183020800] - INFO - /diag/eval status on 172.23.123.62:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-12 23:06:03,774] - [rest_client] [140097183020800] - INFO - settings/indexes params : storageMode=plasma [2021-02-12 23:06:03,792] - [task] [140097183020800] - INFO - server: ip:172.23.122.90 port:8091 ssh_username:root, nodes/self [2021-02-12 23:06:03,795] - [task] [140097183020800] - INFO - {'uptime': '84', 'memoryTotal': 4201684992, 'memoryFree': 3566862336, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.122.90:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4454-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.122.90', 'ip': '172.23.122.90', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} [2021-02-12 23:06:03,796] - [rest_client] [140097183020800] - ERROR - GET http://172.23.122.90:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-02-12 23:06:03,796] - [rest_client] [140097183020800] - INFO - pools/default params : memoryQuota=2147 [2021-02-12 23:06:03,801] - [rest_client] [140097183020800] - INFO - --> in init_cluster...Administrator,password,8091 [2021-02-12 23:06:03,802] - [rest_client] [140097183020800] - INFO - settings/web params on 172.23.122.90:8091:port=8091&username=Administrator&password=password [2021-02-12 23:06:03,855] - [rest_client] [140097183020800] - INFO - --> status:True [2021-02-12 23:06:03,857] - [remote_util] [140097183020800] - INFO - SSH Connecting to 172.23.122.90 with username:root, attempt#1 of 5 [2021-02-12 23:06:03,960] - [remote_util] [140097183020800] - INFO - SSH Connected to 172.23.122.90 as root [2021-02-12 23:06:04,243] - [remote_util] [140097183020800] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:06:04,539] - [remote_util] [140097183020800] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:06:04,547] - [remote_util] [140097183020800] - INFO - running command.raw on 172.23.122.90: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-02-12 23:06:04,603] - [remote_util] [140097183020800] - INFO - command executed successfully with root [2021-02-12 23:06:04,603] - [remote_util] [140097183020800] - INFO - ['ok'] [2021-02-12 23:06:04,605] - [rest_client] [140097183020800] - INFO - /diag/eval status on 172.23.122.90:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-12 23:06:04,607] - [rest_client] [140097183020800] - INFO - /diag/eval status on 172.23.122.90:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-12 23:06:04,610] - [rest_client] [140097183020800] - INFO - settings/indexes params : storageMode=plasma [2021-02-12 23:06:04,628] - [task] [140097183020800] - INFO - server: ip:172.23.122.117 port:8091 ssh_username:root, nodes/self [2021-02-12 23:06:04,631] - [task] [140097183020800] - INFO - {'uptime': '69', 'memoryTotal': 4201684992, 'memoryFree': 3531931648, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.122.117:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4454-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.122.117', 'ip': '172.23.122.117', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} [2021-02-12 23:06:04,633] - [rest_client] [140097183020800] - ERROR - GET http://172.23.122.117:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-02-12 23:06:04,633] - [rest_client] [140097183020800] - INFO - pools/default params : memoryQuota=2147 [2021-02-12 23:06:04,639] - [rest_client] [140097183020800] - INFO - --> in init_cluster...Administrator,password,8091 [2021-02-12 23:06:04,639] - [rest_client] [140097183020800] - INFO - settings/web params on 172.23.122.117:8091:port=8091&username=Administrator&password=password [2021-02-12 23:06:04,694] - [rest_client] [140097183020800] - INFO - --> status:True [2021-02-12 23:06:04,695] - [remote_util] [140097183020800] - INFO - SSH Connecting to 172.23.122.117 with username:root, attempt#1 of 5 [2021-02-12 23:06:04,800] - [remote_util] [140097183020800] - INFO - SSH Connected to 172.23.122.117 as root [2021-02-12 23:06:05,053] - [remote_util] [140097183020800] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:06:05,349] - [remote_util] [140097183020800] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:06:05,357] - [remote_util] [140097183020800] - INFO - running command.raw on 172.23.122.117: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-02-12 23:06:05,413] - [remote_util] [140097183020800] - INFO - command executed successfully with root [2021-02-12 23:06:05,413] - [remote_util] [140097183020800] - INFO - ['ok'] [2021-02-12 23:06:05,415] - [rest_client] [140097183020800] - INFO - /diag/eval status on 172.23.122.117:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-12 23:06:05,417] - [rest_client] [140097183020800] - INFO - /diag/eval status on 172.23.122.117:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-12 23:06:05,421] - [rest_client] [140097183020800] - INFO - settings/indexes params : storageMode=plasma [2021-02-12 23:06:05,439] - [task] [140097183020800] - INFO - server: ip:172.23.122.97 port:8091 ssh_username:root, nodes/self [2021-02-12 23:06:05,442] - [task] [140097183020800] - INFO - {'uptime': '54', 'memoryTotal': 4201684992, 'memoryFree': 3552964608, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.122.97:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4454-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.122.97', 'ip': '172.23.122.97', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} [2021-02-12 23:06:05,444] - [rest_client] [140097183020800] - ERROR - GET http://172.23.122.97:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-02-12 23:06:05,444] - [rest_client] [140097183020800] - INFO - pools/default params : memoryQuota=2147 [2021-02-12 23:06:05,450] - [rest_client] [140097183020800] - INFO - --> in init_cluster...Administrator,password,8091 [2021-02-12 23:06:05,450] - [rest_client] [140097183020800] - INFO - settings/web params on 172.23.122.97:8091:port=8091&username=Administrator&password=password [2021-02-12 23:06:05,506] - [rest_client] [140097183020800] - INFO - --> status:True [2021-02-12 23:06:05,507] - [remote_util] [140097183020800] - INFO - SSH Connecting to 172.23.122.97 with username:root, attempt#1 of 5 [2021-02-12 23:06:05,613] - [remote_util] [140097183020800] - INFO - SSH Connected to 172.23.122.97 as root [2021-02-12 23:06:05,876] - [remote_util] [140097183020800] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:06:06,170] - [remote_util] [140097183020800] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:06:06,178] - [remote_util] [140097183020800] - INFO - running command.raw on 172.23.122.97: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-02-12 23:06:06,233] - [remote_util] [140097183020800] - INFO - command executed successfully with root [2021-02-12 23:06:06,233] - [remote_util] [140097183020800] - INFO - ['ok'] [2021-02-12 23:06:06,235] - [rest_client] [140097183020800] - INFO - /diag/eval status on 172.23.122.97:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-12 23:06:06,237] - [rest_client] [140097183020800] - INFO - /diag/eval status on 172.23.122.97:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-12 23:06:06,240] - [rest_client] [140097183020800] - INFO - settings/indexes params : storageMode=plasma [2021-02-12 23:06:06,258] - [task] [140097183020800] - INFO - server: ip:172.23.122.82 port:8091 ssh_username:root, nodes/self [2021-02-12 23:06:06,261] - [task] [140097183020800] - INFO - {'uptime': '44', 'memoryTotal': 4201684992, 'memoryFree': 3578302464, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.122.82:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4454-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.122.82', 'ip': '172.23.122.82', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} [2021-02-12 23:06:06,263] - [rest_client] [140097183020800] - ERROR - GET http://172.23.122.82:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-02-12 23:06:06,263] - [rest_client] [140097183020800] - INFO - pools/default params : memoryQuota=2147 [2021-02-12 23:06:06,269] - [rest_client] [140097183020800] - INFO - --> in init_cluster...Administrator,password,8091 [2021-02-12 23:06:06,269] - [rest_client] [140097183020800] - INFO - settings/web params on 172.23.122.82:8091:port=8091&username=Administrator&password=password [2021-02-12 23:06:06,327] - [rest_client] [140097183020800] - INFO - --> status:True [2021-02-12 23:06:06,329] - [remote_util] [140097183020800] - INFO - SSH Connecting to 172.23.122.82 with username:root, attempt#1 of 5 [2021-02-12 23:06:06,438] - [remote_util] [140097183020800] - INFO - SSH Connected to 172.23.122.82 as root [2021-02-12 23:06:06,720] - [remote_util] [140097183020800] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:06:07,025] - [remote_util] [140097183020800] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:06:07,032] - [remote_util] [140097183020800] - INFO - running command.raw on 172.23.122.82: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-02-12 23:06:07,091] - [remote_util] [140097183020800] - INFO - command executed successfully with root [2021-02-12 23:06:07,091] - [remote_util] [140097183020800] - INFO - ['ok'] [2021-02-12 23:06:07,094] - [rest_client] [140097183020800] - INFO - /diag/eval status on 172.23.122.82:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-12 23:06:07,095] - [rest_client] [140097183020800] - INFO - /diag/eval status on 172.23.122.82:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-12 23:06:07,100] - [rest_client] [140097183020800] - INFO - settings/indexes params : storageMode=plasma [2021-02-12 23:06:07,123] - [task] [140097183020800] - INFO - server: ip:172.23.122.118 port:8091 ssh_username:root, nodes/self [2021-02-12 23:06:07,125] - [task] [140097183020800] - INFO - {'uptime': '29', 'memoryTotal': 4201689088, 'memoryFree': 3546267648, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.122.118:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4454-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.122.118', 'ip': '172.23.122.118', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} [2021-02-12 23:06:07,127] - [rest_client] [140097183020800] - ERROR - GET http://172.23.122.118:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-02-12 23:06:07,127] - [rest_client] [140097183020800] - INFO - pools/default params : memoryQuota=2147 [2021-02-12 23:06:07,132] - [rest_client] [140097183020800] - INFO - --> in init_cluster...Administrator,password,8091 [2021-02-12 23:06:07,132] - [rest_client] [140097183020800] - INFO - settings/web params on 172.23.122.118:8091:port=8091&username=Administrator&password=password [2021-02-12 23:06:07,189] - [rest_client] [140097183020800] - INFO - --> status:True [2021-02-12 23:06:07,190] - [remote_util] [140097183020800] - INFO - SSH Connecting to 172.23.122.118 with username:root, attempt#1 of 5 [2021-02-12 23:06:07,292] - [remote_util] [140097183020800] - INFO - SSH Connected to 172.23.122.118 as root [2021-02-12 23:06:07,545] - [remote_util] [140097183020800] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:06:07,839] - [remote_util] [140097183020800] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:06:07,847] - [remote_util] [140097183020800] - INFO - running command.raw on 172.23.122.118: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-02-12 23:06:07,903] - [remote_util] [140097183020800] - INFO - command executed successfully with root [2021-02-12 23:06:07,904] - [remote_util] [140097183020800] - INFO - ['ok'] [2021-02-12 23:06:07,906] - [rest_client] [140097183020800] - INFO - /diag/eval status on 172.23.122.118:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-12 23:06:07,908] - [rest_client] [140097183020800] - INFO - /diag/eval status on 172.23.122.118:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-12 23:06:07,911] - [rest_client] [140097183020800] - INFO - settings/indexes params : storageMode=plasma [2021-02-12 23:06:07,930] - [task] [140097183020800] - INFO - server: ip:172.23.122.116 port:8091 ssh_username:root, nodes/self [2021-02-12 23:06:07,933] - [task] [140097183020800] - INFO - {'uptime': '14', 'memoryTotal': 4201689088, 'memoryFree': 3554889728, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.122.116:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4454-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@cb.local', 'ip': '172.23.122.116', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} [2021-02-12 23:06:07,934] - [rest_client] [140097183020800] - ERROR - GET http://172.23.122.116:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-02-12 23:06:07,935] - [rest_client] [140097183020800] - INFO - pools/default params : memoryQuota=2147 [2021-02-12 23:06:07,940] - [rest_client] [140097183020800] - INFO - --> in init_cluster...Administrator,password,8091 [2021-02-12 23:06:07,940] - [rest_client] [140097183020800] - INFO - settings/web params on 172.23.122.116:8091:port=8091&username=Administrator&password=password [2021-02-12 23:06:07,995] - [rest_client] [140097183020800] - INFO - --> status:True [2021-02-12 23:06:07,997] - [remote_util] [140097183020800] - INFO - SSH Connecting to 172.23.122.116 with username:root, attempt#1 of 5 [2021-02-12 23:06:08,101] - [remote_util] [140097183020800] - INFO - SSH Connected to 172.23.122.116 as root [2021-02-12 23:06:08,378] - [remote_util] [140097183020800] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:06:08,674] - [remote_util] [140097183020800] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:06:08,682] - [remote_util] [140097183020800] - INFO - running command.raw on 172.23.122.116: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-02-12 23:06:08,739] - [remote_util] [140097183020800] - INFO - command executed successfully with root [2021-02-12 23:06:08,739] - [remote_util] [140097183020800] - INFO - ['ok'] [2021-02-12 23:06:08,741] - [rest_client] [140097183020800] - INFO - /diag/eval status on 172.23.122.116:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-12 23:06:08,742] - [rest_client] [140097183020800] - INFO - /diag/eval status on 172.23.122.116:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-12 23:06:08,747] - [rest_client] [140097183020800] - INFO - settings/indexes params : storageMode=plasma [2021-02-12 23:06:08,767] - [basetestcase] [140099324987136] - INFO - **** add built-in 'cbadminbucket' user to node 172.23.123.61 **** [2021-02-12 23:06:08,777] - [rest_client] [140099324987136] - ERROR - DELETE http://172.23.123.61:8091/settings/rbac/users/local/cbadminbucket body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:password [2021-02-12 23:06:08,777] - [internal_user] [140099324987136] - INFO - Exception while deleting user. Exception is -b'"User was not found."' [2021-02-12 23:06:08,838] - [basetestcase] [140099324987136] - INFO - sleep for 5 secs. ... [2021-02-12 23:06:13,843] - [basetestcase] [140099324987136] - INFO - **** add 'admin' role to 'cbadminbucket' user **** [2021-02-12 23:06:13,853] - [basetestcase] [140099324987136] - INFO - done initializing cluster [2021-02-12 23:06:14,767] - [task] [140097183020800] - INFO - adding node 172.23.123.62:8091 to cluster [2021-02-12 23:06:14,767] - [rest_client] [140097183020800] - INFO - adding remote node @172.23.123.62:8091 to this cluster @172.23.123.61:8091 [2021-02-12 23:06:17,260] - [task] [140097183020800] - INFO - adding node 172.23.122.90:8091 to cluster [2021-02-12 23:06:17,261] - [rest_client] [140097183020800] - INFO - adding remote node @172.23.122.90:8091 to this cluster @172.23.123.61:8091 [2021-02-12 23:06:19,616] - [task] [140097183020800] - INFO - adding node 172.23.122.117:8091 to cluster [2021-02-12 23:06:19,616] - [rest_client] [140097183020800] - INFO - adding remote node @172.23.122.117:8091 to this cluster @172.23.123.61:8091 [2021-02-12 23:06:21,862] - [rest_client] [140097183020800] - INFO - rebalance params : {'knownNodes': 'ns_1@172.23.122.117,ns_1@172.23.122.90,ns_1@172.23.123.61,ns_1@172.23.123.62', 'ejectedNodes': '', 'user': 'Administrator', 'password': 'password'} [2021-02-12 23:06:21,890] - [rest_client] [140097183020800] - INFO - rebalance operation started [2021-02-12 23:06:21,892] - [rest_client] [140097183020800] - INFO - rebalance percentage : 0.00 % [2021-02-12 23:06:21,892] - [task] [140097183020800] - INFO - Rebalance - status: running, progress: 0.00% [2021-02-12 23:06:31,909] - [task] [140097183020800] - INFO - Rebalance - status: none, progress: 100.00% [2021-02-12 23:06:31,915] - [task] [140097183020800] - INFO - rebalancing was completed with progress: 100% in 10.024903535842896 sec [2021-02-12 23:06:31,916] - [remote_util] [140099324987136] - INFO - SSH Connecting to 172.23.123.61 with username:root, attempt#1 of 5 [2021-02-12 23:06:32,017] - [remote_util] [140099324987136] - INFO - SSH Connected to 172.23.123.61 as root [2021-02-12 23:06:32,278] - [remote_util] [140099324987136] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:06:32,577] - [remote_util] [140099324987136] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:06:32,583] - [remote_util] [140099324987136] - INFO - running command.raw on 172.23.123.61: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-02-12 23:06:32,641] - [remote_util] [140099324987136] - INFO - command executed successfully with root [2021-02-12 23:06:32,641] - [remote_util] [140099324987136] - INFO - ['ok'] [2021-02-12 23:06:32,641] - [basetestcase] [140099324987136] - INFO - Enabled diag/eval for non-local hosts from 172.23.123.61 [2021-02-12 23:06:32,642] - [basetestcase] [140099324987136] - INFO - ============== basetestcase setup was finished for test #9 test_eventing_rebalance_in_delete_recreate_collections ============== [2021-02-12 23:06:32,648] - [remote_util] [140099324987136] - INFO - SSH Connecting to 172.23.123.61 with username:root, attempt#1 of 5 [2021-02-12 23:06:32,751] - [remote_util] [140099324987136] - INFO - SSH Connected to 172.23.123.61 as root [2021-02-12 23:06:33,006] - [remote_util] [140099324987136] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:06:33,303] - [remote_util] [140099324987136] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:06:33,305] - [remote_util] [140099324987136] - INFO - SSH Connecting to 172.23.123.62 with username:root, attempt#1 of 5 [2021-02-12 23:06:33,407] - [remote_util] [140099324987136] - INFO - SSH Connected to 172.23.123.62 as root [2021-02-12 23:06:33,680] - [remote_util] [140099324987136] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:06:33,979] - [remote_util] [140099324987136] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:06:33,981] - [remote_util] [140099324987136] - INFO - SSH Connecting to 172.23.122.90 with username:root, attempt#1 of 5 [2021-02-12 23:06:34,083] - [remote_util] [140099324987136] - INFO - SSH Connected to 172.23.122.90 as root [2021-02-12 23:06:34,367] - [remote_util] [140099324987136] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:06:34,664] - [remote_util] [140099324987136] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:06:34,666] - [remote_util] [140099324987136] - INFO - SSH Connecting to 172.23.122.117 with username:root, attempt#1 of 5 [2021-02-12 23:06:34,763] - [remote_util] [140099324987136] - INFO - SSH Connected to 172.23.122.117 as root [2021-02-12 23:06:35,009] - [remote_util] [140099324987136] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:06:35,294] - [remote_util] [140099324987136] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:06:35,296] - [remote_util] [140099324987136] - INFO - SSH Connecting to 172.23.122.97 with username:root, attempt#1 of 5 [2021-02-12 23:06:35,401] - [remote_util] [140099324987136] - INFO - SSH Connected to 172.23.122.97 as root [2021-02-12 23:06:35,667] - [remote_util] [140099324987136] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:06:35,967] - [remote_util] [140099324987136] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:06:35,969] - [remote_util] [140099324987136] - INFO - SSH Connecting to 172.23.122.82 with username:root, attempt#1 of 5 [2021-02-12 23:06:36,067] - [remote_util] [140099324987136] - INFO - SSH Connected to 172.23.122.82 as root [2021-02-12 23:06:36,326] - [remote_util] [140099324987136] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:06:36,615] - [remote_util] [140099324987136] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:06:36,617] - [remote_util] [140099324987136] - INFO - SSH Connecting to 172.23.122.118 with username:root, attempt#1 of 5 [2021-02-12 23:06:36,714] - [remote_util] [140099324987136] - INFO - SSH Connected to 172.23.122.118 as root [2021-02-12 23:06:36,960] - [remote_util] [140099324987136] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:06:37,251] - [remote_util] [140099324987136] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:06:37,253] - [remote_util] [140099324987136] - INFO - SSH Connecting to 172.23.122.116 with username:root, attempt#1 of 5 [2021-02-12 23:06:37,350] - [remote_util] [140099324987136] - INFO - SSH Connected to 172.23.122.116 as root [2021-02-12 23:06:37,615] - [remote_util] [140099324987136] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:06:37,907] - [remote_util] [140099324987136] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:07:59,454] - [basetestcase] [140099324987136] - INFO - ------- Cluster statistics ------- [2021-02-12 23:07:59,454] - [basetestcase] [140099324987136] - INFO - 172.23.122.117:8091 => {'services': ['n1ql'], 'cpu_utilization': 0.7533902561526871, 'mem_free': 3488567296, 'mem_total': 4201684992, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-02-12 23:07:59,454] - [basetestcase] [140099324987136] - INFO - 172.23.122.90:8091 => {'services': ['index'], 'cpu_utilization': 0.6033182503770739, 'mem_free': 3521286144, 'mem_total': 4201684992, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-02-12 23:07:59,454] - [basetestcase] [140099324987136] - INFO - 172.23.123.61:8091 => {'services': ['kv'], 'cpu_utilization': 1.076345431789737, 'mem_free': 3486732288, 'mem_total': 4201672704, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-02-12 23:07:59,454] - [basetestcase] [140099324987136] - INFO - 172.23.123.62:8091 => {'services': ['eventing'], 'cpu_utilization': 0.5279034690799397, 'mem_free': 3458584576, 'mem_total': 4201676800, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-02-12 23:07:59,454] - [basetestcase] [140099324987136] - INFO - --- End of cluster statistics --- [2021-02-12 23:07:59,455] - [remote_util] [140099324987136] - INFO - SSH Connecting to 172.23.123.61 with username:root, attempt#1 of 5 [2021-02-12 23:07:59,559] - [remote_util] [140099324987136] - INFO - SSH Connected to 172.23.123.61 as root [2021-02-12 23:07:59,825] - [remote_util] [140099324987136] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:08:00,119] - [remote_util] [140099324987136] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:08:02,933] - [basetestcase] [140099324987136] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.117 port:8091 ssh_username:root] [2021-02-12 23:08:02,939] - [basetestcase] [140099324987136] - INFO - list of eventing nodes in cluster: [ip:172.23.123.62 port:8091 ssh_username:root] [2021-02-12 23:08:02,944] - [rest_client] [140099324987136] - INFO - settings/indexes params : storageMode=plasma [2021-02-12 23:08:02,948] - [eventing_base] [140099324987136] - INFO - Setting the min possible memory quota so that adding mode nodes to the cluster wouldn't be a problem. [2021-02-12 23:08:02,948] - [rest_client] [140099324987136] - INFO - pools/default params : memoryQuota=330 [2021-02-12 23:08:02,954] - [rest_client] [140099324987136] - INFO - pools/default params : indexMemoryQuota=256 [2021-02-12 23:08:02,959] - [rest_client] [140099324987136] - INFO - pools/default params : eventingMemoryQuota=512 [2021-02-12 23:08:02,974] - [basetestcase] [140099324987136] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.117 port:8091 ssh_username:root] [2021-02-12 23:08:02,978] - [rest_client] [140099324987136] - INFO - pools/default params : memoryQuota=700 [2021-02-12 23:08:02,986] - [eventing_rebalance_collection] [140099324987136] - INFO - 200 [2021-02-12 23:08:03,561] - [rest_client] [140097183020800] - INFO - http://172.23.123.61:8091/pools/default/buckets with param: name=src_bucket&ramQuotaMB=200&authType=none&replicaNumber=0&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore [2021-02-12 23:08:03,588] - [rest_client] [140097183020800] - INFO - 0.03 seconds to create bucket src_bucket [2021-02-12 23:08:03,589] - [bucket_helper] [140097183020800] - INFO - waiting for memcached bucket : src_bucket in 172.23.123.61 to accept set ops [2021-02-12 23:08:04,219] - [data_helper] [140097183020800] - INFO - creating direct client 172.23.123.61:11210 src_bucket [2021-02-12 23:08:04,317] - [data_helper] [140097183020800] - INFO - creating direct client 172.23.123.61:11210 src_bucket [2021-02-12 23:08:04,377] - [task] [140097183020800] - WARNING - vbucket map not ready after try 0 [2021-02-12 23:08:04,377] - [bucket_helper] [140097183020800] - INFO - waiting for memcached bucket : src_bucket in 172.23.123.61 to accept set ops [2021-02-12 23:08:04,665] - [data_helper] [140097183020800] - INFO - creating direct client 172.23.123.61:11210 src_bucket [2021-02-12 23:08:04,772] - [data_helper] [140097183020800] - INFO - creating direct client 172.23.123.61:11210 src_bucket [2021-02-12 23:08:04,834] - [task] [140097183020800] - WARNING - vbucket map not ready after try 1 [2021-02-12 23:08:04,834] - [bucket_helper] [140097183020800] - INFO - waiting for memcached bucket : src_bucket in 172.23.123.61 to accept set ops [2021-02-12 23:08:04,942] - [data_helper] [140097183020800] - INFO - creating direct client 172.23.123.61:11210 src_bucket [2021-02-12 23:08:05,223] - [data_helper] [140097183020800] - INFO - creating direct client 172.23.123.61:11210 src_bucket [2021-02-12 23:08:05,290] - [task] [140097183020800] - WARNING - vbucket map not ready after try 2 [2021-02-12 23:08:05,291] - [bucket_helper] [140097183020800] - INFO - waiting for memcached bucket : src_bucket in 172.23.123.61 to accept set ops [2021-02-12 23:08:05,387] - [data_helper] [140097183020800] - INFO - creating direct client 172.23.123.61:11210 src_bucket [2021-02-12 23:08:05,490] - [data_helper] [140097183020800] - INFO - creating direct client 172.23.123.61:11210 src_bucket [2021-02-12 23:08:05,548] - [task] [140097183020800] - WARNING - vbucket map not ready after try 3 [2021-02-12 23:08:05,549] - [bucket_helper] [140097183020800] - INFO - waiting for memcached bucket : src_bucket in 172.23.123.61 to accept set ops [2021-02-12 23:08:05,821] - [data_helper] [140097183020800] - INFO - creating direct client 172.23.123.61:11210 src_bucket [2021-02-12 23:08:05,926] - [data_helper] [140097183020800] - INFO - creating direct client 172.23.123.61:11210 src_bucket [2021-02-12 23:08:05,983] - [task] [140097183020800] - WARNING - vbucket map not ready after try 4 [2021-02-12 23:08:05,983] - [bucket_helper] [140097183020800] - INFO - waiting for memcached bucket : src_bucket in 172.23.123.61 to accept set ops [2021-02-12 23:08:06,079] - [data_helper] [140097183020800] - INFO - creating direct client 172.23.123.61:11210 src_bucket [2021-02-12 23:08:06,175] - [data_helper] [140097183020800] - INFO - creating direct client 172.23.123.61:11210 src_bucket [2021-02-12 23:08:06,403] - [task] [140097183020800] - WARNING - vbucket map not ready after try 5 [2021-02-12 23:08:07,423] - [rest_client] [140097183020800] - INFO - http://172.23.123.61:8091/pools/default/buckets with param: name=dst_bucket&ramQuotaMB=200&authType=none&replicaNumber=0&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore [2021-02-12 23:08:07,451] - [rest_client] [140097183020800] - INFO - 0.03 seconds to create bucket dst_bucket [2021-02-12 23:08:07,451] - [bucket_helper] [140097183020800] - INFO - waiting for memcached bucket : dst_bucket in 172.23.123.61 to accept set ops [2021-02-12 23:08:08,075] - [data_helper] [140097183020800] - INFO - creating direct client 172.23.123.61:11210 dst_bucket [2021-02-12 23:08:08,174] - [data_helper] [140097183020800] - INFO - creating direct client 172.23.123.61:11210 dst_bucket [2021-02-12 23:08:08,237] - [task] [140097183020800] - WARNING - vbucket map not ready after try 0 [2021-02-12 23:08:08,237] - [bucket_helper] [140097183020800] - INFO - waiting for memcached bucket : dst_bucket in 172.23.123.61 to accept set ops [2021-02-12 23:08:08,543] - [data_helper] [140097183020800] - INFO - creating direct client 172.23.123.61:11210 dst_bucket [2021-02-12 23:08:08,652] - [data_helper] [140097183020800] - INFO - creating direct client 172.23.123.61:11210 dst_bucket [2021-02-12 23:08:08,715] - [task] [140097183020800] - WARNING - vbucket map not ready after try 1 [2021-02-12 23:08:08,715] - [bucket_helper] [140097183020800] - INFO - waiting for memcached bucket : dst_bucket in 172.23.123.61 to accept set ops [2021-02-12 23:08:08,814] - [data_helper] [140097183020800] - INFO - creating direct client 172.23.123.61:11210 dst_bucket [2021-02-12 23:08:08,917] - [data_helper] [140097183020800] - INFO - creating direct client 172.23.123.61:11210 dst_bucket [2021-02-12 23:08:09,161] - [task] [140097183020800] - WARNING - vbucket map not ready after try 2 [2021-02-12 23:08:09,161] - [bucket_helper] [140097183020800] - INFO - waiting for memcached bucket : dst_bucket in 172.23.123.61 to accept set ops [2021-02-12 23:08:09,260] - [data_helper] [140097183020800] - INFO - creating direct client 172.23.123.61:11210 dst_bucket [2021-02-12 23:08:09,361] - [data_helper] [140097183020800] - INFO - creating direct client 172.23.123.61:11210 dst_bucket [2021-02-12 23:08:09,419] - [task] [140097183020800] - WARNING - vbucket map not ready after try 3 [2021-02-12 23:08:09,419] - [bucket_helper] [140097183020800] - INFO - waiting for memcached bucket : dst_bucket in 172.23.123.61 to accept set ops [2021-02-12 23:08:09,522] - [data_helper] [140097183020800] - INFO - creating direct client 172.23.123.61:11210 dst_bucket [2021-02-12 23:08:09,802] - [data_helper] [140097183020800] - INFO - creating direct client 172.23.123.61:11210 dst_bucket [2021-02-12 23:08:09,861] - [task] [140097183020800] - WARNING - vbucket map not ready after try 4 [2021-02-12 23:08:09,861] - [bucket_helper] [140097183020800] - INFO - waiting for memcached bucket : dst_bucket in 172.23.123.61 to accept set ops [2021-02-12 23:08:09,961] - [data_helper] [140097183020800] - INFO - creating direct client 172.23.123.61:11210 dst_bucket [2021-02-12 23:08:10,065] - [data_helper] [140097183020800] - INFO - creating direct client 172.23.123.61:11210 dst_bucket [2021-02-12 23:08:10,306] - [task] [140097183020800] - WARNING - vbucket map not ready after try 5 [2021-02-12 23:08:11,329] - [rest_client] [140097183020800] - INFO - http://172.23.123.61:8091/pools/default/buckets with param: name=metadata&ramQuotaMB=200&authType=none&replicaNumber=0&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore [2021-02-12 23:08:11,357] - [rest_client] [140097183020800] - INFO - 0.03 seconds to create bucket metadata [2021-02-12 23:08:11,357] - [bucket_helper] [140097183020800] - INFO - waiting for memcached bucket : metadata in 172.23.123.61 to accept set ops [2021-02-12 23:08:11,985] - [data_helper] [140097183020800] - INFO - creating direct client 172.23.123.61:11210 metadata [2021-02-12 23:08:12,096] - [data_helper] [140097183020800] - INFO - creating direct client 172.23.123.61:11210 metadata [2021-02-12 23:08:12,162] - [task] [140097183020800] - WARNING - vbucket map not ready after try 0 [2021-02-12 23:08:12,162] - [bucket_helper] [140097183020800] - INFO - waiting for memcached bucket : metadata in 172.23.123.61 to accept set ops [2021-02-12 23:08:12,271] - [data_helper] [140097183020800] - INFO - creating direct client 172.23.123.61:11210 metadata [2021-02-12 23:08:12,558] - [data_helper] [140097183020800] - INFO - creating direct client 172.23.123.61:11210 metadata [2021-02-12 23:08:12,621] - [task] [140097183020800] - WARNING - vbucket map not ready after try 1 [2021-02-12 23:08:12,621] - [bucket_helper] [140097183020800] - INFO - waiting for memcached bucket : metadata in 172.23.123.61 to accept set ops [2021-02-12 23:08:12,729] - [data_helper] [140097183020800] - INFO - creating direct client 172.23.123.61:11210 metadata [2021-02-12 23:08:12,837] - [data_helper] [140097183020800] - INFO - creating direct client 172.23.123.61:11210 metadata [2021-02-12 23:08:12,905] - [task] [140097183020800] - WARNING - vbucket map not ready after try 2 [2021-02-12 23:08:12,905] - [bucket_helper] [140097183020800] - INFO - waiting for memcached bucket : metadata in 172.23.123.61 to accept set ops [2021-02-12 23:08:13,217] - [data_helper] [140097183020800] - INFO - creating direct client 172.23.123.61:11210 metadata [2021-02-12 23:08:13,340] - [data_helper] [140097183020800] - INFO - creating direct client 172.23.123.61:11210 metadata [2021-02-12 23:08:13,416] - [task] [140097183020800] - WARNING - vbucket map not ready after try 3 [2021-02-12 23:08:13,417] - [bucket_helper] [140097183020800] - INFO - waiting for memcached bucket : metadata in 172.23.123.61 to accept set ops [2021-02-12 23:08:13,544] - [data_helper] [140097183020800] - INFO - creating direct client 172.23.123.61:11210 metadata [2021-02-12 23:08:13,907] - [data_helper] [140097183020800] - INFO - creating direct client 172.23.123.61:11210 metadata [2021-02-12 23:08:13,980] - [task] [140097183020800] - WARNING - vbucket map not ready after try 4 [2021-02-12 23:08:13,980] - [bucket_helper] [140097183020800] - INFO - waiting for memcached bucket : metadata in 172.23.123.61 to accept set ops [2021-02-12 23:08:14,098] - [data_helper] [140097183020800] - INFO - creating direct client 172.23.123.61:11210 metadata [2021-02-12 23:08:14,232] - [data_helper] [140097183020800] - INFO - creating direct client 172.23.123.61:11210 metadata [2021-02-12 23:08:14,308] - [task] [140097183020800] - WARNING - vbucket map not ready after try 5 [2021-02-12 23:08:14,377] - [rest_client] [140099324987136] - INFO - http://172.23.123.62:8091/pools/default/buckets/dst_bucket/scopes with params: name=scope_0 [2021-02-12 23:08:14,377] - [rest_client] [140099324987136] - INFO - Scope created dst_bucket->scope_0 {'uid': '1'} [2021-02-12 23:08:14,405] - [rest_client] [140099324987136] - INFO - http://172.23.123.62:8091/pools/default/buckets/dst_bucket/scopes with params: name=scope_1 [2021-02-12 23:08:14,405] - [rest_client] [140099324987136] - INFO - Scope created dst_bucket->scope_1 {'uid': '2'} [2021-02-12 23:08:14,436] - [rest_client] [140099324987136] - INFO - http://172.23.123.62:8091/pools/default/buckets/dst_bucket/scopes with params: name=scope_2 [2021-02-12 23:08:14,436] - [rest_client] [140099324987136] - INFO - Scope created dst_bucket->scope_2 {'uid': '3'} [2021-02-12 23:08:14,464] - [rest_client] [140099324987136] - INFO - http://172.23.123.62:8091/pools/default/buckets/dst_bucket/scopes with params: name=scope_3 [2021-02-12 23:08:14,464] - [rest_client] [140099324987136] - INFO - Scope created dst_bucket->scope_3 {'uid': '4'} [2021-02-12 23:08:14,493] - [rest_client] [140099324987136] - INFO - http://172.23.123.62:8091/pools/default/buckets/dst_bucket/scopes with params: name=scope_4 [2021-02-12 23:08:14,494] - [rest_client] [140099324987136] - INFO - Scope created dst_bucket->scope_4 {'uid': '5'} [2021-02-12 23:08:14,524] - [rest_client] [140099324987136] - INFO - http://172.23.123.62:8091/pools/default/buckets/src_bucket/scopes with params: name=scope_0 [2021-02-12 23:08:14,524] - [rest_client] [140099324987136] - INFO - Scope created src_bucket->scope_0 {'uid': '1'} [2021-02-12 23:08:14,553] - [rest_client] [140099324987136] - INFO - http://172.23.123.62:8091/pools/default/buckets/src_bucket/scopes with params: name=scope_1 [2021-02-12 23:08:14,553] - [rest_client] [140099324987136] - INFO - Scope created src_bucket->scope_1 {'uid': '2'} [2021-02-12 23:08:14,583] - [rest_client] [140099324987136] - INFO - http://172.23.123.62:8091/pools/default/buckets/src_bucket/scopes with params: name=scope_2 [2021-02-12 23:08:14,584] - [rest_client] [140099324987136] - INFO - Scope created src_bucket->scope_2 {'uid': '3'} [2021-02-12 23:08:14,612] - [rest_client] [140099324987136] - INFO - http://172.23.123.62:8091/pools/default/buckets/src_bucket/scopes with params: name=scope_3 [2021-02-12 23:08:14,613] - [rest_client] [140099324987136] - INFO - Scope created src_bucket->scope_3 {'uid': '4'} [2021-02-12 23:08:14,639] - [rest_client] [140099324987136] - INFO - http://172.23.123.62:8091/pools/default/buckets/src_bucket/scopes with params: name=scope_4 [2021-02-12 23:08:14,639] - [rest_client] [140099324987136] - INFO - Scope created src_bucket->scope_4 {'uid': '5'} [2021-02-12 23:08:14,675] - [rest_client] [140099324987136] - INFO - http://172.23.123.62:8091/pools/default/buckets/dst_bucket/scopes/scope_1/collections with params: name=coll_0 [2021-02-12 23:08:14,675] - [rest_client] [140099324987136] - INFO - Collection created dst_bucket->scope_1->coll_0 manifest:{'uid': '6'} [2021-02-12 23:08:14,703] - [rest_client] [140099324987136] - INFO - http://172.23.123.62:8091/pools/default/buckets/dst_bucket/scopes/scope_1/collections with params: name=coll_1 [2021-02-12 23:08:14,703] - [rest_client] [140099324987136] - INFO - Collection created dst_bucket->scope_1->coll_1 manifest:{'uid': '7'} [2021-02-12 23:08:14,736] - [rest_client] [140099324987136] - INFO - http://172.23.123.62:8091/pools/default/buckets/dst_bucket/scopes/scope_1/collections with params: name=coll_2 [2021-02-12 23:08:14,736] - [rest_client] [140099324987136] - INFO - Collection created dst_bucket->scope_1->coll_2 manifest:{'uid': '8'} [2021-02-12 23:08:14,766] - [rest_client] [140099324987136] - INFO - http://172.23.123.62:8091/pools/default/buckets/dst_bucket/scopes/scope_1/collections with params: name=coll_3 [2021-02-12 23:08:14,766] - [rest_client] [140099324987136] - INFO - Collection created dst_bucket->scope_1->coll_3 manifest:{'uid': '9'} [2021-02-12 23:08:14,800] - [rest_client] [140099324987136] - INFO - http://172.23.123.62:8091/pools/default/buckets/dst_bucket/scopes/scope_1/collections with params: name=coll_4 [2021-02-12 23:08:14,800] - [rest_client] [140099324987136] - INFO - Collection created dst_bucket->scope_1->coll_4 manifest:{'uid': 'a'} [2021-02-12 23:08:14,827] - [rest_client] [140099324987136] - INFO - http://172.23.123.62:8091/pools/default/buckets/src_bucket/scopes/scope_1/collections with params: name=coll_0 [2021-02-12 23:08:14,827] - [rest_client] [140099324987136] - INFO - Collection created src_bucket->scope_1->coll_0 manifest:{'uid': '6'} [2021-02-12 23:08:14,859] - [rest_client] [140099324987136] - INFO - http://172.23.123.62:8091/pools/default/buckets/src_bucket/scopes/scope_1/collections with params: name=coll_1 [2021-02-12 23:08:14,859] - [rest_client] [140099324987136] - INFO - Collection created src_bucket->scope_1->coll_1 manifest:{'uid': '7'} [2021-02-12 23:08:14,888] - [rest_client] [140099324987136] - INFO - http://172.23.123.62:8091/pools/default/buckets/src_bucket/scopes/scope_1/collections with params: name=coll_2 [2021-02-12 23:08:14,889] - [rest_client] [140099324987136] - INFO - Collection created src_bucket->scope_1->coll_2 manifest:{'uid': '8'} [2021-02-12 23:08:14,921] - [rest_client] [140099324987136] - INFO - http://172.23.123.62:8091/pools/default/buckets/src_bucket/scopes/scope_1/collections with params: name=coll_3 [2021-02-12 23:08:14,921] - [rest_client] [140099324987136] - INFO - Collection created src_bucket->scope_1->coll_3 manifest:{'uid': '9'} [2021-02-12 23:08:14,951] - [rest_client] [140099324987136] - INFO - http://172.23.123.62:8091/pools/default/buckets/src_bucket/scopes/scope_1/collections with params: name=coll_4 [2021-02-12 23:08:14,951] - [rest_client] [140099324987136] - INFO - Collection created src_bucket->scope_1->coll_4 manifest:{'uid': 'a'} [2021-02-12 23:08:16,158] - [eventing_base] [140099324987136] - INFO - saving function bucket_op [2021-02-12 23:08:17,340] - [eventing_base] [140099324987136] - INFO - saving function timers [2021-02-12 23:08:18,527] - [eventing_base] [140099324987136] - INFO - saving function sbm [2021-02-12 23:08:19,701] - [eventing_base] [140099324987136] - INFO - saving function curl [2021-02-12 23:08:20,897] - [eventing_base] [140099324987136] - INFO - saving function n1ql [2021-02-12 23:08:22,095] - [basetestcase] [140099324987136] - INFO - sleep for 5 secs. Waiting for bucket_op to deployed... ... [2021-02-12 23:08:27,119] - [basetestcase] [140099324987136] - INFO - sleep for 5 secs. Waiting for bucket_op to deployed... ... [2021-02-12 23:08:32,141] - [basetestcase] [140099324987136] - INFO - sleep for 5 secs. Waiting for bucket_op to deployed... ... [2021-02-12 23:08:37,167] - [basetestcase] [140099324987136] - INFO - sleep for 5 secs. Waiting for bucket_op to deployed... ... [2021-02-12 23:08:43,411] - [basetestcase] [140099324987136] - INFO - sleep for 5 secs. Waiting for timers to deployed... ... [2021-02-12 23:08:48,436] - [basetestcase] [140099324987136] - INFO - sleep for 5 secs. Waiting for timers to deployed... ... [2021-02-12 23:08:53,463] - [basetestcase] [140099324987136] - INFO - sleep for 5 secs. Waiting for timers to deployed... ... [2021-02-12 23:08:58,490] - [basetestcase] [140099324987136] - INFO - sleep for 5 secs. Waiting for timers to deployed... ... [2021-02-12 23:09:04,804] - [basetestcase] [140099324987136] - INFO - sleep for 5 secs. Waiting for sbm to deployed... ... [2021-02-12 23:09:09,826] - [basetestcase] [140099324987136] - INFO - sleep for 5 secs. Waiting for sbm to deployed... ... [2021-02-12 23:09:14,860] - [basetestcase] [140099324987136] - INFO - sleep for 5 secs. Waiting for sbm to deployed... ... [2021-02-12 23:09:19,885] - [basetestcase] [140099324987136] - INFO - sleep for 5 secs. Waiting for sbm to deployed... ... [2021-02-12 23:09:26,096] - [basetestcase] [140099324987136] - INFO - sleep for 5 secs. Waiting for curl to deployed... ... [2021-02-12 23:09:31,126] - [basetestcase] [140099324987136] - INFO - sleep for 5 secs. Waiting for curl to deployed... ... [2021-02-12 23:09:36,156] - [basetestcase] [140099324987136] - INFO - sleep for 5 secs. Waiting for curl to deployed... ... [2021-02-12 23:09:41,180] - [basetestcase] [140099324987136] - INFO - sleep for 5 secs. Waiting for curl to deployed... ... [2021-02-12 23:09:47,402] - [basetestcase] [140099324987136] - INFO - sleep for 5 secs. Waiting for n1ql to deployed... ... [2021-02-12 23:09:52,432] - [basetestcase] [140099324987136] - INFO - sleep for 5 secs. Waiting for n1ql to deployed... ... [2021-02-12 23:09:57,464] - [basetestcase] [140099324987136] - INFO - sleep for 5 secs. Waiting for n1ql to deployed... ... [2021-02-12 23:10:02,484] - [basetestcase] [140099324987136] - INFO - sleep for 5 secs. Waiting for n1ql to deployed... ... [2021-02-12 23:10:08,428] - [task] [140097183020800] - INFO - java -jar java_sdk_client/collections/target/javaclient/javaclient.jar -i 172.23.123.61 -u Administrator -p password -b src_bucket -s _default -c _default -n 40320 -pc 100 -pu 0 -pd 0 -l uniform -dsn 1 -dpx doc_ -dt Person -de 0 -ds 500 -ac False -st 0 -en 40319 -o False -sd False [2021-02-12 23:10:10,610] - [rest_client] [140099324987136] - INFO - rebalance reached >100% in 3.0052831172943115 seconds [2021-02-12 23:10:20,502] - [task] [140097183020800] - INFO - java -jar java_sdk_client/collections/target/javaclient/javaclient.jar -i 172.23.123.61 -u Administrator -p password -b src_bucket -s scope_1 -c coll_1 -n 40320 -pc 100 -pu 0 -pd 0 -l uniform -dsn 1 -dpx doc_ -dt Person -de 0 -ds 500 -ac False -st 0 -en 40319 -o False -sd False [2021-02-12 23:10:31,645] - [task] [140097183020800] - INFO - adding node 172.23.122.97:8091 to cluster [2021-02-12 23:10:31,645] - [rest_client] [140097183020800] - INFO - adding remote node @172.23.122.97:8091 to this cluster @172.23.123.61:8091 [2021-02-12 23:10:34,032] - [rest_client] [140097183020800] - INFO - rebalance params : {'knownNodes': 'ns_1@172.23.122.117,ns_1@172.23.122.90,ns_1@172.23.122.97,ns_1@172.23.123.61,ns_1@172.23.123.62', 'ejectedNodes': '', 'user': 'Administrator', 'password': 'password'} [2021-02-12 23:10:34,062] - [rest_client] [140097183020800] - INFO - rebalance operation started [2021-02-12 23:10:34,066] - [rest_client] [140097183020800] - INFO - rebalance percentage : 0.00 % [2021-02-12 23:10:34,066] - [task] [140097183020800] - INFO - Rebalance - status: running, progress: 0.00% [2021-02-12 23:10:44,088] - [rest_client] [140097183020800] - INFO - rebalance percentage : 40.00 % [2021-02-12 23:10:44,089] - [task] [140097183020800] - INFO - Rebalance - status: running, progress: 40.00% [2021-02-12 23:10:54,113] - [rest_client] [140097183020800] - INFO - rebalance percentage : 59.00 % [2021-02-12 23:10:54,113] - [task] [140097183020800] - INFO - Rebalance - status: running, progress: 59.00% [2021-02-12 23:11:04,136] - [rest_client] [140097183020800] - INFO - rebalance percentage : 40.00 % [2021-02-12 23:11:04,136] - [task] [140097183020800] - INFO - Rebalance - status: running, progress: 40.00% [2021-02-12 23:11:14,162] - [rest_client] [140097183020800] - INFO - rebalance percentage : 79.00 % [2021-02-12 23:11:14,162] - [task] [140097183020800] - INFO - Rebalance - status: running, progress: 79.00% [2021-02-12 23:11:24,186] - [rest_client] [140097183020800] - INFO - rebalance percentage : 79.00 % [2021-02-12 23:11:24,186] - [task] [140097183020800] - INFO - Rebalance - status: running, progress: 79.00% [2021-02-12 23:11:34,211] - [rest_client] [140097183020800] - INFO - rebalance percentage : 79.00 % [2021-02-12 23:11:34,212] - [task] [140097183020800] - INFO - Rebalance - status: running, progress: 79.00% [2021-02-12 23:11:44,237] - [rest_client] [140097183020800] - INFO - rebalance percentage : 79.00 % [2021-02-12 23:11:44,237] - [task] [140097183020800] - INFO - Rebalance - status: running, progress: 79.00% [2021-02-12 23:11:54,261] - [rest_client] [140097183020800] - INFO - rebalance percentage : 79.00 % [2021-02-12 23:11:54,261] - [task] [140097183020800] - INFO - Rebalance - status: running, progress: 79.00% [2021-02-12 23:12:04,285] - [task] [140097183020800] - INFO - Rebalance - status: none, progress: 100.00% [2021-02-12 23:12:04,296] - [task] [140097183020800] - INFO - rebalancing was completed with progress: 100% in 90.23408055305481 sec [2021-02-12 23:12:04,305] - [basetestcase] [140099324987136] - INFO - list of eventing nodes in cluster: [ip:172.23.122.97 port:8091 ssh_username:root, ip:172.23.123.62 port:8091 ssh_username:root] [2021-02-12 23:12:04,310] - [tuq_helper] [140099324987136] - INFO - RUN QUERY create primary index on dst_bucket.scope_1.coll_3 [2021-02-12 23:12:04,315] - [rest_client] [140099324987136] - INFO - query params : statement=create+primary+index+on+dst_bucket.scope_1.coll_3 [2021-02-12 23:12:06,412] - [tuq_helper] [140099324987136] - INFO - TOTAL ELAPSED TIME: 2.095633598s [2021-02-12 23:12:06,439] - [basetestcase] [140099324987136] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.117 port:8091 ssh_username:root] [2021-02-12 23:12:06,446] - [rest_client] [140099324987136] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_3 [2021-02-12 23:12:06,455] - [eventing_base] [140099324987136] - INFO - Final docs count... Current : 40320 Expected : 40320 [2021-02-12 23:12:06,464] - [basetestcase] [140099324987136] - INFO - list of eventing nodes in cluster: [ip:172.23.122.97 port:8091 ssh_username:root, ip:172.23.123.62 port:8091 ssh_username:root] [2021-02-12 23:12:06,470] - [tuq_helper] [140099324987136] - INFO - RUN QUERY create primary index on dst_bucket.scope_1.coll_4 [2021-02-12 23:12:06,474] - [rest_client] [140099324987136] - INFO - query params : statement=create+primary+index+on+dst_bucket.scope_1.coll_4 [2021-02-12 23:12:09,481] - [tuq_helper] [140099324987136] - INFO - TOTAL ELAPSED TIME: 3.004796013s [2021-02-12 23:12:09,498] - [basetestcase] [140099324987136] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.117 port:8091 ssh_username:root] [2021-02-12 23:12:09,502] - [rest_client] [140099324987136] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-12 23:12:09,512] - [eventing_base] [140099324987136] - INFO - Final docs count... Current : 40320 Expected : 40320 [2021-02-12 23:12:09,520] - [basetestcase] [140099324987136] - INFO - list of eventing nodes in cluster: [ip:172.23.122.97 port:8091 ssh_username:root, ip:172.23.123.62 port:8091 ssh_username:root] [2021-02-12 23:12:09,523] - [tuq_helper] [140099324987136] - INFO - RUN QUERY create primary index on src_bucket.scope_1.coll_1 [2021-02-12 23:12:09,527] - [rest_client] [140099324987136] - INFO - query params : statement=create+primary+index+on+src_bucket.scope_1.coll_1 [2021-02-12 23:12:12,051] - [tuq_helper] [140099324987136] - INFO - TOTAL ELAPSED TIME: 2.522974886s [2021-02-12 23:12:12,073] - [basetestcase] [140099324987136] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.117 port:8091 ssh_username:root] [2021-02-12 23:12:12,079] - [rest_client] [140099324987136] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.scope_1.coll_1 [2021-02-12 23:12:12,087] - [eventing_base] [140099324987136] - INFO - Final docs count... Current : 80640 Expected : 80640 [2021-02-12 23:12:12,121] - [rest_client] [140099324987136] - INFO - http://172.23.123.61:8091/pools/default/buckets/dst_bucket/scopes/scope_1/collections with params: name=coll_0 [2021-02-12 23:12:12,121] - [rest_client] [140099324987136] - INFO - Collection created dst_bucket->scope_1->coll_0 manifest:{'uid': 'e'} [2021-02-12 23:12:12,157] - [rest_client] [140099324987136] - INFO - http://172.23.123.61:8091/pools/default/buckets/dst_bucket/scopes/scope_1/collections with params: name=coll_1 [2021-02-12 23:12:12,157] - [rest_client] [140099324987136] - INFO - Collection created dst_bucket->scope_1->coll_1 manifest:{'uid': 'f'} [2021-02-12 23:12:12,192] - [rest_client] [140099324987136] - INFO - http://172.23.123.61:8091/pools/default/buckets/dst_bucket/scopes/scope_1/collections with params: name=coll_2 [2021-02-12 23:12:12,192] - [rest_client] [140099324987136] - INFO - Collection created dst_bucket->scope_1->coll_2 manifest:{'uid': '10'} [2021-02-12 23:12:12,304] - [task] [140097183020800] - INFO - adding node 172.23.122.82:8091 to cluster [2021-02-12 23:12:12,304] - [rest_client] [140097183020800] - INFO - adding remote node @172.23.122.82:8091 to this cluster @172.23.123.61:8091 [2021-02-12 23:12:14,723] - [rest_client] [140097183020800] - INFO - rebalance params : {'knownNodes': 'ns_1@172.23.122.117,ns_1@172.23.122.82,ns_1@172.23.122.90,ns_1@172.23.122.97,ns_1@172.23.123.61,ns_1@172.23.123.62', 'ejectedNodes': '', 'user': 'Administrator', 'password': 'password'} [2021-02-12 23:12:14,759] - [rest_client] [140097183020800] - INFO - rebalance operation started [2021-02-12 23:12:14,759] - [task] [140097183020800] - INFO - java -jar java_sdk_client/collections/target/javaclient/javaclient.jar -i 172.23.123.61 -u Administrator -p password -b src_bucket -s _default -c _default -n 40320 -pc 0 -pu 0 -pd 100 -l uniform -dsn 1 -dpx doc_ -dt Person -de 0 -ds 500 -ac False -st 0 -en 40319 -o False -sd False [2021-02-12 23:12:19,141] - [rest_client] [140097183020800] - INFO - rebalance percentage : 33.00 % [2021-02-12 23:12:19,141] - [task] [140097183020800] - INFO - Rebalance - status: running, progress: 33.00% [2021-02-12 23:12:19,160] - [task] [140097183020800] - INFO - java -jar java_sdk_client/collections/target/javaclient/javaclient.jar -i 172.23.123.61 -u Administrator -p password -b src_bucket -s scope_1 -c coll_1 -n 40320 -pc 0 -pu 0 -pd 100 -l uniform -dsn 1 -dpx doc_ -dt Person -de 0 -ds 500 -ac False -st 0 -en 40319 -o False -sd False [2021-02-12 23:12:23,046] - [rest_client] [140099324987136] - INFO - rebalance percentage : 33.00 % [2021-02-12 23:12:26,055] - [rest_client] [140099324987136] - INFO - rebalance percentage : 33.00 % [2021-02-12 23:12:29,063] - [rest_client] [140099324987136] - INFO - rebalance percentage : 33.00 % [2021-02-12 23:12:30,051] - [rest_client] [140097183020800] - INFO - rebalance percentage : 33.00 % [2021-02-12 23:12:30,051] - [task] [140097183020800] - INFO - Rebalance - status: running, progress: 33.00% [2021-02-12 23:12:32,069] - [rest_client] [140099324987136] - INFO - rebalance percentage : 41.00 % [2021-02-12 23:12:35,075] - [rest_client] [140099324987136] - INFO - rebalance percentage : 41.00 % [2021-02-12 23:12:38,081] - [rest_client] [140099324987136] - INFO - rebalance percentage : 41.00 % [2021-02-12 23:12:40,078] - [rest_client] [140097183020800] - INFO - rebalance percentage : 41.00 % [2021-02-12 23:12:40,079] - [task] [140097183020800] - INFO - Rebalance - status: running, progress: 41.00% [2021-02-12 23:12:41,087] - [rest_client] [140099324987136] - INFO - rebalance percentage : 50.00 % [2021-02-12 23:12:44,096] - [rest_client] [140099324987136] - INFO - rebalance percentage : 33.00 % [2021-02-12 23:12:47,101] - [rest_client] [140099324987136] - INFO - rebalance percentage : 82.00 % [2021-02-12 23:12:50,108] - [rest_client] [140099324987136] - INFO - rebalance percentage : 82.00 % [2021-02-12 23:12:50,109] - [rest_client] [140097183020800] - INFO - rebalance percentage : 82.00 % [2021-02-12 23:12:50,110] - [task] [140097183020800] - INFO - Rebalance - status: running, progress: 82.00% [2021-02-12 23:12:53,114] - [rest_client] [140099324987136] - INFO - rebalance percentage : 82.00 % [2021-02-12 23:12:56,119] - [rest_client] [140099324987136] - INFO - rebalance percentage : 82.00 % [2021-02-12 23:12:59,125] - [rest_client] [140099324987136] - INFO - rebalance percentage : 82.00 % [2021-02-12 23:13:00,133] - [rest_client] [140097183020800] - INFO - rebalance percentage : 82.00 % [2021-02-12 23:13:00,133] - [task] [140097183020800] - INFO - Rebalance - status: running, progress: 82.00% [2021-02-12 23:13:02,131] - [rest_client] [140099324987136] - INFO - rebalance percentage : 82.00 % [2021-02-12 23:13:05,137] - [rest_client] [140099324987136] - INFO - rebalance percentage : 82.00 % [2021-02-12 23:13:08,141] - [rest_client] [140099324987136] - INFO - rebalance percentage : 82.00 % [2021-02-12 23:13:10,156] - [rest_client] [140097183020800] - INFO - rebalance percentage : 82.00 % [2021-02-12 23:13:10,157] - [task] [140097183020800] - INFO - Rebalance - status: running, progress: 82.00% [2021-02-12 23:13:11,146] - [rest_client] [140099324987136] - INFO - rebalance percentage : 83.00 % [2021-02-12 23:13:14,150] - [rest_client] [140099324987136] - INFO - rebalance percentage : 83.00 % [2021-02-12 23:13:17,158] - [rest_client] [140099324987136] - INFO - rebalance percentage : 76.00 % [2021-02-12 23:13:20,164] - [rest_client] [140099324987136] - INFO - rebalance percentage : 83.00 % [2021-02-12 23:13:20,182] - [rest_client] [140097183020800] - INFO - rebalance percentage : 83.00 % [2021-02-12 23:13:20,182] - [task] [140097183020800] - INFO - Rebalance - status: running, progress: 83.00% [2021-02-12 23:13:23,168] - [rest_client] [140099324987136] - INFO - rebalance percentage : 83.00 % [2021-02-12 23:13:26,174] - [rest_client] [140099324987136] - INFO - rebalance percentage : 83.00 % [2021-02-12 23:13:29,179] - [rest_client] [140099324987136] - INFO - rebalance percentage : 83.00 % [2021-02-12 23:13:30,203] - [rest_client] [140097183020800] - INFO - rebalance percentage : 83.00 % [2021-02-12 23:13:30,204] - [task] [140097183020800] - INFO - Rebalance - status: running, progress: 83.00% [2021-02-12 23:13:32,185] - [rest_client] [140099324987136] - INFO - rebalance percentage : 83.00 % [2021-02-12 23:13:35,191] - [rest_client] [140099324987136] - INFO - rebalance percentage : 83.00 % [2021-02-12 23:13:38,194] - [rest_client] [140099324987136] - INFO - rebalance percentage : 83.00 % [2021-02-12 23:13:40,228] - [rest_client] [140097183020800] - INFO - rebalance percentage : 83.00 % [2021-02-12 23:13:40,228] - [task] [140097183020800] - INFO - Rebalance - status: running, progress: 83.00% [2021-02-12 23:13:41,197] - [rest_client] [140099324987136] - INFO - rebalance percentage : 83.00 % [2021-02-12 23:13:47,205] - [rest_client] [140099324987136] - INFO - rebalance reached >100% in 84.16493082046509 seconds [2021-02-12 23:13:50,252] - [task] [140097183020800] - INFO - Rebalance - status: none, progress: 100.00% [2021-02-12 23:13:50,263] - [task] [140097183020800] - INFO - rebalancing was completed with progress: 100% in 95.50445365905762 sec [2021-02-12 23:13:50,272] - [basetestcase] [140099324987136] - INFO - list of eventing nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root, ip:172.23.122.97 port:8091 ssh_username:root, ip:172.23.123.62 port:8091 ssh_username:root] [2021-02-12 23:13:50,278] - [tuq_helper] [140099324987136] - INFO - RUN QUERY create primary index on dst_bucket.scope_1.coll_3 [2021-02-12 23:13:50,282] - [rest_client] [140099324987136] - INFO - query params : statement=create+primary+index+on+dst_bucket.scope_1.coll_3 [2021-02-12 23:13:50,286] - [rest_client] [140099324987136] - ERROR - POST http://172.23.122.117:8093/query?statement=create+primary+index+on+dst_bucket.scope_1.coll_3 body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 500 reason: unknown b'{\n"requestID": "c1efbd46-5a23-4036-957c-6aae8a99ac67",\n"signature": null,\n"results": [\n],\n"errors": [{"code":5000,"msg":"GSI CreatePrimaryIndex() - cause: Index #primary already exists."}],\n"status": "errors",\n"metrics": {"elapsedTime": "2.73761ms","executionTime": "2.653326ms","resultCount": 0,"resultSize": 0,"serviceLoad": 6,"errorCount": 1}\n}\n' auth: Administrator:password [2021-02-12 23:13:50,294] - [basetestcase] [140099324987136] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.117 port:8091 ssh_username:root] [2021-02-12 23:13:50,298] - [rest_client] [140099324987136] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_3 [2021-02-12 23:13:50,306] - [eventing_base] [140099324987136] - INFO - Final docs count... Current : 0 Expected : 0 [2021-02-12 23:13:50,314] - [basetestcase] [140099324987136] - INFO - list of eventing nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root, ip:172.23.122.97 port:8091 ssh_username:root, ip:172.23.123.62 port:8091 ssh_username:root] [2021-02-12 23:13:50,318] - [tuq_helper] [140099324987136] - INFO - RUN QUERY create primary index on dst_bucket.scope_1.coll_4 [2021-02-12 23:13:50,321] - [rest_client] [140099324987136] - INFO - query params : statement=create+primary+index+on+dst_bucket.scope_1.coll_4 [2021-02-12 23:13:50,324] - [rest_client] [140099324987136] - ERROR - POST http://172.23.122.117:8093/query?statement=create+primary+index+on+dst_bucket.scope_1.coll_4 body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 500 reason: unknown b'{\n"requestID": "a4425f99-ff08-4daf-a7dc-e8fba6f219a8",\n"signature": null,\n"results": [\n],\n"errors": [{"code":5000,"msg":"GSI CreatePrimaryIndex() - cause: Index #primary already exists."}],\n"status": "errors",\n"metrics": {"elapsedTime": "1.65228ms","executionTime": "1.608922ms","resultCount": 0,"resultSize": 0,"serviceLoad": 6,"errorCount": 1}\n}\n' auth: Administrator:password [2021-02-12 23:13:50,332] - [basetestcase] [140099324987136] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.117 port:8091 ssh_username:root] [2021-02-12 23:13:50,336] - [rest_client] [140099324987136] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-12 23:13:50,340] - [eventing_base] [140099324987136] - INFO - Final docs count... Current : 0 Expected : 0 [2021-02-12 23:13:50,348] - [basetestcase] [140099324987136] - INFO - list of eventing nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root, ip:172.23.122.97 port:8091 ssh_username:root, ip:172.23.123.62 port:8091 ssh_username:root] [2021-02-12 23:13:50,352] - [tuq_helper] [140099324987136] - INFO - RUN QUERY create primary index on src_bucket.scope_1.coll_1 [2021-02-12 23:13:50,356] - [rest_client] [140099324987136] - INFO - query params : statement=create+primary+index+on+src_bucket.scope_1.coll_1 [2021-02-12 23:13:50,359] - [rest_client] [140099324987136] - ERROR - POST http://172.23.122.117:8093/query?statement=create+primary+index+on+src_bucket.scope_1.coll_1 body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 500 reason: unknown b'{\n"requestID": "8aeb4216-36fb-4e85-8621-d1804bab48e0",\n"signature": null,\n"results": [\n],\n"errors": [{"code":5000,"msg":"GSI CreatePrimaryIndex() - cause: Index #primary already exists."}],\n"status": "errors",\n"metrics": {"elapsedTime": "1.439078ms","executionTime": "1.366905ms","resultCount": 0,"resultSize": 0,"serviceLoad": 6,"errorCount": 1}\n}\n' auth: Administrator:password [2021-02-12 23:13:50,367] - [basetestcase] [140099324987136] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.117 port:8091 ssh_username:root] [2021-02-12 23:13:50,371] - [rest_client] [140099324987136] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.scope_1.coll_1 [2021-02-12 23:13:50,383] - [basetestcase] [140099324987136] - INFO - sleep for 30 secs. Waiting for handler code src_bucket.scope_1.coll_1 to complete bucket operations... Current : 40571 Expected : 40320 ... [2021-02-12 23:14:20,430] - [basetestcase] [140099324987136] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.117 port:8091 ssh_username:root] [2021-02-12 23:14:20,436] - [rest_client] [140099324987136] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.scope_1.coll_1 [2021-02-12 23:14:20,452] - [basetestcase] [140099324987136] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.117 port:8091 ssh_username:root] [2021-02-12 23:14:20,457] - [rest_client] [140099324987136] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.scope_1.coll_1 [2021-02-12 23:14:20,460] - [basetestcase] [140099324987136] - INFO - sleep for 30 secs. Waiting for handler code src_bucket.scope_1.coll_1 to complete bucket operations... Current : 40571 Expected : 40320 ... [2021-02-12 23:14:50,509] - [basetestcase] [140099324987136] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.117 port:8091 ssh_username:root] [2021-02-12 23:14:50,514] - [rest_client] [140099324987136] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.scope_1.coll_1 [2021-02-12 23:14:50,529] - [basetestcase] [140099324987136] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.117 port:8091 ssh_username:root] [2021-02-12 23:14:50,533] - [rest_client] [140099324987136] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.scope_1.coll_1 [2021-02-12 23:14:50,536] - [basetestcase] [140099324987136] - INFO - sleep for 30 secs. Waiting for handler code src_bucket.scope_1.coll_1 to complete bucket operations... Current : 40571 Expected : 40320 ... [2021-02-12 23:15:20,583] - [basetestcase] [140099324987136] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.117 port:8091 ssh_username:root] [2021-02-12 23:15:20,588] - [rest_client] [140099324987136] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.scope_1.coll_1 [2021-02-12 23:15:20,606] - [basetestcase] [140099324987136] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.117 port:8091 ssh_username:root] [2021-02-12 23:15:20,611] - [rest_client] [140099324987136] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.scope_1.coll_1 [2021-02-12 23:15:20,614] - [basetestcase] [140099324987136] - INFO - sleep for 30 secs. Waiting for handler code src_bucket.scope_1.coll_1 to complete bucket operations... Current : 40571 Expected : 40320 ... [2021-02-12 23:15:50,652] - [basetestcase] [140099324987136] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.117 port:8091 ssh_username:root] [2021-02-12 23:15:50,658] - [rest_client] [140099324987136] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.scope_1.coll_1 [2021-02-12 23:15:50,674] - [basetestcase] [140099324987136] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.117 port:8091 ssh_username:root] [2021-02-12 23:15:50,678] - [rest_client] [140099324987136] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.scope_1.coll_1 [2021-02-12 23:15:50,682] - [basetestcase] [140099324987136] - INFO - sleep for 30 secs. Waiting for handler code src_bucket.scope_1.coll_1 to complete bucket operations... Current : 40571 Expected : 40320 ... [2021-02-12 23:16:20,727] - [basetestcase] [140099324987136] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.117 port:8091 ssh_username:root] [2021-02-12 23:16:20,733] - [rest_client] [140099324987136] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.scope_1.coll_1 [2021-02-12 23:16:20,749] - [basetestcase] [140099324987136] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.117 port:8091 ssh_username:root] [2021-02-12 23:16:20,754] - [rest_client] [140099324987136] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.scope_1.coll_1 [2021-02-12 23:16:20,757] - [basetestcase] [140099324987136] - INFO - sleep for 30 secs. Waiting for handler code src_bucket.scope_1.coll_1 to complete bucket operations... Current : 40571 Expected : 40320 ... [2021-02-12 23:16:50,804] - [basetestcase] [140099324987136] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.117 port:8091 ssh_username:root] [2021-02-12 23:16:50,810] - [rest_client] [140099324987136] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.scope_1.coll_1 [2021-02-12 23:16:50,829] - [basetestcase] [140099324987136] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.117 port:8091 ssh_username:root] [2021-02-12 23:16:50,833] - [rest_client] [140099324987136] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.scope_1.coll_1 [2021-02-12 23:16:50,836] - [basetestcase] [140099324987136] - INFO - sleep for 30 secs. Waiting for handler code src_bucket.scope_1.coll_1 to complete bucket operations... Current : 40571 Expected : 40320 ... [2021-02-12 23:17:20,872] - [basetestcase] [140099324987136] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.117 port:8091 ssh_username:root] [2021-02-12 23:17:20,879] - [rest_client] [140099324987136] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.scope_1.coll_1 [2021-02-12 23:17:20,897] - [basetestcase] [140099324987136] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.117 port:8091 ssh_username:root] [2021-02-12 23:17:20,902] - [rest_client] [140099324987136] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.scope_1.coll_1 [2021-02-12 23:17:20,906] - [basetestcase] [140099324987136] - INFO - sleep for 30 secs. Waiting for handler code src_bucket.scope_1.coll_1 to complete bucket operations... Current : 40571 Expected : 40320 ... [2021-02-12 23:17:50,952] - [basetestcase] [140099324987136] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.117 port:8091 ssh_username:root] [2021-02-12 23:17:50,958] - [rest_client] [140099324987136] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.scope_1.coll_1 [2021-02-12 23:17:50,975] - [basetestcase] [140099324987136] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.117 port:8091 ssh_username:root] [2021-02-12 23:17:50,980] - [rest_client] [140099324987136] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.scope_1.coll_1 [2021-02-12 23:17:50,983] - [basetestcase] [140099324987136] - INFO - sleep for 30 secs. Waiting for handler code src_bucket.scope_1.coll_1 to complete bucket operations... Current : 40571 Expected : 40320 ... [2021-02-12 23:18:21,008] - [basetestcase] [140099324987136] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.117 port:8091 ssh_username:root] [2021-02-12 23:18:21,014] - [rest_client] [140099324987136] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.scope_1.coll_1 [2021-02-12 23:18:21,030] - [basetestcase] [140099324987136] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.117 port:8091 ssh_username:root] [2021-02-12 23:18:21,034] - [rest_client] [140099324987136] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.scope_1.coll_1 [2021-02-12 23:18:21,037] - [basetestcase] [140099324987136] - INFO - sleep for 30 secs. Waiting for handler code src_bucket.scope_1.coll_1 to complete bucket operations... Current : 40571 Expected : 40320 ... [2021-02-12 23:18:51,067] - [basetestcase] [140099324987136] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.117 port:8091 ssh_username:root] [2021-02-12 23:18:51,073] - [rest_client] [140099324987136] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.scope_1.coll_1 [2021-02-12 23:18:51,089] - [basetestcase] [140099324987136] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.117 port:8091 ssh_username:root] [2021-02-12 23:18:51,093] - [rest_client] [140099324987136] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.scope_1.coll_1 [2021-02-12 23:18:51,096] - [basetestcase] [140099324987136] - INFO - sleep for 30 secs. Waiting for handler code src_bucket.scope_1.coll_1 to complete bucket operations... Current : 40571 Expected : 40320 ... [2021-02-12 23:19:21,142] - [basetestcase] [140099324987136] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.117 port:8091 ssh_username:root] [2021-02-12 23:19:21,147] - [rest_client] [140099324987136] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.scope_1.coll_1 [2021-02-12 23:19:21,161] - [eventing_base] [140099324987136] - INFO - Compacting bucket : dst_bucket [2021-02-12 23:19:21,168] - [rest_client] [140099324987136] - INFO - bucket compaction successful [2021-02-12 23:19:21,168] - [eventing_base] [140099324987136] - INFO - Compacting bucket : metadata [2021-02-12 23:19:21,174] - [rest_client] [140099324987136] - INFO - bucket compaction successful [2021-02-12 23:19:21,174] - [eventing_base] [140099324987136] - INFO - Compacting bucket : src_bucket [2021-02-12 23:19:21,177] - [rest_client] [140099324987136] - INFO - bucket compaction successful [2021-02-12 23:19:21,186] - [basetestcase] [140099324987136] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.117 port:8091 ssh_username:root] [2021-02-12 23:19:21,190] - [rest_client] [140099324987136] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.scope_1.coll_1 [2021-02-12 23:19:21,194] - [basetestcase] [140099324987136] - INFO - sleep for 30 secs. Waiting for handler code src_bucket.scope_1.coll_1 to complete bucket operations... Current : 40571 Expected : 40320 ... [2021-02-12 23:19:51,242] - [basetestcase] [140099324987136] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.117 port:8091 ssh_username:root] [2021-02-12 23:19:51,249] - [rest_client] [140099324987136] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.scope_1.coll_1 [2021-02-12 23:19:51,264] - [basetestcase] [140099324987136] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.117 port:8091 ssh_username:root] [2021-02-12 23:19:51,268] - [rest_client] [140099324987136] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.scope_1.coll_1 [2021-02-12 23:19:51,271] - [basetestcase] [140099324987136] - INFO - sleep for 30 secs. Waiting for handler code src_bucket.scope_1.coll_1 to complete bucket operations... Current : 40571 Expected : 40320 ... [2021-02-12 23:20:21,318] - [basetestcase] [140099324987136] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.117 port:8091 ssh_username:root] [2021-02-12 23:20:21,324] - [rest_client] [140099324987136] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.scope_1.coll_1 [2021-02-12 23:20:21,341] - [basetestcase] [140099324987136] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.117 port:8091 ssh_username:root] [2021-02-12 23:20:21,346] - [rest_client] [140099324987136] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.scope_1.coll_1 [2021-02-12 23:20:21,349] - [basetestcase] [140099324987136] - INFO - sleep for 30 secs. Waiting for handler code src_bucket.scope_1.coll_1 to complete bucket operations... Current : 40571 Expected : 40320 ... [2021-02-12 23:20:51,392] - [basetestcase] [140099324987136] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.117 port:8091 ssh_username:root] [2021-02-12 23:20:51,398] - [rest_client] [140099324987136] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.scope_1.coll_1 [2021-02-12 23:20:51,412] - [basetestcase] [140099324987136] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.117 port:8091 ssh_username:root] [2021-02-12 23:20:51,416] - [rest_client] [140099324987136] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.scope_1.coll_1 [2021-02-12 23:20:51,419] - [basetestcase] [140099324987136] - INFO - sleep for 30 secs. Waiting for handler code src_bucket.scope_1.coll_1 to complete bucket operations... Current : 40571 Expected : 40320 ... [2021-02-12 23:21:21,459] - [basetestcase] [140099324987136] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.117 port:8091 ssh_username:root] [2021-02-12 23:21:21,465] - [rest_client] [140099324987136] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.scope_1.coll_1 [2021-02-12 23:21:21,482] - [basetestcase] [140099324987136] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.117 port:8091 ssh_username:root] [2021-02-12 23:21:21,487] - [rest_client] [140099324987136] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.scope_1.coll_1 [2021-02-12 23:21:21,490] - [basetestcase] [140099324987136] - INFO - sleep for 30 secs. Waiting for handler code src_bucket.scope_1.coll_1 to complete bucket operations... Current : 40571 Expected : 40320 ... [2021-02-12 23:21:51,530] - [basetestcase] [140099324987136] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.117 port:8091 ssh_username:root] [2021-02-12 23:21:51,536] - [rest_client] [140099324987136] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.scope_1.coll_1 [2021-02-12 23:21:51,552] - [basetestcase] [140099324987136] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.117 port:8091 ssh_username:root] [2021-02-12 23:21:51,556] - [rest_client] [140099324987136] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.scope_1.coll_1 [2021-02-12 23:21:51,560] - [basetestcase] [140099324987136] - INFO - sleep for 30 secs. Waiting for handler code src_bucket.scope_1.coll_1 to complete bucket operations... Current : 40571 Expected : 40320 ... [2021-02-12 23:22:21,598] - [basetestcase] [140099324987136] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.117 port:8091 ssh_username:root] [2021-02-12 23:22:21,604] - [rest_client] [140099324987136] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.scope_1.coll_1 [2021-02-12 23:22:21,620] - [basetestcase] [140099324987136] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.117 port:8091 ssh_username:root] [2021-02-12 23:22:21,624] - [rest_client] [140099324987136] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.scope_1.coll_1 [2021-02-12 23:22:21,632] - [basetestcase] [140099324987136] - INFO - sleep for 30 secs. Waiting for handler code src_bucket.scope_1.coll_1 to complete bucket operations... Current : 40571 Expected : 40320 ... [2021-02-12 23:22:51,677] - [basetestcase] [140099324987136] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.117 port:8091 ssh_username:root] [2021-02-12 23:22:51,683] - [rest_client] [140099324987136] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.scope_1.coll_1 [2021-02-12 23:22:51,698] - [basetestcase] [140099324987136] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.117 port:8091 ssh_username:root] [2021-02-12 23:22:51,702] - [rest_client] [140099324987136] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.scope_1.coll_1 [2021-02-12 23:22:51,705] - [basetestcase] [140099324987136] - INFO - sleep for 30 secs. Waiting for handler code src_bucket.scope_1.coll_1 to complete bucket operations... Current : 40571 Expected : 40320 ... [2021-02-12 23:23:21,753] - [basetestcase] [140099324987136] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.117 port:8091 ssh_username:root] [2021-02-12 23:23:21,759] - [rest_client] [140099324987136] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.scope_1.coll_1 [2021-02-12 23:23:21,787] - [basetestcase] [140099324987136] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.117 port:8091 ssh_username:root] [2021-02-12 23:23:21,791] - [rest_client] [140099324987136] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.scope_1.coll_1 [2021-02-12 23:23:21,795] - [basetestcase] [140099324987136] - INFO - sleep for 30 secs. Waiting for handler code src_bucket.scope_1.coll_1 to complete bucket operations... Current : 40571 Expected : 40320 ... [2021-02-12 23:23:51,827] - [basetestcase] [140099324987136] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.117 port:8091 ssh_username:root] [2021-02-12 23:23:51,833] - [rest_client] [140099324987136] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.scope_1.coll_1 [2021-02-12 23:23:51,849] - [basetestcase] [140099324987136] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.117 port:8091 ssh_username:root] [2021-02-12 23:23:51,853] - [rest_client] [140099324987136] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.scope_1.coll_1 [2021-02-12 23:23:51,864] - [basetestcase] [140099324987136] - INFO - list of eventing nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root, ip:172.23.122.97 port:8091 ssh_username:root, ip:172.23.123.62 port:8091 ssh_username:root] [2021-02-12 23:23:51,916] - [eventing_base] [140099324987136] - INFO - Stats for Node 172.23.122.82 is [ { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 1, "agg_messages_sent_to_worker": 17283, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_deletion": 26998, "dcp_deletion_sent_to_worker": 13499, "dcp_mutation": 18508, "dcp_mutation_sent_to_worker": 5, "dcp_snapshot": 632, "dcp_stream_req_counter": 342, "dcp_streamreq": 342, "dcp_system_event": 810, "execution_stats": 685, "failure_stats": 685, "latency_stats": 685, "lcb_exception_stats": 685, "log_level": 1, "num_processed_events": 13504, "processed_events_size": 3674104, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 13499, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 5, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 13499, "enqueued_dcp_mutation_msg_counter": 5, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 17279, "num_processed_events": 13504, "on_delete_failure": 0, "on_delete_success": 13499, "on_update_failure": 0, "on_update_success": 5, "processed_events_size": 3674104, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "94178": "2021-02-13T07:23:51Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "94178": "2021-02-13T07:23:51Z" }, "v8worker_events_lost": 0 }, "function_id": 3277597741, "function_name": "bucket_op", "gocb_creds_request_counter": 23, "internal_vb_distribution_stats": { "worker_bucket_op_0": "[0-341]" }, "latency_percentile_stats": { "100": 13400, "50": 400, "80": 800, "90": 1100, "95": 1500, "99": 2800 }, "lcb_creds_request_counter": 24, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.82:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.97:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.123.62:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.82:8096": { "worker_bucket_op_0": "[0-341]" }, "172.23.122.97:8096": { "worker_bucket_op_0": "[342-682]" }, "172.23.123.62:8096": { "worker_bucket_op_0": "[683-1023]" } }, "worker_pids": { "worker_bucket_op_0": 94178 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 3, "agg_messages_sent_to_worker": 15398, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_deletion": 25204, "dcp_deletion_sent_to_worker": 11705, "dcp_mutation": 498, "dcp_snapshot": 342, "dcp_stream_req_counter": 342, "dcp_streamreq": 342, "execution_stats": 668, "failure_stats": 668, "latency_stats": 668, "lcb_exception_stats": 668, "log_level": 1, "num_processed_events": 11705, "processed_events_size": 3183612, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 11705, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 11705, "dcp_delete_msg_counter": 11705, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 0, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 11705, "enqueued_dcp_mutation_msg_counter": 0, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 15394, "num_processed_events": 11705, "on_delete_failure": 0, "on_delete_success": 11705, "on_update_failure": 0, "on_update_success": 0, "processed_events_size": 3183612, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "94208": "2021-02-13T07:23:51Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "94208": "2021-02-13T07:23:51Z" }, "v8worker_events_lost": 0 }, "function_id": 3832248590, "function_name": "curl", "gocb_creds_request_counter": 23, "internal_vb_distribution_stats": { "worker_curl_0": "[0-341]" }, "latency_percentile_stats": { "100": 22900, "50": 1200, "80": 1400, "90": 1600, "95": 1800, "99": 3100 }, "lcb_creds_request_counter": 24, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.82:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.97:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.123.62:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.82:8096": { "worker_curl_0": "[0-341]" }, "172.23.122.97:8096": { "worker_curl_0": "[342-682]" }, "172.23.123.62:8096": { "worker_curl_0": "[683-1023]" } }, "worker_pids": { "worker_curl_0": 94208 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 3, "agg_messages_sent_to_worker": 13245, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_deletion": 23132, "dcp_deletion_sent_to_worker": 9633, "dcp_mutation": 75, "dcp_snapshot": 342, "dcp_stream_req_counter": 342, "dcp_streamreq": 342, "execution_stats": 652, "failure_stats": 652, "latency_stats": 652, "lcb_exception_stats": 652, "log_level": 1, "num_processed_events": 9633, "processed_events_size": 2620176, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 9633, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 0, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 9633, "enqueued_dcp_mutation_msg_counter": 0, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 13241, "num_processed_events": 9633, "on_delete_failure": 0, "on_delete_success": 9633, "on_update_failure": 0, "on_update_success": 0, "processed_events_size": 2620176, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "94236": "2021-02-13T07:23:51Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "94236": "2021-02-13T07:23:51Z" }, "v8worker_events_lost": 0 }, "function_id": 1940050101, "function_name": "n1ql", "gocb_creds_request_counter": 23, "internal_vb_distribution_stats": { "worker_n1ql_0": "[0-341]" }, "latency_percentile_stats": { "100": 12200, "50": 300, "80": 500, "90": 700, "95": 800, "99": 1300 }, "lcb_creds_request_counter": 24, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.82:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.97:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.123.62:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.82:8096": { "worker_n1ql_0": "[0-341]" }, "172.23.122.97:8096": { "worker_n1ql_0": "[342-682]" }, "172.23.123.62:8096": { "worker_n1ql_0": "[683-1023]" } }, "worker_pids": { "worker_n1ql_0": 94236 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 4, "agg_messages_sent_to_worker": 16195, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_deletion": 12664, "dcp_deletion_sent_to_worker": 12664, "dcp_mutation": 75, "dcp_mutation_suppressed_counter": 75, "dcp_snapshot": 342, "dcp_stream_req_counter": 342, "dcp_streamreq": 342, "execution_stats": 636, "failure_stats": 636, "latency_stats": 636, "lcb_exception_stats": 636, "log_level": 1, "num_processed_events": 12664, "processed_events_size": 3444356, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 12664, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 0, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 12664, "enqueued_dcp_mutation_msg_counter": 0, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 16191, "num_processed_events": 12664, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 0, "processed_events_size": 3444356, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "94304": "2021-02-13T07:23:51Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "94304": "2021-02-13T07:23:51Z" }, "v8worker_events_lost": 0 }, "function_id": 2359049777, "function_name": "sbm", "gocb_creds_request_counter": 23, "internal_vb_distribution_stats": { "worker_sbm_0": "[0-341]" }, "latency_percentile_stats": { "100": 600, "50": 100, "80": 100, "90": 100, "95": 100, "99": 100 }, "lcb_creds_request_counter": 24, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.82:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.97:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.123.62:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.82:8096": { "worker_sbm_0": "[0-341]" }, "172.23.122.97:8096": { "worker_sbm_0": "[342-682]" }, "172.23.123.62:8096": { "worker_sbm_0": "[683-1023]" } }, "worker_pids": { "worker_sbm_0": 94304 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 2, "agg_messages_sent_to_worker": 16949, "agg_queue_memory": 0, "agg_queue_memory_cap": 20971520, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_deletion": 26998, "dcp_deletion_sent_to_worker": 13499, "dcp_mutation": 13407, "dcp_snapshot": 342, "dcp_stream_req_counter": 342, "dcp_streamreq": 342, "execution_stats": 620, "failure_stats": 620, "latency_stats": 620, "lcb_exception_stats": 620, "log_level": 1, "num_processed_events": 13499, "processed_events_size": 3669616, "thr_count": 1, "thr_map": 1, "timer_events": 20658, "timer_responses_received": 13499, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 13499, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 0, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 13499, "enqueued_dcp_mutation_msg_counter": 0, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 16945, "num_processed_events": 13499, "on_delete_failure": 0, "on_delete_success": 13499, "on_update_failure": 0, "on_update_success": 0, "processed_events_size": 3669616, "timer_callback_failure": 0, "timer_callback_success": 20658, "timer_cancel_counter": 0, "timer_create_counter": 13499, "timer_create_failure": 0, "timer_msg_counter": 20658, "timer_responses_sent": 0, "timestamp": { "94376": "2021-02-13T07:23:51Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "94376": "2021-02-13T07:23:51Z" }, "v8worker_events_lost": 0 }, "function_id": 146981409, "function_name": "timers", "gocb_creds_request_counter": 23, "internal_vb_distribution_stats": { "worker_timers_0": "[0-341]" }, "latency_percentile_stats": { "100": 15000, "50": 900, "80": 1200, "90": 1400, "95": 1700, "99": 2700 }, "lcb_creds_request_counter": 24, "lcb_exception_stats": {}, "metastore_stats": { "metastore_delete_err": 0, "metastore_deletes": 0, "metastore_not_found": 0, "metastore_scan": 0, "metastore_scan_due": 0, "metastore_scan_err": 0, "metastore_set": 0, "metastore_set_err": 0 }, "planner_stats": [ { "host_name": "172.23.122.82:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.97:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.123.62:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.82:8096": { "worker_timers_0": "[0-341]" }, "172.23.122.97:8096": { "worker_timers_0": "[342-682]" }, "172.23.123.62:8096": { "worker_timers_0": "[683-1023]" } }, "worker_pids": { "worker_timers_0": 94376 } } ] [2021-02-12 23:23:51,961] - [eventing_base] [140099324987136] - INFO - Stats for Node 172.23.122.97 is [ { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 6, "agg_messages_sent_to_worker": 77276, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_deletion": 46224, "dcp_deletion_sent_to_worker": 31661, "dcp_mutation": 125701, "dcp_mutation_sent_to_worker": 39413, "dcp_snapshot": 9597, "dcp_stream_req_counter": 1300, "dcp_streamend": 959, "dcp_streamreq": 1300, "dcp_system_event": 11020, "execution_stats": 786, "failure_stats": 786, "latency_stats": 786, "lcb_exception_stats": 786, "log_level": 1, "num_processed_events": 71074, "processed_events_size": 43898212, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 31661, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 39413, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 31661, "enqueued_dcp_mutation_msg_counter": 39413, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 42808, "lcb_retry_failure": 0, "messages_parsed": 77272, "num_processed_events": 71074, "on_delete_failure": 0, "on_delete_success": 11501, "on_update_failure": 68, "on_update_success": 16697, "processed_events_size": 43898212, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "117017": "2021-02-13T07:23:51Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 68, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "117017": "2021-02-13T07:23:51Z" }, "v8worker_events_lost": 0 }, "function_id": 3277597741, "function_name": "bucket_op", "gocb_creds_request_counter": 23, "internal_vb_distribution_stats": { "worker_bucket_op_0": "[342-682]" }, "latency_percentile_stats": { "100": 2501600, "50": 600, "80": 1100, "90": 1800, "95": 3400, "99": 11300 }, "lcb_creds_request_counter": 27032, "lcb_exception_stats": { "211": 68 }, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.82:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.97:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.123.62:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.82:8096": { "worker_bucket_op_0": "[0-341]" }, "172.23.122.97:8096": { "worker_bucket_op_0": "[342-682]" }, "172.23.123.62:8096": { "worker_bucket_op_0": "[683-1023]" } }, "worker_pids": { "worker_bucket_op_0": 117017 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 4, "agg_messages_sent_to_worker": 43052, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_deletion": 47352, "dcp_deletion_sent_to_worker": 32787, "dcp_mutation": 56297, "dcp_mutation_sent_to_worker": 5043, "dcp_snapshot": 9057, "dcp_stream_req_counter": 853, "dcp_streamend": 512, "dcp_streamreq": 853, "dcp_system_event": 1750, "execution_stats": 769, "failure_stats": 769, "latency_stats": 769, "lcb_exception_stats": 769, "log_level": 1, "num_processed_events": 37830, "processed_events_size": 13429144, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 18196, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 18196, "dcp_delete_msg_counter": 32787, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 5043, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 32787, "enqueued_dcp_mutation_msg_counter": 5043, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 19634, "lcb_retry_failure": 0, "messages_parsed": 43048, "num_processed_events": 37830, "on_delete_failure": 0, "on_delete_success": 14421, "on_update_failure": 0, "on_update_success": 3775, "processed_events_size": 13429144, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "117047": "2021-02-13T07:23:50Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "117047": "2021-02-13T07:23:50Z" }, "v8worker_events_lost": 0 }, "function_id": 3832248590, "function_name": "curl", "gocb_creds_request_counter": 23, "internal_vb_distribution_stats": { "worker_curl_0": "[342-682]" }, "latency_percentile_stats": { "100": 52800, "50": 1300, "80": 1900, "90": 2700, "95": 4600, "99": 13200 }, "lcb_creds_request_counter": 27032, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.82:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.97:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.123.62:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.82:8096": { "worker_curl_0": "[0-341]" }, "172.23.122.97:8096": { "worker_curl_0": "[342-682]" }, "172.23.123.62:8096": { "worker_curl_0": "[683-1023]" } }, "worker_pids": { "worker_curl_0": 117047 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 7, "agg_messages_sent_to_worker": 51788, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_deletion": 46345, "dcp_deletion_sent_to_worker": 31684, "dcp_mutation": 65980, "dcp_mutation_sent_to_worker": 14963, "dcp_snapshot": 9078, "dcp_stream_req_counter": 853, "dcp_streamend": 512, "dcp_streamreq": 853, "dcp_system_event": 1700, "execution_stats": 753, "failure_stats": 753, "latency_stats": 753, "lcb_exception_stats": 753, "log_level": 1, "num_processed_events": 46647, "processed_events_size": 22014392, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 31684, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 14963, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 31684, "enqueued_dcp_mutation_msg_counter": 14963, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 17755, "lcb_retry_failure": 0, "messages_parsed": 51784, "num_processed_events": 46647, "on_delete_failure": 0, "on_delete_success": 15390, "on_update_failure": 0, "on_update_success": 13502, "processed_events_size": 22014392, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "117121": "2021-02-13T07:23:50Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "117121": "2021-02-13T07:23:50Z" }, "v8worker_events_lost": 0 }, "function_id": 1940050101, "function_name": "n1ql", "gocb_creds_request_counter": 23, "internal_vb_distribution_stats": { "worker_n1ql_0": "[342-682]" }, "latency_percentile_stats": { "100": 66500, "50": 2100, "80": 2500, "90": 2900, "95": 4200, "99": 10000 }, "lcb_creds_request_counter": 27032, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.82:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.97:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.123.62:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.82:8096": { "worker_n1ql_0": "[0-341]" }, "172.23.122.97:8096": { "worker_n1ql_0": "[342-682]" }, "172.23.123.62:8096": { "worker_n1ql_0": "[683-1023]" } }, "worker_pids": { "worker_n1ql_0": 117121 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 3, "agg_messages_sent_to_worker": 23108, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_deletion": 40634, "dcp_deletion_sent_to_worker": 14249, "dcp_mutation": 23864, "dcp_mutation_sent_to_worker": 3878, "dcp_mutation_suppressed_counter": 18620, "dcp_snapshot": 9237, "dcp_stream_req_counter": 854, "dcp_stream_req_err_counter": 1, "dcp_streamend": 512, "dcp_streamreq": 853, "dcp_system_event": 1580, "execution_stats": 737, "failure_stats": 737, "latency_stats": 737, "lcb_exception_stats": 737, "log_level": 1, "num_processed_events": 18127, "processed_events_size": 7341016, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 14249, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 3878, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 14249, "enqueued_dcp_mutation_msg_counter": 3878, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 4002, "lcb_retry_failure": 0, "messages_parsed": 23104, "num_processed_events": 18127, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 251, "processed_events_size": 7341016, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "128905": "2021-02-13T07:23:51Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "128905": "2021-02-13T07:23:51Z" }, "v8worker_events_lost": 0 }, "function_id": 2359049777, "function_name": "sbm", "gocb_creds_request_counter": 23, "internal_vb_distribution_stats": { "worker_sbm_0": "[342-682]" }, "latency_percentile_stats": { "100": 18600, "50": 100, "80": 100, "90": 100, "95": 100, "99": 1900 }, "lcb_creds_request_counter": 27032, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.82:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.97:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.123.62:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.82:8096": { "worker_sbm_0": "[0-341]" }, "172.23.122.97:8096": { "worker_sbm_0": "[342-682]" }, "172.23.123.62:8096": { "worker_sbm_0": "[683-1023]" } }, "worker_pids": { "worker_sbm_0": 128905 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 5, "agg_messages_sent_to_worker": 55610, "agg_queue_memory": 0, "agg_queue_memory_cap": 20971520, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_deletion": 48017, "dcp_deletion_sent_to_worker": 33355, "dcp_mutation": 72231, "dcp_mutation_sent_to_worker": 17276, "dcp_snapshot": 8782, "dcp_stream_req_counter": 853, "dcp_streamend": 512, "dcp_streamreq": 853, "dcp_system_event": 2330, "execution_stats": 721, "failure_stats": 721, "latency_stats": 721, "lcb_exception_stats": 721, "log_level": 1, "num_processed_events": 50631, "processed_events_size": 24539592, "thr_count": 1, "thr_map": 1, "timer_events": 21465, "timer_responses_received": 19930, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 33355, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 17276, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 33355, "enqueued_dcp_mutation_msg_counter": 17276, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 30701, "lcb_retry_failure": 0, "messages_parsed": 55606, "num_processed_events": 50631, "on_delete_failure": 0, "on_delete_success": 13195, "on_update_failure": 0, "on_update_success": 6735, "processed_events_size": 24539592, "timer_callback_failure": 14, "timer_callback_success": 21451, "timer_cancel_counter": 0, "timer_create_counter": 19930, "timer_create_failure": 0, "timer_msg_counter": 21465, "timer_responses_sent": 0, "timestamp": { "130681": "2021-02-13T07:23:51Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 14, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "130681": "2021-02-13T07:23:51Z" }, "v8worker_events_lost": 0 }, "function_id": 146981409, "function_name": "timers", "gocb_creds_request_counter": 23, "internal_vb_distribution_stats": { "worker_timers_0": "[342-682]" }, "latency_percentile_stats": { "100": 2501300, "50": 1100, "80": 1500, "90": 1800, "95": 2300, "99": 4900 }, "lcb_creds_request_counter": 27032, "lcb_exception_stats": { "211": 14 }, "metastore_stats": { "metastore_delete_err": 0, "metastore_deletes": 0, "metastore_not_found": 0, "metastore_scan": 0, "metastore_scan_due": 0, "metastore_scan_err": 0, "metastore_set": 0, "metastore_set_err": 0 }, "planner_stats": [ { "host_name": "172.23.122.82:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.97:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.123.62:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.82:8096": { "worker_timers_0": "[0-341]" }, "172.23.122.97:8096": { "worker_timers_0": "[342-682]" }, "172.23.123.62:8096": { "worker_timers_0": "[683-1023]" } }, "worker_pids": { "worker_timers_0": 130681 } } ] [2021-02-12 23:23:52,005] - [eventing_base] [140099324987136] - INFO - Stats for Node 172.23.123.62 is [ { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 8, "agg_messages_sent_to_worker": 93076, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_deletion": 46972, "dcp_deletion_sent_to_worker": 32208, "dcp_mutation": 180841, "dcp_mutation_sent_to_worker": 53170, "dcp_snapshot": 129418, "dcp_stream_req_counter": 1878, "dcp_stream_req_err_counter": 1, "dcp_streamend": 1536, "dcp_streamreq": 1877, "dcp_system_event": 13690, "execution_stats": 855, "failure_stats": 855, "latency_stats": 855, "lcb_exception_stats": 855, "log_level": 1, "num_processed_events": 85378, "processed_events_size": 56371792, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 50993, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 50993, "dcp_delete_msg_counter": 32208, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 53170, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 32208, "enqueued_dcp_mutation_msg_counter": 53170, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 34385, "lcb_retry_failure": 0, "messages_parsed": 93072, "num_processed_events": 85378, "on_delete_failure": 0, "on_delete_success": 14194, "on_update_failure": 0, "on_update_success": 36799, "processed_events_size": 56371792, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "26085": "2021-02-13T07:23:51Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "26085": "2021-02-13T07:23:51Z" }, "v8worker_events_lost": 0 }, "function_id": 3832248590, "function_name": "curl", "gocb_creds_request_counter": 26, "internal_vb_distribution_stats": { "worker_curl_0": "[683-1023]" }, "latency_percentile_stats": { "100": 79200, "50": 1500, "80": 2100, "90": 2800, "95": 4100, "99": 11000 }, "lcb_creds_request_counter": 54595, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.82:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.97:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.123.62:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.82:8096": { "worker_curl_0": "[0-341]" }, "172.23.122.97:8096": { "worker_curl_0": "[342-682]" }, "172.23.123.62:8096": { "worker_curl_0": "[683-1023]" } }, "worker_pids": { "worker_curl_0": 26085 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 6, "agg_messages_sent_to_worker": 91770, "agg_queue_memory": 0, "agg_queue_memory_cap": 20971520, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_deletion": 45053, "dcp_deletion_sent_to_worker": 30310, "dcp_mutation": 100337, "dcp_mutation_sent_to_worker": 53868, "dcp_snapshot": 46337, "dcp_stream_req_counter": 1877, "dcp_streamend": 1536, "dcp_streamreq": 1877, "dcp_system_event": 13660, "execution_stats": 834, "failure_stats": 834, "latency_stats": 834, "lcb_exception_stats": 834, "log_level": 1, "num_processed_events": 84178, "processed_events_size": 56480772, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 30310, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 53868, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 30310, "enqueued_dcp_mutation_msg_counter": 53868, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 41597, "lcb_retry_failure": 0, "messages_parsed": 91766, "num_processed_events": 84178, "on_delete_failure": 0, "on_delete_success": 15297, "on_update_failure": 466, "on_update_success": 26818, "processed_events_size": 56480772, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "26208": "2021-02-13T07:23:51Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 466, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "26208": "2021-02-13T07:23:51Z" }, "v8worker_events_lost": 0 }, "function_id": 1940050101, "function_name": "n1ql", "gocb_creds_request_counter": 26, "internal_vb_distribution_stats": { "worker_n1ql_0": "[683-1023]" }, "latency_percentile_stats": { "100": 105200, "50": 2300, "80": 3900, "90": 5400, "95": 7400, "99": 16800 }, "lcb_creds_request_counter": 54595, "lcb_exception_stats": { "12009": 466, "209": 466 }, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.82:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.97:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.123.62:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.82:8096": { "worker_n1ql_0": "[0-341]" }, "172.23.122.97:8096": { "worker_n1ql_0": "[342-682]" }, "172.23.123.62:8096": { "worker_n1ql_0": "[683-1023]" } }, "worker_pids": { "worker_n1ql_0": 26208 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 6, "agg_messages_sent_to_worker": 103988, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_deletion": 47105, "dcp_deletion_sent_to_worker": 32572, "dcp_mutation": 202141, "dcp_mutation_sent_to_worker": 63395, "dcp_snapshot": 129698, "dcp_stream_req_counter": 1877, "dcp_streamend": 1536, "dcp_streamreq": 1877, "dcp_system_event": 17820, "execution_stats": 919, "failure_stats": 919, "latency_stats": 919, "lcb_exception_stats": 919, "log_level": 1, "num_processed_events": 95967, "processed_events_size": 65625420, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 32572, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 63395, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 32572, "enqueued_dcp_mutation_msg_counter": 63395, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 65184, "lcb_retry_failure": 0, "messages_parsed": 103984, "num_processed_events": 95967, "on_delete_failure": 0, "on_delete_success": 15320, "on_update_failure": 61, "on_update_success": 15402, "processed_events_size": 65625420, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "25929": "2021-02-13T07:23:51Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 61, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "25929": "2021-02-13T07:23:51Z" }, "v8worker_events_lost": 0 }, "function_id": 3277597741, "function_name": "bucket_op", "gocb_creds_request_counter": 26, "internal_vb_distribution_stats": { "worker_bucket_op_0": "[683-1023]" }, "latency_percentile_stats": { "100": 2514100, "50": 500, "80": 1000, "90": 1700, "95": 3300, "99": 11000 }, "lcb_creds_request_counter": 54595, "lcb_exception_stats": { "211": 61 }, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.82:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.97:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.123.62:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.82:8096": { "worker_bucket_op_0": "[0-341]" }, "172.23.122.97:8096": { "worker_bucket_op_0": "[342-682]" }, "172.23.123.62:8096": { "worker_bucket_op_0": "[683-1023]" } }, "worker_pids": { "worker_bucket_op_0": 25929 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 5, "agg_messages_sent_to_worker": 103625, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_deletion": 48251, "dcp_deletion_sent_to_worker": 33442, "dcp_mutation": 203256, "dcp_mutation_sent_to_worker": 62268, "dcp_snapshot": 130212, "dcp_stream_req_counter": 1877, "dcp_streamend": 1536, "dcp_streamreq": 1877, "dcp_system_event": 14120, "execution_stats": 898, "failure_stats": 898, "latency_stats": 898, "lcb_exception_stats": 898, "log_level": 1, "num_processed_events": 95710, "processed_events_size": 64853000, "thr_count": 1, "thr_map": 1, "timer_events": 34722, "timer_responses_received": 36143, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 33442, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 62268, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 33442, "enqueued_dcp_mutation_msg_counter": 62268, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 59567, "lcb_retry_failure": 0, "messages_parsed": 103621, "num_processed_events": 95710, "on_delete_failure": 0, "on_delete_success": 13626, "on_update_failure": 0, "on_update_success": 22517, "processed_events_size": 64853000, "timer_callback_failure": 86, "timer_callback_success": 34636, "timer_cancel_counter": 0, "timer_create_counter": 36143, "timer_create_failure": 0, "timer_msg_counter": 34722, "timer_responses_sent": 0, "timestamp": { "25969": "2021-02-13T07:23:51Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 86, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "25969": "2021-02-13T07:23:51Z" }, "v8worker_events_lost": 0 }, "function_id": 146981409, "function_name": "timers", "gocb_creds_request_counter": 26, "internal_vb_distribution_stats": { "worker_timers_0": "[683-1023]" }, "latency_percentile_stats": { "100": 2504600, "50": 1000, "80": 1600, "90": 2500, "95": 3500, "99": 9600 }, "lcb_creds_request_counter": 54595, "lcb_exception_stats": { "211": 86 }, "metastore_stats": { "metastore_delete_err": 0, "metastore_deletes": 0, "metastore_not_found": 0, "metastore_scan": 0, "metastore_scan_due": 0, "metastore_scan_err": 0, "metastore_set": 0, "metastore_set_err": 0 }, "planner_stats": [ { "host_name": "172.23.122.82:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.97:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.123.62:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.82:8096": { "worker_timers_0": "[0-341]" }, "172.23.122.97:8096": { "worker_timers_0": "[342-682]" }, "172.23.123.62:8096": { "worker_timers_0": "[683-1023]" } }, "worker_pids": { "worker_timers_0": 25969 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 4, "agg_messages_sent_to_worker": 74751, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_deletion": 37550, "dcp_deletion_sent_to_worker": 13972, "dcp_mutation": 164628, "dcp_mutation_sent_to_worker": 53010, "dcp_mutation_suppressed_counter": 60257, "dcp_snapshot": 130180, "dcp_stream_req_counter": 1878, "dcp_stream_req_err_counter": 1, "dcp_streamend": 1536, "dcp_streamreq": 1877, "dcp_system_event": 13670, "execution_stats": 877, "failure_stats": 877, "latency_stats": 877, "lcb_exception_stats": 877, "log_level": 1, "num_processed_events": 66982, "processed_events_size": 51257308, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 13972, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 53010, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 13972, "enqueued_dcp_mutation_msg_counter": 53010, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 12880, "lcb_retry_failure": 0, "messages_parsed": 74747, "num_processed_events": 66982, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 40320, "processed_events_size": 51257308, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "26012": "2021-02-13T07:23:51Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "26012": "2021-02-13T07:23:51Z" }, "v8worker_events_lost": 0 }, "function_id": 2359049777, "function_name": "sbm", "gocb_creds_request_counter": 26, "internal_vb_distribution_stats": { "worker_sbm_0": "[683-1023]" }, "latency_percentile_stats": { "100": 24200, "50": 500, "80": 800, "90": 1100, "95": 1500, "99": 3500 }, "lcb_creds_request_counter": 54595, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.82:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.97:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.123.62:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.82:8096": { "worker_sbm_0": "[0-341]" }, "172.23.122.97:8096": { "worker_sbm_0": "[342-682]" }, "172.23.123.62:8096": { "worker_sbm_0": "[683-1023]" } }, "worker_pids": { "worker_sbm_0": 26012 } } ] [2021-02-12 23:23:57,735] - [eventing_base] [140099324987136] - INFO - Stats for Node 172.23.122.82 is [ { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 2, "agg_messages_sent_to_worker": 16949, "agg_queue_memory": 0, "agg_queue_memory_cap": 20971520, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_deletion": 26998, "dcp_deletion_sent_to_worker": 13499, "dcp_mutation": 13407, "dcp_snapshot": 342, "dcp_stream_req_counter": 342, "dcp_streamreq": 342, "execution_stats": 620, "failure_stats": 620, "latency_stats": 620, "lcb_exception_stats": 620, "log_level": 1, "num_processed_events": 13499, "processed_events_size": 3669616, "thr_count": 1, "thr_map": 1, "timer_events": 20658, "timer_responses_received": 13499, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 13499, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 0, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 13499, "enqueued_dcp_mutation_msg_counter": 0, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 16945, "num_processed_events": 13499, "on_delete_failure": 0, "on_delete_success": 13499, "on_update_failure": 0, "on_update_success": 0, "processed_events_size": 3669616, "timer_callback_failure": 0, "timer_callback_success": 20658, "timer_cancel_counter": 0, "timer_create_counter": 13499, "timer_create_failure": 0, "timer_msg_counter": 20658, "timer_responses_sent": 0, "timestamp": { "94376": "2021-02-13T07:23:51Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "94376": "2021-02-13T07:23:51Z" }, "v8worker_events_lost": 0 }, "function_id": 146981409, "function_name": "timers", "gocb_creds_request_counter": 23, "internal_vb_distribution_stats": { "worker_timers_0": "[0-341]" }, "latency_percentile_stats": { "100": 15000, "50": 900, "80": 1200, "90": 1400, "95": 1700, "99": 2700 }, "lcb_creds_request_counter": 24, "lcb_exception_stats": {}, "metastore_stats": { "metastore_delete_err": 0, "metastore_deletes": 0, "metastore_not_found": 0, "metastore_scan": 0, "metastore_scan_due": 0, "metastore_scan_err": 0, "metastore_set": 0, "metastore_set_err": 0 }, "planner_stats": [ { "host_name": "172.23.122.82:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.97:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.123.62:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.82:8096": { "worker_timers_0": "[0-341]" }, "172.23.122.97:8096": { "worker_timers_0": "[342-682]" }, "172.23.123.62:8096": { "worker_timers_0": "[683-1023]" } }, "worker_pids": { "worker_timers_0": 94376 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 1, "agg_messages_sent_to_worker": 17283, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_deletion": 26998, "dcp_deletion_sent_to_worker": 13499, "dcp_mutation": 18508, "dcp_mutation_sent_to_worker": 5, "dcp_snapshot": 632, "dcp_stream_req_counter": 342, "dcp_streamreq": 342, "dcp_system_event": 810, "execution_stats": 685, "failure_stats": 685, "latency_stats": 685, "lcb_exception_stats": 685, "log_level": 1, "num_processed_events": 13504, "processed_events_size": 3674104, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 13499, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 5, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 13499, "enqueued_dcp_mutation_msg_counter": 5, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 17279, "num_processed_events": 13504, "on_delete_failure": 0, "on_delete_success": 13499, "on_update_failure": 0, "on_update_success": 5, "processed_events_size": 3674104, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "94178": "2021-02-13T07:23:51Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "94178": "2021-02-13T07:23:51Z" }, "v8worker_events_lost": 0 }, "function_id": 3277597741, "function_name": "bucket_op", "gocb_creds_request_counter": 23, "internal_vb_distribution_stats": { "worker_bucket_op_0": "[0-341]" }, "latency_percentile_stats": { "100": 13400, "50": 400, "80": 800, "90": 1100, "95": 1500, "99": 2800 }, "lcb_creds_request_counter": 24, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.82:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.97:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.123.62:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.82:8096": { "worker_bucket_op_0": "[0-341]" }, "172.23.122.97:8096": { "worker_bucket_op_0": "[342-682]" }, "172.23.123.62:8096": { "worker_bucket_op_0": "[683-1023]" } }, "worker_pids": { "worker_bucket_op_0": 94178 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 3, "agg_messages_sent_to_worker": 15398, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_deletion": 25204, "dcp_deletion_sent_to_worker": 11705, "dcp_mutation": 498, "dcp_snapshot": 342, "dcp_stream_req_counter": 342, "dcp_streamreq": 342, "execution_stats": 668, "failure_stats": 668, "latency_stats": 668, "lcb_exception_stats": 668, "log_level": 1, "num_processed_events": 11705, "processed_events_size": 3183612, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 11705, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 11705, "dcp_delete_msg_counter": 11705, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 0, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 11705, "enqueued_dcp_mutation_msg_counter": 0, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 15394, "num_processed_events": 11705, "on_delete_failure": 0, "on_delete_success": 11705, "on_update_failure": 0, "on_update_success": 0, "processed_events_size": 3183612, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "94208": "2021-02-13T07:23:51Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "94208": "2021-02-13T07:23:51Z" }, "v8worker_events_lost": 0 }, "function_id": 3832248590, "function_name": "curl", "gocb_creds_request_counter": 23, "internal_vb_distribution_stats": { "worker_curl_0": "[0-341]" }, "latency_percentile_stats": { "100": 22900, "50": 1200, "80": 1400, "90": 1600, "95": 1800, "99": 3100 }, "lcb_creds_request_counter": 24, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.82:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.97:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.123.62:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.82:8096": { "worker_curl_0": "[0-341]" }, "172.23.122.97:8096": { "worker_curl_0": "[342-682]" }, "172.23.123.62:8096": { "worker_curl_0": "[683-1023]" } }, "worker_pids": { "worker_curl_0": 94208 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 3, "agg_messages_sent_to_worker": 13245, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_deletion": 23132, "dcp_deletion_sent_to_worker": 9633, "dcp_mutation": 75, "dcp_snapshot": 342, "dcp_stream_req_counter": 342, "dcp_streamreq": 342, "execution_stats": 652, "failure_stats": 652, "latency_stats": 652, "lcb_exception_stats": 652, "log_level": 1, "num_processed_events": 9633, "processed_events_size": 2620176, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 9633, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 0, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 9633, "enqueued_dcp_mutation_msg_counter": 0, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 13241, "num_processed_events": 9633, "on_delete_failure": 0, "on_delete_success": 9633, "on_update_failure": 0, "on_update_success": 0, "processed_events_size": 2620176, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "94236": "2021-02-13T07:23:51Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "94236": "2021-02-13T07:23:51Z" }, "v8worker_events_lost": 0 }, "function_id": 1940050101, "function_name": "n1ql", "gocb_creds_request_counter": 23, "internal_vb_distribution_stats": { "worker_n1ql_0": "[0-341]" }, "latency_percentile_stats": { "100": 12200, "50": 300, "80": 500, "90": 700, "95": 800, "99": 1300 }, "lcb_creds_request_counter": 24, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.82:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.97:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.123.62:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.82:8096": { "worker_n1ql_0": "[0-341]" }, "172.23.122.97:8096": { "worker_n1ql_0": "[342-682]" }, "172.23.123.62:8096": { "worker_n1ql_0": "[683-1023]" } }, "worker_pids": { "worker_n1ql_0": 94236 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 4, "agg_messages_sent_to_worker": 16195, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_deletion": 12664, "dcp_deletion_sent_to_worker": 12664, "dcp_mutation": 75, "dcp_mutation_suppressed_counter": 75, "dcp_snapshot": 342, "dcp_stream_req_counter": 342, "dcp_streamreq": 342, "execution_stats": 636, "failure_stats": 636, "latency_stats": 636, "lcb_exception_stats": 636, "log_level": 1, "num_processed_events": 12664, "processed_events_size": 3444356, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 12664, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 0, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 12664, "enqueued_dcp_mutation_msg_counter": 0, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 16191, "num_processed_events": 12664, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 0, "processed_events_size": 3444356, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "94304": "2021-02-13T07:23:51Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "94304": "2021-02-13T07:23:51Z" }, "v8worker_events_lost": 0 }, "function_id": 2359049777, "function_name": "sbm", "gocb_creds_request_counter": 23, "internal_vb_distribution_stats": { "worker_sbm_0": "[0-341]" }, "latency_percentile_stats": { "100": 600, "50": 100, "80": 100, "90": 100, "95": 100, "99": 100 }, "lcb_creds_request_counter": 24, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.82:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.97:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.123.62:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.82:8096": { "worker_sbm_0": "[0-341]" }, "172.23.122.97:8096": { "worker_sbm_0": "[342-682]" }, "172.23.123.62:8096": { "worker_sbm_0": "[683-1023]" } }, "worker_pids": { "worker_sbm_0": 94304 } } ] [2021-02-12 23:24:03,772] - [eventing_base] [140099324987136] - INFO - Stats for Node 172.23.122.97 is [ { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 4, "agg_messages_sent_to_worker": 43087, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_deletion": 47352, "dcp_deletion_sent_to_worker": 32787, "dcp_mutation": 56297, "dcp_mutation_sent_to_worker": 5043, "dcp_snapshot": 9057, "dcp_stream_req_counter": 853, "dcp_streamend": 512, "dcp_streamreq": 853, "dcp_system_event": 1750, "execution_stats": 776, "failure_stats": 776, "latency_stats": 776, "lcb_exception_stats": 776, "log_level": 1, "num_processed_events": 37830, "processed_events_size": 13429144, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 18196, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 18196, "dcp_delete_msg_counter": 32787, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 5043, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 32787, "enqueued_dcp_mutation_msg_counter": 5043, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 19634, "lcb_retry_failure": 0, "messages_parsed": 43083, "num_processed_events": 37830, "on_delete_failure": 0, "on_delete_success": 14421, "on_update_failure": 0, "on_update_success": 3775, "processed_events_size": 13429144, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "117047": "2021-02-13T07:23:57Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "117047": "2021-02-13T07:23:57Z" }, "v8worker_events_lost": 0 }, "function_id": 3832248590, "function_name": "curl", "gocb_creds_request_counter": 23, "internal_vb_distribution_stats": { "worker_curl_0": "[342-682]" }, "latency_percentile_stats": { "100": 52800, "50": 1300, "80": 1900, "90": 2700, "95": 4600, "99": 13200 }, "lcb_creds_request_counter": 27032, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.82:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.97:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.123.62:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.82:8096": { "worker_curl_0": "[0-341]" }, "172.23.122.97:8096": { "worker_curl_0": "[342-682]" }, "172.23.123.62:8096": { "worker_curl_0": "[683-1023]" } }, "worker_pids": { "worker_curl_0": 117047 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 7, "agg_messages_sent_to_worker": 51823, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_deletion": 46345, "dcp_deletion_sent_to_worker": 31684, "dcp_mutation": 65980, "dcp_mutation_sent_to_worker": 14963, "dcp_snapshot": 9078, "dcp_stream_req_counter": 853, "dcp_streamend": 512, "dcp_streamreq": 853, "dcp_system_event": 1700, "execution_stats": 760, "failure_stats": 760, "latency_stats": 760, "lcb_exception_stats": 760, "log_level": 1, "num_processed_events": 46647, "processed_events_size": 22014392, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 31684, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 14963, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 31684, "enqueued_dcp_mutation_msg_counter": 14963, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 17755, "lcb_retry_failure": 0, "messages_parsed": 51819, "num_processed_events": 46647, "on_delete_failure": 0, "on_delete_success": 15390, "on_update_failure": 0, "on_update_success": 13502, "processed_events_size": 22014392, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "117121": "2021-02-13T07:23:57Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "117121": "2021-02-13T07:23:57Z" }, "v8worker_events_lost": 0 }, "function_id": 1940050101, "function_name": "n1ql", "gocb_creds_request_counter": 23, "internal_vb_distribution_stats": { "worker_n1ql_0": "[342-682]" }, "latency_percentile_stats": { "100": 66500, "50": 2100, "80": 2500, "90": 2900, "95": 4200, "99": 10000 }, "lcb_creds_request_counter": 27032, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.82:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.97:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.123.62:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.82:8096": { "worker_n1ql_0": "[0-341]" }, "172.23.122.97:8096": { "worker_n1ql_0": "[342-682]" }, "172.23.123.62:8096": { "worker_n1ql_0": "[683-1023]" } }, "worker_pids": { "worker_n1ql_0": 117121 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 3, "agg_messages_sent_to_worker": 23143, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_deletion": 40634, "dcp_deletion_sent_to_worker": 14249, "dcp_mutation": 23864, "dcp_mutation_sent_to_worker": 3878, "dcp_mutation_suppressed_counter": 18620, "dcp_snapshot": 9237, "dcp_stream_req_counter": 854, "dcp_stream_req_err_counter": 1, "dcp_streamend": 512, "dcp_streamreq": 853, "dcp_system_event": 1580, "execution_stats": 744, "failure_stats": 744, "latency_stats": 744, "lcb_exception_stats": 744, "log_level": 1, "num_processed_events": 18127, "processed_events_size": 7341016, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 14249, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 3878, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 14249, "enqueued_dcp_mutation_msg_counter": 3878, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 4002, "lcb_retry_failure": 0, "messages_parsed": 23139, "num_processed_events": 18127, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 251, "processed_events_size": 7341016, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "128905": "2021-02-13T07:23:58Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "128905": "2021-02-13T07:23:58Z" }, "v8worker_events_lost": 0 }, "function_id": 2359049777, "function_name": "sbm", "gocb_creds_request_counter": 23, "internal_vb_distribution_stats": { "worker_sbm_0": "[342-682]" }, "latency_percentile_stats": { "100": 18600, "50": 100, "80": 100, "90": 100, "95": 100, "99": 1900 }, "lcb_creds_request_counter": 27032, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.82:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.97:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.123.62:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.82:8096": { "worker_sbm_0": "[0-341]" }, "172.23.122.97:8096": { "worker_sbm_0": "[342-682]" }, "172.23.123.62:8096": { "worker_sbm_0": "[683-1023]" } }, "worker_pids": { "worker_sbm_0": 128905 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 5, "agg_messages_sent_to_worker": 55645, "agg_queue_memory": 0, "agg_queue_memory_cap": 20971520, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_deletion": 48017, "dcp_deletion_sent_to_worker": 33355, "dcp_mutation": 72231, "dcp_mutation_sent_to_worker": 17276, "dcp_snapshot": 8782, "dcp_stream_req_counter": 853, "dcp_streamend": 512, "dcp_streamreq": 853, "dcp_system_event": 2330, "execution_stats": 728, "failure_stats": 728, "latency_stats": 728, "lcb_exception_stats": 728, "log_level": 1, "num_processed_events": 50631, "processed_events_size": 24539592, "thr_count": 1, "thr_map": 1, "timer_events": 21465, "timer_responses_received": 19930, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 33355, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 17276, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 33355, "enqueued_dcp_mutation_msg_counter": 17276, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 30701, "lcb_retry_failure": 0, "messages_parsed": 55641, "num_processed_events": 50631, "on_delete_failure": 0, "on_delete_success": 13195, "on_update_failure": 0, "on_update_success": 6735, "processed_events_size": 24539592, "timer_callback_failure": 14, "timer_callback_success": 21451, "timer_cancel_counter": 0, "timer_create_counter": 19930, "timer_create_failure": 0, "timer_msg_counter": 21465, "timer_responses_sent": 0, "timestamp": { "130681": "2021-02-13T07:23:58Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 14, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "130681": "2021-02-13T07:23:58Z" }, "v8worker_events_lost": 0 }, "function_id": 146981409, "function_name": "timers", "gocb_creds_request_counter": 23, "internal_vb_distribution_stats": { "worker_timers_0": "[342-682]" }, "latency_percentile_stats": { "100": 2501300, "50": 1100, "80": 1500, "90": 1800, "95": 2300, "99": 4900 }, "lcb_creds_request_counter": 27032, "lcb_exception_stats": { "211": 14 }, "metastore_stats": { "metastore_delete_err": 0, "metastore_deletes": 0, "metastore_not_found": 0, "metastore_scan": 0, "metastore_scan_due": 0, "metastore_scan_err": 0, "metastore_set": 0, "metastore_set_err": 0 }, "planner_stats": [ { "host_name": "172.23.122.82:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.97:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.123.62:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.82:8096": { "worker_timers_0": "[0-341]" }, "172.23.122.97:8096": { "worker_timers_0": "[342-682]" }, "172.23.123.62:8096": { "worker_timers_0": "[683-1023]" } }, "worker_pids": { "worker_timers_0": 130681 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 6, "agg_messages_sent_to_worker": 77311, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_deletion": 46224, "dcp_deletion_sent_to_worker": 31661, "dcp_mutation": 125701, "dcp_mutation_sent_to_worker": 39413, "dcp_snapshot": 9597, "dcp_stream_req_counter": 1300, "dcp_streamend": 959, "dcp_streamreq": 1300, "dcp_system_event": 11020, "execution_stats": 793, "failure_stats": 793, "latency_stats": 793, "lcb_exception_stats": 793, "log_level": 1, "num_processed_events": 71074, "processed_events_size": 43898212, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 31661, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 39413, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 31661, "enqueued_dcp_mutation_msg_counter": 39413, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 42808, "lcb_retry_failure": 0, "messages_parsed": 77307, "num_processed_events": 71074, "on_delete_failure": 0, "on_delete_success": 11501, "on_update_failure": 68, "on_update_success": 16697, "processed_events_size": 43898212, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "117017": "2021-02-13T07:23:58Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 68, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "117017": "2021-02-13T07:23:58Z" }, "v8worker_events_lost": 0 }, "function_id": 3277597741, "function_name": "bucket_op", "gocb_creds_request_counter": 23, "internal_vb_distribution_stats": { "worker_bucket_op_0": "[342-682]" }, "latency_percentile_stats": { "100": 2501600, "50": 600, "80": 1100, "90": 1800, "95": 3400, "99": 11300 }, "lcb_creds_request_counter": 27032, "lcb_exception_stats": { "211": 68 }, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.82:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.97:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.123.62:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.82:8096": { "worker_bucket_op_0": "[0-341]" }, "172.23.122.97:8096": { "worker_bucket_op_0": "[342-682]" }, "172.23.123.62:8096": { "worker_bucket_op_0": "[683-1023]" } }, "worker_pids": { "worker_bucket_op_0": 117017 } } ] [2021-02-12 23:24:09,171] - [eventing_base] [140099324987136] - INFO - Stats for Node 172.23.123.62 is [ { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 6, "agg_messages_sent_to_worker": 91835, "agg_queue_memory": 0, "agg_queue_memory_cap": 20971520, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_deletion": 45053, "dcp_deletion_sent_to_worker": 30310, "dcp_mutation": 100337, "dcp_mutation_sent_to_worker": 53868, "dcp_snapshot": 46337, "dcp_stream_req_counter": 1877, "dcp_streamend": 1536, "dcp_streamreq": 1877, "dcp_system_event": 13660, "execution_stats": 847, "failure_stats": 847, "latency_stats": 847, "lcb_exception_stats": 847, "log_level": 1, "num_processed_events": 84178, "processed_events_size": 56480772, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 30310, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 53868, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 30310, "enqueued_dcp_mutation_msg_counter": 53868, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 41597, "lcb_retry_failure": 0, "messages_parsed": 91831, "num_processed_events": 84178, "on_delete_failure": 0, "on_delete_success": 15297, "on_update_failure": 466, "on_update_success": 26818, "processed_events_size": 56480772, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "26208": "2021-02-13T07:24:04Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 466, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "26208": "2021-02-13T07:24:04Z" }, "v8worker_events_lost": 0 }, "function_id": 1940050101, "function_name": "n1ql", "gocb_creds_request_counter": 26, "internal_vb_distribution_stats": { "worker_n1ql_0": "[683-1023]" }, "latency_percentile_stats": { "100": 105200, "50": 2300, "80": 3900, "90": 5400, "95": 7400, "99": 16800 }, "lcb_creds_request_counter": 54595, "lcb_exception_stats": { "12009": 466, "209": 466 }, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.82:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.97:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.123.62:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.82:8096": { "worker_n1ql_0": "[0-341]" }, "172.23.122.97:8096": { "worker_n1ql_0": "[342-682]" }, "172.23.123.62:8096": { "worker_n1ql_0": "[683-1023]" } }, "worker_pids": { "worker_n1ql_0": 26208 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 6, "agg_messages_sent_to_worker": 104053, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_deletion": 47105, "dcp_deletion_sent_to_worker": 32572, "dcp_mutation": 202141, "dcp_mutation_sent_to_worker": 63395, "dcp_snapshot": 129698, "dcp_stream_req_counter": 1877, "dcp_streamend": 1536, "dcp_streamreq": 1877, "dcp_system_event": 17820, "execution_stats": 932, "failure_stats": 932, "latency_stats": 932, "lcb_exception_stats": 932, "log_level": 1, "num_processed_events": 95967, "processed_events_size": 65625420, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 32572, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 63395, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 32572, "enqueued_dcp_mutation_msg_counter": 63395, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 65184, "lcb_retry_failure": 0, "messages_parsed": 104049, "num_processed_events": 95967, "on_delete_failure": 0, "on_delete_success": 15320, "on_update_failure": 61, "on_update_success": 15402, "processed_events_size": 65625420, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "25929": "2021-02-13T07:24:04Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 61, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "25929": "2021-02-13T07:24:04Z" }, "v8worker_events_lost": 0 }, "function_id": 3277597741, "function_name": "bucket_op", "gocb_creds_request_counter": 26, "internal_vb_distribution_stats": { "worker_bucket_op_0": "[683-1023]" }, "latency_percentile_stats": { "100": 2514100, "50": 500, "80": 1000, "90": 1700, "95": 3300, "99": 11000 }, "lcb_creds_request_counter": 54595, "lcb_exception_stats": { "211": 61 }, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.82:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.97:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.123.62:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.82:8096": { "worker_bucket_op_0": "[0-341]" }, "172.23.122.97:8096": { "worker_bucket_op_0": "[342-682]" }, "172.23.123.62:8096": { "worker_bucket_op_0": "[683-1023]" } }, "worker_pids": { "worker_bucket_op_0": 25929 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 5, "agg_messages_sent_to_worker": 103690, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_deletion": 48251, "dcp_deletion_sent_to_worker": 33442, "dcp_mutation": 203256, "dcp_mutation_sent_to_worker": 62268, "dcp_snapshot": 130212, "dcp_stream_req_counter": 1877, "dcp_streamend": 1536, "dcp_streamreq": 1877, "dcp_system_event": 14120, "execution_stats": 911, "failure_stats": 911, "latency_stats": 911, "lcb_exception_stats": 911, "log_level": 1, "num_processed_events": 95710, "processed_events_size": 64853000, "thr_count": 1, "thr_map": 1, "timer_events": 34722, "timer_responses_received": 36143, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 33442, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 62268, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 33442, "enqueued_dcp_mutation_msg_counter": 62268, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 59567, "lcb_retry_failure": 0, "messages_parsed": 103686, "num_processed_events": 95710, "on_delete_failure": 0, "on_delete_success": 13626, "on_update_failure": 0, "on_update_success": 22517, "processed_events_size": 64853000, "timer_callback_failure": 86, "timer_callback_success": 34636, "timer_cancel_counter": 0, "timer_create_counter": 36143, "timer_create_failure": 0, "timer_msg_counter": 34722, "timer_responses_sent": 0, "timestamp": { "25969": "2021-02-13T07:24:04Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 86, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "25969": "2021-02-13T07:24:04Z" }, "v8worker_events_lost": 0 }, "function_id": 146981409, "function_name": "timers", "gocb_creds_request_counter": 26, "internal_vb_distribution_stats": { "worker_timers_0": "[683-1023]" }, "latency_percentile_stats": { "100": 2504600, "50": 1000, "80": 1600, "90": 2500, "95": 3500, "99": 9600 }, "lcb_creds_request_counter": 54595, "lcb_exception_stats": { "211": 86 }, "metastore_stats": { "metastore_delete_err": 0, "metastore_deletes": 0, "metastore_not_found": 0, "metastore_scan": 0, "metastore_scan_due": 0, "metastore_scan_err": 0, "metastore_set": 0, "metastore_set_err": 0 }, "planner_stats": [ { "host_name": "172.23.122.82:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.97:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.123.62:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.82:8096": { "worker_timers_0": "[0-341]" }, "172.23.122.97:8096": { "worker_timers_0": "[342-682]" }, "172.23.123.62:8096": { "worker_timers_0": "[683-1023]" } }, "worker_pids": { "worker_timers_0": 25969 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 4, "agg_messages_sent_to_worker": 74811, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_deletion": 37550, "dcp_deletion_sent_to_worker": 13972, "dcp_mutation": 164628, "dcp_mutation_sent_to_worker": 53010, "dcp_mutation_suppressed_counter": 60257, "dcp_snapshot": 130180, "dcp_stream_req_counter": 1878, "dcp_stream_req_err_counter": 1, "dcp_streamend": 1536, "dcp_streamreq": 1877, "dcp_system_event": 13670, "execution_stats": 889, "failure_stats": 889, "latency_stats": 889, "lcb_exception_stats": 889, "log_level": 1, "num_processed_events": 66982, "processed_events_size": 51257308, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 13972, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 53010, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 13972, "enqueued_dcp_mutation_msg_counter": 53010, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 12880, "lcb_retry_failure": 0, "messages_parsed": 74807, "num_processed_events": 66982, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 40320, "processed_events_size": 51257308, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "26012": "2021-02-13T07:24:03Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "26012": "2021-02-13T07:24:03Z" }, "v8worker_events_lost": 0 }, "function_id": 2359049777, "function_name": "sbm", "gocb_creds_request_counter": 26, "internal_vb_distribution_stats": { "worker_sbm_0": "[683-1023]" }, "latency_percentile_stats": { "100": 24200, "50": 500, "80": 800, "90": 1100, "95": 1500, "99": 3500 }, "lcb_creds_request_counter": 54595, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.82:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.97:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.123.62:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.82:8096": { "worker_sbm_0": "[0-341]" }, "172.23.122.97:8096": { "worker_sbm_0": "[342-682]" }, "172.23.123.62:8096": { "worker_sbm_0": "[683-1023]" } }, "worker_pids": { "worker_sbm_0": 26012 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 8, "agg_messages_sent_to_worker": 93141, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_deletion": 46972, "dcp_deletion_sent_to_worker": 32208, "dcp_mutation": 180841, "dcp_mutation_sent_to_worker": 53170, "dcp_snapshot": 129418, "dcp_stream_req_counter": 1878, "dcp_stream_req_err_counter": 1, "dcp_streamend": 1536, "dcp_streamreq": 1877, "dcp_system_event": 13690, "execution_stats": 868, "failure_stats": 868, "latency_stats": 868, "lcb_exception_stats": 868, "log_level": 1, "num_processed_events": 85378, "processed_events_size": 56371792, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 50993, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 50993, "dcp_delete_msg_counter": 32208, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 53170, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 32208, "enqueued_dcp_mutation_msg_counter": 53170, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 34385, "lcb_retry_failure": 0, "messages_parsed": 93137, "num_processed_events": 85378, "on_delete_failure": 0, "on_delete_success": 14194, "on_update_failure": 0, "on_update_success": 36799, "processed_events_size": 56371792, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "26085": "2021-02-13T07:24:04Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "26085": "2021-02-13T07:24:04Z" }, "v8worker_events_lost": 0 }, "function_id": 3832248590, "function_name": "curl", "gocb_creds_request_counter": 26, "internal_vb_distribution_stats": { "worker_curl_0": "[683-1023]" }, "latency_percentile_stats": { "100": 79200, "50": 1500, "80": 2100, "90": 2800, "95": 4100, "99": 11000 }, "lcb_creds_request_counter": 54595, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.82:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.97:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.123.62:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.82:8096": { "worker_curl_0": "[0-341]" }, "172.23.122.97:8096": { "worker_curl_0": "[342-682]" }, "172.23.123.62:8096": { "worker_curl_0": "[683-1023]" } }, "worker_pids": { "worker_curl_0": 26085 } } ] [2021-02-12 23:24:10,271] - [eventing_base] [140099324987136] - INFO - Execution stats update_success: 30593 update_failure: 466 delete_success: 43217 delete_failure: 0 [2021-02-12 23:24:10,287] - [basetestcase] [140099324987136] - INFO - list of eventing nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root, ip:172.23.122.97 port:8091 ssh_username:root, ip:172.23.123.62 port:8091 ssh_username:root] [2021-02-12 23:24:10,330] - [eventing_base] [140099324987136] - INFO - Stats for Node 172.23.122.82 is [ { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 1, "agg_messages_sent_to_worker": 17373, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_deletion": 26998, "dcp_deletion_sent_to_worker": 13499, "dcp_mutation": 18508, "dcp_mutation_sent_to_worker": 5, "dcp_snapshot": 632, "dcp_stream_req_counter": 342, "dcp_streamreq": 342, "dcp_system_event": 810, "execution_stats": 703, "failure_stats": 703, "latency_stats": 703, "lcb_exception_stats": 703, "log_level": 1, "num_processed_events": 13504, "processed_events_size": 3674104, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 13499, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 5, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 13499, "enqueued_dcp_mutation_msg_counter": 5, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 17369, "num_processed_events": 13504, "on_delete_failure": 0, "on_delete_success": 13499, "on_update_failure": 0, "on_update_success": 5, "processed_events_size": 3674104, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "94178": "2021-02-13T07:24:09Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "94178": "2021-02-13T07:24:09Z" }, "v8worker_events_lost": 0 }, "function_id": 3277597741, "function_name": "bucket_op", "gocb_creds_request_counter": 23, "internal_vb_distribution_stats": { "worker_bucket_op_0": "[0-341]" }, "latency_percentile_stats": { "100": 13400, "50": 400, "80": 800, "90": 1100, "95": 1500, "99": 2800 }, "lcb_creds_request_counter": 24, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.82:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.97:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.123.62:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.82:8096": { "worker_bucket_op_0": "[0-341]" }, "172.23.122.97:8096": { "worker_bucket_op_0": "[342-682]" }, "172.23.123.62:8096": { "worker_bucket_op_0": "[683-1023]" } }, "worker_pids": { "worker_bucket_op_0": 94178 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 3, "agg_messages_sent_to_worker": 15493, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_deletion": 25204, "dcp_deletion_sent_to_worker": 11705, "dcp_mutation": 498, "dcp_snapshot": 342, "dcp_stream_req_counter": 342, "dcp_streamreq": 342, "execution_stats": 687, "failure_stats": 687, "latency_stats": 687, "lcb_exception_stats": 687, "log_level": 1, "num_processed_events": 11705, "processed_events_size": 3183612, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 11705, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 11705, "dcp_delete_msg_counter": 11705, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 0, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 11705, "enqueued_dcp_mutation_msg_counter": 0, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 15489, "num_processed_events": 11705, "on_delete_failure": 0, "on_delete_success": 11705, "on_update_failure": 0, "on_update_success": 0, "processed_events_size": 3183612, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "94208": "2021-02-13T07:24:10Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "94208": "2021-02-13T07:24:10Z" }, "v8worker_events_lost": 0 }, "function_id": 3832248590, "function_name": "curl", "gocb_creds_request_counter": 23, "internal_vb_distribution_stats": { "worker_curl_0": "[0-341]" }, "latency_percentile_stats": { "100": 22900, "50": 1200, "80": 1400, "90": 1600, "95": 1800, "99": 3100 }, "lcb_creds_request_counter": 24, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.82:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.97:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.123.62:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.82:8096": { "worker_curl_0": "[0-341]" }, "172.23.122.97:8096": { "worker_curl_0": "[342-682]" }, "172.23.123.62:8096": { "worker_curl_0": "[683-1023]" } }, "worker_pids": { "worker_curl_0": 94208 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 3, "agg_messages_sent_to_worker": 13340, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_deletion": 23132, "dcp_deletion_sent_to_worker": 9633, "dcp_mutation": 75, "dcp_snapshot": 342, "dcp_stream_req_counter": 342, "dcp_streamreq": 342, "execution_stats": 671, "failure_stats": 671, "latency_stats": 671, "lcb_exception_stats": 671, "log_level": 1, "num_processed_events": 9633, "processed_events_size": 2620176, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 9633, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 0, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 9633, "enqueued_dcp_mutation_msg_counter": 0, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 13336, "num_processed_events": 9633, "on_delete_failure": 0, "on_delete_success": 9633, "on_update_failure": 0, "on_update_success": 0, "processed_events_size": 2620176, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "94236": "2021-02-13T07:24:10Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "94236": "2021-02-13T07:24:10Z" }, "v8worker_events_lost": 0 }, "function_id": 1940050101, "function_name": "n1ql", "gocb_creds_request_counter": 23, "internal_vb_distribution_stats": { "worker_n1ql_0": "[0-341]" }, "latency_percentile_stats": { "100": 12200, "50": 300, "80": 500, "90": 700, "95": 800, "99": 1300 }, "lcb_creds_request_counter": 24, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.82:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.97:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.123.62:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.82:8096": { "worker_n1ql_0": "[0-341]" }, "172.23.122.97:8096": { "worker_n1ql_0": "[342-682]" }, "172.23.123.62:8096": { "worker_n1ql_0": "[683-1023]" } }, "worker_pids": { "worker_n1ql_0": 94236 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 4, "agg_messages_sent_to_worker": 16290, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_deletion": 12664, "dcp_deletion_sent_to_worker": 12664, "dcp_mutation": 75, "dcp_mutation_suppressed_counter": 75, "dcp_snapshot": 342, "dcp_stream_req_counter": 342, "dcp_streamreq": 342, "execution_stats": 655, "failure_stats": 655, "latency_stats": 655, "lcb_exception_stats": 655, "log_level": 1, "num_processed_events": 12664, "processed_events_size": 3444356, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 12664, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 0, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 12664, "enqueued_dcp_mutation_msg_counter": 0, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 16286, "num_processed_events": 12664, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 0, "processed_events_size": 3444356, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "94304": "2021-02-13T07:24:10Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "94304": "2021-02-13T07:24:10Z" }, "v8worker_events_lost": 0 }, "function_id": 2359049777, "function_name": "sbm", "gocb_creds_request_counter": 23, "internal_vb_distribution_stats": { "worker_sbm_0": "[0-341]" }, "latency_percentile_stats": { "100": 600, "50": 100, "80": 100, "90": 100, "95": 100, "99": 100 }, "lcb_creds_request_counter": 24, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.82:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.97:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.123.62:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.82:8096": { "worker_sbm_0": "[0-341]" }, "172.23.122.97:8096": { "worker_sbm_0": "[342-682]" }, "172.23.123.62:8096": { "worker_sbm_0": "[683-1023]" } }, "worker_pids": { "worker_sbm_0": 94304 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 2, "agg_messages_sent_to_worker": 17044, "agg_queue_memory": 0, "agg_queue_memory_cap": 20971520, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_deletion": 26998, "dcp_deletion_sent_to_worker": 13499, "dcp_mutation": 13407, "dcp_snapshot": 342, "dcp_stream_req_counter": 342, "dcp_streamreq": 342, "execution_stats": 639, "failure_stats": 639, "latency_stats": 639, "lcb_exception_stats": 639, "log_level": 1, "num_processed_events": 13499, "processed_events_size": 3669616, "thr_count": 1, "thr_map": 1, "timer_events": 20658, "timer_responses_received": 13499, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 13499, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 0, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 13499, "enqueued_dcp_mutation_msg_counter": 0, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 17040, "num_processed_events": 13499, "on_delete_failure": 0, "on_delete_success": 13499, "on_update_failure": 0, "on_update_success": 0, "processed_events_size": 3669616, "timer_callback_failure": 0, "timer_callback_success": 20658, "timer_cancel_counter": 0, "timer_create_counter": 13499, "timer_create_failure": 0, "timer_msg_counter": 20658, "timer_responses_sent": 0, "timestamp": { "94376": "2021-02-13T07:24:10Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "94376": "2021-02-13T07:24:10Z" }, "v8worker_events_lost": 0 }, "function_id": 146981409, "function_name": "timers", "gocb_creds_request_counter": 23, "internal_vb_distribution_stats": { "worker_timers_0": "[0-341]" }, "latency_percentile_stats": { "100": 15000, "50": 900, "80": 1200, "90": 1400, "95": 1700, "99": 2700 }, "lcb_creds_request_counter": 24, "lcb_exception_stats": {}, "metastore_stats": { "metastore_delete_err": 0, "metastore_deletes": 0, "metastore_not_found": 0, "metastore_scan": 0, "metastore_scan_due": 0, "metastore_scan_err": 0, "metastore_set": 0, "metastore_set_err": 0 }, "planner_stats": [ { "host_name": "172.23.122.82:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.97:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.123.62:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.82:8096": { "worker_timers_0": "[0-341]" }, "172.23.122.97:8096": { "worker_timers_0": "[342-682]" }, "172.23.123.62:8096": { "worker_timers_0": "[683-1023]" } }, "worker_pids": { "worker_timers_0": 94376 } } ] [2021-02-12 23:24:10,373] - [eventing_base] [140099324987136] - INFO - Stats for Node 172.23.122.97 is [ { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 6, "agg_messages_sent_to_worker": 77366, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_deletion": 46224, "dcp_deletion_sent_to_worker": 31661, "dcp_mutation": 125701, "dcp_mutation_sent_to_worker": 39413, "dcp_snapshot": 9597, "dcp_stream_req_counter": 1300, "dcp_streamend": 959, "dcp_streamreq": 1300, "dcp_system_event": 11020, "execution_stats": 804, "failure_stats": 804, "latency_stats": 804, "lcb_exception_stats": 804, "log_level": 1, "num_processed_events": 71074, "processed_events_size": 43898212, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 31661, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 39413, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 31661, "enqueued_dcp_mutation_msg_counter": 39413, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 42808, "lcb_retry_failure": 0, "messages_parsed": 77362, "num_processed_events": 71074, "on_delete_failure": 0, "on_delete_success": 11501, "on_update_failure": 68, "on_update_success": 16697, "processed_events_size": 43898212, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "117017": "2021-02-13T07:24:09Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 68, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "117017": "2021-02-13T07:24:09Z" }, "v8worker_events_lost": 0 }, "function_id": 3277597741, "function_name": "bucket_op", "gocb_creds_request_counter": 23, "internal_vb_distribution_stats": { "worker_bucket_op_0": "[342-682]" }, "latency_percentile_stats": { "100": 2501600, "50": 600, "80": 1100, "90": 1800, "95": 3400, "99": 11300 }, "lcb_creds_request_counter": 27032, "lcb_exception_stats": { "211": 68 }, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.82:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.97:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.123.62:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.82:8096": { "worker_bucket_op_0": "[0-341]" }, "172.23.122.97:8096": { "worker_bucket_op_0": "[342-682]" }, "172.23.123.62:8096": { "worker_bucket_op_0": "[683-1023]" } }, "worker_pids": { "worker_bucket_op_0": 117017 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 4, "agg_messages_sent_to_worker": 43147, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_deletion": 47352, "dcp_deletion_sent_to_worker": 32787, "dcp_mutation": 56297, "dcp_mutation_sent_to_worker": 5043, "dcp_snapshot": 9057, "dcp_stream_req_counter": 853, "dcp_streamend": 512, "dcp_streamreq": 853, "dcp_system_event": 1750, "execution_stats": 788, "failure_stats": 788, "latency_stats": 788, "lcb_exception_stats": 788, "log_level": 1, "num_processed_events": 37830, "processed_events_size": 13429144, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 18196, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 18196, "dcp_delete_msg_counter": 32787, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 5043, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 32787, "enqueued_dcp_mutation_msg_counter": 5043, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 19634, "lcb_retry_failure": 0, "messages_parsed": 43143, "num_processed_events": 37830, "on_delete_failure": 0, "on_delete_success": 14421, "on_update_failure": 0, "on_update_success": 3775, "processed_events_size": 13429144, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "117047": "2021-02-13T07:24:09Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "117047": "2021-02-13T07:24:09Z" }, "v8worker_events_lost": 0 }, "function_id": 3832248590, "function_name": "curl", "gocb_creds_request_counter": 23, "internal_vb_distribution_stats": { "worker_curl_0": "[342-682]" }, "latency_percentile_stats": { "100": 52800, "50": 1300, "80": 1900, "90": 2700, "95": 4600, "99": 13200 }, "lcb_creds_request_counter": 27032, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.82:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.97:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.123.62:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.82:8096": { "worker_curl_0": "[0-341]" }, "172.23.122.97:8096": { "worker_curl_0": "[342-682]" }, "172.23.123.62:8096": { "worker_curl_0": "[683-1023]" } }, "worker_pids": { "worker_curl_0": 117047 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 7, "agg_messages_sent_to_worker": 51883, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_deletion": 46345, "dcp_deletion_sent_to_worker": 31684, "dcp_mutation": 65980, "dcp_mutation_sent_to_worker": 14963, "dcp_snapshot": 9078, "dcp_stream_req_counter": 853, "dcp_streamend": 512, "dcp_streamreq": 853, "dcp_system_event": 1700, "execution_stats": 772, "failure_stats": 772, "latency_stats": 772, "lcb_exception_stats": 772, "log_level": 1, "num_processed_events": 46647, "processed_events_size": 22014392, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 31684, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 14963, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 31684, "enqueued_dcp_mutation_msg_counter": 14963, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 17755, "lcb_retry_failure": 0, "messages_parsed": 51879, "num_processed_events": 46647, "on_delete_failure": 0, "on_delete_success": 15390, "on_update_failure": 0, "on_update_success": 13502, "processed_events_size": 22014392, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "117121": "2021-02-13T07:24:09Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "117121": "2021-02-13T07:24:09Z" }, "v8worker_events_lost": 0 }, "function_id": 1940050101, "function_name": "n1ql", "gocb_creds_request_counter": 23, "internal_vb_distribution_stats": { "worker_n1ql_0": "[342-682]" }, "latency_percentile_stats": { "100": 66500, "50": 2100, "80": 2500, "90": 2900, "95": 4200, "99": 10000 }, "lcb_creds_request_counter": 27032, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.82:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.97:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.123.62:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.82:8096": { "worker_n1ql_0": "[0-341]" }, "172.23.122.97:8096": { "worker_n1ql_0": "[342-682]" }, "172.23.123.62:8096": { "worker_n1ql_0": "[683-1023]" } }, "worker_pids": { "worker_n1ql_0": 117121 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 3, "agg_messages_sent_to_worker": 23203, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_deletion": 40634, "dcp_deletion_sent_to_worker": 14249, "dcp_mutation": 23864, "dcp_mutation_sent_to_worker": 3878, "dcp_mutation_suppressed_counter": 18620, "dcp_snapshot": 9237, "dcp_stream_req_counter": 854, "dcp_stream_req_err_counter": 1, "dcp_streamend": 512, "dcp_streamreq": 853, "dcp_system_event": 1580, "execution_stats": 756, "failure_stats": 756, "latency_stats": 756, "lcb_exception_stats": 756, "log_level": 1, "num_processed_events": 18127, "processed_events_size": 7341016, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 14249, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 3878, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 14249, "enqueued_dcp_mutation_msg_counter": 3878, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 4002, "lcb_retry_failure": 0, "messages_parsed": 23199, "num_processed_events": 18127, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 251, "processed_events_size": 7341016, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "128905": "2021-02-13T07:24:10Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "128905": "2021-02-13T07:24:10Z" }, "v8worker_events_lost": 0 }, "function_id": 2359049777, "function_name": "sbm", "gocb_creds_request_counter": 23, "internal_vb_distribution_stats": { "worker_sbm_0": "[342-682]" }, "latency_percentile_stats": { "100": 18600, "50": 100, "80": 100, "90": 100, "95": 100, "99": 1900 }, "lcb_creds_request_counter": 27032, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.82:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.97:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.123.62:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.82:8096": { "worker_sbm_0": "[0-341]" }, "172.23.122.97:8096": { "worker_sbm_0": "[342-682]" }, "172.23.123.62:8096": { "worker_sbm_0": "[683-1023]" } }, "worker_pids": { "worker_sbm_0": 128905 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 5, "agg_messages_sent_to_worker": 55705, "agg_queue_memory": 0, "agg_queue_memory_cap": 20971520, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_deletion": 48017, "dcp_deletion_sent_to_worker": 33355, "dcp_mutation": 72231, "dcp_mutation_sent_to_worker": 17276, "dcp_snapshot": 8782, "dcp_stream_req_counter": 853, "dcp_streamend": 512, "dcp_streamreq": 853, "dcp_system_event": 2330, "execution_stats": 740, "failure_stats": 740, "latency_stats": 740, "lcb_exception_stats": 740, "log_level": 1, "num_processed_events": 50631, "processed_events_size": 24539592, "thr_count": 1, "thr_map": 1, "timer_events": 21465, "timer_responses_received": 19930, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 33355, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 17276, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 33355, "enqueued_dcp_mutation_msg_counter": 17276, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 30701, "lcb_retry_failure": 0, "messages_parsed": 55701, "num_processed_events": 50631, "on_delete_failure": 0, "on_delete_success": 13195, "on_update_failure": 0, "on_update_success": 6735, "processed_events_size": 24539592, "timer_callback_failure": 14, "timer_callback_success": 21451, "timer_cancel_counter": 0, "timer_create_counter": 19930, "timer_create_failure": 0, "timer_msg_counter": 21465, "timer_responses_sent": 0, "timestamp": { "130681": "2021-02-13T07:24:10Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 14, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "130681": "2021-02-13T07:24:10Z" }, "v8worker_events_lost": 0 }, "function_id": 146981409, "function_name": "timers", "gocb_creds_request_counter": 23, "internal_vb_distribution_stats": { "worker_timers_0": "[342-682]" }, "latency_percentile_stats": { "100": 2501300, "50": 1100, "80": 1500, "90": 1800, "95": 2300, "99": 4900 }, "lcb_creds_request_counter": 27032, "lcb_exception_stats": { "211": 14 }, "metastore_stats": { "metastore_delete_err": 0, "metastore_deletes": 0, "metastore_not_found": 0, "metastore_scan": 0, "metastore_scan_due": 0, "metastore_scan_err": 0, "metastore_set": 0, "metastore_set_err": 0 }, "planner_stats": [ { "host_name": "172.23.122.82:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.97:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.123.62:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.82:8096": { "worker_timers_0": "[0-341]" }, "172.23.122.97:8096": { "worker_timers_0": "[342-682]" }, "172.23.123.62:8096": { "worker_timers_0": "[683-1023]" } }, "worker_pids": { "worker_timers_0": 130681 } } ] [2021-02-12 23:24:10,415] - [eventing_base] [140099324987136] - INFO - Stats for Node 172.23.123.62 is [ { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 6, "agg_messages_sent_to_worker": 104083, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_deletion": 47105, "dcp_deletion_sent_to_worker": 32572, "dcp_mutation": 202141, "dcp_mutation_sent_to_worker": 63395, "dcp_snapshot": 129698, "dcp_stream_req_counter": 1877, "dcp_streamend": 1536, "dcp_streamreq": 1877, "dcp_system_event": 17820, "execution_stats": 938, "failure_stats": 938, "latency_stats": 938, "lcb_exception_stats": 938, "log_level": 1, "num_processed_events": 95967, "processed_events_size": 65625420, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 32572, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 63395, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 32572, "enqueued_dcp_mutation_msg_counter": 63395, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 65184, "lcb_retry_failure": 0, "messages_parsed": 104079, "num_processed_events": 95967, "on_delete_failure": 0, "on_delete_success": 15320, "on_update_failure": 61, "on_update_success": 15402, "processed_events_size": 65625420, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "25929": "2021-02-13T07:24:10Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 61, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "25929": "2021-02-13T07:24:10Z" }, "v8worker_events_lost": 0 }, "function_id": 3277597741, "function_name": "bucket_op", "gocb_creds_request_counter": 26, "internal_vb_distribution_stats": { "worker_bucket_op_0": "[683-1023]" }, "latency_percentile_stats": { "100": 2514100, "50": 500, "80": 1000, "90": 1700, "95": 3300, "99": 11000 }, "lcb_creds_request_counter": 54595, "lcb_exception_stats": { "211": 61 }, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.82:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.97:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.123.62:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.82:8096": { "worker_bucket_op_0": "[0-341]" }, "172.23.122.97:8096": { "worker_bucket_op_0": "[342-682]" }, "172.23.123.62:8096": { "worker_bucket_op_0": "[683-1023]" } }, "worker_pids": { "worker_bucket_op_0": 25929 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 5, "agg_messages_sent_to_worker": 103715, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_deletion": 48251, "dcp_deletion_sent_to_worker": 33442, "dcp_mutation": 203256, "dcp_mutation_sent_to_worker": 62268, "dcp_snapshot": 130212, "dcp_stream_req_counter": 1877, "dcp_streamend": 1536, "dcp_streamreq": 1877, "dcp_system_event": 14120, "execution_stats": 916, "failure_stats": 916, "latency_stats": 916, "lcb_exception_stats": 916, "log_level": 1, "num_processed_events": 95710, "processed_events_size": 64853000, "thr_count": 1, "thr_map": 1, "timer_events": 34722, "timer_responses_received": 36143, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 33442, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 62268, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 33442, "enqueued_dcp_mutation_msg_counter": 62268, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 59567, "lcb_retry_failure": 0, "messages_parsed": 103711, "num_processed_events": 95710, "on_delete_failure": 0, "on_delete_success": 13626, "on_update_failure": 0, "on_update_success": 22517, "processed_events_size": 64853000, "timer_callback_failure": 86, "timer_callback_success": 34636, "timer_cancel_counter": 0, "timer_create_counter": 36143, "timer_create_failure": 0, "timer_msg_counter": 34722, "timer_responses_sent": 0, "timestamp": { "25969": "2021-02-13T07:24:09Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 86, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "25969": "2021-02-13T07:24:09Z" }, "v8worker_events_lost": 0 }, "function_id": 146981409, "function_name": "timers", "gocb_creds_request_counter": 26, "internal_vb_distribution_stats": { "worker_timers_0": "[683-1023]" }, "latency_percentile_stats": { "100": 2504600, "50": 1000, "80": 1600, "90": 2500, "95": 3500, "99": 9600 }, "lcb_creds_request_counter": 54595, "lcb_exception_stats": { "211": 86 }, "metastore_stats": { "metastore_delete_err": 0, "metastore_deletes": 0, "metastore_not_found": 0, "metastore_scan": 0, "metastore_scan_due": 0, "metastore_scan_err": 0, "metastore_set": 0, "metastore_set_err": 0 }, "planner_stats": [ { "host_name": "172.23.122.82:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.97:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.123.62:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.82:8096": { "worker_timers_0": "[0-341]" }, "172.23.122.97:8096": { "worker_timers_0": "[342-682]" }, "172.23.123.62:8096": { "worker_timers_0": "[683-1023]" } }, "worker_pids": { "worker_timers_0": 25969 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 4, "agg_messages_sent_to_worker": 74841, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_deletion": 37550, "dcp_deletion_sent_to_worker": 13972, "dcp_mutation": 164628, "dcp_mutation_sent_to_worker": 53010, "dcp_mutation_suppressed_counter": 60257, "dcp_snapshot": 130180, "dcp_stream_req_counter": 1878, "dcp_stream_req_err_counter": 1, "dcp_streamend": 1536, "dcp_streamreq": 1877, "dcp_system_event": 13670, "execution_stats": 895, "failure_stats": 895, "latency_stats": 895, "lcb_exception_stats": 895, "log_level": 1, "num_processed_events": 66982, "processed_events_size": 51257308, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 13972, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 53010, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 13972, "enqueued_dcp_mutation_msg_counter": 53010, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 12880, "lcb_retry_failure": 0, "messages_parsed": 74837, "num_processed_events": 66982, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 40320, "processed_events_size": 51257308, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "26012": "2021-02-13T07:24:09Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "26012": "2021-02-13T07:24:09Z" }, "v8worker_events_lost": 0 }, "function_id": 2359049777, "function_name": "sbm", "gocb_creds_request_counter": 26, "internal_vb_distribution_stats": { "worker_sbm_0": "[683-1023]" }, "latency_percentile_stats": { "100": 24200, "50": 500, "80": 800, "90": 1100, "95": 1500, "99": 3500 }, "lcb_creds_request_counter": 54595, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.82:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.97:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.123.62:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.82:8096": { "worker_sbm_0": "[0-341]" }, "172.23.122.97:8096": { "worker_sbm_0": "[342-682]" }, "172.23.123.62:8096": { "worker_sbm_0": "[683-1023]" } }, "worker_pids": { "worker_sbm_0": 26012 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 8, "agg_messages_sent_to_worker": 93171, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_deletion": 46972, "dcp_deletion_sent_to_worker": 32208, "dcp_mutation": 180841, "dcp_mutation_sent_to_worker": 53170, "dcp_snapshot": 129418, "dcp_stream_req_counter": 1878, "dcp_stream_req_err_counter": 1, "dcp_streamend": 1536, "dcp_streamreq": 1877, "dcp_system_event": 13690, "execution_stats": 874, "failure_stats": 874, "latency_stats": 874, "lcb_exception_stats": 874, "log_level": 1, "num_processed_events": 85378, "processed_events_size": 56371792, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 50993, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 50993, "dcp_delete_msg_counter": 32208, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 53170, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 32208, "enqueued_dcp_mutation_msg_counter": 53170, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 34385, "lcb_retry_failure": 0, "messages_parsed": 93167, "num_processed_events": 85378, "on_delete_failure": 0, "on_delete_success": 14194, "on_update_failure": 0, "on_update_success": 36799, "processed_events_size": 56371792, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "26085": "2021-02-13T07:24:10Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "26085": "2021-02-13T07:24:10Z" }, "v8worker_events_lost": 0 }, "function_id": 3832248590, "function_name": "curl", "gocb_creds_request_counter": 26, "internal_vb_distribution_stats": { "worker_curl_0": "[683-1023]" }, "latency_percentile_stats": { "100": 79200, "50": 1500, "80": 2100, "90": 2800, "95": 4100, "99": 11000 }, "lcb_creds_request_counter": 54595, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.82:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.97:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.123.62:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.82:8096": { "worker_curl_0": "[0-341]" }, "172.23.122.97:8096": { "worker_curl_0": "[342-682]" }, "172.23.123.62:8096": { "worker_curl_0": "[683-1023]" } }, "worker_pids": { "worker_curl_0": 26085 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 6, "agg_messages_sent_to_worker": 91860, "agg_queue_memory": 0, "agg_queue_memory_cap": 20971520, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_deletion": 45053, "dcp_deletion_sent_to_worker": 30310, "dcp_mutation": 100337, "dcp_mutation_sent_to_worker": 53868, "dcp_snapshot": 46337, "dcp_stream_req_counter": 1877, "dcp_streamend": 1536, "dcp_streamreq": 1877, "dcp_system_event": 13660, "execution_stats": 852, "failure_stats": 852, "latency_stats": 852, "lcb_exception_stats": 852, "log_level": 1, "num_processed_events": 84178, "processed_events_size": 56480772, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 30310, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 53868, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 30310, "enqueued_dcp_mutation_msg_counter": 53868, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 41597, "lcb_retry_failure": 0, "messages_parsed": 91856, "num_processed_events": 84178, "on_delete_failure": 0, "on_delete_success": 15297, "on_update_failure": 466, "on_update_success": 26818, "processed_events_size": 56480772, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "26208": "2021-02-13T07:24:09Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 466, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "26208": "2021-02-13T07:24:09Z" }, "v8worker_events_lost": 0 }, "function_id": 1940050101, "function_name": "n1ql", "gocb_creds_request_counter": 26, "internal_vb_distribution_stats": { "worker_n1ql_0": "[683-1023]" }, "latency_percentile_stats": { "100": 105200, "50": 2300, "80": 3900, "90": 5400, "95": 7400, "99": 16800 }, "lcb_creds_request_counter": 54595, "lcb_exception_stats": { "12009": 466, "209": 466 }, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.82:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.97:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.123.62:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.82:8096": { "worker_n1ql_0": "[0-341]" }, "172.23.122.97:8096": { "worker_n1ql_0": "[342-682]" }, "172.23.123.62:8096": { "worker_n1ql_0": "[683-1023]" } }, "worker_pids": { "worker_n1ql_0": 26208 } } ] [2021-02-12 23:24:10,431] - [basetestcase] [140099324987136] - INFO - list of eventing nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root, ip:172.23.122.97 port:8091 ssh_username:root, ip:172.23.123.62 port:8091 ssh_username:root] [2021-02-12 23:24:10,432] - [remote_util] [140099324987136] - INFO - SSH Connecting to 172.23.122.82 with username:root, attempt#1 of 5 [2021-02-12 23:24:10,588] - [remote_util] [140099324987136] - INFO - SSH Connected to 172.23.122.82 as root [2021-02-12 23:24:10,836] - [remote_util] [140099324987136] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:24:11,150] - [remote_util] [140099324987136] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:24:11,158] - [rest_client] [140099324987136] - INFO - /diag/eval status on 172.23.122.82:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). [2021-02-12 23:24:11,158] - [remote_util] [140099324987136] - INFO - running command.raw on 172.23.122.82: zgrep "panic" "/opt/couchbase/var/lib/couchbase/logs"/eventing.log* | wc -l [2021-02-12 23:24:11,451] - [remote_util] [140099324987136] - INFO - command executed successfully with root [2021-02-12 23:24:11,451] - [remote_util] [140099324987136] - INFO - running command.raw on 172.23.122.82: ls "/opt/couchbase/var/lib/couchbase/logs"/../crash/| wc -l [2021-02-12 23:24:11,466] - [remote_util] [140099324987136] - INFO - command executed successfully with root [2021-02-12 23:24:11,468] - [remote_util] [140099324987136] - INFO - SSH Connecting to 172.23.122.97 with username:root, attempt#1 of 5 [2021-02-12 23:24:11,574] - [remote_util] [140099324987136] - INFO - SSH Connected to 172.23.122.97 as root [2021-02-12 23:24:11,840] - [remote_util] [140099324987136] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:24:12,145] - [remote_util] [140099324987136] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:24:12,153] - [rest_client] [140099324987136] - INFO - /diag/eval status on 172.23.122.97:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). [2021-02-12 23:24:12,153] - [remote_util] [140099324987136] - INFO - running command.raw on 172.23.122.97: zgrep "panic" "/opt/couchbase/var/lib/couchbase/logs"/eventing.log* | wc -l [2021-02-12 23:24:13,818] - [remote_util] [140099324987136] - INFO - command executed successfully with root [2021-02-12 23:24:13,819] - [remote_util] [140099324987136] - INFO - running command.raw on 172.23.122.97: ls "/opt/couchbase/var/lib/couchbase/logs"/../crash/| wc -l [2021-02-12 23:24:13,833] - [remote_util] [140099324987136] - INFO - command executed successfully with root [2021-02-12 23:24:13,835] - [remote_util] [140099324987136] - INFO - SSH Connecting to 172.23.123.62 with username:root, attempt#1 of 5 [2021-02-12 23:24:13,940] - [remote_util] [140099324987136] - INFO - SSH Connected to 172.23.123.62 as root [2021-02-12 23:24:14,219] - [remote_util] [140099324987136] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:24:14,522] - [remote_util] [140099324987136] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:24:14,531] - [rest_client] [140099324987136] - INFO - /diag/eval status on 172.23.123.62:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). [2021-02-12 23:24:14,531] - [remote_util] [140099324987136] - INFO - running command.raw on 172.23.123.62: zgrep "panic" "/opt/couchbase/var/lib/couchbase/logs"/eventing.log* | wc -l [2021-02-12 23:24:15,403] - [remote_util] [140099324987136] - INFO - command executed successfully with root [2021-02-12 23:24:15,403] - [remote_util] [140099324987136] - INFO - running command.raw on 172.23.123.62: ls "/opt/couchbase/var/lib/couchbase/logs"/../crash/| wc -l [2021-02-12 23:24:15,459] - [remote_util] [140099324987136] - INFO - command executed successfully with root [2021-02-12 23:24:15,513] - [basetestcase] [140099324987136] - INFO - ------- Cluster statistics ------- [2021-02-12 23:24:15,513] - [basetestcase] [140099324987136] - INFO - 172.23.122.117:8091 => {'services': ['n1ql'], 'cpu_utilization': 0.7288263382759488, 'mem_free': 3432210432, 'mem_total': 4201684992, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-02-12 23:24:15,513] - [basetestcase] [140099324987136] - INFO - 172.23.122.82:8091 => {'services': ['eventing'], 'cpu_utilization': 6.544704972945118, 'mem_free': 3036938240, 'mem_total': 4201684992, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-02-12 23:24:15,513] - [basetestcase] [140099324987136] - INFO - 172.23.122.90:8091 => {'services': ['index'], 'cpu_utilization': 1.018070755917536, 'mem_free': 3291537408, 'mem_total': 4201684992, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-02-12 23:24:15,513] - [basetestcase] [140099324987136] - INFO - 172.23.122.97:8091 => {'services': ['eventing'], 'cpu_utilization': 11.34181157539579, 'mem_free': 2947158016, 'mem_total': 4201684992, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-02-12 23:24:15,513] - [basetestcase] [140099324987136] - INFO - 172.23.123.61:8091 => {'services': ['kv'], 'cpu_utilization': 11.23450413223141, 'mem_free': 2675073024, 'mem_total': 4201672704, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-02-12 23:24:15,513] - [basetestcase] [140099324987136] - INFO - 172.23.123.62:8091 => {'services': ['eventing'], 'cpu_utilization': 12.25554106910039, 'mem_free': 2737479680, 'mem_total': 4201676800, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-02-12 23:24:15,513] - [basetestcase] [140099324987136] - INFO - --- End of cluster statistics --- [2021-02-12 23:24:15,515] - [remote_util] [140099324987136] - INFO - SSH Connecting to 172.23.123.61 with username:root, attempt#1 of 5 [2021-02-12 23:24:15,625] - [remote_util] [140099324987136] - INFO - SSH Connected to 172.23.123.61 as root [2021-02-12 23:24:15,909] - [remote_util] [140099324987136] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:24:16,218] - [remote_util] [140099324987136] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:24:16,220] - [remote_util] [140099324987136] - INFO - SSH Connecting to 172.23.123.62 with username:root, attempt#1 of 5 [2021-02-12 23:24:16,325] - [remote_util] [140099324987136] - INFO - SSH Connected to 172.23.123.62 as root [2021-02-12 23:24:16,617] - [remote_util] [140099324987136] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:24:16,928] - [remote_util] [140099324987136] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:24:16,929] - [remote_util] [140099324987136] - INFO - SSH Connecting to 172.23.122.90 with username:root, attempt#1 of 5 [2021-02-12 23:24:17,041] - [remote_util] [140099324987136] - INFO - SSH Connected to 172.23.122.90 as root [2021-02-12 23:24:17,339] - [remote_util] [140099324987136] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:24:17,638] - [remote_util] [140099324987136] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:24:17,640] - [remote_util] [140099324987136] - INFO - SSH Connecting to 172.23.122.117 with username:root, attempt#1 of 5 [2021-02-12 23:24:17,747] - [remote_util] [140099324987136] - INFO - SSH Connected to 172.23.122.117 as root [2021-02-12 23:24:18,008] - [remote_util] [140099324987136] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:24:18,304] - [remote_util] [140099324987136] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:24:18,306] - [remote_util] [140099324987136] - INFO - SSH Connecting to 172.23.122.97 with username:root, attempt#1 of 5 [2021-02-12 23:24:18,408] - [remote_util] [140099324987136] - INFO - SSH Connected to 172.23.122.97 as root [2021-02-12 23:24:18,672] - [remote_util] [140099324987136] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:24:18,974] - [remote_util] [140099324987136] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:24:18,977] - [remote_util] [140099324987136] - INFO - SSH Connecting to 172.23.122.82 with username:root, attempt#1 of 5 [2021-02-12 23:24:19,081] - [remote_util] [140099324987136] - INFO - SSH Connected to 172.23.122.82 as root [2021-02-12 23:24:19,342] - [remote_util] [140099324987136] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:24:19,645] - [remote_util] [140099324987136] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:24:19,647] - [remote_util] [140099324987136] - INFO - SSH Connecting to 172.23.122.118 with username:root, attempt#1 of 5 [2021-02-12 23:24:19,748] - [remote_util] [140099324987136] - INFO - SSH Connected to 172.23.122.118 as root [2021-02-12 23:24:19,993] - [remote_util] [140099324987136] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:24:20,287] - [remote_util] [140099324987136] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:24:20,289] - [remote_util] [140099324987136] - INFO - SSH Connecting to 172.23.122.116 with username:root, attempt#1 of 5 [2021-02-12 23:24:20,394] - [remote_util] [140099324987136] - INFO - SSH Connected to 172.23.122.116 as root [2021-02-12 23:24:20,673] - [remote_util] [140099324987136] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:24:20,965] - [remote_util] [140099324987136] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:25:46,466] - [basetestcase] [140099324987136] - INFO - ============== basetestcase cleanup was started for test #9 test_eventing_rebalance_in_delete_recreate_collections ============== [2021-02-12 23:25:46,513] - [bucket_helper] [140099324987136] - INFO - deleting existing buckets ['dst_bucket', 'metadata', 'src_bucket'] on 172.23.123.61 [2021-02-12 23:25:47,474] - [bucket_helper] [140099324987136] - INFO - waiting for bucket deletion to complete.... [2021-02-12 23:25:47,492] - [rest_client] [140099324987136] - INFO - node 172.23.123.61 existing buckets : ['metadata', 'src_bucket'] [2021-02-12 23:25:47,493] - [bucket_helper] [140099324987136] - INFO - deleted bucket : dst_bucket from 172.23.123.61 [2021-02-12 23:25:48,142] - [bucket_helper] [140099324987136] - INFO - waiting for bucket deletion to complete.... [2021-02-12 23:25:48,152] - [rest_client] [140099324987136] - INFO - node 172.23.123.61 existing buckets : ['src_bucket'] [2021-02-12 23:25:48,153] - [bucket_helper] [140099324987136] - INFO - deleted bucket : metadata from 172.23.123.61 [2021-02-12 23:25:49,287] - [bucket_helper] [140099324987136] - INFO - waiting for bucket deletion to complete.... [2021-02-12 23:25:49,290] - [rest_client] [140099324987136] - INFO - node 172.23.123.61 existing buckets : [] [2021-02-12 23:25:49,290] - [bucket_helper] [140099324987136] - INFO - deleted bucket : src_bucket from 172.23.123.61 [2021-02-12 23:25:49,298] - [bucket_helper] [140099324987136] - INFO - Could not find any buckets for node 172.23.123.62, nothing to delete [2021-02-12 23:25:49,305] - [bucket_helper] [140099324987136] - INFO - Could not find any buckets for node 172.23.122.90, nothing to delete [2021-02-12 23:25:49,312] - [bucket_helper] [140099324987136] - INFO - Could not find any buckets for node 172.23.122.117, nothing to delete [2021-02-12 23:25:49,319] - [bucket_helper] [140099324987136] - INFO - Could not find any buckets for node 172.23.122.97, nothing to delete [2021-02-12 23:25:49,326] - [bucket_helper] [140099324987136] - INFO - Could not find any buckets for node 172.23.122.82, nothing to delete [2021-02-12 23:25:49,331] - [bucket_helper] [140099324987136] - INFO - Could not find any buckets for node 172.23.122.118, nothing to delete [2021-02-12 23:25:49,336] - [bucket_helper] [140099324987136] - INFO - Could not find any buckets for node 172.23.122.116, nothing to delete [2021-02-12 23:25:49,341] - [rest_client] [140099324987136] - INFO - -->is_ns_server_running? [2021-02-12 23:25:49,355] - [cluster_helper] [140099324987136] - INFO - rebalancing all nodes in order to remove nodes [2021-02-12 23:25:49,357] - [rest_client] [140099324987136] - INFO - rebalance params : {'knownNodes': 'ns_1@172.23.122.117,ns_1@172.23.122.82,ns_1@172.23.122.90,ns_1@172.23.122.97,ns_1@172.23.123.61,ns_1@172.23.123.62', 'ejectedNodes': 'ns_1@172.23.122.117,ns_1@172.23.122.82,ns_1@172.23.122.90,ns_1@172.23.122.97,ns_1@172.23.123.62', 'user': 'Administrator', 'password': 'password'} [2021-02-12 23:25:49,384] - [rest_client] [140099324987136] - INFO - rebalance operation started [2021-02-12 23:25:49,386] - [rest_client] [140099324987136] - INFO - rebalance percentage : 0.00 % [2021-02-12 23:25:59,399] - [rest_client] [140099324987136] - INFO - rebalance percentage : 33.00 % [2021-02-12 23:26:09,412] - [rest_client] [140099324987136] - INFO - rebalance percentage : 83.00 % [2021-02-12 23:26:29,438] - [rest_client] [140099324987136] - INFO - rebalance progress took 40.05 seconds [2021-02-12 23:26:29,438] - [rest_client] [140099324987136] - INFO - sleep for 10 seconds after rebalance... [2021-02-12 23:26:39,479] - [cluster_helper] [140099324987136] - INFO - removed all the nodes from cluster associated with ip:172.23.123.61 port:8091 ssh_username:root ? [('ns_1@172.23.122.117', 8091), ('ns_1@172.23.122.82', 8091), ('ns_1@172.23.122.90', 8091), ('ns_1@172.23.122.97', 8091), ('ns_1@172.23.123.62', 8091)] [2021-02-12 23:26:39,484] - [cluster_helper] [140099324987136] - INFO - waiting for ns_server @ 172.23.123.61:8091 [2021-02-12 23:26:39,485] - [rest_client] [140099324987136] - INFO - -->is_ns_server_running? [2021-02-12 23:26:39,489] - [cluster_helper] [140099324987136] - INFO - ns_server @ 172.23.123.61:8091 is running [2021-02-12 23:26:39,492] - [cluster_helper] [140099324987136] - INFO - waiting for ns_server @ 172.23.123.62:8091 [2021-02-12 23:26:39,492] - [rest_client] [140099324987136] - INFO - -->is_ns_server_running? [2021-02-12 23:26:39,495] - [cluster_helper] [140099324987136] - INFO - ns_server @ 172.23.123.62:8091 is running [2021-02-12 23:26:39,498] - [cluster_helper] [140099324987136] - INFO - waiting for ns_server @ 172.23.122.90:8091 [2021-02-12 23:26:39,498] - [rest_client] [140099324987136] - INFO - -->is_ns_server_running? [2021-02-12 23:26:39,500] - [cluster_helper] [140099324987136] - INFO - ns_server @ 172.23.122.90:8091 is running [2021-02-12 23:26:39,503] - [cluster_helper] [140099324987136] - INFO - waiting for ns_server @ 172.23.122.117:8091 [2021-02-12 23:26:39,503] - [rest_client] [140099324987136] - INFO - -->is_ns_server_running? [2021-02-12 23:26:39,506] - [cluster_helper] [140099324987136] - INFO - ns_server @ 172.23.122.117:8091 is running [2021-02-12 23:26:39,509] - [cluster_helper] [140099324987136] - INFO - waiting for ns_server @ 172.23.122.97:8091 [2021-02-12 23:26:39,510] - [rest_client] [140099324987136] - INFO - -->is_ns_server_running? [2021-02-12 23:26:39,512] - [cluster_helper] [140099324987136] - INFO - ns_server @ 172.23.122.97:8091 is running [2021-02-12 23:26:39,515] - [cluster_helper] [140099324987136] - INFO - waiting for ns_server @ 172.23.122.82:8091 [2021-02-12 23:26:39,515] - [rest_client] [140099324987136] - INFO - -->is_ns_server_running? [2021-02-12 23:26:39,518] - [cluster_helper] [140099324987136] - INFO - ns_server @ 172.23.122.82:8091 is running [2021-02-12 23:26:39,521] - [cluster_helper] [140099324987136] - INFO - waiting for ns_server @ 172.23.122.118:8091 [2021-02-12 23:26:39,521] - [rest_client] [140099324987136] - INFO - -->is_ns_server_running? [2021-02-12 23:26:39,524] - [cluster_helper] [140099324987136] - INFO - ns_server @ 172.23.122.118:8091 is running [2021-02-12 23:26:39,528] - [cluster_helper] [140099324987136] - INFO - waiting for ns_server @ 172.23.122.116:8091 [2021-02-12 23:26:39,528] - [rest_client] [140099324987136] - INFO - -->is_ns_server_running? [2021-02-12 23:26:39,531] - [cluster_helper] [140099324987136] - INFO - ns_server @ 172.23.122.116:8091 is running [2021-02-12 23:26:39,531] - [basetestcase] [140099324987136] - INFO - ============== basetestcase cleanup was finished for test #9 test_eventing_rebalance_in_delete_recreate_collections ============== [2021-02-12 23:26:39,531] - [basetestcase] [140099324987136] - INFO - closing all ssh connections [2021-02-12 23:26:39,533] - [basetestcase] [140099324987136] - INFO - closing all memcached connections [2021-02-12 23:26:39,534] - [remote_util] [140099324987136] - INFO - SSH Connecting to 172.23.123.61 with username:root, attempt#1 of 5 [2021-02-12 23:26:39,635] - [remote_util] [140099324987136] - INFO - SSH Connected to 172.23.123.61 as root [2021-02-12 23:26:39,905] - [remote_util] [140099324987136] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:26:40,205] - [remote_util] [140099324987136] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:26:40,215] - [remote_util] [140099324987136] - INFO - SSH Connecting to 172.23.123.61 with username:root, attempt#1 of 5 [2021-02-12 23:26:40,313] - [remote_util] [140099324987136] - INFO - SSH Connected to 172.23.123.61 as root [2021-02-12 23:26:40,566] - [remote_util] [140099324987136] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:26:40,863] - [remote_util] [140099324987136] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:26:40,920] - [remote_util] [140099324987136] - INFO - 172.23.123.61 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 23:26:40,971] - [remote_util] [140099324987136] - INFO - Running systemd command on this server [2021-02-12 23:26:40,971] - [remote_util] [140099324987136] - INFO - running command.raw on 172.23.123.61: systemctl stop couchbase-server.service [2021-02-12 23:26:42,838] - [remote_util] [140099324987136] - INFO - command executed successfully with root [2021-02-12 23:26:42,838] - [basetestcase] [140099324987136] - INFO - Couchbase stopped [2021-02-12 23:26:42,838] - [basetestcase] [140099324987136] - INFO - sleep for 5 secs. ... [2021-02-12 23:26:47,843] - [remote_util] [140099324987136] - INFO - running command.raw on 172.23.123.61: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-12 23:26:47,861] - [remote_util] [140099324987136] - INFO - command executed successfully with root [2021-02-12 23:26:47,861] - [remote_util] [140099324987136] - INFO - running command.raw on 172.23.123.61: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-12 23:26:47,919] - [remote_util] [140099324987136] - INFO - command executed successfully with root [2021-02-12 23:26:47,920] - [remote_util] [140099324987136] - INFO - SSH Connecting to 172.23.123.61 with username:root, attempt#1 of 5 [2021-02-12 23:26:48,030] - [remote_util] [140099324987136] - INFO - SSH Connected to 172.23.123.61 as root [2021-02-12 23:26:48,302] - [remote_util] [140099324987136] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:26:48,609] - [remote_util] [140099324987136] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:26:48,628] - [remote_util] [140099324987136] - INFO - 172.23.123.61 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 23:26:48,628] - [remote_util] [140099324987136] - INFO - Checking for process beam.smp on linux [2021-02-12 23:26:48,668] - [remote_util] [140099324987136] - INFO - Starting couchbase server [2021-02-12 23:26:48,769] - [remote_util] [140099324987136] - INFO - Running systemd command on this server [2021-02-12 23:26:48,769] - [remote_util] [140099324987136] - INFO - running command.raw on 172.23.123.61: systemctl start couchbase-server.service [2021-02-12 23:26:48,795] - [remote_util] [140099324987136] - INFO - command executed successfully with root [2021-02-12 23:26:48,795] - [remote_util] [140099324987136] - INFO - 172.23.123.61:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-12 23:26:53,800] - [remote_util] [140099324987136] - INFO - running command.raw on 172.23.123.61: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-12 23:26:53,824] - [remote_util] [140099324987136] - INFO - command executed successfully with root [2021-02-12 23:26:53,825] - [remote_util] [140099324987136] - INFO - Couchbase server status: [' Process: 11347 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-12 23:26:53,825] - [remote_util] [140099324987136] - INFO - Checking for process beam.smp on linux [2021-02-12 23:26:53,892] - [remote_util] [140099324987136] - INFO - process beam.smp is running on 172.23.123.61: with pid 11490 [2021-02-12 23:26:53,892] - [basetestcase] [140099324987136] - INFO - Couchbase started [2021-02-12 23:26:53,894] - [remote_util] [140099324987136] - INFO - SSH Connecting to 172.23.123.62 with username:root, attempt#1 of 5 [2021-02-12 23:26:54,003] - [remote_util] [140099324987136] - INFO - SSH Connected to 172.23.123.62 as root [2021-02-12 23:26:54,298] - [remote_util] [140099324987136] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:26:54,608] - [remote_util] [140099324987136] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:26:54,616] - [remote_util] [140099324987136] - INFO - SSH Connecting to 172.23.123.62 with username:root, attempt#1 of 5 [2021-02-12 23:26:54,718] - [remote_util] [140099324987136] - INFO - SSH Connected to 172.23.123.62 as root [2021-02-12 23:26:55,007] - [remote_util] [140099324987136] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:26:55,318] - [remote_util] [140099324987136] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:26:55,379] - [remote_util] [140099324987136] - INFO - 172.23.123.62 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 23:26:55,439] - [remote_util] [140099324987136] - INFO - Running systemd command on this server [2021-02-12 23:26:55,439] - [remote_util] [140099324987136] - INFO - running command.raw on 172.23.123.62: systemctl stop couchbase-server.service [2021-02-12 23:26:57,084] - [remote_util] [140099324987136] - INFO - command executed successfully with root [2021-02-12 23:26:57,085] - [basetestcase] [140099324987136] - INFO - Couchbase stopped [2021-02-12 23:26:57,085] - [basetestcase] [140099324987136] - INFO - sleep for 5 secs. ... [2021-02-12 23:27:02,090] - [remote_util] [140099324987136] - INFO - running command.raw on 172.23.123.62: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-12 23:27:02,123] - [remote_util] [140099324987136] - INFO - command executed successfully with root [2021-02-12 23:27:02,123] - [remote_util] [140099324987136] - INFO - running command.raw on 172.23.123.62: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-12 23:27:02,178] - [remote_util] [140099324987136] - INFO - command executed successfully with root [2021-02-12 23:27:02,180] - [remote_util] [140099324987136] - INFO - SSH Connecting to 172.23.123.62 with username:root, attempt#1 of 5 [2021-02-12 23:27:02,281] - [remote_util] [140099324987136] - INFO - SSH Connected to 172.23.123.62 as root [2021-02-12 23:27:02,550] - [remote_util] [140099324987136] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:27:02,849] - [remote_util] [140099324987136] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:27:02,907] - [remote_util] [140099324987136] - INFO - 172.23.123.62 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 23:27:02,907] - [remote_util] [140099324987136] - INFO - Checking for process beam.smp on linux [2021-02-12 23:27:02,944] - [remote_util] [140099324987136] - INFO - Starting couchbase server [2021-02-12 23:27:03,035] - [remote_util] [140099324987136] - INFO - Running systemd command on this server [2021-02-12 23:27:03,035] - [remote_util] [140099324987136] - INFO - running command.raw on 172.23.123.62: systemctl start couchbase-server.service [2021-02-12 23:27:03,057] - [remote_util] [140099324987136] - INFO - command executed successfully with root [2021-02-12 23:27:03,057] - [remote_util] [140099324987136] - INFO - 172.23.123.62:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-12 23:27:08,062] - [remote_util] [140099324987136] - INFO - running command.raw on 172.23.123.62: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-12 23:27:08,087] - [remote_util] [140099324987136] - INFO - command executed successfully with root [2021-02-12 23:27:08,087] - [remote_util] [140099324987136] - INFO - Couchbase server status: [' Process: 58267 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-12 23:27:08,087] - [remote_util] [140099324987136] - INFO - Checking for process beam.smp on linux [2021-02-12 23:27:08,155] - [remote_util] [140099324987136] - INFO - process beam.smp is running on 172.23.123.62: with pid 58409 [2021-02-12 23:27:08,155] - [basetestcase] [140099324987136] - INFO - Couchbase started [2021-02-12 23:27:08,157] - [remote_util] [140099324987136] - INFO - SSH Connecting to 172.23.122.90 with username:root, attempt#1 of 5 [2021-02-12 23:27:08,258] - [remote_util] [140099324987136] - INFO - SSH Connected to 172.23.122.90 as root [2021-02-12 23:27:08,539] - [remote_util] [140099324987136] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:27:08,836] - [remote_util] [140099324987136] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:27:08,844] - [remote_util] [140099324987136] - INFO - SSH Connecting to 172.23.122.90 with username:root, attempt#1 of 5 [2021-02-12 23:27:08,940] - [remote_util] [140099324987136] - INFO - SSH Connected to 172.23.122.90 as root [2021-02-12 23:27:09,213] - [remote_util] [140099324987136] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:27:09,512] - [remote_util] [140099324987136] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:27:09,533] - [remote_util] [140099324987136] - INFO - 172.23.122.90 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 23:27:09,585] - [remote_util] [140099324987136] - INFO - Running systemd command on this server [2021-02-12 23:27:09,586] - [remote_util] [140099324987136] - INFO - running command.raw on 172.23.122.90: systemctl stop couchbase-server.service [2021-02-12 23:27:11,248] - [remote_util] [140099324987136] - INFO - command executed successfully with root [2021-02-12 23:27:11,249] - [basetestcase] [140099324987136] - INFO - Couchbase stopped [2021-02-12 23:27:11,249] - [basetestcase] [140099324987136] - INFO - sleep for 5 secs. ... [2021-02-12 23:27:16,252] - [remote_util] [140099324987136] - INFO - running command.raw on 172.23.122.90: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-12 23:27:16,271] - [remote_util] [140099324987136] - INFO - command executed successfully with root [2021-02-12 23:27:16,271] - [remote_util] [140099324987136] - INFO - running command.raw on 172.23.122.90: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-12 23:27:16,324] - [remote_util] [140099324987136] - INFO - command executed successfully with root [2021-02-12 23:27:16,325] - [remote_util] [140099324987136] - INFO - SSH Connecting to 172.23.122.90 with username:root, attempt#1 of 5 [2021-02-12 23:27:16,426] - [remote_util] [140099324987136] - INFO - SSH Connected to 172.23.122.90 as root [2021-02-12 23:27:16,714] - [remote_util] [140099324987136] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:27:17,014] - [remote_util] [140099324987136] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:27:17,073] - [remote_util] [140099324987136] - INFO - 172.23.122.90 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 23:27:17,073] - [remote_util] [140099324987136] - INFO - Checking for process beam.smp on linux [2021-02-12 23:27:17,112] - [remote_util] [140099324987136] - INFO - Starting couchbase server [2021-02-12 23:27:17,210] - [remote_util] [140099324987136] - INFO - Running systemd command on this server [2021-02-12 23:27:17,210] - [remote_util] [140099324987136] - INFO - running command.raw on 172.23.122.90: systemctl start couchbase-server.service [2021-02-12 23:27:17,235] - [remote_util] [140099324987136] - INFO - command executed successfully with root [2021-02-12 23:27:17,235] - [remote_util] [140099324987136] - INFO - 172.23.122.90:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-12 23:27:22,240] - [remote_util] [140099324987136] - INFO - running command.raw on 172.23.122.90: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-12 23:27:22,266] - [remote_util] [140099324987136] - INFO - command executed successfully with root [2021-02-12 23:27:22,266] - [remote_util] [140099324987136] - INFO - Couchbase server status: [' Process: 9168 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-12 23:27:22,266] - [remote_util] [140099324987136] - INFO - Checking for process beam.smp on linux [2021-02-12 23:27:22,336] - [remote_util] [140099324987136] - INFO - process beam.smp is running on 172.23.122.90: with pid 9313 [2021-02-12 23:27:22,336] - [basetestcase] [140099324987136] - INFO - Couchbase started [2021-02-12 23:27:22,337] - [remote_util] [140099324987136] - INFO - SSH Connecting to 172.23.122.117 with username:root, attempt#1 of 5 [2021-02-12 23:27:22,439] - [remote_util] [140099324987136] - INFO - SSH Connected to 172.23.122.117 as root [2021-02-12 23:27:22,687] - [remote_util] [140099324987136] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:27:22,973] - [remote_util] [140099324987136] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:27:22,981] - [remote_util] [140099324987136] - INFO - SSH Connecting to 172.23.122.117 with username:root, attempt#1 of 5 [2021-02-12 23:27:23,080] - [remote_util] [140099324987136] - INFO - SSH Connected to 172.23.122.117 as root [2021-02-12 23:27:23,333] - [remote_util] [140099324987136] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:27:23,627] - [remote_util] [140099324987136] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:27:23,647] - [remote_util] [140099324987136] - INFO - 172.23.122.117 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 23:27:23,700] - [remote_util] [140099324987136] - INFO - Running systemd command on this server [2021-02-12 23:27:23,700] - [remote_util] [140099324987136] - INFO - running command.raw on 172.23.122.117: systemctl stop couchbase-server.service [2021-02-12 23:27:25,437] - [remote_util] [140099324987136] - INFO - command executed successfully with root [2021-02-12 23:27:25,438] - [basetestcase] [140099324987136] - INFO - Couchbase stopped [2021-02-12 23:27:25,438] - [basetestcase] [140099324987136] - INFO - sleep for 5 secs. ... [2021-02-12 23:27:30,444] - [remote_util] [140099324987136] - INFO - running command.raw on 172.23.122.117: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-12 23:27:30,458] - [remote_util] [140099324987136] - INFO - command executed successfully with root [2021-02-12 23:27:30,458] - [remote_util] [140099324987136] - INFO - running command.raw on 172.23.122.117: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-12 23:27:30,510] - [remote_util] [140099324987136] - INFO - command executed successfully with root [2021-02-12 23:27:30,511] - [remote_util] [140099324987136] - INFO - SSH Connecting to 172.23.122.117 with username:root, attempt#1 of 5 [2021-02-12 23:27:30,607] - [remote_util] [140099324987136] - INFO - SSH Connected to 172.23.122.117 as root [2021-02-12 23:27:30,861] - [remote_util] [140099324987136] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:27:31,154] - [remote_util] [140099324987136] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:27:31,174] - [remote_util] [140099324987136] - INFO - 172.23.122.117 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 23:27:31,174] - [remote_util] [140099324987136] - INFO - Checking for process beam.smp on linux [2021-02-12 23:27:31,209] - [remote_util] [140099324987136] - INFO - Starting couchbase server [2021-02-12 23:27:31,305] - [remote_util] [140099324987136] - INFO - Running systemd command on this server [2021-02-12 23:27:31,305] - [remote_util] [140099324987136] - INFO - running command.raw on 172.23.122.117: systemctl start couchbase-server.service [2021-02-12 23:27:31,326] - [remote_util] [140099324987136] - INFO - command executed successfully with root [2021-02-12 23:27:31,326] - [remote_util] [140099324987136] - INFO - 172.23.122.117:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-12 23:27:36,331] - [remote_util] [140099324987136] - INFO - running command.raw on 172.23.122.117: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-12 23:27:36,355] - [remote_util] [140099324987136] - INFO - command executed successfully with root [2021-02-12 23:27:36,356] - [remote_util] [140099324987136] - INFO - Couchbase server status: [' Process: 9781 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-12 23:27:36,356] - [remote_util] [140099324987136] - INFO - Checking for process beam.smp on linux [2021-02-12 23:27:36,422] - [remote_util] [140099324987136] - INFO - process beam.smp is running on 172.23.122.117: with pid 9925 [2021-02-12 23:27:36,422] - [basetestcase] [140099324987136] - INFO - Couchbase started [2021-02-12 23:27:36,424] - [remote_util] [140099324987136] - INFO - SSH Connecting to 172.23.122.97 with username:root, attempt#1 of 5 [2021-02-12 23:27:36,527] - [remote_util] [140099324987136] - INFO - SSH Connected to 172.23.122.97 as root [2021-02-12 23:27:36,790] - [remote_util] [140099324987136] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:27:37,093] - [remote_util] [140099324987136] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:27:37,100] - [remote_util] [140099324987136] - INFO - SSH Connecting to 172.23.122.97 with username:root, attempt#1 of 5 [2021-02-12 23:27:37,202] - [remote_util] [140099324987136] - INFO - SSH Connected to 172.23.122.97 as root [2021-02-12 23:27:37,463] - [remote_util] [140099324987136] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:27:37,764] - [remote_util] [140099324987136] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:27:37,782] - [remote_util] [140099324987136] - INFO - 172.23.122.97 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 23:27:37,835] - [remote_util] [140099324987136] - INFO - Running systemd command on this server [2021-02-12 23:27:37,835] - [remote_util] [140099324987136] - INFO - running command.raw on 172.23.122.97: systemctl stop couchbase-server.service [2021-02-12 23:27:39,635] - [remote_util] [140099324987136] - INFO - command executed successfully with root [2021-02-12 23:27:39,635] - [basetestcase] [140099324987136] - INFO - Couchbase stopped [2021-02-12 23:27:39,635] - [basetestcase] [140099324987136] - INFO - sleep for 5 secs. ... [2021-02-12 23:27:44,640] - [remote_util] [140099324987136] - INFO - running command.raw on 172.23.122.97: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-12 23:27:44,660] - [remote_util] [140099324987136] - INFO - command executed successfully with root [2021-02-12 23:27:44,661] - [remote_util] [140099324987136] - INFO - running command.raw on 172.23.122.97: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-12 23:27:44,714] - [remote_util] [140099324987136] - INFO - command executed successfully with root [2021-02-12 23:27:44,716] - [remote_util] [140099324987136] - INFO - SSH Connecting to 172.23.122.97 with username:root, attempt#1 of 5 [2021-02-12 23:27:44,815] - [remote_util] [140099324987136] - INFO - SSH Connected to 172.23.122.97 as root [2021-02-12 23:27:45,073] - [remote_util] [140099324987136] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:27:45,372] - [remote_util] [140099324987136] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:27:45,430] - [remote_util] [140099324987136] - INFO - 172.23.122.97 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 23:27:45,430] - [remote_util] [140099324987136] - INFO - Checking for process beam.smp on linux [2021-02-12 23:27:45,466] - [remote_util] [140099324987136] - INFO - Starting couchbase server [2021-02-12 23:27:45,559] - [remote_util] [140099324987136] - INFO - Running systemd command on this server [2021-02-12 23:27:45,559] - [remote_util] [140099324987136] - INFO - running command.raw on 172.23.122.97: systemctl start couchbase-server.service [2021-02-12 23:27:45,582] - [remote_util] [140099324987136] - INFO - command executed successfully with root [2021-02-12 23:27:45,582] - [remote_util] [140099324987136] - INFO - 172.23.122.97:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-12 23:27:50,587] - [remote_util] [140099324987136] - INFO - running command.raw on 172.23.122.97: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-12 23:27:50,614] - [remote_util] [140099324987136] - INFO - command executed successfully with root [2021-02-12 23:27:50,614] - [remote_util] [140099324987136] - INFO - Couchbase server status: [' Process: 4792 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-12 23:27:50,614] - [remote_util] [140099324987136] - INFO - Checking for process beam.smp on linux [2021-02-12 23:27:50,684] - [remote_util] [140099324987136] - INFO - process beam.smp is running on 172.23.122.97: with pid 4936 [2021-02-12 23:27:50,684] - [basetestcase] [140099324987136] - INFO - Couchbase started [2021-02-12 23:27:50,685] - [remote_util] [140099324987136] - INFO - SSH Connecting to 172.23.122.82 with username:root, attempt#1 of 5 [2021-02-12 23:27:50,791] - [remote_util] [140099324987136] - INFO - SSH Connected to 172.23.122.82 as root [2021-02-12 23:27:51,075] - [remote_util] [140099324987136] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:27:51,381] - [remote_util] [140099324987136] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:27:51,389] - [remote_util] [140099324987136] - INFO - SSH Connecting to 172.23.122.82 with username:root, attempt#1 of 5 [2021-02-12 23:27:51,488] - [remote_util] [140099324987136] - INFO - SSH Connected to 172.23.122.82 as root [2021-02-12 23:27:51,750] - [remote_util] [140099324987136] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:27:52,048] - [remote_util] [140099324987136] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:27:52,067] - [remote_util] [140099324987136] - INFO - 172.23.122.82 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 23:27:52,121] - [remote_util] [140099324987136] - INFO - Running systemd command on this server [2021-02-12 23:27:52,121] - [remote_util] [140099324987136] - INFO - running command.raw on 172.23.122.82: systemctl stop couchbase-server.service [2021-02-12 23:27:53,927] - [remote_util] [140099324987136] - INFO - command executed successfully with root [2021-02-12 23:27:53,928] - [basetestcase] [140099324987136] - INFO - Couchbase stopped [2021-02-12 23:27:53,928] - [basetestcase] [140099324987136] - INFO - sleep for 5 secs. ... [2021-02-12 23:27:58,933] - [remote_util] [140099324987136] - INFO - running command.raw on 172.23.122.82: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-12 23:27:58,952] - [remote_util] [140099324987136] - INFO - command executed successfully with root [2021-02-12 23:27:58,953] - [remote_util] [140099324987136] - INFO - running command.raw on 172.23.122.82: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-12 23:27:59,007] - [remote_util] [140099324987136] - INFO - command executed successfully with root [2021-02-12 23:27:59,009] - [remote_util] [140099324987136] - INFO - SSH Connecting to 172.23.122.82 with username:root, attempt#1 of 5 [2021-02-12 23:27:59,115] - [remote_util] [140099324987136] - INFO - SSH Connected to 172.23.122.82 as root [2021-02-12 23:27:59,390] - [remote_util] [140099324987136] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:27:59,695] - [remote_util] [140099324987136] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:27:59,714] - [remote_util] [140099324987136] - INFO - 172.23.122.82 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 23:27:59,714] - [remote_util] [140099324987136] - INFO - Checking for process beam.smp on linux [2021-02-12 23:27:59,754] - [remote_util] [140099324987136] - INFO - Starting couchbase server [2021-02-12 23:27:59,852] - [remote_util] [140099324987136] - INFO - Running systemd command on this server [2021-02-12 23:27:59,852] - [remote_util] [140099324987136] - INFO - running command.raw on 172.23.122.82: systemctl start couchbase-server.service [2021-02-12 23:27:59,876] - [remote_util] [140099324987136] - INFO - command executed successfully with root [2021-02-12 23:27:59,876] - [remote_util] [140099324987136] - INFO - 172.23.122.82:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-12 23:28:04,881] - [remote_util] [140099324987136] - INFO - running command.raw on 172.23.122.82: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-12 23:28:04,905] - [remote_util] [140099324987136] - INFO - command executed successfully with root [2021-02-12 23:28:04,905] - [remote_util] [140099324987136] - INFO - Couchbase server status: [' Process: 98604 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-12 23:28:04,905] - [remote_util] [140099324987136] - INFO - Checking for process beam.smp on linux [2021-02-12 23:28:04,973] - [remote_util] [140099324987136] - INFO - process beam.smp is running on 172.23.122.82: with pid 98748 [2021-02-12 23:28:04,973] - [basetestcase] [140099324987136] - INFO - Couchbase started [2021-02-12 23:28:04,975] - [remote_util] [140099324987136] - INFO - SSH Connecting to 172.23.122.118 with username:root, attempt#1 of 5 [2021-02-12 23:28:05,080] - [remote_util] [140099324987136] - INFO - SSH Connected to 172.23.122.118 as root [2021-02-12 23:28:05,339] - [remote_util] [140099324987136] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:28:05,640] - [remote_util] [140099324987136] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:28:05,649] - [remote_util] [140099324987136] - INFO - SSH Connecting to 172.23.122.118 with username:root, attempt#1 of 5 [2021-02-12 23:28:05,748] - [remote_util] [140099324987136] - INFO - SSH Connected to 172.23.122.118 as root [2021-02-12 23:28:05,996] - [remote_util] [140099324987136] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:28:06,287] - [remote_util] [140099324987136] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:28:06,306] - [remote_util] [140099324987136] - INFO - 172.23.122.118 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 23:28:06,357] - [remote_util] [140099324987136] - INFO - Running systemd command on this server [2021-02-12 23:28:06,357] - [remote_util] [140099324987136] - INFO - running command.raw on 172.23.122.118: systemctl stop couchbase-server.service [2021-02-12 23:28:08,058] - [remote_util] [140099324987136] - INFO - command executed successfully with root [2021-02-12 23:28:08,059] - [basetestcase] [140099324987136] - INFO - Couchbase stopped [2021-02-12 23:28:08,059] - [basetestcase] [140099324987136] - INFO - sleep for 5 secs. ... [2021-02-12 23:28:13,064] - [remote_util] [140099324987136] - INFO - running command.raw on 172.23.122.118: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-12 23:28:13,079] - [remote_util] [140099324987136] - INFO - command executed successfully with root [2021-02-12 23:28:13,079] - [remote_util] [140099324987136] - INFO - running command.raw on 172.23.122.118: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-12 23:28:13,133] - [remote_util] [140099324987136] - INFO - command executed successfully with root [2021-02-12 23:28:13,134] - [remote_util] [140099324987136] - INFO - SSH Connecting to 172.23.122.118 with username:root, attempt#1 of 5 [2021-02-12 23:28:13,231] - [remote_util] [140099324987136] - INFO - SSH Connected to 172.23.122.118 as root [2021-02-12 23:28:13,472] - [remote_util] [140099324987136] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:28:13,760] - [remote_util] [140099324987136] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:28:13,817] - [remote_util] [140099324987136] - INFO - 172.23.122.118 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 23:28:13,818] - [remote_util] [140099324987136] - INFO - Checking for process beam.smp on linux [2021-02-12 23:28:13,852] - [remote_util] [140099324987136] - INFO - Starting couchbase server [2021-02-12 23:28:13,944] - [remote_util] [140099324987136] - INFO - Running systemd command on this server [2021-02-12 23:28:13,944] - [remote_util] [140099324987136] - INFO - running command.raw on 172.23.122.118: systemctl start couchbase-server.service [2021-02-12 23:28:13,965] - [remote_util] [140099324987136] - INFO - command executed successfully with root [2021-02-12 23:28:13,966] - [remote_util] [140099324987136] - INFO - 172.23.122.118:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-12 23:28:18,966] - [remote_util] [140099324987136] - INFO - running command.raw on 172.23.122.118: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-12 23:28:18,990] - [remote_util] [140099324987136] - INFO - command executed successfully with root [2021-02-12 23:28:18,990] - [remote_util] [140099324987136] - INFO - Couchbase server status: [' Process: 124188 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-12 23:28:18,990] - [remote_util] [140099324987136] - INFO - Checking for process beam.smp on linux [2021-02-12 23:28:19,059] - [remote_util] [140099324987136] - INFO - process beam.smp is running on 172.23.122.118: with pid 124332 [2021-02-12 23:28:19,059] - [basetestcase] [140099324987136] - INFO - Couchbase started [2021-02-12 23:28:19,061] - [remote_util] [140099324987136] - INFO - SSH Connecting to 172.23.122.116 with username:root, attempt#1 of 5 [2021-02-12 23:28:19,163] - [remote_util] [140099324987136] - INFO - SSH Connected to 172.23.122.116 as root [2021-02-12 23:28:19,430] - [remote_util] [140099324987136] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:28:19,721] - [remote_util] [140099324987136] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:28:19,729] - [remote_util] [140099324987136] - INFO - SSH Connecting to 172.23.122.116 with username:root, attempt#1 of 5 [2021-02-12 23:28:19,825] - [remote_util] [140099324987136] - INFO - SSH Connected to 172.23.122.116 as root [2021-02-12 23:28:20,093] - [remote_util] [140099324987136] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:28:20,382] - [remote_util] [140099324987136] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:28:20,400] - [remote_util] [140099324987136] - INFO - 172.23.122.116 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 23:28:20,450] - [remote_util] [140099324987136] - INFO - Running systemd command on this server [2021-02-12 23:28:20,451] - [remote_util] [140099324987136] - INFO - running command.raw on 172.23.122.116: systemctl stop couchbase-server.service [2021-02-12 23:28:22,283] - [remote_util] [140099324987136] - INFO - command executed successfully with root [2021-02-12 23:28:22,284] - [basetestcase] [140099324987136] - INFO - Couchbase stopped [2021-02-12 23:28:22,284] - [basetestcase] [140099324987136] - INFO - sleep for 5 secs. ... [2021-02-12 23:28:27,289] - [remote_util] [140099324987136] - INFO - running command.raw on 172.23.122.116: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-12 23:28:27,305] - [remote_util] [140099324987136] - INFO - command executed successfully with root [2021-02-12 23:28:27,305] - [remote_util] [140099324987136] - INFO - running command.raw on 172.23.122.116: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-12 23:28:27,358] - [remote_util] [140099324987136] - INFO - command executed successfully with root [2021-02-12 23:28:27,360] - [remote_util] [140099324987136] - INFO - SSH Connecting to 172.23.122.116 with username:root, attempt#1 of 5 [2021-02-12 23:28:27,457] - [remote_util] [140099324987136] - INFO - SSH Connected to 172.23.122.116 as root [2021-02-12 23:28:27,727] - [remote_util] [140099324987136] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:28:28,019] - [remote_util] [140099324987136] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:28:28,038] - [remote_util] [140099324987136] - INFO - 172.23.122.116 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 23:28:28,038] - [remote_util] [140099324987136] - INFO - Checking for process beam.smp on linux [2021-02-12 23:28:28,074] - [remote_util] [140099324987136] - INFO - Starting couchbase server [2021-02-12 23:28:28,166] - [remote_util] [140099324987136] - INFO - Running systemd command on this server [2021-02-12 23:28:28,166] - [remote_util] [140099324987136] - INFO - running command.raw on 172.23.122.116: systemctl start couchbase-server.service [2021-02-12 23:28:28,189] - [remote_util] [140099324987136] - INFO - command executed successfully with root [2021-02-12 23:28:28,189] - [remote_util] [140099324987136] - INFO - 172.23.122.116:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-12 23:28:33,194] - [remote_util] [140099324987136] - INFO - running command.raw on 172.23.122.116: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-12 23:28:33,220] - [remote_util] [140099324987136] - INFO - command executed successfully with root [2021-02-12 23:28:33,220] - [remote_util] [140099324987136] - INFO - Couchbase server status: [' Process: 83858 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-12 23:28:33,220] - [remote_util] [140099324987136] - INFO - Checking for process beam.smp on linux [2021-02-12 23:28:33,285] - [remote_util] [140099324987136] - INFO - process beam.smp is running on 172.23.122.116: with pid 84001 [2021-02-12 23:28:33,285] - [basetestcase] [140099324987136] - INFO - Couchbase started [2021-02-12 23:28:33,286] - [basetestcase] [140099324987136] - INFO - sleep for 10 secs. ... [2021-02-12 23:28:43,331] - [remote_util] [140099324987136] - INFO - SSH Connecting to 172.23.123.61 with username:root, attempt#1 of 5 [2021-02-12 23:28:43,333] - [remote_util] [140099341772544] - INFO - SSH Connecting to 172.23.123.62 with username:root, attempt#1 of 5 [2021-02-12 23:28:43,335] - [remote_util] [140099333379840] - INFO - SSH Connecting to 172.23.122.90 with username:root, attempt#1 of 5 [2021-02-12 23:28:43,338] - [remote_util] [140098737792768] - INFO - SSH Connecting to 172.23.122.117 with username:root, attempt#1 of 5 [2021-02-12 23:28:43,339] - [remote_util] [140098712618752] - INFO - SSH Connecting to 172.23.122.97 with username:root, attempt#1 of 5 [2021-02-12 23:28:43,342] - [remote_util] [140097149204224] - INFO - SSH Connecting to 172.23.122.118 with username:root, attempt#1 of 5 [2021-02-12 23:28:43,344] - [remote_util] [140097168602880] - INFO - SSH Connecting to 172.23.122.82 with username:root, attempt#1 of 5 [2021-02-12 23:28:43,345] - [remote_util] [140096925595392] - INFO - SSH Connecting to 172.23.122.116 with username:root, attempt#1 of 5 [2021-02-12 23:28:43,436] - [remote_util] [140099333379840] - INFO - SSH Connected to 172.23.122.90 as root [2021-02-12 23:28:43,443] - [remote_util] [140098712618752] - INFO - SSH Connected to 172.23.122.97 as root [2021-02-12 23:28:43,443] - [remote_util] [140097149204224] - INFO - SSH Connected to 172.23.122.118 as root [2021-02-12 23:28:43,444] - [remote_util] [140098737792768] - INFO - SSH Connected to 172.23.122.117 as root [2021-02-12 23:28:43,445] - [remote_util] [140099324987136] - INFO - SSH Connected to 172.23.123.61 as root [2021-02-12 23:28:43,448] - [remote_util] [140097168602880] - INFO - SSH Connected to 172.23.122.82 as root [2021-02-12 23:28:43,449] - [remote_util] [140096925595392] - INFO - SSH Connected to 172.23.122.116 as root [2021-02-12 23:28:43,451] - [remote_util] [140099341772544] - INFO - SSH Connected to 172.23.123.62 as root [2021-02-12 23:28:43,703] - [remote_util] [140097149204224] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:28:43,732] - [remote_util] [140098712618752] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:28:43,747] - [remote_util] [140099333379840] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:28:43,754] - [remote_util] [140098737792768] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:28:43,767] - [remote_util] [140099324987136] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:28:43,768] - [remote_util] [140097168602880] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:28:43,769] - [remote_util] [140096925595392] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:28:43,778] - [remote_util] [140099341772544] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:28:44,024] - [remote_util] [140097149204224] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:28:44,025] - [remote_util] [140097149204224] - INFO - running command.raw on 172.23.122.118: /opt/couchbase/bin/cbcollect_info 172.23.122.118-20210212-2328-diag.zip [2021-02-12 23:28:44,051] - [remote_util] [140099333379840] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:28:44,052] - [remote_util] [140099333379840] - INFO - running command.raw on 172.23.122.90: /opt/couchbase/bin/cbcollect_info 172.23.122.90-20210212-2328-diag.zip [2021-02-12 23:28:44,060] - [remote_util] [140096925595392] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:28:44,060] - [remote_util] [140096925595392] - INFO - running command.raw on 172.23.122.116: /opt/couchbase/bin/cbcollect_info 172.23.122.116-20210212-2328-diag.zip [2021-02-12 23:28:44,061] - [remote_util] [140098737792768] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:28:44,062] - [remote_util] [140098737792768] - INFO - running command.raw on 172.23.122.117: /opt/couchbase/bin/cbcollect_info 172.23.122.117-20210212-2328-diag.zip [2021-02-12 23:28:44,071] - [remote_util] [140097168602880] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:28:44,072] - [remote_util] [140097168602880] - INFO - running command.raw on 172.23.122.82: /opt/couchbase/bin/cbcollect_info 172.23.122.82-20210212-2328-diag.zip [2021-02-12 23:28:44,082] - [remote_util] [140099324987136] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:28:44,083] - [remote_util] [140099324987136] - INFO - running command.raw on 172.23.123.61: /opt/couchbase/bin/cbcollect_info 172.23.123.61-20210212-2328-diag.zip [2021-02-12 23:28:44,084] - [remote_util] [140099341772544] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:28:44,084] - [remote_util] [140099341772544] - INFO - running command.raw on 172.23.123.62: /opt/couchbase/bin/cbcollect_info 172.23.123.62-20210212-2328-diag.zip [2021-02-12 23:28:44,090] - [remote_util] [140098712618752] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:28:44,091] - [remote_util] [140098712618752] - INFO - running command.raw on 172.23.122.97: /opt/couchbase/bin/cbcollect_info 172.23.122.97-20210212-2328-diag.zip [2021-02-12 23:30:12,447] - [remote_util] [140096925595392] - INFO - command executed successfully with root [2021-02-12 23:30:12,687] - [remote_util] [140096925595392] - INFO - running command.raw on 172.23.122.116: rm -f /root/172.23.122.116-20210212-2328-diag.zip [2021-02-12 23:30:12,703] - [remote_util] [140096925595392] - INFO - command executed successfully with root [2021-02-12 23:30:21,554] - [remote_util] [140098737792768] - INFO - command executed successfully with root [2021-02-12 23:30:22,062] - [remote_util] [140098737792768] - INFO - running command.raw on 172.23.122.117: rm -f /root/172.23.122.117-20210212-2328-diag.zip [2021-02-12 23:30:22,082] - [remote_util] [140098737792768] - INFO - command executed successfully with root [2021-02-12 23:30:22,462] - [remote_util] [140097149204224] - INFO - command executed successfully with root [2021-02-12 23:30:22,947] - [remote_util] [140097149204224] - INFO - running command.raw on 172.23.122.118: rm -f /root/172.23.122.118-20210212-2328-diag.zip [2021-02-12 23:30:22,964] - [remote_util] [140097149204224] - INFO - command executed successfully with root [2021-02-12 23:30:25,441] - [remote_util] [140099341772544] - INFO - command executed successfully with root [2021-02-12 23:30:26,163] - [remote_util] [140099341772544] - INFO - running command.raw on 172.23.123.62: rm -f /root/172.23.123.62-20210212-2328-diag.zip [2021-02-12 23:30:26,186] - [remote_util] [140099341772544] - INFO - command executed successfully with root [2021-02-12 23:30:28,295] - [remote_util] [140097168602880] - INFO - command executed successfully with root [2021-02-12 23:30:28,769] - [remote_util] [140099333379840] - INFO - command executed successfully with root [2021-02-12 23:30:28,953] - [remote_util] [140097168602880] - INFO - running command.raw on 172.23.122.82: rm -f /root/172.23.122.82-20210212-2328-diag.zip [2021-02-12 23:30:28,976] - [remote_util] [140097168602880] - INFO - command executed successfully with root [2021-02-12 23:30:29,033] - [remote_util] [140099324987136] - INFO - command executed successfully with root [2021-02-12 23:30:30,112] - [remote_util] [140099333379840] - INFO - running command.raw on 172.23.122.90: rm -f /root/172.23.122.90-20210212-2328-diag.zip [2021-02-12 23:30:30,134] - [remote_util] [140099333379840] - INFO - command executed successfully with root [2021-02-12 23:30:30,184] - [remote_util] [140099324987136] - INFO - running command.raw on 172.23.123.61: rm -f /root/172.23.123.61-20210212-2328-diag.zip [2021-02-12 23:30:30,205] - [remote_util] [140099324987136] - INFO - command executed successfully with root [2021-02-12 23:30:34,781] - [remote_util] [140098712618752] - INFO - command executed successfully with root [2021-02-12 23:30:35,558] - [remote_util] [140098712618752] - INFO - running command.raw on 172.23.122.97: rm -f /root/172.23.122.97-20210212-2328-diag.zip [2021-02-12 23:30:35,580] - [remote_util] [140098712618752] - INFO - command executed successfully with root