[2021-02-12 21:30:10,079] - [eventing_base] [140099341772544] - INFO - Starting Test: test_kv_rebalance_out_when_existing_eventing_node_is_processing_mutations [2021-02-12 21:30:10,081] - [remote_util] [140099341772544] - INFO - SSH Connecting to 172.23.123.61 with username:root, attempt#1 of 5 [2021-02-12 21:30:10,185] - [remote_util] [140099341772544] - INFO - SSH Connected to 172.23.123.61 as root [2021-02-12 21:30:10,456] - [remote_util] [140099341772544] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:30:10,760] - [remote_util] [140099341772544] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:30:10,765] - [rest_client] [140099341772544] - INFO - -->is_ns_server_running? [2021-02-12 21:30:10,773] - [rest_client] [140099341772544] - INFO - Node version in cluster 7.0.0-4454-enterprise [2021-02-12 21:30:10,777] - [rest_client] [140099341772544] - ERROR - GET http://172.23.123.61:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-02-12 21:30:10,777] - [rest_client] [140099341772544] - INFO - Node versions in cluster [] [2021-02-12 21:30:10,777] - [basetestcase] [140099341772544] - INFO - ============== basetestcase setup was started for test #5 test_kv_rebalance_out_when_existing_eventing_node_is_processing_mutations============== [2021-02-12 21:30:10,782] - [rest_client] [140099341772544] - ERROR - GET http://172.23.123.61:8091/pools/default/ body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-02-12 21:30:10,796] - [bucket_helper] [140099341772544] - INFO - Could not find any buckets for node 172.23.123.61, nothing to delete [2021-02-12 21:30:10,806] - [bucket_helper] [140099341772544] - INFO - Could not find any buckets for node 172.23.123.62, nothing to delete [2021-02-12 21:30:10,815] - [bucket_helper] [140099341772544] - INFO - Could not find any buckets for node 172.23.122.90, nothing to delete [2021-02-12 21:30:10,825] - [bucket_helper] [140099341772544] - INFO - Could not find any buckets for node 172.23.122.117, nothing to delete [2021-02-12 21:30:10,835] - [bucket_helper] [140099341772544] - INFO - Could not find any buckets for node 172.23.122.97, nothing to delete [2021-02-12 21:30:10,845] - [bucket_helper] [140099341772544] - INFO - Could not find any buckets for node 172.23.122.82, nothing to delete [2021-02-12 21:30:10,855] - [bucket_helper] [140099341772544] - INFO - Could not find any buckets for node 172.23.122.118, nothing to delete [2021-02-12 21:30:10,864] - [bucket_helper] [140099341772544] - INFO - Could not find any buckets for node 172.23.122.116, nothing to delete [2021-02-12 21:30:10,868] - [rest_client] [140099341772544] - INFO - -->is_ns_server_running? [2021-02-12 21:30:10,879] - [cluster_helper] [140099341772544] - INFO - waiting for ns_server @ 172.23.123.61:8091 [2021-02-12 21:30:10,879] - [rest_client] [140099341772544] - INFO - -->is_ns_server_running? [2021-02-12 21:30:10,881] - [cluster_helper] [140099341772544] - INFO - ns_server @ 172.23.123.61:8091 is running [2021-02-12 21:30:10,885] - [cluster_helper] [140099341772544] - INFO - waiting for ns_server @ 172.23.123.62:8091 [2021-02-12 21:30:10,885] - [rest_client] [140099341772544] - INFO - -->is_ns_server_running? [2021-02-12 21:30:10,887] - [cluster_helper] [140099341772544] - INFO - ns_server @ 172.23.123.62:8091 is running [2021-02-12 21:30:10,890] - [cluster_helper] [140099341772544] - INFO - waiting for ns_server @ 172.23.122.90:8091 [2021-02-12 21:30:10,890] - [rest_client] [140099341772544] - INFO - -->is_ns_server_running? [2021-02-12 21:30:10,893] - [cluster_helper] [140099341772544] - INFO - ns_server @ 172.23.122.90:8091 is running [2021-02-12 21:30:10,896] - [cluster_helper] [140099341772544] - INFO - waiting for ns_server @ 172.23.122.117:8091 [2021-02-12 21:30:10,896] - [rest_client] [140099341772544] - INFO - -->is_ns_server_running? [2021-02-12 21:30:10,899] - [cluster_helper] [140099341772544] - INFO - ns_server @ 172.23.122.117:8091 is running [2021-02-12 21:30:10,902] - [cluster_helper] [140099341772544] - INFO - waiting for ns_server @ 172.23.122.97:8091 [2021-02-12 21:30:10,902] - [rest_client] [140099341772544] - INFO - -->is_ns_server_running? [2021-02-12 21:30:10,905] - [cluster_helper] [140099341772544] - INFO - ns_server @ 172.23.122.97:8091 is running [2021-02-12 21:30:10,909] - [cluster_helper] [140099341772544] - INFO - waiting for ns_server @ 172.23.122.82:8091 [2021-02-12 21:30:10,909] - [rest_client] [140099341772544] - INFO - -->is_ns_server_running? [2021-02-12 21:30:10,911] - [cluster_helper] [140099341772544] - INFO - ns_server @ 172.23.122.82:8091 is running [2021-02-12 21:30:10,914] - [cluster_helper] [140099341772544] - INFO - waiting for ns_server @ 172.23.122.118:8091 [2021-02-12 21:30:10,914] - [rest_client] [140099341772544] - INFO - -->is_ns_server_running? [2021-02-12 21:30:10,917] - [cluster_helper] [140099341772544] - INFO - ns_server @ 172.23.122.118:8091 is running [2021-02-12 21:30:10,920] - [cluster_helper] [140099341772544] - INFO - waiting for ns_server @ 172.23.122.116:8091 [2021-02-12 21:30:10,920] - [rest_client] [140099341772544] - INFO - -->is_ns_server_running? [2021-02-12 21:30:10,923] - [cluster_helper] [140099341772544] - INFO - ns_server @ 172.23.122.116:8091 is running [2021-02-12 21:30:10,923] - [basetestcase] [140099341772544] - INFO - Building docker image with java sdk client [2021-02-12 21:30:14,467] - [basetestcase] [140099341772544] - INFO - initializing cluster [2021-02-12 21:30:14,469] - [remote_util] [140099341772544] - INFO - SSH Connecting to 172.23.123.61 with username:root, attempt#1 of 5 [2021-02-12 21:30:14,571] - [remote_util] [140099341772544] - INFO - SSH Connected to 172.23.123.61 as root [2021-02-12 21:30:14,836] - [remote_util] [140099341772544] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:30:15,136] - [remote_util] [140099341772544] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:30:15,143] - [remote_util] [140099341772544] - INFO - SSH Connecting to 172.23.123.61 with username:root, attempt#1 of 5 [2021-02-12 21:30:15,247] - [remote_util] [140099341772544] - INFO - SSH Connected to 172.23.123.61 as root [2021-02-12 21:30:15,512] - [remote_util] [140099341772544] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:30:15,813] - [remote_util] [140099341772544] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:30:15,834] - [remote_util] [140099341772544] - INFO - 172.23.123.61 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 21:30:15,891] - [remote_util] [140099341772544] - INFO - Running systemd command on this server [2021-02-12 21:30:15,891] - [remote_util] [140099341772544] - INFO - running command.raw on 172.23.123.61: systemctl stop couchbase-server.service [2021-02-12 21:30:17,545] - [remote_util] [140099341772544] - INFO - command executed successfully with root [2021-02-12 21:30:17,545] - [basetestcase] [140099341772544] - INFO - Couchbase stopped [2021-02-12 21:30:17,546] - [basetestcase] [140099341772544] - INFO - sleep for 5 secs. ... [2021-02-12 21:30:22,551] - [remote_util] [140099341772544] - INFO - running command.raw on 172.23.123.61: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-12 21:30:22,566] - [remote_util] [140099341772544] - INFO - command executed successfully with root [2021-02-12 21:30:22,566] - [remote_util] [140099341772544] - INFO - running command.raw on 172.23.123.61: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-12 21:30:22,621] - [remote_util] [140099341772544] - INFO - command executed successfully with root [2021-02-12 21:30:22,622] - [remote_util] [140099341772544] - INFO - SSH Connecting to 172.23.123.61 with username:root, attempt#1 of 5 [2021-02-12 21:30:22,727] - [remote_util] [140099341772544] - INFO - SSH Connected to 172.23.123.61 as root [2021-02-12 21:30:22,995] - [remote_util] [140099341772544] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:30:23,297] - [remote_util] [140099341772544] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:30:23,358] - [remote_util] [140099341772544] - INFO - 172.23.123.61 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 21:30:23,358] - [remote_util] [140099341772544] - INFO - Checking for process beam.smp on linux [2021-02-12 21:30:23,397] - [remote_util] [140099341772544] - INFO - Starting couchbase server [2021-02-12 21:30:23,495] - [remote_util] [140099341772544] - INFO - Running systemd command on this server [2021-02-12 21:30:23,495] - [remote_util] [140099341772544] - INFO - running command.raw on 172.23.123.61: systemctl start couchbase-server.service [2021-02-12 21:30:23,519] - [remote_util] [140099341772544] - INFO - command executed successfully with root [2021-02-12 21:30:23,519] - [remote_util] [140099341772544] - INFO - 172.23.123.61:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-12 21:30:28,524] - [remote_util] [140099341772544] - INFO - running command.raw on 172.23.123.61: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-12 21:30:28,551] - [remote_util] [140099341772544] - INFO - command executed successfully with root [2021-02-12 21:30:28,551] - [remote_util] [140099341772544] - INFO - Couchbase server status: [' Process: 88880 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-12 21:30:28,551] - [remote_util] [140099341772544] - INFO - Checking for process beam.smp on linux [2021-02-12 21:30:28,619] - [remote_util] [140099341772544] - INFO - process beam.smp is running on 172.23.123.61: with pid 89021 [2021-02-12 21:30:28,620] - [basetestcase] [140099341772544] - INFO - Couchbase started [2021-02-12 21:30:28,621] - [remote_util] [140099341772544] - INFO - SSH Connecting to 172.23.123.62 with username:root, attempt#1 of 5 [2021-02-12 21:30:28,727] - [remote_util] [140099341772544] - INFO - SSH Connected to 172.23.123.62 as root [2021-02-12 21:30:29,015] - [remote_util] [140099341772544] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:30:29,319] - [remote_util] [140099341772544] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:30:29,327] - [remote_util] [140099341772544] - INFO - SSH Connecting to 172.23.123.62 with username:root, attempt#1 of 5 [2021-02-12 21:30:29,428] - [remote_util] [140099341772544] - INFO - SSH Connected to 172.23.123.62 as root [2021-02-12 21:30:29,703] - [remote_util] [140099341772544] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:30:30,003] - [remote_util] [140099341772544] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:30:30,062] - [remote_util] [140099341772544] - INFO - 172.23.123.62 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 21:30:30,115] - [remote_util] [140099341772544] - INFO - Running systemd command on this server [2021-02-12 21:30:30,115] - [remote_util] [140099341772544] - INFO - running command.raw on 172.23.123.62: systemctl stop couchbase-server.service [2021-02-12 21:30:31,861] - [remote_util] [140099341772544] - INFO - command executed successfully with root [2021-02-12 21:30:31,861] - [basetestcase] [140099341772544] - INFO - Couchbase stopped [2021-02-12 21:30:31,862] - [basetestcase] [140099341772544] - INFO - sleep for 5 secs. ... [2021-02-12 21:30:36,867] - [remote_util] [140099341772544] - INFO - running command.raw on 172.23.123.62: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-12 21:30:36,885] - [remote_util] [140099341772544] - INFO - command executed successfully with root [2021-02-12 21:30:36,885] - [remote_util] [140099341772544] - INFO - running command.raw on 172.23.123.62: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-12 21:30:36,938] - [remote_util] [140099341772544] - INFO - command executed successfully with root [2021-02-12 21:30:36,939] - [remote_util] [140099341772544] - INFO - SSH Connecting to 172.23.123.62 with username:root, attempt#1 of 5 [2021-02-12 21:30:37,040] - [remote_util] [140099341772544] - INFO - SSH Connected to 172.23.123.62 as root [2021-02-12 21:30:37,321] - [remote_util] [140099341772544] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:30:37,620] - [remote_util] [140099341772544] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:30:37,679] - [remote_util] [140099341772544] - INFO - 172.23.123.62 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 21:30:37,679] - [remote_util] [140099341772544] - INFO - Checking for process beam.smp on linux [2021-02-12 21:30:37,717] - [remote_util] [140099341772544] - INFO - Starting couchbase server [2021-02-12 21:30:37,813] - [remote_util] [140099341772544] - INFO - Running systemd command on this server [2021-02-12 21:30:37,813] - [remote_util] [140099341772544] - INFO - running command.raw on 172.23.123.62: systemctl start couchbase-server.service [2021-02-12 21:30:37,836] - [remote_util] [140099341772544] - INFO - command executed successfully with root [2021-02-12 21:30:37,836] - [remote_util] [140099341772544] - INFO - 172.23.123.62:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-12 21:30:42,841] - [remote_util] [140099341772544] - INFO - running command.raw on 172.23.123.62: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-12 21:30:42,868] - [remote_util] [140099341772544] - INFO - command executed successfully with root [2021-02-12 21:30:42,868] - [remote_util] [140099341772544] - INFO - Couchbase server status: [' Process: 102171 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-12 21:30:42,868] - [remote_util] [140099341772544] - INFO - Checking for process beam.smp on linux [2021-02-12 21:30:42,940] - [remote_util] [140099341772544] - INFO - process beam.smp is running on 172.23.123.62: with pid 102315 [2021-02-12 21:30:42,940] - [basetestcase] [140099341772544] - INFO - Couchbase started [2021-02-12 21:30:42,941] - [remote_util] [140099341772544] - INFO - SSH Connecting to 172.23.122.90 with username:root, attempt#1 of 5 [2021-02-12 21:30:43,046] - [remote_util] [140099341772544] - INFO - SSH Connected to 172.23.122.90 as root [2021-02-12 21:30:43,339] - [remote_util] [140099341772544] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:30:43,637] - [remote_util] [140099341772544] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:30:43,649] - [remote_util] [140099341772544] - INFO - SSH Connecting to 172.23.122.90 with username:root, attempt#1 of 5 [2021-02-12 21:30:43,749] - [remote_util] [140099341772544] - INFO - SSH Connected to 172.23.122.90 as root [2021-02-12 21:30:44,028] - [remote_util] [140099341772544] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:30:44,327] - [remote_util] [140099341772544] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:30:44,387] - [remote_util] [140099341772544] - INFO - 172.23.122.90 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 21:30:44,441] - [remote_util] [140099341772544] - INFO - Running systemd command on this server [2021-02-12 21:30:44,442] - [remote_util] [140099341772544] - INFO - running command.raw on 172.23.122.90: systemctl stop couchbase-server.service [2021-02-12 21:30:46,228] - [remote_util] [140099341772544] - INFO - command executed successfully with root [2021-02-12 21:30:46,228] - [basetestcase] [140099341772544] - INFO - Couchbase stopped [2021-02-12 21:30:46,229] - [basetestcase] [140099341772544] - INFO - sleep for 5 secs. ... [2021-02-12 21:30:51,234] - [remote_util] [140099341772544] - INFO - running command.raw on 172.23.122.90: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-12 21:30:51,249] - [remote_util] [140099341772544] - INFO - command executed successfully with root [2021-02-12 21:30:51,249] - [remote_util] [140099341772544] - INFO - running command.raw on 172.23.122.90: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-12 21:30:51,302] - [remote_util] [140099341772544] - INFO - command executed successfully with root [2021-02-12 21:30:51,304] - [remote_util] [140099341772544] - INFO - SSH Connecting to 172.23.122.90 with username:root, attempt#1 of 5 [2021-02-12 21:30:51,403] - [remote_util] [140099341772544] - INFO - SSH Connected to 172.23.122.90 as root [2021-02-12 21:30:51,681] - [remote_util] [140099341772544] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:30:51,967] - [remote_util] [140099341772544] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:30:52,025] - [remote_util] [140099341772544] - INFO - 172.23.122.90 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 21:30:52,025] - [remote_util] [140099341772544] - INFO - Checking for process beam.smp on linux [2021-02-12 21:30:52,062] - [remote_util] [140099341772544] - INFO - Starting couchbase server [2021-02-12 21:30:52,152] - [remote_util] [140099341772544] - INFO - Running systemd command on this server [2021-02-12 21:30:52,152] - [remote_util] [140099341772544] - INFO - running command.raw on 172.23.122.90: systemctl start couchbase-server.service [2021-02-12 21:30:52,173] - [remote_util] [140099341772544] - INFO - command executed successfully with root [2021-02-12 21:30:52,173] - [remote_util] [140099341772544] - INFO - 172.23.122.90:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-12 21:30:57,178] - [remote_util] [140099341772544] - INFO - running command.raw on 172.23.122.90: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-12 21:30:57,201] - [remote_util] [140099341772544] - INFO - command executed successfully with root [2021-02-12 21:30:57,201] - [remote_util] [140099341772544] - INFO - Couchbase server status: [' Process: 67469 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-12 21:30:57,201] - [remote_util] [140099341772544] - INFO - Checking for process beam.smp on linux [2021-02-12 21:30:57,271] - [remote_util] [140099341772544] - INFO - process beam.smp is running on 172.23.122.90: with pid 67613 [2021-02-12 21:30:57,271] - [basetestcase] [140099341772544] - INFO - Couchbase started [2021-02-12 21:30:57,273] - [remote_util] [140099341772544] - INFO - SSH Connecting to 172.23.122.117 with username:root, attempt#1 of 5 [2021-02-12 21:30:57,374] - [remote_util] [140099341772544] - INFO - SSH Connected to 172.23.122.117 as root [2021-02-12 21:30:57,631] - [remote_util] [140099341772544] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:30:57,925] - [remote_util] [140099341772544] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:30:57,932] - [remote_util] [140099341772544] - INFO - SSH Connecting to 172.23.122.117 with username:root, attempt#1 of 5 [2021-02-12 21:30:58,027] - [remote_util] [140099341772544] - INFO - SSH Connected to 172.23.122.117 as root [2021-02-12 21:30:58,271] - [remote_util] [140099341772544] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:30:58,562] - [remote_util] [140099341772544] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:30:58,581] - [remote_util] [140099341772544] - INFO - 172.23.122.117 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 21:30:58,632] - [remote_util] [140099341772544] - INFO - Running systemd command on this server [2021-02-12 21:30:58,632] - [remote_util] [140099341772544] - INFO - running command.raw on 172.23.122.117: systemctl stop couchbase-server.service [2021-02-12 21:31:00,373] - [remote_util] [140099341772544] - INFO - command executed successfully with root [2021-02-12 21:31:00,373] - [basetestcase] [140099341772544] - INFO - Couchbase stopped [2021-02-12 21:31:00,373] - [basetestcase] [140099341772544] - INFO - sleep for 5 secs. ... [2021-02-12 21:31:05,378] - [remote_util] [140099341772544] - INFO - running command.raw on 172.23.122.117: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-12 21:31:05,394] - [remote_util] [140099341772544] - INFO - command executed successfully with root [2021-02-12 21:31:05,394] - [remote_util] [140099341772544] - INFO - running command.raw on 172.23.122.117: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-12 21:31:05,450] - [remote_util] [140099341772544] - INFO - command executed successfully with root [2021-02-12 21:31:05,451] - [remote_util] [140099341772544] - INFO - SSH Connecting to 172.23.122.117 with username:root, attempt#1 of 5 [2021-02-12 21:31:05,551] - [remote_util] [140099341772544] - INFO - SSH Connected to 172.23.122.117 as root [2021-02-12 21:31:05,809] - [remote_util] [140099341772544] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:31:06,106] - [remote_util] [140099341772544] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:31:06,126] - [remote_util] [140099341772544] - INFO - 172.23.122.117 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 21:31:06,126] - [remote_util] [140099341772544] - INFO - Checking for process beam.smp on linux [2021-02-12 21:31:06,161] - [remote_util] [140099341772544] - INFO - Starting couchbase server [2021-02-12 21:31:06,254] - [remote_util] [140099341772544] - INFO - Running systemd command on this server [2021-02-12 21:31:06,254] - [remote_util] [140099341772544] - INFO - running command.raw on 172.23.122.117: systemctl start couchbase-server.service [2021-02-12 21:31:06,276] - [remote_util] [140099341772544] - INFO - command executed successfully with root [2021-02-12 21:31:06,277] - [remote_util] [140099341772544] - INFO - 172.23.122.117:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-12 21:31:11,282] - [remote_util] [140099341772544] - INFO - running command.raw on 172.23.122.117: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-12 21:31:11,305] - [remote_util] [140099341772544] - INFO - command executed successfully with root [2021-02-12 21:31:11,305] - [remote_util] [140099341772544] - INFO - Couchbase server status: [' Process: 13945 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-12 21:31:11,305] - [remote_util] [140099341772544] - INFO - Checking for process beam.smp on linux [2021-02-12 21:31:11,373] - [remote_util] [140099341772544] - INFO - process beam.smp is running on 172.23.122.117: with pid 14090 [2021-02-12 21:31:11,373] - [basetestcase] [140099341772544] - INFO - Couchbase started [2021-02-12 21:31:11,375] - [remote_util] [140099341772544] - INFO - SSH Connecting to 172.23.122.97 with username:root, attempt#1 of 5 [2021-02-12 21:31:11,480] - [remote_util] [140099341772544] - INFO - SSH Connected to 172.23.122.97 as root [2021-02-12 21:31:11,754] - [remote_util] [140099341772544] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:31:12,056] - [remote_util] [140099341772544] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:31:12,063] - [remote_util] [140099341772544] - INFO - SSH Connecting to 172.23.122.97 with username:root, attempt#1 of 5 [2021-02-12 21:31:12,164] - [remote_util] [140099341772544] - INFO - SSH Connected to 172.23.122.97 as root [2021-02-12 21:31:12,427] - [remote_util] [140099341772544] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:31:12,732] - [remote_util] [140099341772544] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:31:12,752] - [remote_util] [140099341772544] - INFO - 172.23.122.97 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 21:31:12,809] - [remote_util] [140099341772544] - INFO - Running systemd command on this server [2021-02-12 21:31:12,809] - [remote_util] [140099341772544] - INFO - running command.raw on 172.23.122.97: systemctl stop couchbase-server.service [2021-02-12 21:31:14,630] - [remote_util] [140099341772544] - INFO - command executed successfully with root [2021-02-12 21:31:14,631] - [basetestcase] [140099341772544] - INFO - Couchbase stopped [2021-02-12 21:31:14,631] - [basetestcase] [140099341772544] - INFO - sleep for 5 secs. ... [2021-02-12 21:31:19,636] - [remote_util] [140099341772544] - INFO - running command.raw on 172.23.122.97: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-12 21:31:19,651] - [remote_util] [140099341772544] - INFO - command executed successfully with root [2021-02-12 21:31:19,651] - [remote_util] [140099341772544] - INFO - running command.raw on 172.23.122.97: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-12 21:31:19,703] - [remote_util] [140099341772544] - INFO - command executed successfully with root [2021-02-12 21:31:19,705] - [remote_util] [140099341772544] - INFO - SSH Connecting to 172.23.122.97 with username:root, attempt#1 of 5 [2021-02-12 21:31:19,806] - [remote_util] [140099341772544] - INFO - SSH Connected to 172.23.122.97 as root [2021-02-12 21:31:20,068] - [remote_util] [140099341772544] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:31:20,367] - [remote_util] [140099341772544] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:31:20,427] - [remote_util] [140099341772544] - INFO - 172.23.122.97 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 21:31:20,428] - [remote_util] [140099341772544] - INFO - Checking for process beam.smp on linux [2021-02-12 21:31:20,464] - [remote_util] [140099341772544] - INFO - Starting couchbase server [2021-02-12 21:31:20,561] - [remote_util] [140099341772544] - INFO - Running systemd command on this server [2021-02-12 21:31:20,561] - [remote_util] [140099341772544] - INFO - running command.raw on 172.23.122.97: systemctl start couchbase-server.service [2021-02-12 21:31:20,583] - [remote_util] [140099341772544] - INFO - command executed successfully with root [2021-02-12 21:31:20,583] - [remote_util] [140099341772544] - INFO - 172.23.122.97:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-12 21:31:25,589] - [remote_util] [140099341772544] - INFO - running command.raw on 172.23.122.97: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-12 21:31:25,630] - [remote_util] [140099341772544] - INFO - command executed successfully with root [2021-02-12 21:31:25,630] - [remote_util] [140099341772544] - INFO - Couchbase server status: [' Process: 47049 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-12 21:31:25,630] - [remote_util] [140099341772544] - INFO - Checking for process beam.smp on linux [2021-02-12 21:31:25,702] - [remote_util] [140099341772544] - INFO - process beam.smp is running on 172.23.122.97: with pid 47194 [2021-02-12 21:31:25,702] - [basetestcase] [140099341772544] - INFO - Couchbase started [2021-02-12 21:31:25,704] - [remote_util] [140099341772544] - INFO - SSH Connecting to 172.23.122.82 with username:root, attempt#1 of 5 [2021-02-12 21:31:25,818] - [remote_util] [140099341772544] - INFO - SSH Connected to 172.23.122.82 as root [2021-02-12 21:31:26,116] - [remote_util] [140099341772544] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:31:26,446] - [remote_util] [140099341772544] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:31:26,456] - [remote_util] [140099341772544] - INFO - SSH Connecting to 172.23.122.82 with username:root, attempt#1 of 5 [2021-02-12 21:31:26,561] - [remote_util] [140099341772544] - INFO - SSH Connected to 172.23.122.82 as root [2021-02-12 21:31:26,868] - [remote_util] [140099341772544] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:31:27,190] - [remote_util] [140099341772544] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:31:27,212] - [remote_util] [140099341772544] - INFO - 172.23.122.82 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 21:31:27,278] - [remote_util] [140099341772544] - INFO - Running systemd command on this server [2021-02-12 21:31:27,278] - [remote_util] [140099341772544] - INFO - running command.raw on 172.23.122.82: systemctl stop couchbase-server.service [2021-02-12 21:31:29,154] - [remote_util] [140099341772544] - INFO - command executed successfully with root [2021-02-12 21:31:29,155] - [basetestcase] [140099341772544] - INFO - Couchbase stopped [2021-02-12 21:31:29,155] - [basetestcase] [140099341772544] - INFO - sleep for 5 secs. ... [2021-02-12 21:31:34,158] - [remote_util] [140099341772544] - INFO - running command.raw on 172.23.122.82: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-12 21:31:34,177] - [remote_util] [140099341772544] - INFO - command executed successfully with root [2021-02-12 21:31:34,177] - [remote_util] [140099341772544] - INFO - running command.raw on 172.23.122.82: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-12 21:31:34,233] - [remote_util] [140099341772544] - INFO - command executed successfully with root [2021-02-12 21:31:34,235] - [remote_util] [140099341772544] - INFO - SSH Connecting to 172.23.122.82 with username:root, attempt#1 of 5 [2021-02-12 21:31:34,346] - [remote_util] [140099341772544] - INFO - SSH Connected to 172.23.122.82 as root [2021-02-12 21:31:34,663] - [remote_util] [140099341772544] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:31:34,983] - [remote_util] [140099341772544] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:31:35,007] - [remote_util] [140099341772544] - INFO - 172.23.122.82 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 21:31:35,007] - [remote_util] [140099341772544] - INFO - Checking for process beam.smp on linux [2021-02-12 21:31:35,054] - [remote_util] [140099341772544] - INFO - Starting couchbase server [2021-02-12 21:31:35,164] - [remote_util] [140099341772544] - INFO - Running systemd command on this server [2021-02-12 21:31:35,164] - [remote_util] [140099341772544] - INFO - running command.raw on 172.23.122.82: systemctl start couchbase-server.service [2021-02-12 21:31:35,190] - [remote_util] [140099341772544] - INFO - command executed successfully with root [2021-02-12 21:31:35,190] - [remote_util] [140099341772544] - INFO - 172.23.122.82:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-12 21:31:40,196] - [remote_util] [140099341772544] - INFO - running command.raw on 172.23.122.82: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-12 21:31:40,228] - [remote_util] [140099341772544] - INFO - command executed successfully with root [2021-02-12 21:31:40,228] - [remote_util] [140099341772544] - INFO - Couchbase server status: [' Process: 38944 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-12 21:31:40,228] - [remote_util] [140099341772544] - INFO - Checking for process beam.smp on linux [2021-02-12 21:31:40,304] - [remote_util] [140099341772544] - INFO - process beam.smp is running on 172.23.122.82: with pid 39088 [2021-02-12 21:31:40,304] - [basetestcase] [140099341772544] - INFO - Couchbase started [2021-02-12 21:31:40,306] - [remote_util] [140099341772544] - INFO - SSH Connecting to 172.23.122.118 with username:root, attempt#1 of 5 [2021-02-12 21:31:40,408] - [remote_util] [140099341772544] - INFO - SSH Connected to 172.23.122.118 as root [2021-02-12 21:31:40,657] - [remote_util] [140099341772544] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:31:40,949] - [remote_util] [140099341772544] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:31:40,957] - [remote_util] [140099341772544] - INFO - SSH Connecting to 172.23.122.118 with username:root, attempt#1 of 5 [2021-02-12 21:31:41,053] - [remote_util] [140099341772544] - INFO - SSH Connected to 172.23.122.118 as root [2021-02-12 21:31:41,292] - [remote_util] [140099341772544] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:31:41,584] - [remote_util] [140099341772544] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:31:41,642] - [remote_util] [140099341772544] - INFO - 172.23.122.118 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 21:31:41,695] - [remote_util] [140099341772544] - INFO - Running systemd command on this server [2021-02-12 21:31:41,695] - [remote_util] [140099341772544] - INFO - running command.raw on 172.23.122.118: systemctl stop couchbase-server.service [2021-02-12 21:31:43,517] - [remote_util] [140099341772544] - INFO - command executed successfully with root [2021-02-12 21:31:43,517] - [basetestcase] [140099341772544] - INFO - Couchbase stopped [2021-02-12 21:31:43,517] - [basetestcase] [140099341772544] - INFO - sleep for 5 secs. ... [2021-02-12 21:31:48,522] - [remote_util] [140099341772544] - INFO - running command.raw on 172.23.122.118: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-12 21:31:48,537] - [remote_util] [140099341772544] - INFO - command executed successfully with root [2021-02-12 21:31:48,538] - [remote_util] [140099341772544] - INFO - running command.raw on 172.23.122.118: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-12 21:31:48,591] - [remote_util] [140099341772544] - INFO - command executed successfully with root [2021-02-12 21:31:48,593] - [remote_util] [140099341772544] - INFO - SSH Connecting to 172.23.122.118 with username:root, attempt#1 of 5 [2021-02-12 21:31:48,692] - [remote_util] [140099341772544] - INFO - SSH Connected to 172.23.122.118 as root [2021-02-12 21:31:48,939] - [remote_util] [140099341772544] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:31:49,230] - [remote_util] [140099341772544] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:31:49,249] - [remote_util] [140099341772544] - INFO - 172.23.122.118 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 21:31:49,249] - [remote_util] [140099341772544] - INFO - Checking for process beam.smp on linux [2021-02-12 21:31:49,284] - [remote_util] [140099341772544] - INFO - Starting couchbase server [2021-02-12 21:31:49,376] - [remote_util] [140099341772544] - INFO - Running systemd command on this server [2021-02-12 21:31:49,376] - [remote_util] [140099341772544] - INFO - running command.raw on 172.23.122.118: systemctl start couchbase-server.service [2021-02-12 21:31:49,398] - [remote_util] [140099341772544] - INFO - command executed successfully with root [2021-02-12 21:31:49,398] - [remote_util] [140099341772544] - INFO - 172.23.122.118:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-12 21:31:54,403] - [remote_util] [140099341772544] - INFO - running command.raw on 172.23.122.118: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-12 21:31:54,426] - [remote_util] [140099341772544] - INFO - command executed successfully with root [2021-02-12 21:31:54,426] - [remote_util] [140099341772544] - INFO - Couchbase server status: [' Process: 79684 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-12 21:31:54,427] - [remote_util] [140099341772544] - INFO - Checking for process beam.smp on linux [2021-02-12 21:31:54,494] - [remote_util] [140099341772544] - INFO - process beam.smp is running on 172.23.122.118: with pid 79860 [2021-02-12 21:31:54,494] - [basetestcase] [140099341772544] - INFO - Couchbase started [2021-02-12 21:31:54,496] - [remote_util] [140099341772544] - INFO - SSH Connecting to 172.23.122.116 with username:root, attempt#1 of 5 [2021-02-12 21:31:54,599] - [remote_util] [140099341772544] - INFO - SSH Connected to 172.23.122.116 as root [2021-02-12 21:31:54,884] - [remote_util] [140099341772544] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:31:55,179] - [remote_util] [140099341772544] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:31:55,187] - [remote_util] [140099341772544] - INFO - SSH Connecting to 172.23.122.116 with username:root, attempt#1 of 5 [2021-02-12 21:31:55,285] - [remote_util] [140099341772544] - INFO - SSH Connected to 172.23.122.116 as root [2021-02-12 21:31:55,554] - [remote_util] [140099341772544] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:31:55,853] - [remote_util] [140099341772544] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:31:55,910] - [remote_util] [140099341772544] - INFO - 172.23.122.116 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 21:31:55,962] - [remote_util] [140099341772544] - INFO - Running systemd command on this server [2021-02-12 21:31:55,962] - [remote_util] [140099341772544] - INFO - running command.raw on 172.23.122.116: systemctl stop couchbase-server.service [2021-02-12 21:31:57,635] - [remote_util] [140099341772544] - INFO - command executed successfully with root [2021-02-12 21:31:57,636] - [basetestcase] [140099341772544] - INFO - Couchbase stopped [2021-02-12 21:31:57,636] - [basetestcase] [140099341772544] - INFO - sleep for 5 secs. ... [2021-02-12 21:32:02,637] - [remote_util] [140099341772544] - INFO - running command.raw on 172.23.122.116: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-12 21:32:02,652] - [remote_util] [140099341772544] - INFO - command executed successfully with root [2021-02-12 21:32:02,652] - [remote_util] [140099341772544] - INFO - running command.raw on 172.23.122.116: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-12 21:32:02,705] - [remote_util] [140099341772544] - INFO - command executed successfully with root [2021-02-12 21:32:02,707] - [remote_util] [140099341772544] - INFO - SSH Connecting to 172.23.122.116 with username:root, attempt#1 of 5 [2021-02-12 21:32:02,809] - [remote_util] [140099341772544] - INFO - SSH Connected to 172.23.122.116 as root [2021-02-12 21:32:03,088] - [remote_util] [140099341772544] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:32:03,384] - [remote_util] [140099341772544] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:32:03,443] - [remote_util] [140099341772544] - INFO - 172.23.122.116 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 21:32:03,444] - [remote_util] [140099341772544] - INFO - Checking for process beam.smp on linux [2021-02-12 21:32:03,481] - [remote_util] [140099341772544] - INFO - Starting couchbase server [2021-02-12 21:32:03,574] - [remote_util] [140099341772544] - INFO - Running systemd command on this server [2021-02-12 21:32:03,574] - [remote_util] [140099341772544] - INFO - running command.raw on 172.23.122.116: systemctl start couchbase-server.service [2021-02-12 21:32:03,596] - [remote_util] [140099341772544] - INFO - command executed successfully with root [2021-02-12 21:32:03,596] - [remote_util] [140099341772544] - INFO - 172.23.122.116:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-12 21:32:08,601] - [remote_util] [140099341772544] - INFO - running command.raw on 172.23.122.116: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-12 21:32:08,623] - [remote_util] [140099341772544] - INFO - command executed successfully with root [2021-02-12 21:32:08,623] - [remote_util] [140099341772544] - INFO - Couchbase server status: [' Process: 43894 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-12 21:32:08,624] - [remote_util] [140099341772544] - INFO - Checking for process beam.smp on linux [2021-02-12 21:32:08,689] - [remote_util] [140099341772544] - INFO - process beam.smp is running on 172.23.122.116: with pid 44068 [2021-02-12 21:32:08,689] - [basetestcase] [140099341772544] - INFO - Couchbase started [2021-02-12 21:32:08,690] - [basetestcase] [140099341772544] - INFO - sleep for 10 secs. ... [2021-02-12 21:32:19,220] - [task] [140098737792768] - INFO - server: ip:172.23.123.61 port:8091 ssh_username:root, nodes/self [2021-02-12 21:32:19,223] - [task] [140098737792768] - INFO - {'uptime': '110', 'memoryTotal': 4201672704, 'memoryFree': 3512049664, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.123.61:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4454-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.123.61', 'ip': '172.23.123.61', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} [2021-02-12 21:32:19,225] - [rest_client] [140098737792768] - ERROR - GET http://172.23.123.61:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-02-12 21:32:19,225] - [rest_client] [140098737792768] - INFO - pools/default params : memoryQuota=2147 [2021-02-12 21:32:19,232] - [rest_client] [140098737792768] - INFO - --> init_node_services(Administrator,password,172.23.123.61,8091,['kv']) [2021-02-12 21:32:19,232] - [rest_client] [140098737792768] - INFO - /node/controller/setupServices params on 172.23.123.61: 8091:hostname=172.23.123.61&user=Administrator&password=password&services=kv [2021-02-12 21:32:19,263] - [rest_client] [140098737792768] - INFO - --> in init_cluster...Administrator,password,8091 [2021-02-12 21:32:19,263] - [rest_client] [140098737792768] - INFO - settings/web params on 172.23.123.61:8091:port=8091&username=Administrator&password=password [2021-02-12 21:32:19,322] - [rest_client] [140098737792768] - INFO - --> status:True [2021-02-12 21:32:19,323] - [remote_util] [140098737792768] - INFO - SSH Connecting to 172.23.123.61 with username:root, attempt#1 of 5 [2021-02-12 21:32:19,434] - [remote_util] [140098737792768] - INFO - SSH Connected to 172.23.123.61 as root [2021-02-12 21:32:19,720] - [remote_util] [140098737792768] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:32:20,020] - [remote_util] [140098737792768] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:32:20,028] - [remote_util] [140098737792768] - INFO - running command.raw on 172.23.123.61: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-02-12 21:32:20,085] - [remote_util] [140098737792768] - INFO - command executed successfully with root [2021-02-12 21:32:20,085] - [remote_util] [140098737792768] - INFO - ['ok'] [2021-02-12 21:32:20,087] - [rest_client] [140098737792768] - INFO - /diag/eval status on 172.23.123.61:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-12 21:32:20,090] - [rest_client] [140098737792768] - INFO - /diag/eval status on 172.23.123.61:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-12 21:32:20,094] - [rest_client] [140098737792768] - INFO - settings/indexes params : storageMode=plasma [2021-02-12 21:32:20,113] - [task] [140098737792768] - INFO - server: ip:172.23.123.62 port:8091 ssh_username:root, nodes/self [2021-02-12 21:32:20,116] - [task] [140098737792768] - INFO - {'uptime': '99', 'memoryTotal': 4201676800, 'memoryFree': 3476553728, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.123.62:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4454-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.123.62', 'ip': '172.23.123.62', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} [2021-02-12 21:32:20,118] - [rest_client] [140098737792768] - ERROR - GET http://172.23.123.62:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-02-12 21:32:20,118] - [rest_client] [140098737792768] - INFO - pools/default params : memoryQuota=2147 [2021-02-12 21:32:20,125] - [rest_client] [140098737792768] - INFO - --> in init_cluster...Administrator,password,8091 [2021-02-12 21:32:20,125] - [rest_client] [140098737792768] - INFO - settings/web params on 172.23.123.62:8091:port=8091&username=Administrator&password=password [2021-02-12 21:32:20,179] - [rest_client] [140098737792768] - INFO - --> status:True [2021-02-12 21:32:20,181] - [remote_util] [140098737792768] - INFO - SSH Connecting to 172.23.123.62 with username:root, attempt#1 of 5 [2021-02-12 21:32:20,287] - [remote_util] [140098737792768] - INFO - SSH Connected to 172.23.123.62 as root [2021-02-12 21:32:20,559] - [remote_util] [140098737792768] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:32:20,857] - [remote_util] [140098737792768] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:32:20,865] - [remote_util] [140098737792768] - INFO - running command.raw on 172.23.123.62: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-02-12 21:32:20,929] - [remote_util] [140098737792768] - INFO - command executed successfully with root [2021-02-12 21:32:20,929] - [remote_util] [140098737792768] - INFO - ['ok'] [2021-02-12 21:32:20,931] - [rest_client] [140098737792768] - INFO - /diag/eval status on 172.23.123.62:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-12 21:32:20,933] - [rest_client] [140098737792768] - INFO - /diag/eval status on 172.23.123.62:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-12 21:32:20,936] - [rest_client] [140098737792768] - INFO - settings/indexes params : storageMode=plasma [2021-02-12 21:32:20,957] - [task] [140098737792768] - INFO - server: ip:172.23.122.90 port:8091 ssh_username:root, nodes/self [2021-02-12 21:32:20,960] - [task] [140098737792768] - INFO - {'uptime': '84', 'memoryTotal': 4201684992, 'memoryFree': 3562430464, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.122.90:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4454-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.122.90', 'ip': '172.23.122.90', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} [2021-02-12 21:32:20,962] - [rest_client] [140098737792768] - ERROR - GET http://172.23.122.90:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-02-12 21:32:20,962] - [rest_client] [140098737792768] - INFO - pools/default params : memoryQuota=2147 [2021-02-12 21:32:20,967] - [rest_client] [140098737792768] - INFO - --> in init_cluster...Administrator,password,8091 [2021-02-12 21:32:20,967] - [rest_client] [140098737792768] - INFO - settings/web params on 172.23.122.90:8091:port=8091&username=Administrator&password=password [2021-02-12 21:32:21,024] - [rest_client] [140098737792768] - INFO - --> status:True [2021-02-12 21:32:21,026] - [remote_util] [140098737792768] - INFO - SSH Connecting to 172.23.122.90 with username:root, attempt#1 of 5 [2021-02-12 21:32:21,128] - [remote_util] [140098737792768] - INFO - SSH Connected to 172.23.122.90 as root [2021-02-12 21:32:21,422] - [remote_util] [140098737792768] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:32:21,719] - [remote_util] [140098737792768] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:32:21,727] - [remote_util] [140098737792768] - INFO - running command.raw on 172.23.122.90: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-02-12 21:32:21,785] - [remote_util] [140098737792768] - INFO - command executed successfully with root [2021-02-12 21:32:21,785] - [remote_util] [140098737792768] - INFO - ['ok'] [2021-02-12 21:32:21,788] - [rest_client] [140098737792768] - INFO - /diag/eval status on 172.23.122.90:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-12 21:32:21,789] - [rest_client] [140098737792768] - INFO - /diag/eval status on 172.23.122.90:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-12 21:32:21,793] - [rest_client] [140098737792768] - INFO - settings/indexes params : storageMode=plasma [2021-02-12 21:32:21,816] - [task] [140098737792768] - INFO - server: ip:172.23.122.117 port:8091 ssh_username:root, nodes/self [2021-02-12 21:32:21,818] - [task] [140098737792768] - INFO - {'uptime': '70', 'memoryTotal': 4201684992, 'memoryFree': 3539292160, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.122.117:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4454-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.122.117', 'ip': '172.23.122.117', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} [2021-02-12 21:32:21,820] - [rest_client] [140098737792768] - ERROR - GET http://172.23.122.117:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-02-12 21:32:21,820] - [rest_client] [140098737792768] - INFO - pools/default params : memoryQuota=2147 [2021-02-12 21:32:21,826] - [rest_client] [140098737792768] - INFO - --> in init_cluster...Administrator,password,8091 [2021-02-12 21:32:21,826] - [rest_client] [140098737792768] - INFO - settings/web params on 172.23.122.117:8091:port=8091&username=Administrator&password=password [2021-02-12 21:32:21,880] - [rest_client] [140098737792768] - INFO - --> status:True [2021-02-12 21:32:21,881] - [remote_util] [140098737792768] - INFO - SSH Connecting to 172.23.122.117 with username:root, attempt#1 of 5 [2021-02-12 21:32:21,981] - [remote_util] [140098737792768] - INFO - SSH Connected to 172.23.122.117 as root [2021-02-12 21:32:22,232] - [remote_util] [140098737792768] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:32:22,524] - [remote_util] [140098737792768] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:32:22,532] - [remote_util] [140098737792768] - INFO - running command.raw on 172.23.122.117: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-02-12 21:32:22,587] - [remote_util] [140098737792768] - INFO - command executed successfully with root [2021-02-12 21:32:22,587] - [remote_util] [140098737792768] - INFO - ['ok'] [2021-02-12 21:32:22,589] - [rest_client] [140098737792768] - INFO - /diag/eval status on 172.23.122.117:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-12 21:32:22,591] - [rest_client] [140098737792768] - INFO - /diag/eval status on 172.23.122.117:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-12 21:32:22,594] - [rest_client] [140098737792768] - INFO - settings/indexes params : storageMode=plasma [2021-02-12 21:32:22,615] - [task] [140098737792768] - INFO - server: ip:172.23.122.97 port:8091 ssh_username:root, nodes/self [2021-02-12 21:32:22,618] - [task] [140098737792768] - INFO - {'uptime': '55', 'memoryTotal': 4201684992, 'memoryFree': 3553415168, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.122.97:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4454-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.122.97', 'ip': '172.23.122.97', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} [2021-02-12 21:32:22,620] - [rest_client] [140098737792768] - ERROR - GET http://172.23.122.97:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-02-12 21:32:22,620] - [rest_client] [140098737792768] - INFO - pools/default params : memoryQuota=2147 [2021-02-12 21:32:22,626] - [rest_client] [140098737792768] - INFO - --> in init_cluster...Administrator,password,8091 [2021-02-12 21:32:22,626] - [rest_client] [140098737792768] - INFO - settings/web params on 172.23.122.97:8091:port=8091&username=Administrator&password=password [2021-02-12 21:32:22,683] - [rest_client] [140098737792768] - INFO - --> status:True [2021-02-12 21:32:22,684] - [remote_util] [140098737792768] - INFO - SSH Connecting to 172.23.122.97 with username:root, attempt#1 of 5 [2021-02-12 21:32:22,796] - [remote_util] [140098737792768] - INFO - SSH Connected to 172.23.122.97 as root [2021-02-12 21:32:23,066] - [remote_util] [140098737792768] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:32:23,373] - [remote_util] [140098737792768] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:32:23,381] - [remote_util] [140098737792768] - INFO - running command.raw on 172.23.122.97: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-02-12 21:32:23,440] - [remote_util] [140098737792768] - INFO - command executed successfully with root [2021-02-12 21:32:23,440] - [remote_util] [140098737792768] - INFO - ['ok'] [2021-02-12 21:32:23,442] - [rest_client] [140098737792768] - INFO - /diag/eval status on 172.23.122.97:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-12 21:32:23,444] - [rest_client] [140098737792768] - INFO - /diag/eval status on 172.23.122.97:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-12 21:32:23,448] - [rest_client] [140098737792768] - INFO - settings/indexes params : storageMode=plasma [2021-02-12 21:32:23,468] - [task] [140098737792768] - INFO - server: ip:172.23.122.82 port:8091 ssh_username:root, nodes/self [2021-02-12 21:32:23,471] - [task] [140098737792768] - INFO - {'uptime': '45', 'memoryTotal': 4201684992, 'memoryFree': 3588718592, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.122.82:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4454-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.122.82', 'ip': '172.23.122.82', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} [2021-02-12 21:32:23,472] - [rest_client] [140098737792768] - ERROR - GET http://172.23.122.82:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-02-12 21:32:23,473] - [rest_client] [140098737792768] - INFO - pools/default params : memoryQuota=2147 [2021-02-12 21:32:23,478] - [rest_client] [140098737792768] - INFO - --> in init_cluster...Administrator,password,8091 [2021-02-12 21:32:23,478] - [rest_client] [140098737792768] - INFO - settings/web params on 172.23.122.82:8091:port=8091&username=Administrator&password=password [2021-02-12 21:32:23,535] - [rest_client] [140098737792768] - INFO - --> status:True [2021-02-12 21:32:23,536] - [remote_util] [140098737792768] - INFO - SSH Connecting to 172.23.122.82 with username:root, attempt#1 of 5 [2021-02-12 21:32:23,641] - [remote_util] [140098737792768] - INFO - SSH Connected to 172.23.122.82 as root [2021-02-12 21:32:23,919] - [remote_util] [140098737792768] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:32:24,222] - [remote_util] [140098737792768] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:32:24,230] - [remote_util] [140098737792768] - INFO - running command.raw on 172.23.122.82: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-02-12 21:32:24,285] - [remote_util] [140098737792768] - INFO - command executed successfully with root [2021-02-12 21:32:24,285] - [remote_util] [140098737792768] - INFO - ['ok'] [2021-02-12 21:32:24,287] - [rest_client] [140098737792768] - INFO - /diag/eval status on 172.23.122.82:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-12 21:32:24,289] - [rest_client] [140098737792768] - INFO - /diag/eval status on 172.23.122.82:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-12 21:32:24,294] - [rest_client] [140098737792768] - INFO - settings/indexes params : storageMode=plasma [2021-02-12 21:32:24,312] - [task] [140098737792768] - INFO - server: ip:172.23.122.118 port:8091 ssh_username:root, nodes/self [2021-02-12 21:32:24,315] - [task] [140098737792768] - INFO - {'uptime': '29', 'memoryTotal': 4201689088, 'memoryFree': 3549962240, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.122.118:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4454-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.122.118', 'ip': '172.23.122.118', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} [2021-02-12 21:32:24,317] - [rest_client] [140098737792768] - ERROR - GET http://172.23.122.118:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-02-12 21:32:24,317] - [rest_client] [140098737792768] - INFO - pools/default params : memoryQuota=2147 [2021-02-12 21:32:24,323] - [rest_client] [140098737792768] - INFO - --> in init_cluster...Administrator,password,8091 [2021-02-12 21:32:24,323] - [rest_client] [140098737792768] - INFO - settings/web params on 172.23.122.118:8091:port=8091&username=Administrator&password=password [2021-02-12 21:32:24,378] - [rest_client] [140098737792768] - INFO - --> status:True [2021-02-12 21:32:24,379] - [remote_util] [140098737792768] - INFO - SSH Connecting to 172.23.122.118 with username:root, attempt#1 of 5 [2021-02-12 21:32:24,478] - [remote_util] [140098737792768] - INFO - SSH Connected to 172.23.122.118 as root [2021-02-12 21:32:24,723] - [remote_util] [140098737792768] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:32:25,010] - [remote_util] [140098737792768] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:32:25,017] - [remote_util] [140098737792768] - INFO - running command.raw on 172.23.122.118: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-02-12 21:32:25,072] - [remote_util] [140098737792768] - INFO - command executed successfully with root [2021-02-12 21:32:25,072] - [remote_util] [140098737792768] - INFO - ['ok'] [2021-02-12 21:32:25,075] - [rest_client] [140098737792768] - INFO - /diag/eval status on 172.23.122.118:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-12 21:32:25,076] - [rest_client] [140098737792768] - INFO - /diag/eval status on 172.23.122.118:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-12 21:32:25,082] - [rest_client] [140098737792768] - INFO - settings/indexes params : storageMode=plasma [2021-02-12 21:32:25,100] - [task] [140098737792768] - INFO - server: ip:172.23.122.116 port:8091 ssh_username:root, nodes/self [2021-02-12 21:32:25,103] - [task] [140098737792768] - INFO - {'uptime': '19', 'memoryTotal': 4201689088, 'memoryFree': 3562549248, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.122.116:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4454-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@cb.local', 'ip': '172.23.122.116', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} [2021-02-12 21:32:25,105] - [rest_client] [140098737792768] - ERROR - GET http://172.23.122.116:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-02-12 21:32:25,105] - [rest_client] [140098737792768] - INFO - pools/default params : memoryQuota=2147 [2021-02-12 21:32:25,110] - [rest_client] [140098737792768] - INFO - --> in init_cluster...Administrator,password,8091 [2021-02-12 21:32:25,110] - [rest_client] [140098737792768] - INFO - settings/web params on 172.23.122.116:8091:port=8091&username=Administrator&password=password [2021-02-12 21:32:25,168] - [rest_client] [140098737792768] - INFO - --> status:True [2021-02-12 21:32:25,169] - [remote_util] [140098737792768] - INFO - SSH Connecting to 172.23.122.116 with username:root, attempt#1 of 5 [2021-02-12 21:32:25,272] - [remote_util] [140098737792768] - INFO - SSH Connected to 172.23.122.116 as root [2021-02-12 21:32:25,541] - [remote_util] [140098737792768] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:32:25,831] - [remote_util] [140098737792768] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:32:25,838] - [remote_util] [140098737792768] - INFO - running command.raw on 172.23.122.116: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-02-12 21:32:25,893] - [remote_util] [140098737792768] - INFO - command executed successfully with root [2021-02-12 21:32:25,893] - [remote_util] [140098737792768] - INFO - ['ok'] [2021-02-12 21:32:25,895] - [rest_client] [140098737792768] - INFO - /diag/eval status on 172.23.122.116:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-12 21:32:25,897] - [rest_client] [140098737792768] - INFO - /diag/eval status on 172.23.122.116:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-12 21:32:25,901] - [rest_client] [140098737792768] - INFO - settings/indexes params : storageMode=plasma [2021-02-12 21:32:25,923] - [basetestcase] [140099341772544] - INFO - **** add built-in 'cbadminbucket' user to node 172.23.123.61 **** [2021-02-12 21:32:25,932] - [rest_client] [140099341772544] - ERROR - DELETE http://172.23.123.61:8091/settings/rbac/users/local/cbadminbucket body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:password [2021-02-12 21:32:25,933] - [internal_user] [140099341772544] - INFO - Exception while deleting user. Exception is -b'"User was not found."' [2021-02-12 21:32:25,994] - [basetestcase] [140099341772544] - INFO - sleep for 5 secs. ... [2021-02-12 21:32:30,999] - [basetestcase] [140099341772544] - INFO - **** add 'admin' role to 'cbadminbucket' user **** [2021-02-12 21:32:31,010] - [basetestcase] [140099341772544] - INFO - done initializing cluster [2021-02-12 21:32:31,920] - [task] [140098737792768] - INFO - adding node 172.23.123.62:8091 to cluster [2021-02-12 21:32:31,921] - [rest_client] [140098737792768] - INFO - adding remote node @172.23.123.62:8091 to this cluster @172.23.123.61:8091 [2021-02-12 21:32:34,277] - [task] [140098737792768] - INFO - adding node 172.23.122.90:8091 to cluster [2021-02-12 21:32:34,278] - [rest_client] [140098737792768] - INFO - adding remote node @172.23.122.90:8091 to this cluster @172.23.123.61:8091 [2021-02-12 21:32:36,570] - [task] [140098737792768] - INFO - adding node 172.23.122.117:8091 to cluster [2021-02-12 21:32:36,570] - [rest_client] [140098737792768] - INFO - adding remote node @172.23.122.117:8091 to this cluster @172.23.123.61:8091 [2021-02-12 21:32:38,746] - [task] [140098737792768] - INFO - adding node 172.23.122.97:8091 to cluster [2021-02-12 21:32:38,746] - [rest_client] [140098737792768] - INFO - adding remote node @172.23.122.97:8091 to this cluster @172.23.123.61:8091 [2021-02-12 21:32:41,148] - [task] [140098737792768] - INFO - adding node 172.23.122.82:8091 to cluster [2021-02-12 21:32:41,148] - [rest_client] [140098737792768] - INFO - adding remote node @172.23.122.82:8091 to this cluster @172.23.123.61:8091 [2021-02-12 21:32:43,574] - [task] [140098737792768] - INFO - adding node 172.23.122.118:8091 to cluster [2021-02-12 21:32:43,574] - [rest_client] [140098737792768] - INFO - adding remote node @172.23.122.118:8091 to this cluster @172.23.123.61:8091 [2021-02-12 21:32:45,843] - [rest_client] [140098737792768] - INFO - rebalance params : {'knownNodes': 'ns_1@172.23.122.117,ns_1@172.23.122.118,ns_1@172.23.122.82,ns_1@172.23.122.90,ns_1@172.23.122.97,ns_1@172.23.123.61,ns_1@172.23.123.62', 'ejectedNodes': '', 'user': 'Administrator', 'password': 'password'} [2021-02-12 21:32:45,871] - [rest_client] [140098737792768] - INFO - rebalance operation started [2021-02-12 21:32:45,873] - [rest_client] [140098737792768] - INFO - rebalance percentage : 0.00 % [2021-02-12 21:32:45,873] - [task] [140098737792768] - INFO - Rebalance - status: running, progress: 0.00% [2021-02-12 21:32:55,898] - [task] [140098737792768] - INFO - Rebalance - status: none, progress: 100.00% [2021-02-12 21:32:55,907] - [task] [140098737792768] - INFO - rebalancing was completed with progress: 100% in 10.035835266113281 sec [2021-02-12 21:32:55,908] - [remote_util] [140099341772544] - INFO - SSH Connecting to 172.23.123.61 with username:root, attempt#1 of 5 [2021-02-12 21:32:56,017] - [remote_util] [140099341772544] - INFO - SSH Connected to 172.23.123.61 as root [2021-02-12 21:32:56,313] - [remote_util] [140099341772544] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:32:56,628] - [remote_util] [140099341772544] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:32:56,636] - [remote_util] [140099341772544] - INFO - running command.raw on 172.23.123.61: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-02-12 21:32:56,699] - [remote_util] [140099341772544] - INFO - command executed successfully with root [2021-02-12 21:32:56,699] - [remote_util] [140099341772544] - INFO - ['ok'] [2021-02-12 21:32:56,699] - [basetestcase] [140099341772544] - INFO - Enabled diag/eval for non-local hosts from 172.23.123.61 [2021-02-12 21:32:56,700] - [basetestcase] [140099341772544] - INFO - ============== basetestcase setup was finished for test #5 test_kv_rebalance_out_when_existing_eventing_node_is_processing_mutations ============== [2021-02-12 21:32:56,708] - [remote_util] [140099341772544] - INFO - SSH Connecting to 172.23.123.61 with username:root, attempt#1 of 5 [2021-02-12 21:32:56,812] - [remote_util] [140099341772544] - INFO - SSH Connected to 172.23.123.61 as root [2021-02-12 21:32:57,081] - [remote_util] [140099341772544] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:32:57,383] - [remote_util] [140099341772544] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:32:57,384] - [remote_util] [140099341772544] - INFO - SSH Connecting to 172.23.123.62 with username:root, attempt#1 of 5 [2021-02-12 21:32:57,490] - [remote_util] [140099341772544] - INFO - SSH Connected to 172.23.123.62 as root [2021-02-12 21:32:57,780] - [remote_util] [140099341772544] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:32:58,092] - [remote_util] [140099341772544] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:32:58,094] - [remote_util] [140099341772544] - INFO - SSH Connecting to 172.23.122.90 with username:root, attempt#1 of 5 [2021-02-12 21:32:58,199] - [remote_util] [140099341772544] - INFO - SSH Connected to 172.23.122.90 as root [2021-02-12 21:32:58,491] - [remote_util] [140099341772544] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:32:58,788] - [remote_util] [140099341772544] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:32:58,790] - [remote_util] [140099341772544] - INFO - SSH Connecting to 172.23.122.117 with username:root, attempt#1 of 5 [2021-02-12 21:32:58,891] - [remote_util] [140099341772544] - INFO - SSH Connected to 172.23.122.117 as root [2021-02-12 21:32:59,139] - [remote_util] [140099341772544] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:32:59,426] - [remote_util] [140099341772544] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:32:59,429] - [remote_util] [140099341772544] - INFO - SSH Connecting to 172.23.122.97 with username:root, attempt#1 of 5 [2021-02-12 21:32:59,534] - [remote_util] [140099341772544] - INFO - SSH Connected to 172.23.122.97 as root [2021-02-12 21:32:59,806] - [remote_util] [140099341772544] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:33:00,108] - [remote_util] [140099341772544] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:33:00,110] - [remote_util] [140099341772544] - INFO - SSH Connecting to 172.23.122.82 with username:root, attempt#1 of 5 [2021-02-12 21:33:00,216] - [remote_util] [140099341772544] - INFO - SSH Connected to 172.23.122.82 as root [2021-02-12 21:33:00,492] - [remote_util] [140099341772544] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:33:00,796] - [remote_util] [140099341772544] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:33:00,798] - [remote_util] [140099341772544] - INFO - SSH Connecting to 172.23.122.118 with username:root, attempt#1 of 5 [2021-02-12 21:33:00,897] - [remote_util] [140099341772544] - INFO - SSH Connected to 172.23.122.118 as root [2021-02-12 21:33:01,139] - [remote_util] [140099341772544] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:33:01,429] - [remote_util] [140099341772544] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:33:01,431] - [remote_util] [140099341772544] - INFO - SSH Connecting to 172.23.122.116 with username:root, attempt#1 of 5 [2021-02-12 21:33:01,534] - [remote_util] [140099341772544] - INFO - SSH Connected to 172.23.122.116 as root [2021-02-12 21:33:01,815] - [remote_util] [140099341772544] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:33:02,109] - [remote_util] [140099341772544] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:33:27,594] - [basetestcase] [140099341772544] - INFO - ------- Cluster statistics ------- [2021-02-12 21:33:27,594] - [basetestcase] [140099341772544] - INFO - 172.23.122.117:8091 => {'services': ['eventing'], 'cpu_utilization': 9.467306235799041, 'mem_free': 3497701376, 'mem_total': 4201684992, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-02-12 21:33:27,594] - [basetestcase] [140099341772544] - INFO - 172.23.122.118:8091 => {'services': ['index', 'n1ql'], 'cpu_utilization': 0.5781799899446959, 'mem_free': 3467833344, 'mem_total': 4201689088, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-02-12 21:33:27,594] - [basetestcase] [140099341772544] - INFO - 172.23.122.82:8091 => {'services': ['eventing'], 'cpu_utilization': 0.8040201005025126, 'mem_free': 3535798272, 'mem_total': 4201684992, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-02-12 21:33:27,594] - [basetestcase] [140099341772544] - INFO - 172.23.122.90:8091 => {'services': ['kv'], 'cpu_utilization': 11.99898528665652, 'mem_free': 3528253440, 'mem_total': 4201684992, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-02-12 21:33:27,594] - [basetestcase] [140099341772544] - INFO - 172.23.122.97:8091 => {'services': ['eventing'], 'cpu_utilization': 4.351106639839034, 'mem_free': 3510300672, 'mem_total': 4201684992, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-02-12 21:33:27,594] - [basetestcase] [140099341772544] - INFO - 172.23.123.61:8091 => {'services': ['kv'], 'cpu_utilization': 1.671656686626747, 'mem_free': 3466338304, 'mem_total': 4201672704, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-02-12 21:33:27,594] - [basetestcase] [140099341772544] - INFO - 172.23.123.62:8091 => {'services': ['kv'], 'cpu_utilization': 0.9038413256339443, 'mem_free': 3449044992, 'mem_total': 4201676800, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-02-12 21:33:27,594] - [basetestcase] [140099341772544] - INFO - --- End of cluster statistics --- [2021-02-12 21:33:27,596] - [remote_util] [140099341772544] - INFO - SSH Connecting to 172.23.123.61 with username:root, attempt#1 of 5 [2021-02-12 21:33:27,705] - [remote_util] [140099341772544] - INFO - SSH Connected to 172.23.123.61 as root [2021-02-12 21:33:27,989] - [remote_util] [140099341772544] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:33:28,299] - [remote_util] [140099341772544] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:33:31,095] - [basetestcase] [140099341772544] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.118 port:8091 ssh_username:root] [2021-02-12 21:33:31,102] - [basetestcase] [140099341772544] - INFO - list of eventing nodes in cluster: [ip:172.23.122.117 port:8091 ssh_username:root, ip:172.23.122.82 port:8091 ssh_username:root, ip:172.23.122.97 port:8091 ssh_username:root] [2021-02-12 21:33:31,109] - [rest_client] [140099341772544] - INFO - settings/indexes params : storageMode=plasma [2021-02-12 21:33:31,115] - [eventing_base] [140099341772544] - INFO - Setting the min possible memory quota so that adding mode nodes to the cluster wouldn't be a problem. [2021-02-12 21:33:31,116] - [rest_client] [140099341772544] - INFO - pools/default params : memoryQuota=330 [2021-02-12 21:33:31,122] - [rest_client] [140099341772544] - INFO - pools/default params : indexMemoryQuota=256 [2021-02-12 21:33:31,128] - [rest_client] [140099341772544] - INFO - pools/default params : eventingMemoryQuota=512 [2021-02-12 21:33:31,149] - [basetestcase] [140099341772544] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.118 port:8091 ssh_username:root] [2021-02-12 21:33:31,154] - [rest_client] [140099341772544] - INFO - pools/default params : memoryQuota=700 [2021-02-12 21:33:31,159] - [eventing_rebalance_collection] [140099341772544] - INFO - 200 [2021-02-12 21:33:31,728] - [rest_client] [140098737792768] - INFO - http://172.23.123.61:8091/pools/default/buckets with param: name=src_bucket&ramQuotaMB=200&authType=none&replicaNumber=0&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore [2021-02-12 21:33:31,757] - [rest_client] [140098737792768] - INFO - 0.03 seconds to create bucket src_bucket [2021-02-12 21:33:31,757] - [bucket_helper] [140098737792768] - INFO - waiting for memcached bucket : src_bucket in 172.23.123.61 to accept set ops [2021-02-12 21:33:32,416] - [data_helper] [140098737792768] - INFO - creating direct client 172.23.122.90:11210 src_bucket [2021-02-12 21:33:32,503] - [data_helper] [140098737792768] - INFO - creating direct client 172.23.123.61:11210 src_bucket [2021-02-12 21:33:32,584] - [data_helper] [140098737792768] - INFO - creating direct client 172.23.123.62:11210 src_bucket [2021-02-12 21:33:32,884] - [data_helper] [140098737792768] - INFO - creating direct client 172.23.122.90:11210 src_bucket [2021-02-12 21:33:32,963] - [data_helper] [140098737792768] - INFO - creating direct client 172.23.123.61:11210 src_bucket [2021-02-12 21:33:33,066] - [data_helper] [140098737792768] - INFO - creating direct client 172.23.123.62:11210 src_bucket [2021-02-12 21:33:33,144] - [task] [140098737792768] - WARNING - vbucket map not ready after try 0 [2021-02-12 21:33:33,144] - [bucket_helper] [140098737792768] - INFO - waiting for memcached bucket : src_bucket in 172.23.123.61 to accept set ops [2021-02-12 21:33:33,267] - [data_helper] [140098737792768] - INFO - creating direct client 172.23.122.90:11210 src_bucket [2021-02-12 21:33:33,514] - [data_helper] [140098737792768] - INFO - creating direct client 172.23.123.61:11210 src_bucket [2021-02-12 21:33:33,595] - [data_helper] [140098737792768] - INFO - creating direct client 172.23.123.62:11210 src_bucket [2021-02-12 21:33:33,712] - [data_helper] [140098737792768] - INFO - creating direct client 172.23.122.90:11210 src_bucket [2021-02-12 21:33:33,792] - [data_helper] [140098737792768] - INFO - creating direct client 172.23.123.61:11210 src_bucket [2021-02-12 21:33:33,871] - [data_helper] [140098737792768] - INFO - creating direct client 172.23.123.62:11210 src_bucket [2021-02-12 21:33:34,096] - [task] [140098737792768] - WARNING - vbucket map not ready after try 1 [2021-02-12 21:33:34,097] - [bucket_helper] [140098737792768] - INFO - waiting for memcached bucket : src_bucket in 172.23.123.61 to accept set ops [2021-02-12 21:33:34,226] - [data_helper] [140098737792768] - INFO - creating direct client 172.23.122.90:11210 src_bucket [2021-02-12 21:33:34,310] - [data_helper] [140098737792768] - INFO - creating direct client 172.23.123.61:11210 src_bucket [2021-02-12 21:33:34,394] - [data_helper] [140098737792768] - INFO - creating direct client 172.23.123.62:11210 src_bucket [2021-02-12 21:33:34,519] - [data_helper] [140098737792768] - INFO - creating direct client 172.23.122.90:11210 src_bucket [2021-02-12 21:33:34,602] - [data_helper] [140098737792768] - INFO - creating direct client 172.23.123.61:11210 src_bucket [2021-02-12 21:33:34,833] - [data_helper] [140098737792768] - INFO - creating direct client 172.23.123.62:11210 src_bucket [2021-02-12 21:33:34,897] - [task] [140098737792768] - WARNING - vbucket map not ready after try 2 [2021-02-12 21:33:34,897] - [bucket_helper] [140098737792768] - INFO - waiting for memcached bucket : src_bucket in 172.23.123.61 to accept set ops [2021-02-12 21:33:35,009] - [data_helper] [140098737792768] - INFO - creating direct client 172.23.122.90:11210 src_bucket [2021-02-12 21:33:35,094] - [data_helper] [140098737792768] - INFO - creating direct client 172.23.123.61:11210 src_bucket [2021-02-12 21:33:35,173] - [data_helper] [140098737792768] - INFO - creating direct client 172.23.123.62:11210 src_bucket [2021-02-12 21:33:35,459] - [data_helper] [140098737792768] - INFO - creating direct client 172.23.122.90:11210 src_bucket [2021-02-12 21:33:35,540] - [data_helper] [140098737792768] - INFO - creating direct client 172.23.123.61:11210 src_bucket [2021-02-12 21:33:35,614] - [data_helper] [140098737792768] - INFO - creating direct client 172.23.123.62:11210 src_bucket [2021-02-12 21:33:35,686] - [task] [140098737792768] - WARNING - vbucket map not ready after try 3 [2021-02-12 21:33:35,687] - [bucket_helper] [140098737792768] - INFO - waiting for memcached bucket : src_bucket in 172.23.123.61 to accept set ops [2021-02-12 21:33:35,804] - [data_helper] [140098737792768] - INFO - creating direct client 172.23.122.90:11210 src_bucket [2021-02-12 21:33:35,880] - [data_helper] [140098737792768] - INFO - creating direct client 172.23.123.61:11210 src_bucket [2021-02-12 21:33:36,122] - [data_helper] [140098737792768] - INFO - creating direct client 172.23.123.62:11210 src_bucket [2021-02-12 21:33:36,236] - [data_helper] [140098737792768] - INFO - creating direct client 172.23.122.90:11210 src_bucket [2021-02-12 21:33:36,311] - [data_helper] [140098737792768] - INFO - creating direct client 172.23.123.61:11210 src_bucket [2021-02-12 21:33:36,393] - [data_helper] [140098737792768] - INFO - creating direct client 172.23.123.62:11210 src_bucket [2021-02-12 21:33:36,465] - [task] [140098737792768] - WARNING - vbucket map not ready after try 4 [2021-02-12 21:33:36,465] - [bucket_helper] [140098737792768] - INFO - waiting for memcached bucket : src_bucket in 172.23.123.61 to accept set ops [2021-02-12 21:33:36,751] - [data_helper] [140098737792768] - INFO - creating direct client 172.23.122.90:11210 src_bucket [2021-02-12 21:33:36,838] - [data_helper] [140098737792768] - INFO - creating direct client 172.23.123.61:11210 src_bucket [2021-02-12 21:33:36,915] - [data_helper] [140098737792768] - INFO - creating direct client 172.23.123.62:11210 src_bucket [2021-02-12 21:33:37,037] - [data_helper] [140098737792768] - INFO - creating direct client 172.23.122.90:11210 src_bucket [2021-02-12 21:33:37,116] - [data_helper] [140098737792768] - INFO - creating direct client 172.23.123.61:11210 src_bucket [2021-02-12 21:33:37,194] - [data_helper] [140098737792768] - INFO - creating direct client 172.23.123.62:11210 src_bucket [2021-02-12 21:33:37,421] - [task] [140098737792768] - WARNING - vbucket map not ready after try 5 [2021-02-12 21:33:38,446] - [rest_client] [140098737792768] - INFO - http://172.23.123.61:8091/pools/default/buckets with param: name=dst_bucket&ramQuotaMB=200&authType=none&replicaNumber=0&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore [2021-02-12 21:33:38,475] - [rest_client] [140098737792768] - INFO - 0.03 seconds to create bucket dst_bucket [2021-02-12 21:33:38,475] - [bucket_helper] [140098737792768] - INFO - waiting for memcached bucket : dst_bucket in 172.23.123.61 to accept set ops [2021-02-12 21:33:39,122] - [data_helper] [140098737792768] - INFO - creating direct client 172.23.122.90:11210 dst_bucket [2021-02-12 21:33:39,210] - [data_helper] [140098737792768] - INFO - creating direct client 172.23.123.61:11210 dst_bucket [2021-02-12 21:33:39,297] - [data_helper] [140098737792768] - INFO - creating direct client 172.23.123.62:11210 dst_bucket [2021-02-12 21:33:39,625] - [data_helper] [140098737792768] - INFO - creating direct client 172.23.122.90:11210 dst_bucket [2021-02-12 21:33:39,716] - [data_helper] [140098737792768] - INFO - creating direct client 172.23.123.61:11210 dst_bucket [2021-02-12 21:33:39,798] - [data_helper] [140098737792768] - INFO - creating direct client 172.23.123.62:11210 dst_bucket [2021-02-12 21:33:39,868] - [task] [140098737792768] - WARNING - vbucket map not ready after try 0 [2021-02-12 21:33:39,868] - [bucket_helper] [140098737792768] - INFO - waiting for memcached bucket : dst_bucket in 172.23.123.61 to accept set ops [2021-02-12 21:33:39,983] - [data_helper] [140098737792768] - INFO - creating direct client 172.23.122.90:11210 dst_bucket [2021-02-12 21:33:40,059] - [data_helper] [140098737792768] - INFO - creating direct client 172.23.123.61:11210 dst_bucket [2021-02-12 21:33:40,309] - [data_helper] [140098737792768] - INFO - creating direct client 172.23.123.62:11210 dst_bucket [2021-02-12 21:33:40,437] - [data_helper] [140098737792768] - INFO - creating direct client 172.23.122.90:11210 dst_bucket [2021-02-12 21:33:40,518] - [data_helper] [140098737792768] - INFO - creating direct client 172.23.123.61:11210 dst_bucket [2021-02-12 21:33:40,597] - [data_helper] [140098737792768] - INFO - creating direct client 172.23.123.62:11210 dst_bucket [2021-02-12 21:33:40,677] - [task] [140098737792768] - WARNING - vbucket map not ready after try 1 [2021-02-12 21:33:40,677] - [bucket_helper] [140098737792768] - INFO - waiting for memcached bucket : dst_bucket in 172.23.123.61 to accept set ops [2021-02-12 21:33:40,804] - [data_helper] [140098737792768] - INFO - creating direct client 172.23.122.90:11210 dst_bucket [2021-02-12 21:33:41,074] - [data_helper] [140098737792768] - INFO - creating direct client 172.23.123.61:11210 dst_bucket [2021-02-12 21:33:41,159] - [data_helper] [140098737792768] - INFO - creating direct client 172.23.123.62:11210 dst_bucket [2021-02-12 21:33:41,298] - [data_helper] [140098737792768] - INFO - creating direct client 172.23.122.90:11210 dst_bucket [2021-02-12 21:33:41,391] - [data_helper] [140098737792768] - INFO - creating direct client 172.23.123.61:11210 dst_bucket [2021-02-12 21:33:41,473] - [data_helper] [140098737792768] - INFO - creating direct client 172.23.123.62:11210 dst_bucket [2021-02-12 21:33:41,714] - [task] [140098737792768] - WARNING - vbucket map not ready after try 2 [2021-02-12 21:33:41,714] - [bucket_helper] [140098737792768] - INFO - waiting for memcached bucket : dst_bucket in 172.23.123.61 to accept set ops [2021-02-12 21:33:41,837] - [data_helper] [140098737792768] - INFO - creating direct client 172.23.122.90:11210 dst_bucket [2021-02-12 21:33:41,921] - [data_helper] [140098737792768] - INFO - creating direct client 172.23.123.61:11210 dst_bucket [2021-02-12 21:33:42,006] - [data_helper] [140098737792768] - INFO - creating direct client 172.23.123.62:11210 dst_bucket [2021-02-12 21:33:42,138] - [data_helper] [140098737792768] - INFO - creating direct client 172.23.122.90:11210 dst_bucket [2021-02-12 21:33:42,226] - [data_helper] [140098737792768] - INFO - creating direct client 172.23.123.61:11210 dst_bucket [2021-02-12 21:33:42,474] - [data_helper] [140098737792768] - INFO - creating direct client 172.23.123.62:11210 dst_bucket [2021-02-12 21:33:42,563] - [task] [140098737792768] - WARNING - vbucket map not ready after try 3 [2021-02-12 21:33:42,563] - [bucket_helper] [140098737792768] - INFO - waiting for memcached bucket : dst_bucket in 172.23.123.61 to accept set ops [2021-02-12 21:33:42,688] - [data_helper] [140098737792768] - INFO - creating direct client 172.23.122.90:11210 dst_bucket [2021-02-12 21:33:42,770] - [data_helper] [140098737792768] - INFO - creating direct client 172.23.123.61:11210 dst_bucket [2021-02-12 21:33:42,855] - [data_helper] [140098737792768] - INFO - creating direct client 172.23.123.62:11210 dst_bucket [2021-02-12 21:33:42,985] - [data_helper] [140098737792768] - INFO - creating direct client 172.23.122.90:11210 dst_bucket [2021-02-12 21:33:43,250] - [data_helper] [140098737792768] - INFO - creating direct client 172.23.123.61:11210 dst_bucket [2021-02-12 21:33:43,333] - [data_helper] [140098737792768] - INFO - creating direct client 172.23.123.62:11210 dst_bucket [2021-02-12 21:33:43,412] - [task] [140098737792768] - WARNING - vbucket map not ready after try 4 [2021-02-12 21:33:43,412] - [bucket_helper] [140098737792768] - INFO - waiting for memcached bucket : dst_bucket in 172.23.123.61 to accept set ops [2021-02-12 21:33:43,554] - [data_helper] [140098737792768] - INFO - creating direct client 172.23.122.90:11210 dst_bucket [2021-02-12 21:33:43,641] - [data_helper] [140098737792768] - INFO - creating direct client 172.23.123.61:11210 dst_bucket [2021-02-12 21:33:43,893] - [data_helper] [140098737792768] - INFO - creating direct client 172.23.123.62:11210 dst_bucket [2021-02-12 21:33:44,023] - [data_helper] [140098737792768] - INFO - creating direct client 172.23.122.90:11210 dst_bucket [2021-02-12 21:33:44,107] - [data_helper] [140098737792768] - INFO - creating direct client 172.23.123.61:11210 dst_bucket [2021-02-12 21:33:44,190] - [data_helper] [140098737792768] - INFO - creating direct client 172.23.123.62:11210 dst_bucket [2021-02-12 21:33:44,272] - [task] [140098737792768] - WARNING - vbucket map not ready after try 5 [2021-02-12 21:33:45,300] - [rest_client] [140098737792768] - INFO - http://172.23.123.61:8091/pools/default/buckets with param: name=metadata&ramQuotaMB=200&authType=none&replicaNumber=0&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore [2021-02-12 21:33:45,328] - [rest_client] [140098737792768] - INFO - 0.03 seconds to create bucket metadata [2021-02-12 21:33:45,329] - [bucket_helper] [140098737792768] - INFO - waiting for memcached bucket : metadata in 172.23.123.61 to accept set ops [2021-02-12 21:33:46,151] - [data_helper] [140098737792768] - INFO - creating direct client 172.23.122.90:11210 metadata [2021-02-12 21:33:46,273] - [data_helper] [140098737792768] - INFO - creating direct client 172.23.123.61:11210 metadata [2021-02-12 21:33:46,360] - [data_helper] [140098737792768] - INFO - creating direct client 172.23.123.62:11210 metadata [2021-02-12 21:33:46,508] - [data_helper] [140098737792768] - INFO - creating direct client 172.23.122.90:11210 metadata [2021-02-12 21:33:46,608] - [data_helper] [140098737792768] - INFO - creating direct client 172.23.123.61:11210 metadata [2021-02-12 21:33:46,708] - [data_helper] [140098737792768] - INFO - creating direct client 172.23.123.62:11210 metadata [2021-02-12 21:33:46,966] - [task] [140098737792768] - WARNING - vbucket map not ready after try 0 [2021-02-12 21:33:46,966] - [bucket_helper] [140098737792768] - INFO - waiting for memcached bucket : metadata in 172.23.123.61 to accept set ops [2021-02-12 21:33:47,105] - [data_helper] [140098737792768] - INFO - creating direct client 172.23.122.90:11210 metadata [2021-02-12 21:33:47,206] - [data_helper] [140098737792768] - INFO - creating direct client 172.23.123.61:11210 metadata [2021-02-12 21:33:47,299] - [data_helper] [140098737792768] - INFO - creating direct client 172.23.123.62:11210 metadata [2021-02-12 21:33:47,433] - [data_helper] [140098737792768] - INFO - creating direct client 172.23.122.90:11210 metadata [2021-02-12 21:33:47,683] - [data_helper] [140098737792768] - INFO - creating direct client 172.23.123.61:11210 metadata [2021-02-12 21:33:47,778] - [data_helper] [140098737792768] - INFO - creating direct client 172.23.123.62:11210 metadata [2021-02-12 21:33:47,850] - [task] [140098737792768] - WARNING - vbucket map not ready after try 1 [2021-02-12 21:33:47,850] - [bucket_helper] [140098737792768] - INFO - waiting for memcached bucket : metadata in 172.23.123.61 to accept set ops [2021-02-12 21:33:47,977] - [data_helper] [140098737792768] - INFO - creating direct client 172.23.122.90:11210 metadata [2021-02-12 21:33:48,068] - [data_helper] [140098737792768] - INFO - creating direct client 172.23.123.61:11210 metadata [2021-02-12 21:33:48,156] - [data_helper] [140098737792768] - INFO - creating direct client 172.23.123.62:11210 metadata [2021-02-12 21:33:48,484] - [data_helper] [140098737792768] - INFO - creating direct client 172.23.122.90:11210 metadata [2021-02-12 21:33:48,587] - [data_helper] [140098737792768] - INFO - creating direct client 172.23.123.61:11210 metadata [2021-02-12 21:33:48,675] - [data_helper] [140098737792768] - INFO - creating direct client 172.23.123.62:11210 metadata [2021-02-12 21:33:48,749] - [task] [140098737792768] - WARNING - vbucket map not ready after try 2 [2021-02-12 21:33:48,749] - [bucket_helper] [140098737792768] - INFO - waiting for memcached bucket : metadata in 172.23.123.61 to accept set ops [2021-02-12 21:33:48,891] - [data_helper] [140098737792768] - INFO - creating direct client 172.23.122.90:11210 metadata [2021-02-12 21:33:48,981] - [data_helper] [140098737792768] - INFO - creating direct client 172.23.123.61:11210 metadata [2021-02-12 21:33:49,232] - [data_helper] [140098737792768] - INFO - creating direct client 172.23.123.62:11210 metadata [2021-02-12 21:33:49,371] - [data_helper] [140098737792768] - INFO - creating direct client 172.23.122.90:11210 metadata [2021-02-12 21:33:49,467] - [data_helper] [140098737792768] - INFO - creating direct client 172.23.123.61:11210 metadata [2021-02-12 21:33:49,556] - [data_helper] [140098737792768] - INFO - creating direct client 172.23.123.62:11210 metadata [2021-02-12 21:33:49,629] - [task] [140098737792768] - WARNING - vbucket map not ready after try 3 [2021-02-12 21:33:49,629] - [bucket_helper] [140098737792768] - INFO - waiting for memcached bucket : metadata in 172.23.123.61 to accept set ops [2021-02-12 21:33:49,932] - [data_helper] [140098737792768] - INFO - creating direct client 172.23.122.90:11210 metadata [2021-02-12 21:33:50,023] - [data_helper] [140098737792768] - INFO - creating direct client 172.23.123.61:11210 metadata [2021-02-12 21:33:50,107] - [data_helper] [140098737792768] - INFO - creating direct client 172.23.123.62:11210 metadata [2021-02-12 21:33:50,234] - [data_helper] [140098737792768] - INFO - creating direct client 172.23.122.90:11210 metadata [2021-02-12 21:33:50,322] - [data_helper] [140098737792768] - INFO - creating direct client 172.23.123.61:11210 metadata [2021-02-12 21:33:50,418] - [data_helper] [140098737792768] - INFO - creating direct client 172.23.123.62:11210 metadata [2021-02-12 21:33:50,681] - [task] [140098737792768] - WARNING - vbucket map not ready after try 4 [2021-02-12 21:33:50,682] - [bucket_helper] [140098737792768] - INFO - waiting for memcached bucket : metadata in 172.23.123.61 to accept set ops [2021-02-12 21:33:50,809] - [data_helper] [140098737792768] - INFO - creating direct client 172.23.122.90:11210 metadata [2021-02-12 21:33:50,901] - [data_helper] [140098737792768] - INFO - creating direct client 172.23.123.61:11210 metadata [2021-02-12 21:33:50,998] - [data_helper] [140098737792768] - INFO - creating direct client 172.23.123.62:11210 metadata [2021-02-12 21:33:51,134] - [data_helper] [140098737792768] - INFO - creating direct client 172.23.122.90:11210 metadata [2021-02-12 21:33:51,232] - [data_helper] [140098737792768] - INFO - creating direct client 172.23.123.61:11210 metadata [2021-02-12 21:33:51,515] - [data_helper] [140098737792768] - INFO - creating direct client 172.23.123.62:11210 metadata [2021-02-12 21:33:51,594] - [task] [140098737792768] - WARNING - vbucket map not ready after try 5 [2021-02-12 21:33:51,673] - [rest_client] [140099341772544] - INFO - http://172.23.122.117:8091/pools/default/buckets/dst_bucket/scopes with params: name=scope_0 [2021-02-12 21:33:51,673] - [rest_client] [140099341772544] - INFO - Scope created dst_bucket->scope_0 {'uid': '1'} [2021-02-12 21:33:51,702] - [rest_client] [140099341772544] - INFO - http://172.23.122.117:8091/pools/default/buckets/dst_bucket/scopes with params: name=scope_1 [2021-02-12 21:33:51,702] - [rest_client] [140099341772544] - INFO - Scope created dst_bucket->scope_1 {'uid': '2'} [2021-02-12 21:33:51,731] - [rest_client] [140099341772544] - INFO - http://172.23.122.117:8091/pools/default/buckets/dst_bucket/scopes with params: name=scope_2 [2021-02-12 21:33:51,731] - [rest_client] [140099341772544] - INFO - Scope created dst_bucket->scope_2 {'uid': '3'} [2021-02-12 21:33:51,758] - [rest_client] [140099341772544] - INFO - http://172.23.122.117:8091/pools/default/buckets/dst_bucket/scopes with params: name=scope_3 [2021-02-12 21:33:51,759] - [rest_client] [140099341772544] - INFO - Scope created dst_bucket->scope_3 {'uid': '4'} [2021-02-12 21:33:51,787] - [rest_client] [140099341772544] - INFO - http://172.23.122.117:8091/pools/default/buckets/dst_bucket/scopes with params: name=scope_4 [2021-02-12 21:33:51,787] - [rest_client] [140099341772544] - INFO - Scope created dst_bucket->scope_4 {'uid': '5'} [2021-02-12 21:33:51,818] - [rest_client] [140099341772544] - INFO - http://172.23.122.117:8091/pools/default/buckets/src_bucket/scopes with params: name=scope_0 [2021-02-12 21:33:51,818] - [rest_client] [140099341772544] - INFO - Scope created src_bucket->scope_0 {'uid': '1'} [2021-02-12 21:33:51,845] - [rest_client] [140099341772544] - INFO - http://172.23.122.117:8091/pools/default/buckets/src_bucket/scopes with params: name=scope_1 [2021-02-12 21:33:51,845] - [rest_client] [140099341772544] - INFO - Scope created src_bucket->scope_1 {'uid': '2'} [2021-02-12 21:33:51,873] - [rest_client] [140099341772544] - INFO - http://172.23.122.117:8091/pools/default/buckets/src_bucket/scopes with params: name=scope_2 [2021-02-12 21:33:51,873] - [rest_client] [140099341772544] - INFO - Scope created src_bucket->scope_2 {'uid': '3'} [2021-02-12 21:33:51,901] - [rest_client] [140099341772544] - INFO - http://172.23.122.117:8091/pools/default/buckets/src_bucket/scopes with params: name=scope_3 [2021-02-12 21:33:51,902] - [rest_client] [140099341772544] - INFO - Scope created src_bucket->scope_3 {'uid': '4'} [2021-02-12 21:33:51,932] - [rest_client] [140099341772544] - INFO - http://172.23.122.117:8091/pools/default/buckets/src_bucket/scopes with params: name=scope_4 [2021-02-12 21:33:51,932] - [rest_client] [140099341772544] - INFO - Scope created src_bucket->scope_4 {'uid': '5'} [2021-02-12 21:33:51,960] - [rest_client] [140099341772544] - INFO - http://172.23.122.117:8091/pools/default/buckets/dst_bucket/scopes/scope_1/collections with params: name=coll_0 [2021-02-12 21:33:51,960] - [rest_client] [140099341772544] - INFO - Collection created dst_bucket->scope_1->coll_0 manifest:{'uid': '6'} [2021-02-12 21:33:51,993] - [rest_client] [140099341772544] - INFO - http://172.23.122.117:8091/pools/default/buckets/dst_bucket/scopes/scope_1/collections with params: name=coll_1 [2021-02-12 21:33:51,993] - [rest_client] [140099341772544] - INFO - Collection created dst_bucket->scope_1->coll_1 manifest:{'uid': '7'} [2021-02-12 21:33:52,021] - [rest_client] [140099341772544] - INFO - http://172.23.122.117:8091/pools/default/buckets/dst_bucket/scopes/scope_1/collections with params: name=coll_2 [2021-02-12 21:33:52,021] - [rest_client] [140099341772544] - INFO - Collection created dst_bucket->scope_1->coll_2 manifest:{'uid': '8'} [2021-02-12 21:33:52,049] - [rest_client] [140099341772544] - INFO - http://172.23.122.117:8091/pools/default/buckets/dst_bucket/scopes/scope_1/collections with params: name=coll_3 [2021-02-12 21:33:52,049] - [rest_client] [140099341772544] - INFO - Collection created dst_bucket->scope_1->coll_3 manifest:{'uid': '9'} [2021-02-12 21:33:52,084] - [rest_client] [140099341772544] - INFO - http://172.23.122.117:8091/pools/default/buckets/dst_bucket/scopes/scope_1/collections with params: name=coll_4 [2021-02-12 21:33:52,084] - [rest_client] [140099341772544] - INFO - Collection created dst_bucket->scope_1->coll_4 manifest:{'uid': 'a'} [2021-02-12 21:33:52,114] - [rest_client] [140099341772544] - INFO - http://172.23.122.117:8091/pools/default/buckets/src_bucket/scopes/scope_1/collections with params: name=coll_0 [2021-02-12 21:33:52,114] - [rest_client] [140099341772544] - INFO - Collection created src_bucket->scope_1->coll_0 manifest:{'uid': '6'} [2021-02-12 21:33:52,141] - [rest_client] [140099341772544] - INFO - http://172.23.122.117:8091/pools/default/buckets/src_bucket/scopes/scope_1/collections with params: name=coll_1 [2021-02-12 21:33:52,141] - [rest_client] [140099341772544] - INFO - Collection created src_bucket->scope_1->coll_1 manifest:{'uid': '7'} [2021-02-12 21:33:52,175] - [rest_client] [140099341772544] - INFO - http://172.23.122.117:8091/pools/default/buckets/src_bucket/scopes/scope_1/collections with params: name=coll_2 [2021-02-12 21:33:52,175] - [rest_client] [140099341772544] - INFO - Collection created src_bucket->scope_1->coll_2 manifest:{'uid': '8'} [2021-02-12 21:33:52,203] - [rest_client] [140099341772544] - INFO - http://172.23.122.117:8091/pools/default/buckets/src_bucket/scopes/scope_1/collections with params: name=coll_3 [2021-02-12 21:33:52,203] - [rest_client] [140099341772544] - INFO - Collection created src_bucket->scope_1->coll_3 manifest:{'uid': '9'} [2021-02-12 21:33:52,236] - [rest_client] [140099341772544] - INFO - http://172.23.122.117:8091/pools/default/buckets/src_bucket/scopes/scope_1/collections with params: name=coll_4 [2021-02-12 21:33:52,236] - [rest_client] [140099341772544] - INFO - Collection created src_bucket->scope_1->coll_4 manifest:{'uid': 'a'} [2021-02-12 21:33:53,465] - [eventing_base] [140099341772544] - INFO - saving function bucket_op [2021-02-12 21:33:54,656] - [eventing_base] [140099341772544] - INFO - saving function timers [2021-02-12 21:33:55,927] - [eventing_base] [140099341772544] - INFO - saving function sbm [2021-02-12 21:33:57,160] - [eventing_base] [140099341772544] - INFO - saving function curl [2021-02-12 21:33:58,387] - [eventing_base] [140099341772544] - INFO - saving function n1ql [2021-02-12 21:33:59,665] - [basetestcase] [140099341772544] - INFO - sleep for 5 secs. Waiting for bucket_op to deployed... ... [2021-02-12 21:34:04,710] - [basetestcase] [140099341772544] - INFO - sleep for 5 secs. Waiting for bucket_op to deployed... ... [2021-02-12 21:34:09,756] - [basetestcase] [140099341772544] - INFO - sleep for 5 secs. Waiting for bucket_op to deployed... ... [2021-02-12 21:34:14,807] - [basetestcase] [140099341772544] - INFO - sleep for 5 secs. Waiting for bucket_op to deployed... ... [2021-02-12 21:34:21,110] - [basetestcase] [140099341772544] - INFO - sleep for 5 secs. Waiting for timers to deployed... ... [2021-02-12 21:34:26,160] - [basetestcase] [140099341772544] - INFO - sleep for 5 secs. Waiting for timers to deployed... ... [2021-02-12 21:34:31,221] - [basetestcase] [140099341772544] - INFO - sleep for 5 secs. Waiting for timers to deployed... ... [2021-02-12 21:34:36,267] - [basetestcase] [140099341772544] - INFO - sleep for 5 secs. Waiting for timers to deployed... ... [2021-02-12 21:34:42,644] - [basetestcase] [140099341772544] - INFO - sleep for 5 secs. Waiting for sbm to deployed... ... [2021-02-12 21:34:47,687] - [basetestcase] [140099341772544] - INFO - sleep for 5 secs. Waiting for sbm to deployed... ... [2021-02-12 21:34:52,759] - [basetestcase] [140099341772544] - INFO - sleep for 5 secs. Waiting for sbm to deployed... ... [2021-02-12 21:34:57,802] - [basetestcase] [140099341772544] - INFO - sleep for 5 secs. Waiting for sbm to deployed... ... [2021-02-12 21:35:04,061] - [basetestcase] [140099341772544] - INFO - sleep for 5 secs. Waiting for curl to deployed... ... [2021-02-12 21:35:09,110] - [basetestcase] [140099341772544] - INFO - sleep for 5 secs. Waiting for curl to deployed... ... [2021-02-12 21:35:14,191] - [basetestcase] [140099341772544] - INFO - sleep for 5 secs. Waiting for curl to deployed... ... [2021-02-12 21:35:19,234] - [basetestcase] [140099341772544] - INFO - sleep for 5 secs. Waiting for curl to deployed... ... [2021-02-12 21:35:25,510] - [basetestcase] [140099341772544] - INFO - sleep for 5 secs. Waiting for n1ql to deployed... ... [2021-02-12 21:35:30,553] - [basetestcase] [140099341772544] - INFO - sleep for 5 secs. Waiting for n1ql to deployed... ... [2021-02-12 21:35:35,618] - [basetestcase] [140099341772544] - INFO - sleep for 5 secs. Waiting for n1ql to deployed... ... [2021-02-12 21:35:40,661] - [basetestcase] [140099341772544] - INFO - sleep for 5 secs. Waiting for n1ql to deployed... ... [2021-02-12 21:35:46,714] - [task] [140098737792768] - INFO - java -jar java_sdk_client/collections/target/javaclient/javaclient.jar -i 172.23.123.61 -u Administrator -p password -b src_bucket -s _default -c _default -n 40320 -pc 100 -pu 0 -pd 0 -l uniform -dsn 1 -dpx doc_ -dt Person -de 0 -ds 500 -ac False -st 0 -en 40319 -o False -sd False [2021-02-12 21:35:48,728] - [rest_client] [140099341772544] - INFO - rebalance reached >100% in 3.004424571990967 seconds [2021-02-12 21:35:57,617] - [task] [140098737792768] - INFO - java -jar java_sdk_client/collections/target/javaclient/javaclient.jar -i 172.23.123.61 -u Administrator -p password -b src_bucket -s scope_1 -c coll_1 -n 40320 -pc 100 -pu 0 -pd 0 -l uniform -dsn 1 -dpx doc_ -dt Person -de 0 -ds 500 -ac False -st 0 -en 40319 -o False -sd False [2021-02-12 21:36:08,796] - [rest_client] [140098737792768] - INFO - rebalance params : {'knownNodes': 'ns_1@172.23.122.117,ns_1@172.23.122.118,ns_1@172.23.122.82,ns_1@172.23.122.90,ns_1@172.23.122.97,ns_1@172.23.123.61,ns_1@172.23.123.62', 'ejectedNodes': 'ns_1@172.23.123.62', 'user': 'Administrator', 'password': 'password'} [2021-02-12 21:36:08,826] - [rest_client] [140098737792768] - INFO - rebalance operation started [2021-02-12 21:36:08,828] - [rest_client] [140098737792768] - INFO - rebalance percentage : 0.00 % [2021-02-12 21:36:08,829] - [task] [140098737792768] - INFO - Rebalance - status: running, progress: 0.00% [2021-02-12 21:36:18,861] - [rest_client] [140098737792768] - INFO - rebalance percentage : 14.00 % [2021-02-12 21:36:18,861] - [task] [140098737792768] - INFO - Rebalance - status: running, progress: 14.00% [2021-02-12 21:36:28,889] - [rest_client] [140098737792768] - INFO - rebalance percentage : 27.00 % [2021-02-12 21:36:28,890] - [task] [140098737792768] - INFO - Rebalance - status: running, progress: 27.00% [2021-02-12 21:36:38,917] - [rest_client] [140098737792768] - INFO - rebalance percentage : 28.00 % [2021-02-12 21:36:38,917] - [task] [140098737792768] - INFO - Rebalance - status: running, progress: 28.00% [2021-02-12 21:36:48,942] - [rest_client] [140098737792768] - INFO - rebalance percentage : 42.00 % [2021-02-12 21:36:48,942] - [task] [140098737792768] - INFO - Rebalance - status: running, progress: 42.00% [2021-02-12 21:36:58,974] - [rest_client] [140098737792768] - INFO - rebalance percentage : 50.00 % [2021-02-12 21:36:58,974] - [task] [140098737792768] - INFO - Rebalance - status: running, progress: 50.00% [2021-02-12 21:37:08,999] - [rest_client] [140098737792768] - INFO - rebalance percentage : 50.00 % [2021-02-12 21:37:08,999] - [task] [140098737792768] - INFO - Rebalance - status: running, progress: 50.00% [2021-02-12 21:37:19,025] - [rest_client] [140098737792768] - INFO - rebalance percentage : 55.00 % [2021-02-12 21:37:19,025] - [task] [140098737792768] - INFO - Rebalance - status: running, progress: 55.00% [2021-02-12 21:37:29,050] - [rest_client] [140098737792768] - INFO - rebalance percentage : 59.00 % [2021-02-12 21:37:29,050] - [task] [140098737792768] - INFO - Rebalance - status: running, progress: 59.00% [2021-02-12 21:37:39,073] - [rest_client] [140098737792768] - INFO - rebalance percentage : 63.00 % [2021-02-12 21:37:39,074] - [task] [140098737792768] - INFO - Rebalance - status: running, progress: 63.00% [2021-02-12 21:37:49,099] - [rest_client] [140098737792768] - INFO - rebalance percentage : 69.00 % [2021-02-12 21:37:49,099] - [task] [140098737792768] - INFO - Rebalance - status: running, progress: 69.00% [2021-02-12 21:37:59,122] - [rest_client] [140098737792768] - INFO - rebalance percentage : 73.00 % [2021-02-12 21:37:59,122] - [task] [140098737792768] - INFO - Rebalance - status: running, progress: 73.00% [2021-02-12 21:38:09,147] - [rest_client] [140098737792768] - INFO - rebalance percentage : 77.00 % [2021-02-12 21:38:09,147] - [task] [140098737792768] - INFO - Rebalance - status: running, progress: 77.00% [2021-02-12 21:38:19,172] - [rest_client] [140098737792768] - INFO - rebalance percentage : 83.00 % [2021-02-12 21:38:19,173] - [task] [140098737792768] - INFO - Rebalance - status: running, progress: 83.00% [2021-02-12 21:38:29,198] - [rest_client] [140098737792768] - INFO - rebalance percentage : 88.00 % [2021-02-12 21:38:29,198] - [task] [140098737792768] - INFO - Rebalance - status: running, progress: 88.00% [2021-02-12 21:38:39,222] - [rest_client] [140098737792768] - INFO - rebalance percentage : 91.00 % [2021-02-12 21:38:39,222] - [task] [140098737792768] - INFO - Rebalance - status: running, progress: 91.00% [2021-02-12 21:38:49,245] - [task] [140098737792768] - INFO - Rebalance - status: none, progress: 100.00% [2021-02-12 21:38:49,259] - [rest_client] [140098737792768] - ERROR - socket error while connecting to http://172.23.123.62:8091/nodes/self error [Errno 111] Connection refused [2021-02-12 21:38:52,271] - [task] [140098737792768] - INFO - rebalancing was completed with progress: 100% in 163.44422268867493 sec [2021-02-12 21:38:52,292] - [basetestcase] [140099341772544] - INFO - list of eventing nodes in cluster: [ip:172.23.122.117 port:8091 ssh_username:root, ip:172.23.122.82 port:8091 ssh_username:root, ip:172.23.122.97 port:8091 ssh_username:root] [2021-02-12 21:38:52,298] - [tuq_helper] [140099341772544] - INFO - RUN QUERY create primary index on dst_bucket.scope_1.coll_0 [2021-02-12 21:38:52,302] - [rest_client] [140099341772544] - INFO - query params : statement=create+primary+index+on+dst_bucket.scope_1.coll_0 [2021-02-12 21:38:54,792] - [tuq_helper] [140099341772544] - INFO - TOTAL ELAPSED TIME: 2.487589408s [2021-02-12 21:38:54,815] - [basetestcase] [140099341772544] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.118 port:8091 ssh_username:root] [2021-02-12 21:38:54,821] - [rest_client] [140099341772544] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_0 [2021-02-12 21:38:54,837] - [eventing_base] [140099341772544] - INFO - Final docs count... Current : 40320 Expected : 40320 [2021-02-12 21:38:54,848] - [basetestcase] [140099341772544] - INFO - list of eventing nodes in cluster: [ip:172.23.122.117 port:8091 ssh_username:root, ip:172.23.122.82 port:8091 ssh_username:root, ip:172.23.122.97 port:8091 ssh_username:root] [2021-02-12 21:38:54,852] - [tuq_helper] [140099341772544] - INFO - RUN QUERY create primary index on dst_bucket.scope_1.coll_1 [2021-02-12 21:38:54,857] - [rest_client] [140099341772544] - INFO - query params : statement=create+primary+index+on+dst_bucket.scope_1.coll_1 [2021-02-12 21:38:57,887] - [tuq_helper] [140099341772544] - INFO - TOTAL ELAPSED TIME: 3.028213699s [2021-02-12 21:38:57,906] - [basetestcase] [140099341772544] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.118 port:8091 ssh_username:root] [2021-02-12 21:38:57,913] - [rest_client] [140099341772544] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_1 [2021-02-12 21:38:57,924] - [eventing_base] [140099341772544] - INFO - Final docs count... Current : 40320 Expected : 40320 [2021-02-12 21:38:57,932] - [basetestcase] [140099341772544] - INFO - list of eventing nodes in cluster: [ip:172.23.122.117 port:8091 ssh_username:root, ip:172.23.122.82 port:8091 ssh_username:root, ip:172.23.122.97 port:8091 ssh_username:root] [2021-02-12 21:38:57,937] - [tuq_helper] [140099341772544] - INFO - RUN QUERY create primary index on dst_bucket.scope_1.coll_3 [2021-02-12 21:38:57,942] - [rest_client] [140099341772544] - INFO - query params : statement=create+primary+index+on+dst_bucket.scope_1.coll_3 [2021-02-12 21:39:00,914] - [tuq_helper] [140099341772544] - INFO - TOTAL ELAPSED TIME: 2.970521046s [2021-02-12 21:39:00,942] - [basetestcase] [140099341772544] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.118 port:8091 ssh_username:root] [2021-02-12 21:39:00,947] - [rest_client] [140099341772544] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_3 [2021-02-12 21:39:00,957] - [eventing_base] [140099341772544] - INFO - Final docs count... Current : 40320 Expected : 40320 [2021-02-12 21:39:00,966] - [basetestcase] [140099341772544] - INFO - list of eventing nodes in cluster: [ip:172.23.122.117 port:8091 ssh_username:root, ip:172.23.122.82 port:8091 ssh_username:root, ip:172.23.122.97 port:8091 ssh_username:root] [2021-02-12 21:39:00,971] - [tuq_helper] [140099341772544] - INFO - RUN QUERY create primary index on dst_bucket.scope_1.coll_4 [2021-02-12 21:39:00,975] - [rest_client] [140099341772544] - INFO - query params : statement=create+primary+index+on+dst_bucket.scope_1.coll_4 [2021-02-12 21:39:03,838] - [tuq_helper] [140099341772544] - INFO - TOTAL ELAPSED TIME: 2.861384732s [2021-02-12 21:39:03,859] - [basetestcase] [140099341772544] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.118 port:8091 ssh_username:root] [2021-02-12 21:39:03,865] - [rest_client] [140099341772544] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-12 21:39:03,875] - [basetestcase] [140099341772544] - INFO - sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_4 to complete bucket operations... Current : 40311 Expected : 40320 ... [2021-02-12 21:39:33,925] - [basetestcase] [140099341772544] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.118 port:8091 ssh_username:root] [2021-02-12 21:39:33,930] - [rest_client] [140099341772544] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-12 21:39:33,950] - [basetestcase] [140099341772544] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.118 port:8091 ssh_username:root] [2021-02-12 21:39:33,955] - [rest_client] [140099341772544] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-12 21:39:33,958] - [basetestcase] [140099341772544] - INFO - sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_4 to complete bucket operations... Current : 40311 Expected : 40320 ... [2021-02-12 21:40:03,988] - [basetestcase] [140099341772544] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.118 port:8091 ssh_username:root] [2021-02-12 21:40:03,995] - [rest_client] [140099341772544] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-12 21:40:04,013] - [basetestcase] [140099341772544] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.118 port:8091 ssh_username:root] [2021-02-12 21:40:04,018] - [rest_client] [140099341772544] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-12 21:40:04,021] - [basetestcase] [140099341772544] - INFO - sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_4 to complete bucket operations... Current : 40311 Expected : 40320 ... [2021-02-12 21:40:34,071] - [basetestcase] [140099341772544] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.118 port:8091 ssh_username:root] [2021-02-12 21:40:34,078] - [rest_client] [140099341772544] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-12 21:40:34,094] - [basetestcase] [140099341772544] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.118 port:8091 ssh_username:root] [2021-02-12 21:40:34,100] - [rest_client] [140099341772544] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-12 21:40:34,103] - [basetestcase] [140099341772544] - INFO - sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_4 to complete bucket operations... Current : 40311 Expected : 40320 ... [2021-02-12 21:41:04,153] - [basetestcase] [140099341772544] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.118 port:8091 ssh_username:root] [2021-02-12 21:41:04,159] - [rest_client] [140099341772544] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-12 21:41:04,176] - [basetestcase] [140099341772544] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.118 port:8091 ssh_username:root] [2021-02-12 21:41:04,181] - [rest_client] [140099341772544] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-12 21:41:04,184] - [basetestcase] [140099341772544] - INFO - sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_4 to complete bucket operations... Current : 40311 Expected : 40320 ... [2021-02-12 21:41:34,233] - [basetestcase] [140099341772544] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.118 port:8091 ssh_username:root] [2021-02-12 21:41:34,240] - [rest_client] [140099341772544] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-12 21:41:34,257] - [basetestcase] [140099341772544] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.118 port:8091 ssh_username:root] [2021-02-12 21:41:34,262] - [rest_client] [140099341772544] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-12 21:41:34,265] - [basetestcase] [140099341772544] - INFO - sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_4 to complete bucket operations... Current : 40311 Expected : 40320 ... [2021-02-12 21:42:04,315] - [basetestcase] [140099341772544] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.118 port:8091 ssh_username:root] [2021-02-12 21:42:04,322] - [rest_client] [140099341772544] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-12 21:42:04,340] - [basetestcase] [140099341772544] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.118 port:8091 ssh_username:root] [2021-02-12 21:42:04,346] - [rest_client] [140099341772544] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-12 21:42:04,349] - [basetestcase] [140099341772544] - INFO - sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_4 to complete bucket operations... Current : 40311 Expected : 40320 ... [2021-02-12 21:42:34,398] - [basetestcase] [140099341772544] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.118 port:8091 ssh_username:root] [2021-02-12 21:42:34,405] - [rest_client] [140099341772544] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-12 21:42:34,423] - [basetestcase] [140099341772544] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.118 port:8091 ssh_username:root] [2021-02-12 21:42:34,427] - [rest_client] [140099341772544] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-12 21:42:34,430] - [basetestcase] [140099341772544] - INFO - sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_4 to complete bucket operations... Current : 40311 Expected : 40320 ... [2021-02-12 21:43:04,468] - [basetestcase] [140099341772544] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.118 port:8091 ssh_username:root] [2021-02-12 21:43:04,474] - [rest_client] [140099341772544] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-12 21:43:04,490] - [basetestcase] [140099341772544] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.118 port:8091 ssh_username:root] [2021-02-12 21:43:04,496] - [rest_client] [140099341772544] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-12 21:43:04,500] - [basetestcase] [140099341772544] - INFO - sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_4 to complete bucket operations... Current : 40311 Expected : 40320 ... [2021-02-12 21:43:34,537] - [basetestcase] [140099341772544] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.118 port:8091 ssh_username:root] [2021-02-12 21:43:34,543] - [rest_client] [140099341772544] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-12 21:43:34,561] - [basetestcase] [140099341772544] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.118 port:8091 ssh_username:root] [2021-02-12 21:43:34,567] - [rest_client] [140099341772544] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-12 21:43:34,570] - [basetestcase] [140099341772544] - INFO - sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_4 to complete bucket operations... Current : 40311 Expected : 40320 ... [2021-02-12 21:44:04,620] - [basetestcase] [140099341772544] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.118 port:8091 ssh_username:root] [2021-02-12 21:44:04,626] - [rest_client] [140099341772544] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-12 21:44:04,642] - [basetestcase] [140099341772544] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.118 port:8091 ssh_username:root] [2021-02-12 21:44:04,647] - [rest_client] [140099341772544] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-12 21:44:04,650] - [basetestcase] [140099341772544] - INFO - sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_4 to complete bucket operations... Current : 40311 Expected : 40320 ... [2021-02-12 21:44:34,685] - [basetestcase] [140099341772544] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.118 port:8091 ssh_username:root] [2021-02-12 21:44:34,692] - [rest_client] [140099341772544] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-12 21:44:34,707] - [eventing_base] [140099341772544] - INFO - Compacting bucket : dst_bucket [2021-02-12 21:44:34,714] - [rest_client] [140099341772544] - INFO - bucket compaction successful [2021-02-12 21:44:34,714] - [eventing_base] [140099341772544] - INFO - Compacting bucket : metadata [2021-02-12 21:44:34,719] - [rest_client] [140099341772544] - INFO - bucket compaction successful [2021-02-12 21:44:34,719] - [eventing_base] [140099341772544] - INFO - Compacting bucket : src_bucket [2021-02-12 21:44:34,728] - [rest_client] [140099341772544] - INFO - bucket compaction successful [2021-02-12 21:44:34,742] - [basetestcase] [140099341772544] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.118 port:8091 ssh_username:root] [2021-02-12 21:44:34,748] - [rest_client] [140099341772544] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-12 21:44:34,751] - [basetestcase] [140099341772544] - INFO - sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_4 to complete bucket operations... Current : 40311 Expected : 40320 ... [2021-02-12 21:45:04,800] - [basetestcase] [140099341772544] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.118 port:8091 ssh_username:root] [2021-02-12 21:45:04,809] - [rest_client] [140099341772544] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-12 21:45:04,825] - [basetestcase] [140099341772544] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.118 port:8091 ssh_username:root] [2021-02-12 21:45:04,830] - [rest_client] [140099341772544] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-12 21:45:04,833] - [basetestcase] [140099341772544] - INFO - sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_4 to complete bucket operations... Current : 40311 Expected : 40320 ... [2021-02-12 21:45:34,883] - [basetestcase] [140099341772544] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.118 port:8091 ssh_username:root] [2021-02-12 21:45:34,890] - [rest_client] [140099341772544] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-12 21:45:34,908] - [basetestcase] [140099341772544] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.118 port:8091 ssh_username:root] [2021-02-12 21:45:34,914] - [rest_client] [140099341772544] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-12 21:45:34,917] - [basetestcase] [140099341772544] - INFO - sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_4 to complete bucket operations... Current : 40311 Expected : 40320 ... [2021-02-12 21:46:04,968] - [basetestcase] [140099341772544] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.118 port:8091 ssh_username:root] [2021-02-12 21:46:04,974] - [rest_client] [140099341772544] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-12 21:46:04,990] - [basetestcase] [140099341772544] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.118 port:8091 ssh_username:root] [2021-02-12 21:46:04,995] - [rest_client] [140099341772544] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-12 21:46:04,998] - [basetestcase] [140099341772544] - INFO - sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_4 to complete bucket operations... Current : 40311 Expected : 40320 ... [2021-02-12 21:46:35,046] - [basetestcase] [140099341772544] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.118 port:8091 ssh_username:root] [2021-02-12 21:46:35,052] - [rest_client] [140099341772544] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-12 21:46:35,069] - [basetestcase] [140099341772544] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.118 port:8091 ssh_username:root] [2021-02-12 21:46:35,074] - [rest_client] [140099341772544] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-12 21:46:35,077] - [basetestcase] [140099341772544] - INFO - sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_4 to complete bucket operations... Current : 40311 Expected : 40320 ... [2021-02-12 21:47:05,107] - [basetestcase] [140099341772544] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.118 port:8091 ssh_username:root] [2021-02-12 21:47:05,114] - [rest_client] [140099341772544] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-12 21:47:05,131] - [basetestcase] [140099341772544] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.118 port:8091 ssh_username:root] [2021-02-12 21:47:05,136] - [rest_client] [140099341772544] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-12 21:47:05,141] - [basetestcase] [140099341772544] - INFO - sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_4 to complete bucket operations... Current : 40311 Expected : 40320 ... [2021-02-12 21:47:35,191] - [basetestcase] [140099341772544] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.118 port:8091 ssh_username:root] [2021-02-12 21:47:35,198] - [rest_client] [140099341772544] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-12 21:47:35,217] - [basetestcase] [140099341772544] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.118 port:8091 ssh_username:root] [2021-02-12 21:47:35,223] - [rest_client] [140099341772544] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-12 21:47:35,226] - [basetestcase] [140099341772544] - INFO - sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_4 to complete bucket operations... Current : 40311 Expected : 40320 ... [2021-02-12 21:48:05,279] - [basetestcase] [140099341772544] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.118 port:8091 ssh_username:root] [2021-02-12 21:48:05,285] - [rest_client] [140099341772544] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-12 21:48:05,302] - [basetestcase] [140099341772544] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.118 port:8091 ssh_username:root] [2021-02-12 21:48:05,306] - [rest_client] [140099341772544] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-12 21:48:05,310] - [basetestcase] [140099341772544] - INFO - sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_4 to complete bucket operations... Current : 40311 Expected : 40320 ... [2021-02-12 21:48:35,360] - [basetestcase] [140099341772544] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.118 port:8091 ssh_username:root] [2021-02-12 21:48:35,367] - [rest_client] [140099341772544] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-12 21:48:35,384] - [basetestcase] [140099341772544] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.118 port:8091 ssh_username:root] [2021-02-12 21:48:35,388] - [rest_client] [140099341772544] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-12 21:48:35,391] - [basetestcase] [140099341772544] - INFO - sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_4 to complete bucket operations... Current : 40311 Expected : 40320 ... [2021-02-12 21:49:05,443] - [basetestcase] [140099341772544] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.118 port:8091 ssh_username:root] [2021-02-12 21:49:05,449] - [rest_client] [140099341772544] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-12 21:49:05,468] - [basetestcase] [140099341772544] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.118 port:8091 ssh_username:root] [2021-02-12 21:49:05,473] - [rest_client] [140099341772544] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-12 21:49:05,484] - [basetestcase] [140099341772544] - INFO - list of eventing nodes in cluster: [ip:172.23.122.117 port:8091 ssh_username:root, ip:172.23.122.82 port:8091 ssh_username:root, ip:172.23.122.97 port:8091 ssh_username:root] [2021-02-12 21:49:05,533] - [eventing_base] [140099341772544] - INFO - Stats for Node 172.23.122.117 is [ { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 2, "agg_messages_sent_to_worker": 18005, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 40516, "dcp_mutation_sent_to_worker": 13499, "dcp_snapshot": 40699, "dcp_stream_req_counter": 342, "dcp_streamreq": 342, "dcp_system_event": 3420, "execution_stats": 831, "failure_stats": 831, "latency_stats": 831, "lcb_exception_stats": 831, "log_level": 1, "num_processed_events": 13499, "processed_events_size": 12086936, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 13499, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 13499, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13499, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13499, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 18001, "num_processed_events": 13499, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13499, "processed_events_size": 12086936, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "16937": "2021-02-13T05:49:05Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "16937": "2021-02-13T05:49:05Z" }, "v8worker_events_lost": 0 }, "function_id": 3301735981, "function_name": "curl", "gocb_creds_request_counter": 98, "internal_vb_distribution_stats": { "worker_curl_0": "[0-341]" }, "latency_percentile_stats": { "100": 92400, "50": 1400, "80": 1700, "90": 1900, "95": 2200, "99": 3500 }, "lcb_creds_request_counter": 27076, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.82:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.97:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_curl_0": "[0-341]" }, "172.23.122.82:8096": { "worker_curl_0": "[342-682]" }, "172.23.122.97:8096": { "worker_curl_0": "[683-1023]" } }, "worker_pids": { "worker_curl_0": 16937 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 2, "agg_messages_sent_to_worker": 17894, "agg_queue_memory": 0, "agg_queue_memory_cap": 20971520, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 40516, "dcp_mutation_sent_to_worker": 13499, "dcp_snapshot": 40700, "dcp_stream_req_counter": 342, "dcp_streamreq": 342, "dcp_system_event": 3420, "execution_stats": 809, "failure_stats": 809, "latency_stats": 809, "lcb_exception_stats": 809, "log_level": 1, "num_processed_events": 13499, "processed_events_size": 12086936, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13499, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13499, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 17890, "num_processed_events": 13499, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13499, "processed_events_size": 12086936, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "16978": "2021-02-13T05:49:04Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "16978": "2021-02-13T05:49:04Z" }, "v8worker_events_lost": 0 }, "function_id": 1994061678, "function_name": "n1ql", "gocb_creds_request_counter": 98, "internal_vb_distribution_stats": { "worker_n1ql_0": "[0-341]" }, "latency_percentile_stats": { "100": 106800, "50": 3200, "80": 4300, "90": 5300, "95": 6600, "99": 11400 }, "lcb_creds_request_counter": 27076, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.82:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.97:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_n1ql_0": "[0-341]" }, "172.23.122.82:8096": { "worker_n1ql_0": "[342-682]" }, "172.23.122.97:8096": { "worker_n1ql_0": "[683-1023]" } }, "worker_pids": { "worker_n1ql_0": 16978 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 3, "agg_messages_sent_to_worker": 18328, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 40516, "dcp_mutation_sent_to_worker": 13499, "dcp_snapshot": 40712, "dcp_stream_req_counter": 342, "dcp_streamreq": 342, "dcp_system_event": 3420, "execution_stats": 895, "failure_stats": 895, "latency_stats": 895, "lcb_exception_stats": 895, "log_level": 1, "num_processed_events": 13499, "processed_events_size": 12086936, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13499, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13499, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 18324, "num_processed_events": 13499, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13499, "processed_events_size": 12086936, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "16782": "2021-02-13T05:49:04Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "16782": "2021-02-13T05:49:04Z" }, "v8worker_events_lost": 0 }, "function_id": 881891323, "function_name": "bucket_op", "gocb_creds_request_counter": 98, "internal_vb_distribution_stats": { "worker_bucket_op_0": "[0-341]" }, "latency_percentile_stats": { "100": 18300, "50": 500, "80": 800, "90": 1000, "95": 1200, "99": 2200 }, "lcb_creds_request_counter": 27076, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.82:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.97:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_bucket_op_0": "[0-341]" }, "172.23.122.82:8096": { "worker_bucket_op_0": "[342-682]" }, "172.23.122.97:8096": { "worker_bucket_op_0": "[683-1023]" } }, "worker_pids": { "worker_bucket_op_0": 16782 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 3, "agg_messages_sent_to_worker": 18222, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 40516, "dcp_mutation_sent_to_worker": 13499, "dcp_snapshot": 40714, "dcp_stream_req_counter": 342, "dcp_streamreq": 342, "dcp_system_event": 3420, "execution_stats": 874, "failure_stats": 874, "latency_stats": 874, "lcb_exception_stats": 874, "log_level": 1, "num_processed_events": 13499, "processed_events_size": 12086936, "thr_count": 1, "thr_map": 1, "timer_events": 13528, "timer_responses_received": 13499, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13499, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13499, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 18218, "num_processed_events": 13499, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13499, "processed_events_size": 12086936, "timer_callback_failure": 0, "timer_callback_success": 13528, "timer_cancel_counter": 0, "timer_create_counter": 13499, "timer_create_failure": 0, "timer_msg_counter": 13528, "timer_responses_sent": 0, "timestamp": { "16822": "2021-02-13T05:49:05Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "16822": "2021-02-13T05:49:05Z" }, "v8worker_events_lost": 0 }, "function_id": 2473559842, "function_name": "timers", "gocb_creds_request_counter": 98, "internal_vb_distribution_stats": { "worker_timers_0": "[0-341]" }, "latency_percentile_stats": { "100": 16700, "50": 1500, "80": 1900, "90": 2300, "95": 2700, "99": 4400 }, "lcb_creds_request_counter": 27076, "lcb_exception_stats": {}, "metastore_stats": { "metastore_delete_err": 0, "metastore_deletes": 0, "metastore_not_found": 0, "metastore_scan": 0, "metastore_scan_due": 0, "metastore_scan_err": 0, "metastore_set": 0, "metastore_set_err": 0 }, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.82:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.97:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_timers_0": "[0-341]" }, "172.23.122.82:8096": { "worker_timers_0": "[342-682]" }, "172.23.122.97:8096": { "worker_timers_0": "[683-1023]" } }, "worker_pids": { "worker_timers_0": 16822 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 2, "agg_messages_sent_to_worker": 18111, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 40516, "dcp_mutation_sent_to_worker": 13499, "dcp_mutation_suppressed_counter": 13518, "dcp_snapshot": 40707, "dcp_stream_req_counter": 342, "dcp_streamreq": 342, "dcp_system_event": 3420, "execution_stats": 852, "failure_stats": 852, "latency_stats": 852, "lcb_exception_stats": 852, "log_level": 1, "num_processed_events": 13499, "processed_events_size": 12084340, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13499, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13499, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 18107, "num_processed_events": 13499, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13499, "processed_events_size": 12084340, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "16895": "2021-02-13T05:49:04Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "16895": "2021-02-13T05:49:04Z" }, "v8worker_events_lost": 0 }, "function_id": 327234242, "function_name": "sbm", "gocb_creds_request_counter": 98, "internal_vb_distribution_stats": { "worker_sbm_0": "[0-341]" }, "latency_percentile_stats": { "100": 15300, "50": 600, "80": 900, "90": 1200, "95": 1500, "99": 2900 }, "lcb_creds_request_counter": 27076, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.82:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.97:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_sbm_0": "[0-341]" }, "172.23.122.82:8096": { "worker_sbm_0": "[342-682]" }, "172.23.122.97:8096": { "worker_sbm_0": "[683-1023]" } }, "worker_pids": { "worker_sbm_0": 16895 } } ] [2021-02-12 21:49:05,578] - [eventing_base] [140099341772544] - INFO - Stats for Node 172.23.122.82 is [ { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 2, "agg_messages_sent_to_worker": 18260, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 40278, "dcp_mutation_sent_to_worker": 13432, "dcp_snapshot": 40428, "dcp_stream_req_counter": 341, "dcp_streamreq": 341, "dcp_system_event": 3410, "execution_stats": 895, "failure_stats": 895, "latency_stats": 895, "lcb_exception_stats": 895, "log_level": 1, "num_processed_events": 13432, "processed_events_size": 12024888, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13432, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13432, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 18256, "num_processed_events": 13432, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13432, "processed_events_size": 12024888, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "41748": "2021-02-13T05:49:04Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "41748": "2021-02-13T05:49:04Z" }, "v8worker_events_lost": 0 }, "function_id": 881891323, "function_name": "bucket_op", "gocb_creds_request_counter": 89, "internal_vb_distribution_stats": { "worker_bucket_op_0": "[342-682]" }, "latency_percentile_stats": { "100": 23100, "50": 600, "80": 900, "90": 1100, "95": 1400, "99": 2500 }, "lcb_creds_request_counter": 26954, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.82:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.97:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_bucket_op_0": "[0-341]" }, "172.23.122.82:8096": { "worker_bucket_op_0": "[342-682]" }, "172.23.122.97:8096": { "worker_bucket_op_0": "[683-1023]" } }, "worker_pids": { "worker_bucket_op_0": 41748 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 2, "agg_messages_sent_to_worker": 18154, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 40278, "dcp_mutation_sent_to_worker": 13432, "dcp_snapshot": 40439, "dcp_stream_req_counter": 341, "dcp_streamreq": 341, "dcp_system_event": 3410, "execution_stats": 874, "failure_stats": 874, "latency_stats": 874, "lcb_exception_stats": 874, "log_level": 1, "num_processed_events": 13432, "processed_events_size": 12024888, "thr_count": 1, "thr_map": 1, "timer_events": 13558, "timer_responses_received": 13432, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13432, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13432, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 18150, "num_processed_events": 13432, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13432, "processed_events_size": 12024888, "timer_callback_failure": 0, "timer_callback_success": 13558, "timer_cancel_counter": 0, "timer_create_counter": 13432, "timer_create_failure": 0, "timer_msg_counter": 13558, "timer_responses_sent": 0, "timestamp": { "41811": "2021-02-13T05:49:05Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "41811": "2021-02-13T05:49:05Z" }, "v8worker_events_lost": 0 }, "function_id": 2473559842, "function_name": "timers", "gocb_creds_request_counter": 89, "internal_vb_distribution_stats": { "worker_timers_0": "[342-682]" }, "latency_percentile_stats": { "100": 24900, "50": 1600, "80": 2100, "90": 2600, "95": 3100, "99": 5500 }, "lcb_creds_request_counter": 26954, "lcb_exception_stats": {}, "metastore_stats": { "metastore_delete_err": 0, "metastore_deletes": 0, "metastore_not_found": 0, "metastore_scan": 0, "metastore_scan_due": 0, "metastore_scan_err": 0, "metastore_set": 0, "metastore_set_err": 0 }, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.82:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.97:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_timers_0": "[0-341]" }, "172.23.122.82:8096": { "worker_timers_0": "[342-682]" }, "172.23.122.97:8096": { "worker_timers_0": "[683-1023]" } }, "worker_pids": { "worker_timers_0": 41811 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 2, "agg_messages_sent_to_worker": 18043, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 40278, "dcp_mutation_sent_to_worker": 13432, "dcp_mutation_suppressed_counter": 13414, "dcp_snapshot": 40407, "dcp_stream_req_counter": 341, "dcp_streamreq": 341, "dcp_system_event": 3410, "execution_stats": 852, "failure_stats": 852, "latency_stats": 852, "lcb_exception_stats": 852, "log_level": 1, "num_processed_events": 13432, "processed_events_size": 12026424, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13432, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13432, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 18039, "num_processed_events": 13432, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13432, "processed_events_size": 12026424, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "41844": "2021-02-13T05:49:04Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "41844": "2021-02-13T05:49:04Z" }, "v8worker_events_lost": 0 }, "function_id": 327234242, "function_name": "sbm", "gocb_creds_request_counter": 89, "internal_vb_distribution_stats": { "worker_sbm_0": "[342-682]" }, "latency_percentile_stats": { "100": 21600, "50": 700, "80": 1000, "90": 1300, "95": 1600, "99": 3000 }, "lcb_creds_request_counter": 26954, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.82:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.97:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_sbm_0": "[0-341]" }, "172.23.122.82:8096": { "worker_sbm_0": "[342-682]" }, "172.23.122.97:8096": { "worker_sbm_0": "[683-1023]" } }, "worker_pids": { "worker_sbm_0": 41844 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 1, "agg_messages_sent_to_worker": 17937, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 40278, "dcp_mutation_sent_to_worker": 13432, "dcp_snapshot": 40421, "dcp_stream_req_counter": 341, "dcp_streamreq": 341, "dcp_system_event": 3410, "execution_stats": 831, "failure_stats": 831, "latency_stats": 831, "lcb_exception_stats": 831, "log_level": 1, "num_processed_events": 13432, "processed_events_size": 12024888, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 13432, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 13432, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13432, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13432, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 17933, "num_processed_events": 13432, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13432, "processed_events_size": 12024888, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "41920": "2021-02-13T05:49:05Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "41920": "2021-02-13T05:49:05Z" }, "v8worker_events_lost": 0 }, "function_id": 3301735981, "function_name": "curl", "gocb_creds_request_counter": 89, "internal_vb_distribution_stats": { "worker_curl_0": "[342-682]" }, "latency_percentile_stats": { "100": 92400, "50": 1600, "80": 2000, "90": 2300, "95": 2800, "99": 5200 }, "lcb_creds_request_counter": 26954, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.82:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.97:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_curl_0": "[0-341]" }, "172.23.122.82:8096": { "worker_curl_0": "[342-682]" }, "172.23.122.97:8096": { "worker_curl_0": "[683-1023]" } }, "worker_pids": { "worker_curl_0": 41920 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 3, "agg_messages_sent_to_worker": 17826, "agg_queue_memory": 0, "agg_queue_memory_cap": 20971520, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 40278, "dcp_mutation_sent_to_worker": 13432, "dcp_snapshot": 40406, "dcp_stream_req_counter": 341, "dcp_streamreq": 341, "dcp_system_event": 3410, "execution_stats": 809, "failure_stats": 809, "latency_stats": 809, "lcb_exception_stats": 809, "log_level": 1, "num_processed_events": 13432, "processed_events_size": 12024888, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13432, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13432, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 17822, "num_processed_events": 13432, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13432, "processed_events_size": 12024888, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "41950": "2021-02-13T05:49:04Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "41950": "2021-02-13T05:49:04Z" }, "v8worker_events_lost": 0 }, "function_id": 1994061678, "function_name": "n1ql", "gocb_creds_request_counter": 89, "internal_vb_distribution_stats": { "worker_n1ql_0": "[342-682]" }, "latency_percentile_stats": { "100": 83800, "50": 3500, "80": 4900, "90": 6200, "95": 7700, "99": 12800 }, "lcb_creds_request_counter": 26954, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.82:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.97:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_n1ql_0": "[0-341]" }, "172.23.122.82:8096": { "worker_n1ql_0": "[342-682]" }, "172.23.122.97:8096": { "worker_n1ql_0": "[683-1023]" } }, "worker_pids": { "worker_n1ql_0": 41950 } } ] [2021-02-12 21:49:05,625] - [eventing_base] [140099341772544] - INFO - Stats for Node 172.23.122.97 is [ { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 1, "agg_messages_sent_to_worker": 18909, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 66943, "dcp_mutation_sent_to_worker": 13389, "dcp_snapshot": 40670, "dcp_stream_req_counter": 4771, "dcp_stream_req_err_counter": 3769, "dcp_streamend": 351, "dcp_streamreq": 1002, "dcp_system_event": 3410, "execution_stats": 895, "failure_stats": 895, "latency_stats": 895, "lcb_exception_stats": 895, "log_level": 1, "num_processed_events": 13389, "processed_events_size": 11986244, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13389, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13389, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 18905, "num_processed_events": 13389, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13389, "processed_events_size": 11986244, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "49851": "2021-02-13T05:49:04Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "49851": "2021-02-13T05:49:04Z" }, "v8worker_events_lost": 0 }, "function_id": 881891323, "function_name": "bucket_op", "gocb_creds_request_counter": 88, "internal_vb_distribution_stats": { "worker_bucket_op_0": "[683-1023]" }, "latency_percentile_stats": { "100": 21500, "50": 700, "80": 900, "90": 1100, "95": 1300, "99": 2300 }, "lcb_creds_request_counter": 26866, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.82:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.97:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_bucket_op_0": "[0-341]" }, "172.23.122.82:8096": { "worker_bucket_op_0": "[342-682]" }, "172.23.122.97:8096": { "worker_bucket_op_0": "[683-1023]" } }, "worker_pids": { "worker_bucket_op_0": 49851 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 2, "agg_messages_sent_to_worker": 18803, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 66943, "dcp_mutation_sent_to_worker": 13389, "dcp_snapshot": 40655, "dcp_stream_req_counter": 11043, "dcp_stream_req_err_counter": 10058, "dcp_streamend": 351, "dcp_streamreq": 985, "dcp_system_event": 3410, "execution_stats": 874, "failure_stats": 874, "latency_stats": 874, "lcb_exception_stats": 874, "log_level": 1, "num_processed_events": 13389, "processed_events_size": 11986244, "thr_count": 1, "thr_map": 1, "timer_events": 13234, "timer_responses_received": 13389, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13389, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13389, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 18799, "num_processed_events": 13389, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13389, "processed_events_size": 11986244, "timer_callback_failure": 0, "timer_callback_success": 13234, "timer_cancel_counter": 0, "timer_create_counter": 13389, "timer_create_failure": 0, "timer_msg_counter": 13234, "timer_responses_sent": 0, "timestamp": { "49884": "2021-02-13T05:49:05Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "49884": "2021-02-13T05:49:05Z" }, "v8worker_events_lost": 0 }, "function_id": 2473559842, "function_name": "timers", "gocb_creds_request_counter": 88, "internal_vb_distribution_stats": { "worker_timers_0": "[683-1023]" }, "latency_percentile_stats": { "100": 23300, "50": 1600, "80": 2100, "90": 2400, "95": 2900, "99": 4700 }, "lcb_creds_request_counter": 26866, "lcb_exception_stats": {}, "metastore_stats": { "metastore_delete_err": 0, "metastore_deletes": 0, "metastore_not_found": 0, "metastore_scan": 0, "metastore_scan_due": 0, "metastore_scan_err": 0, "metastore_set": 0, "metastore_set_err": 0 }, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.82:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.97:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_timers_0": "[0-341]" }, "172.23.122.82:8096": { "worker_timers_0": "[342-682]" }, "172.23.122.97:8096": { "worker_timers_0": "[683-1023]" } }, "worker_pids": { "worker_timers_0": 49884 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "agg_messages_sent_to_worker": 18692, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 40608, "dcp_mutation_sent_to_worker": 13389, "dcp_mutation_suppressed_counter": 13830, "dcp_snapshot": 40529, "dcp_stream_req_counter": 10404, "dcp_stream_req_err_counter": 9419, "dcp_streamend": 351, "dcp_streamreq": 985, "dcp_system_event": 3410, "execution_stats": 852, "failure_stats": 852, "latency_stats": 852, "lcb_exception_stats": 852, "log_level": 1, "num_processed_events": 13389, "processed_events_size": 11994668, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13389, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13389, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 18688, "num_processed_events": 13389, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13389, "processed_events_size": 11994668, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "49917": "2021-02-13T05:49:04Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "49917": "2021-02-13T05:49:04Z" }, "v8worker_events_lost": 0 }, "function_id": 327234242, "function_name": "sbm", "gocb_creds_request_counter": 88, "internal_vb_distribution_stats": { "worker_sbm_0": "[683-1023]" }, "latency_percentile_stats": { "100": 18600, "50": 700, "80": 1000, "90": 1200, "95": 1600, "99": 3000 }, "lcb_creds_request_counter": 26866, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.82:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.97:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_sbm_0": "[0-341]" }, "172.23.122.82:8096": { "worker_sbm_0": "[342-682]" }, "172.23.122.97:8096": { "worker_sbm_0": "[683-1023]" } }, "worker_pids": { "worker_sbm_0": 49917 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 2, "agg_messages_sent_to_worker": 18586, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 66943, "dcp_mutation_sent_to_worker": 13389, "dcp_snapshot": 40637, "dcp_stream_req_counter": 10254, "dcp_stream_req_err_counter": 9269, "dcp_streamend": 351, "dcp_streamreq": 985, "dcp_system_event": 3410, "execution_stats": 831, "failure_stats": 831, "latency_stats": 831, "lcb_exception_stats": 831, "log_level": 1, "num_processed_events": 13389, "processed_events_size": 11986244, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 13389, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 13389, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13389, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13389, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 18582, "num_processed_events": 13389, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13389, "processed_events_size": 11986244, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "49993": "2021-02-13T05:49:05Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "49993": "2021-02-13T05:49:05Z" }, "v8worker_events_lost": 0 }, "function_id": 3301735981, "function_name": "curl", "gocb_creds_request_counter": 88, "internal_vb_distribution_stats": { "worker_curl_0": "[683-1023]" }, "latency_percentile_stats": { "100": 92600, "50": 1500, "80": 1900, "90": 2200, "95": 2600, "99": 4200 }, "lcb_creds_request_counter": 26866, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.82:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.97:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_curl_0": "[0-341]" }, "172.23.122.82:8096": { "worker_curl_0": "[342-682]" }, "172.23.122.97:8096": { "worker_curl_0": "[683-1023]" } }, "worker_pids": { "worker_curl_0": 49993 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 3, "agg_messages_sent_to_worker": 18480, "agg_queue_memory": 0, "agg_queue_memory_cap": 20971520, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 66943, "dcp_mutation_sent_to_worker": 13389, "dcp_snapshot": 40639, "dcp_stream_req_counter": 5805, "dcp_stream_req_err_counter": 4820, "dcp_streamend": 351, "dcp_streamreq": 985, "dcp_system_event": 3410, "execution_stats": 810, "failure_stats": 810, "latency_stats": 810, "lcb_exception_stats": 810, "log_level": 1, "num_processed_events": 13389, "processed_events_size": 11986244, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13389, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13389, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 18476, "num_processed_events": 13389, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 9, "on_update_success": 13380, "processed_events_size": 11986244, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "50026": "2021-02-13T05:49:05Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 9, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "50026": "2021-02-13T05:49:05Z" }, "v8worker_events_lost": 0 }, "function_id": 1994061678, "function_name": "n1ql", "gocb_creds_request_counter": 88, "internal_vb_distribution_stats": { "worker_n1ql_0": "[683-1023]" }, "latency_percentile_stats": { "100": 107900, "50": 3400, "80": 4700, "90": 6000, "95": 7600, "99": 13800 }, "lcb_creds_request_counter": 26866, "lcb_exception_stats": { "12009": 9, "209": 9 }, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.82:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.97:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_n1ql_0": "[0-341]" }, "172.23.122.82:8096": { "worker_n1ql_0": "[342-682]" }, "172.23.122.97:8096": { "worker_n1ql_0": "[683-1023]" } }, "worker_pids": { "worker_n1ql_0": 50026 } } ] [2021-02-12 21:49:10,103] - [eventing_base] [140099341772544] - INFO - Stats for Node 172.23.122.117 is [ { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 3, "agg_messages_sent_to_worker": 18328, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 40516, "dcp_mutation_sent_to_worker": 13499, "dcp_snapshot": 40712, "dcp_stream_req_counter": 342, "dcp_streamreq": 342, "dcp_system_event": 3420, "execution_stats": 895, "failure_stats": 895, "latency_stats": 895, "lcb_exception_stats": 895, "log_level": 1, "num_processed_events": 13499, "processed_events_size": 12086936, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13499, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13499, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 18324, "num_processed_events": 13499, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13499, "processed_events_size": 12086936, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "16782": "2021-02-13T05:49:04Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "16782": "2021-02-13T05:49:04Z" }, "v8worker_events_lost": 0 }, "function_id": 881891323, "function_name": "bucket_op", "gocb_creds_request_counter": 98, "internal_vb_distribution_stats": { "worker_bucket_op_0": "[0-341]" }, "latency_percentile_stats": { "100": 18300, "50": 500, "80": 800, "90": 1000, "95": 1200, "99": 2200 }, "lcb_creds_request_counter": 27076, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.82:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.97:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_bucket_op_0": "[0-341]" }, "172.23.122.82:8096": { "worker_bucket_op_0": "[342-682]" }, "172.23.122.97:8096": { "worker_bucket_op_0": "[683-1023]" } }, "worker_pids": { "worker_bucket_op_0": 16782 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 3, "agg_messages_sent_to_worker": 18222, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 40516, "dcp_mutation_sent_to_worker": 13499, "dcp_snapshot": 40714, "dcp_stream_req_counter": 342, "dcp_streamreq": 342, "dcp_system_event": 3420, "execution_stats": 874, "failure_stats": 874, "latency_stats": 874, "lcb_exception_stats": 874, "log_level": 1, "num_processed_events": 13499, "processed_events_size": 12086936, "thr_count": 1, "thr_map": 1, "timer_events": 13528, "timer_responses_received": 13499, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13499, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13499, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 18218, "num_processed_events": 13499, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13499, "processed_events_size": 12086936, "timer_callback_failure": 0, "timer_callback_success": 13528, "timer_cancel_counter": 0, "timer_create_counter": 13499, "timer_create_failure": 0, "timer_msg_counter": 13528, "timer_responses_sent": 0, "timestamp": { "16822": "2021-02-13T05:49:05Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "16822": "2021-02-13T05:49:05Z" }, "v8worker_events_lost": 0 }, "function_id": 2473559842, "function_name": "timers", "gocb_creds_request_counter": 98, "internal_vb_distribution_stats": { "worker_timers_0": "[0-341]" }, "latency_percentile_stats": { "100": 16700, "50": 1500, "80": 1900, "90": 2300, "95": 2700, "99": 4400 }, "lcb_creds_request_counter": 27076, "lcb_exception_stats": {}, "metastore_stats": { "metastore_delete_err": 0, "metastore_deletes": 0, "metastore_not_found": 0, "metastore_scan": 0, "metastore_scan_due": 0, "metastore_scan_err": 0, "metastore_set": 0, "metastore_set_err": 0 }, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.82:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.97:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_timers_0": "[0-341]" }, "172.23.122.82:8096": { "worker_timers_0": "[342-682]" }, "172.23.122.97:8096": { "worker_timers_0": "[683-1023]" } }, "worker_pids": { "worker_timers_0": 16822 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 2, "agg_messages_sent_to_worker": 18111, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 40516, "dcp_mutation_sent_to_worker": 13499, "dcp_mutation_suppressed_counter": 13518, "dcp_snapshot": 40707, "dcp_stream_req_counter": 342, "dcp_streamreq": 342, "dcp_system_event": 3420, "execution_stats": 852, "failure_stats": 852, "latency_stats": 852, "lcb_exception_stats": 852, "log_level": 1, "num_processed_events": 13499, "processed_events_size": 12084340, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13499, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13499, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 18107, "num_processed_events": 13499, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13499, "processed_events_size": 12084340, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "16895": "2021-02-13T05:49:04Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "16895": "2021-02-13T05:49:04Z" }, "v8worker_events_lost": 0 }, "function_id": 327234242, "function_name": "sbm", "gocb_creds_request_counter": 98, "internal_vb_distribution_stats": { "worker_sbm_0": "[0-341]" }, "latency_percentile_stats": { "100": 15300, "50": 600, "80": 900, "90": 1200, "95": 1500, "99": 2900 }, "lcb_creds_request_counter": 27076, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.82:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.97:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_sbm_0": "[0-341]" }, "172.23.122.82:8096": { "worker_sbm_0": "[342-682]" }, "172.23.122.97:8096": { "worker_sbm_0": "[683-1023]" } }, "worker_pids": { "worker_sbm_0": 16895 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 2, "agg_messages_sent_to_worker": 18005, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 40516, "dcp_mutation_sent_to_worker": 13499, "dcp_snapshot": 40699, "dcp_stream_req_counter": 342, "dcp_streamreq": 342, "dcp_system_event": 3420, "execution_stats": 831, "failure_stats": 831, "latency_stats": 831, "lcb_exception_stats": 831, "log_level": 1, "num_processed_events": 13499, "processed_events_size": 12086936, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 13499, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 13499, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13499, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13499, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 18001, "num_processed_events": 13499, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13499, "processed_events_size": 12086936, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "16937": "2021-02-13T05:49:05Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "16937": "2021-02-13T05:49:05Z" }, "v8worker_events_lost": 0 }, "function_id": 3301735981, "function_name": "curl", "gocb_creds_request_counter": 98, "internal_vb_distribution_stats": { "worker_curl_0": "[0-341]" }, "latency_percentile_stats": { "100": 92400, "50": 1400, "80": 1700, "90": 1900, "95": 2200, "99": 3500 }, "lcb_creds_request_counter": 27076, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.82:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.97:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_curl_0": "[0-341]" }, "172.23.122.82:8096": { "worker_curl_0": "[342-682]" }, "172.23.122.97:8096": { "worker_curl_0": "[683-1023]" } }, "worker_pids": { "worker_curl_0": 16937 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 2, "agg_messages_sent_to_worker": 17899, "agg_queue_memory": 0, "agg_queue_memory_cap": 20971520, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 40516, "dcp_mutation_sent_to_worker": 13499, "dcp_snapshot": 40700, "dcp_stream_req_counter": 342, "dcp_streamreq": 342, "dcp_system_event": 3420, "execution_stats": 810, "failure_stats": 810, "latency_stats": 810, "lcb_exception_stats": 810, "log_level": 1, "num_processed_events": 13499, "processed_events_size": 12086936, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13499, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13499, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 17895, "num_processed_events": 13499, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13499, "processed_events_size": 12086936, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "16978": "2021-02-13T05:49:05Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "16978": "2021-02-13T05:49:05Z" }, "v8worker_events_lost": 0 }, "function_id": 1994061678, "function_name": "n1ql", "gocb_creds_request_counter": 98, "internal_vb_distribution_stats": { "worker_n1ql_0": "[0-341]" }, "latency_percentile_stats": { "100": 106800, "50": 3200, "80": 4300, "90": 5300, "95": 6600, "99": 11400 }, "lcb_creds_request_counter": 27076, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.82:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.97:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_n1ql_0": "[0-341]" }, "172.23.122.82:8096": { "worker_n1ql_0": "[342-682]" }, "172.23.122.97:8096": { "worker_n1ql_0": "[683-1023]" } }, "worker_pids": { "worker_n1ql_0": 16978 } } ] [2021-02-12 21:49:15,295] - [eventing_base] [140099341772544] - INFO - Stats for Node 172.23.122.82 is [ { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 2, "agg_messages_sent_to_worker": 18073, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 40278, "dcp_mutation_sent_to_worker": 13432, "dcp_mutation_suppressed_counter": 13414, "dcp_snapshot": 40407, "dcp_stream_req_counter": 341, "dcp_streamreq": 341, "dcp_system_event": 3410, "execution_stats": 858, "failure_stats": 858, "latency_stats": 858, "lcb_exception_stats": 858, "log_level": 1, "num_processed_events": 13432, "processed_events_size": 12026424, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13432, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13432, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 18069, "num_processed_events": 13432, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13432, "processed_events_size": 12026424, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "41844": "2021-02-13T05:49:10Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "41844": "2021-02-13T05:49:10Z" }, "v8worker_events_lost": 0 }, "function_id": 327234242, "function_name": "sbm", "gocb_creds_request_counter": 89, "internal_vb_distribution_stats": { "worker_sbm_0": "[342-682]" }, "latency_percentile_stats": { "100": 21600, "50": 700, "80": 1000, "90": 1300, "95": 1600, "99": 3000 }, "lcb_creds_request_counter": 26954, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.82:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.97:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_sbm_0": "[0-341]" }, "172.23.122.82:8096": { "worker_sbm_0": "[342-682]" }, "172.23.122.97:8096": { "worker_sbm_0": "[683-1023]" } }, "worker_pids": { "worker_sbm_0": 41844 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 1, "agg_messages_sent_to_worker": 17962, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 40278, "dcp_mutation_sent_to_worker": 13432, "dcp_snapshot": 40421, "dcp_stream_req_counter": 341, "dcp_streamreq": 341, "dcp_system_event": 3410, "execution_stats": 836, "failure_stats": 836, "latency_stats": 836, "lcb_exception_stats": 836, "log_level": 1, "num_processed_events": 13432, "processed_events_size": 12024888, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 13432, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 13432, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13432, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13432, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 17958, "num_processed_events": 13432, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13432, "processed_events_size": 12024888, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "41920": "2021-02-13T05:49:10Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "41920": "2021-02-13T05:49:10Z" }, "v8worker_events_lost": 0 }, "function_id": 3301735981, "function_name": "curl", "gocb_creds_request_counter": 89, "internal_vb_distribution_stats": { "worker_curl_0": "[342-682]" }, "latency_percentile_stats": { "100": 92400, "50": 1600, "80": 2000, "90": 2300, "95": 2800, "99": 5200 }, "lcb_creds_request_counter": 26954, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.82:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.97:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_curl_0": "[0-341]" }, "172.23.122.82:8096": { "worker_curl_0": "[342-682]" }, "172.23.122.97:8096": { "worker_curl_0": "[683-1023]" } }, "worker_pids": { "worker_curl_0": 41920 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 3, "agg_messages_sent_to_worker": 17856, "agg_queue_memory": 0, "agg_queue_memory_cap": 20971520, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 40278, "dcp_mutation_sent_to_worker": 13432, "dcp_snapshot": 40406, "dcp_stream_req_counter": 341, "dcp_streamreq": 341, "dcp_system_event": 3410, "execution_stats": 815, "failure_stats": 815, "latency_stats": 815, "lcb_exception_stats": 815, "log_level": 1, "num_processed_events": 13432, "processed_events_size": 12024888, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13432, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13432, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 17852, "num_processed_events": 13432, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13432, "processed_events_size": 12024888, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "41950": "2021-02-13T05:49:10Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "41950": "2021-02-13T05:49:10Z" }, "v8worker_events_lost": 0 }, "function_id": 1994061678, "function_name": "n1ql", "gocb_creds_request_counter": 89, "internal_vb_distribution_stats": { "worker_n1ql_0": "[342-682]" }, "latency_percentile_stats": { "100": 83800, "50": 3500, "80": 4900, "90": 6200, "95": 7700, "99": 12800 }, "lcb_creds_request_counter": 26954, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.82:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.97:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_n1ql_0": "[0-341]" }, "172.23.122.82:8096": { "worker_n1ql_0": "[342-682]" }, "172.23.122.97:8096": { "worker_n1ql_0": "[683-1023]" } }, "worker_pids": { "worker_n1ql_0": 41950 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 2, "agg_messages_sent_to_worker": 18285, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 40278, "dcp_mutation_sent_to_worker": 13432, "dcp_snapshot": 40428, "dcp_stream_req_counter": 341, "dcp_streamreq": 341, "dcp_system_event": 3410, "execution_stats": 900, "failure_stats": 900, "latency_stats": 900, "lcb_exception_stats": 900, "log_level": 1, "num_processed_events": 13432, "processed_events_size": 12024888, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13432, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13432, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 18281, "num_processed_events": 13432, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13432, "processed_events_size": 12024888, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "41748": "2021-02-13T05:49:09Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "41748": "2021-02-13T05:49:09Z" }, "v8worker_events_lost": 0 }, "function_id": 881891323, "function_name": "bucket_op", "gocb_creds_request_counter": 89, "internal_vb_distribution_stats": { "worker_bucket_op_0": "[342-682]" }, "latency_percentile_stats": { "100": 23100, "50": 600, "80": 900, "90": 1100, "95": 1400, "99": 2500 }, "lcb_creds_request_counter": 26954, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.82:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.97:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_bucket_op_0": "[0-341]" }, "172.23.122.82:8096": { "worker_bucket_op_0": "[342-682]" }, "172.23.122.97:8096": { "worker_bucket_op_0": "[683-1023]" } }, "worker_pids": { "worker_bucket_op_0": 41748 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 2, "agg_messages_sent_to_worker": 18179, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 40278, "dcp_mutation_sent_to_worker": 13432, "dcp_snapshot": 40439, "dcp_stream_req_counter": 341, "dcp_streamreq": 341, "dcp_system_event": 3410, "execution_stats": 879, "failure_stats": 879, "latency_stats": 879, "lcb_exception_stats": 879, "log_level": 1, "num_processed_events": 13432, "processed_events_size": 12024888, "thr_count": 1, "thr_map": 1, "timer_events": 13558, "timer_responses_received": 13432, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13432, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13432, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 18175, "num_processed_events": 13432, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13432, "processed_events_size": 12024888, "timer_callback_failure": 0, "timer_callback_success": 13558, "timer_cancel_counter": 0, "timer_create_counter": 13432, "timer_create_failure": 0, "timer_msg_counter": 13558, "timer_responses_sent": 0, "timestamp": { "41811": "2021-02-13T05:49:10Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "41811": "2021-02-13T05:49:10Z" }, "v8worker_events_lost": 0 }, "function_id": 2473559842, "function_name": "timers", "gocb_creds_request_counter": 89, "internal_vb_distribution_stats": { "worker_timers_0": "[342-682]" }, "latency_percentile_stats": { "100": 24900, "50": 1600, "80": 2100, "90": 2600, "95": 3100, "99": 5500 }, "lcb_creds_request_counter": 26954, "lcb_exception_stats": {}, "metastore_stats": { "metastore_delete_err": 0, "metastore_deletes": 0, "metastore_not_found": 0, "metastore_scan": 0, "metastore_scan_due": 0, "metastore_scan_err": 0, "metastore_set": 0, "metastore_set_err": 0 }, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.82:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.97:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_timers_0": "[0-341]" }, "172.23.122.82:8096": { "worker_timers_0": "[342-682]" }, "172.23.122.97:8096": { "worker_timers_0": "[683-1023]" } }, "worker_pids": { "worker_timers_0": 41811 } } ] [2021-02-12 21:49:20,203] - [eventing_base] [140099341772544] - INFO - Stats for Node 172.23.122.97 is [ { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 2, "agg_messages_sent_to_worker": 18636, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 66943, "dcp_mutation_sent_to_worker": 13389, "dcp_snapshot": 40637, "dcp_stream_req_counter": 10254, "dcp_stream_req_err_counter": 9269, "dcp_streamend": 351, "dcp_streamreq": 985, "dcp_system_event": 3410, "execution_stats": 841, "failure_stats": 841, "latency_stats": 841, "lcb_exception_stats": 841, "log_level": 1, "num_processed_events": 13389, "processed_events_size": 11986244, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 13389, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 13389, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13389, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13389, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 18632, "num_processed_events": 13389, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13389, "processed_events_size": 11986244, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "49993": "2021-02-13T05:49:15Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "49993": "2021-02-13T05:49:15Z" }, "v8worker_events_lost": 0 }, "function_id": 3301735981, "function_name": "curl", "gocb_creds_request_counter": 88, "internal_vb_distribution_stats": { "worker_curl_0": "[683-1023]" }, "latency_percentile_stats": { "100": 92600, "50": 1500, "80": 1900, "90": 2200, "95": 2600, "99": 4200 }, "lcb_creds_request_counter": 26866, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.82:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.97:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_curl_0": "[0-341]" }, "172.23.122.82:8096": { "worker_curl_0": "[342-682]" }, "172.23.122.97:8096": { "worker_curl_0": "[683-1023]" } }, "worker_pids": { "worker_curl_0": 49993 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 3, "agg_messages_sent_to_worker": 18530, "agg_queue_memory": 0, "agg_queue_memory_cap": 20971520, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 66943, "dcp_mutation_sent_to_worker": 13389, "dcp_snapshot": 40639, "dcp_stream_req_counter": 5805, "dcp_stream_req_err_counter": 4820, "dcp_streamend": 351, "dcp_streamreq": 985, "dcp_system_event": 3410, "execution_stats": 820, "failure_stats": 820, "latency_stats": 820, "lcb_exception_stats": 820, "log_level": 1, "num_processed_events": 13389, "processed_events_size": 11986244, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13389, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13389, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 18526, "num_processed_events": 13389, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 9, "on_update_success": 13380, "processed_events_size": 11986244, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "50026": "2021-02-13T05:49:15Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 9, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "50026": "2021-02-13T05:49:15Z" }, "v8worker_events_lost": 0 }, "function_id": 1994061678, "function_name": "n1ql", "gocb_creds_request_counter": 88, "internal_vb_distribution_stats": { "worker_n1ql_0": "[683-1023]" }, "latency_percentile_stats": { "100": 107900, "50": 3400, "80": 4700, "90": 6000, "95": 7600, "99": 13800 }, "lcb_creds_request_counter": 26866, "lcb_exception_stats": { "12009": 9, "209": 9 }, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.82:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.97:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_n1ql_0": "[0-341]" }, "172.23.122.82:8096": { "worker_n1ql_0": "[342-682]" }, "172.23.122.97:8096": { "worker_n1ql_0": "[683-1023]" } }, "worker_pids": { "worker_n1ql_0": 50026 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 1, "agg_messages_sent_to_worker": 18964, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 66943, "dcp_mutation_sent_to_worker": 13389, "dcp_snapshot": 40670, "dcp_stream_req_counter": 4771, "dcp_stream_req_err_counter": 3769, "dcp_streamend": 351, "dcp_streamreq": 1002, "dcp_system_event": 3410, "execution_stats": 906, "failure_stats": 906, "latency_stats": 906, "lcb_exception_stats": 906, "log_level": 1, "num_processed_events": 13389, "processed_events_size": 11986244, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13389, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13389, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 18960, "num_processed_events": 13389, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13389, "processed_events_size": 11986244, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "49851": "2021-02-13T05:49:15Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "49851": "2021-02-13T05:49:15Z" }, "v8worker_events_lost": 0 }, "function_id": 881891323, "function_name": "bucket_op", "gocb_creds_request_counter": 88, "internal_vb_distribution_stats": { "worker_bucket_op_0": "[683-1023]" }, "latency_percentile_stats": { "100": 21500, "50": 700, "80": 900, "90": 1100, "95": 1300, "99": 2300 }, "lcb_creds_request_counter": 26866, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.82:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.97:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_bucket_op_0": "[0-341]" }, "172.23.122.82:8096": { "worker_bucket_op_0": "[342-682]" }, "172.23.122.97:8096": { "worker_bucket_op_0": "[683-1023]" } }, "worker_pids": { "worker_bucket_op_0": 49851 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 2, "agg_messages_sent_to_worker": 18853, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 66943, "dcp_mutation_sent_to_worker": 13389, "dcp_snapshot": 40655, "dcp_stream_req_counter": 11043, "dcp_stream_req_err_counter": 10058, "dcp_streamend": 351, "dcp_streamreq": 985, "dcp_system_event": 3410, "execution_stats": 884, "failure_stats": 884, "latency_stats": 884, "lcb_exception_stats": 884, "log_level": 1, "num_processed_events": 13389, "processed_events_size": 11986244, "thr_count": 1, "thr_map": 1, "timer_events": 13234, "timer_responses_received": 13389, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13389, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13389, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 18849, "num_processed_events": 13389, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13389, "processed_events_size": 11986244, "timer_callback_failure": 0, "timer_callback_success": 13234, "timer_cancel_counter": 0, "timer_create_counter": 13389, "timer_create_failure": 0, "timer_msg_counter": 13234, "timer_responses_sent": 0, "timestamp": { "49884": "2021-02-13T05:49:15Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "49884": "2021-02-13T05:49:15Z" }, "v8worker_events_lost": 0 }, "function_id": 2473559842, "function_name": "timers", "gocb_creds_request_counter": 88, "internal_vb_distribution_stats": { "worker_timers_0": "[683-1023]" }, "latency_percentile_stats": { "100": 23300, "50": 1600, "80": 2100, "90": 2400, "95": 2900, "99": 4700 }, "lcb_creds_request_counter": 26866, "lcb_exception_stats": {}, "metastore_stats": { "metastore_delete_err": 0, "metastore_deletes": 0, "metastore_not_found": 0, "metastore_scan": 0, "metastore_scan_due": 0, "metastore_scan_err": 0, "metastore_set": 0, "metastore_set_err": 0 }, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.82:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.97:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_timers_0": "[0-341]" }, "172.23.122.82:8096": { "worker_timers_0": "[342-682]" }, "172.23.122.97:8096": { "worker_timers_0": "[683-1023]" } }, "worker_pids": { "worker_timers_0": 49884 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "agg_messages_sent_to_worker": 18747, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 40608, "dcp_mutation_sent_to_worker": 13389, "dcp_mutation_suppressed_counter": 13830, "dcp_snapshot": 40529, "dcp_stream_req_counter": 10404, "dcp_stream_req_err_counter": 9419, "dcp_streamend": 351, "dcp_streamreq": 985, "dcp_system_event": 3410, "execution_stats": 863, "failure_stats": 863, "latency_stats": 863, "lcb_exception_stats": 863, "log_level": 1, "num_processed_events": 13389, "processed_events_size": 11994668, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13389, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13389, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 18743, "num_processed_events": 13389, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13389, "processed_events_size": 11994668, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "49917": "2021-02-13T05:49:15Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "49917": "2021-02-13T05:49:15Z" }, "v8worker_events_lost": 0 }, "function_id": 327234242, "function_name": "sbm", "gocb_creds_request_counter": 88, "internal_vb_distribution_stats": { "worker_sbm_0": "[683-1023]" }, "latency_percentile_stats": { "100": 18600, "50": 700, "80": 1000, "90": 1200, "95": 1600, "99": 3000 }, "lcb_creds_request_counter": 26866, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.82:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.97:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_sbm_0": "[0-341]" }, "172.23.122.82:8096": { "worker_sbm_0": "[342-682]" }, "172.23.122.97:8096": { "worker_sbm_0": "[683-1023]" } }, "worker_pids": { "worker_sbm_0": 49917 } } ] [2021-02-12 21:49:20,897] - [eventing_base] [140099341772544] - INFO - Execution stats update_success: 40320 update_failure: 0 delete_success: 0 delete_failure: 0 [2021-02-12 21:49:20,916] - [basetestcase] [140099341772544] - INFO - list of eventing nodes in cluster: [ip:172.23.122.117 port:8091 ssh_username:root, ip:172.23.122.82 port:8091 ssh_username:root, ip:172.23.122.97 port:8091 ssh_username:root] [2021-02-12 21:49:20,961] - [eventing_base] [140099341772544] - INFO - Stats for Node 172.23.122.117 is [ { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 2, "agg_messages_sent_to_worker": 18191, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 40516, "dcp_mutation_sent_to_worker": 13499, "dcp_mutation_suppressed_counter": 13518, "dcp_snapshot": 40707, "dcp_stream_req_counter": 342, "dcp_streamreq": 342, "dcp_system_event": 3420, "execution_stats": 868, "failure_stats": 868, "latency_stats": 868, "lcb_exception_stats": 868, "log_level": 1, "num_processed_events": 13499, "processed_events_size": 12084340, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13499, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13499, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 18187, "num_processed_events": 13499, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13499, "processed_events_size": 12084340, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "16895": "2021-02-13T05:49:20Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "16895": "2021-02-13T05:49:20Z" }, "v8worker_events_lost": 0 }, "function_id": 327234242, "function_name": "sbm", "gocb_creds_request_counter": 98, "internal_vb_distribution_stats": { "worker_sbm_0": "[0-341]" }, "latency_percentile_stats": { "100": 15300, "50": 600, "80": 900, "90": 1200, "95": 1500, "99": 2900 }, "lcb_creds_request_counter": 27076, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.82:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.97:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_sbm_0": "[0-341]" }, "172.23.122.82:8096": { "worker_sbm_0": "[342-682]" }, "172.23.122.97:8096": { "worker_sbm_0": "[683-1023]" } }, "worker_pids": { "worker_sbm_0": 16895 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 2, "agg_messages_sent_to_worker": 18080, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 40516, "dcp_mutation_sent_to_worker": 13499, "dcp_snapshot": 40699, "dcp_stream_req_counter": 342, "dcp_streamreq": 342, "dcp_system_event": 3420, "execution_stats": 846, "failure_stats": 846, "latency_stats": 846, "lcb_exception_stats": 846, "log_level": 1, "num_processed_events": 13499, "processed_events_size": 12086936, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 13499, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 13499, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13499, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13499, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 18076, "num_processed_events": 13499, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13499, "processed_events_size": 12086936, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "16937": "2021-02-13T05:49:20Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "16937": "2021-02-13T05:49:20Z" }, "v8worker_events_lost": 0 }, "function_id": 3301735981, "function_name": "curl", "gocb_creds_request_counter": 98, "internal_vb_distribution_stats": { "worker_curl_0": "[0-341]" }, "latency_percentile_stats": { "100": 92400, "50": 1400, "80": 1700, "90": 1900, "95": 2200, "99": 3500 }, "lcb_creds_request_counter": 27076, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.82:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.97:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_curl_0": "[0-341]" }, "172.23.122.82:8096": { "worker_curl_0": "[342-682]" }, "172.23.122.97:8096": { "worker_curl_0": "[683-1023]" } }, "worker_pids": { "worker_curl_0": 16937 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 2, "agg_messages_sent_to_worker": 17974, "agg_queue_memory": 0, "agg_queue_memory_cap": 20971520, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 40516, "dcp_mutation_sent_to_worker": 13499, "dcp_snapshot": 40700, "dcp_stream_req_counter": 342, "dcp_streamreq": 342, "dcp_system_event": 3420, "execution_stats": 825, "failure_stats": 825, "latency_stats": 825, "lcb_exception_stats": 825, "log_level": 1, "num_processed_events": 13499, "processed_events_size": 12086936, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13499, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13499, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 17970, "num_processed_events": 13499, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13499, "processed_events_size": 12086936, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "16978": "2021-02-13T05:49:20Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "16978": "2021-02-13T05:49:20Z" }, "v8worker_events_lost": 0 }, "function_id": 1994061678, "function_name": "n1ql", "gocb_creds_request_counter": 98, "internal_vb_distribution_stats": { "worker_n1ql_0": "[0-341]" }, "latency_percentile_stats": { "100": 106800, "50": 3200, "80": 4300, "90": 5300, "95": 6600, "99": 11400 }, "lcb_creds_request_counter": 27076, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.82:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.97:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_n1ql_0": "[0-341]" }, "172.23.122.82:8096": { "worker_n1ql_0": "[342-682]" }, "172.23.122.97:8096": { "worker_n1ql_0": "[683-1023]" } }, "worker_pids": { "worker_n1ql_0": 16978 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 3, "agg_messages_sent_to_worker": 18408, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 40516, "dcp_mutation_sent_to_worker": 13499, "dcp_snapshot": 40712, "dcp_stream_req_counter": 342, "dcp_streamreq": 342, "dcp_system_event": 3420, "execution_stats": 911, "failure_stats": 911, "latency_stats": 911, "lcb_exception_stats": 911, "log_level": 1, "num_processed_events": 13499, "processed_events_size": 12086936, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13499, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13499, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 18404, "num_processed_events": 13499, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13499, "processed_events_size": 12086936, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "16782": "2021-02-13T05:49:20Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "16782": "2021-02-13T05:49:20Z" }, "v8worker_events_lost": 0 }, "function_id": 881891323, "function_name": "bucket_op", "gocb_creds_request_counter": 98, "internal_vb_distribution_stats": { "worker_bucket_op_0": "[0-341]" }, "latency_percentile_stats": { "100": 18300, "50": 500, "80": 800, "90": 1000, "95": 1200, "99": 2200 }, "lcb_creds_request_counter": 27076, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.82:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.97:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_bucket_op_0": "[0-341]" }, "172.23.122.82:8096": { "worker_bucket_op_0": "[342-682]" }, "172.23.122.97:8096": { "worker_bucket_op_0": "[683-1023]" } }, "worker_pids": { "worker_bucket_op_0": 16782 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 3, "agg_messages_sent_to_worker": 18297, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 40516, "dcp_mutation_sent_to_worker": 13499, "dcp_snapshot": 40714, "dcp_stream_req_counter": 342, "dcp_streamreq": 342, "dcp_system_event": 3420, "execution_stats": 889, "failure_stats": 889, "latency_stats": 889, "lcb_exception_stats": 889, "log_level": 1, "num_processed_events": 13499, "processed_events_size": 12086936, "thr_count": 1, "thr_map": 1, "timer_events": 13528, "timer_responses_received": 13499, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13499, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13499, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 18293, "num_processed_events": 13499, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13499, "processed_events_size": 12086936, "timer_callback_failure": 0, "timer_callback_success": 13528, "timer_cancel_counter": 0, "timer_create_counter": 13499, "timer_create_failure": 0, "timer_msg_counter": 13528, "timer_responses_sent": 0, "timestamp": { "16822": "2021-02-13T05:49:20Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "16822": "2021-02-13T05:49:20Z" }, "v8worker_events_lost": 0 }, "function_id": 2473559842, "function_name": "timers", "gocb_creds_request_counter": 98, "internal_vb_distribution_stats": { "worker_timers_0": "[0-341]" }, "latency_percentile_stats": { "100": 16700, "50": 1500, "80": 1900, "90": 2300, "95": 2700, "99": 4400 }, "lcb_creds_request_counter": 27076, "lcb_exception_stats": {}, "metastore_stats": { "metastore_delete_err": 0, "metastore_deletes": 0, "metastore_not_found": 0, "metastore_scan": 0, "metastore_scan_due": 0, "metastore_scan_err": 0, "metastore_set": 0, "metastore_set_err": 0 }, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.82:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.97:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_timers_0": "[0-341]" }, "172.23.122.82:8096": { "worker_timers_0": "[342-682]" }, "172.23.122.97:8096": { "worker_timers_0": "[683-1023]" } }, "worker_pids": { "worker_timers_0": 16822 } } ] [2021-02-12 21:49:21,005] - [eventing_base] [140099341772544] - INFO - Stats for Node 172.23.122.82 is [ { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 2, "agg_messages_sent_to_worker": 18229, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 40278, "dcp_mutation_sent_to_worker": 13432, "dcp_snapshot": 40439, "dcp_stream_req_counter": 341, "dcp_streamreq": 341, "dcp_system_event": 3410, "execution_stats": 889, "failure_stats": 889, "latency_stats": 889, "lcb_exception_stats": 889, "log_level": 1, "num_processed_events": 13432, "processed_events_size": 12024888, "thr_count": 1, "thr_map": 1, "timer_events": 13558, "timer_responses_received": 13432, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13432, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13432, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 18225, "num_processed_events": 13432, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13432, "processed_events_size": 12024888, "timer_callback_failure": 0, "timer_callback_success": 13558, "timer_cancel_counter": 0, "timer_create_counter": 13432, "timer_create_failure": 0, "timer_msg_counter": 13558, "timer_responses_sent": 0, "timestamp": { "41811": "2021-02-13T05:49:20Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "41811": "2021-02-13T05:49:20Z" }, "v8worker_events_lost": 0 }, "function_id": 2473559842, "function_name": "timers", "gocb_creds_request_counter": 89, "internal_vb_distribution_stats": { "worker_timers_0": "[342-682]" }, "latency_percentile_stats": { "100": 24900, "50": 1600, "80": 2100, "90": 2600, "95": 3100, "99": 5500 }, "lcb_creds_request_counter": 26954, "lcb_exception_stats": {}, "metastore_stats": { "metastore_delete_err": 0, "metastore_deletes": 0, "metastore_not_found": 0, "metastore_scan": 0, "metastore_scan_due": 0, "metastore_scan_err": 0, "metastore_set": 0, "metastore_set_err": 0 }, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.82:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.97:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_timers_0": "[0-341]" }, "172.23.122.82:8096": { "worker_timers_0": "[342-682]" }, "172.23.122.97:8096": { "worker_timers_0": "[683-1023]" } }, "worker_pids": { "worker_timers_0": 41811 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 2, "agg_messages_sent_to_worker": 18123, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 40278, "dcp_mutation_sent_to_worker": 13432, "dcp_mutation_suppressed_counter": 13414, "dcp_snapshot": 40407, "dcp_stream_req_counter": 341, "dcp_streamreq": 341, "dcp_system_event": 3410, "execution_stats": 868, "failure_stats": 868, "latency_stats": 868, "lcb_exception_stats": 868, "log_level": 1, "num_processed_events": 13432, "processed_events_size": 12026424, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13432, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13432, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 18119, "num_processed_events": 13432, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13432, "processed_events_size": 12026424, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "41844": "2021-02-13T05:49:20Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "41844": "2021-02-13T05:49:20Z" }, "v8worker_events_lost": 0 }, "function_id": 327234242, "function_name": "sbm", "gocb_creds_request_counter": 89, "internal_vb_distribution_stats": { "worker_sbm_0": "[342-682]" }, "latency_percentile_stats": { "100": 21600, "50": 700, "80": 1000, "90": 1300, "95": 1600, "99": 3000 }, "lcb_creds_request_counter": 26954, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.82:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.97:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_sbm_0": "[0-341]" }, "172.23.122.82:8096": { "worker_sbm_0": "[342-682]" }, "172.23.122.97:8096": { "worker_sbm_0": "[683-1023]" } }, "worker_pids": { "worker_sbm_0": 41844 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 1, "agg_messages_sent_to_worker": 18012, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 40278, "dcp_mutation_sent_to_worker": 13432, "dcp_snapshot": 40421, "dcp_stream_req_counter": 341, "dcp_streamreq": 341, "dcp_system_event": 3410, "execution_stats": 846, "failure_stats": 846, "latency_stats": 846, "lcb_exception_stats": 846, "log_level": 1, "num_processed_events": 13432, "processed_events_size": 12024888, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 13432, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 13432, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13432, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13432, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 18008, "num_processed_events": 13432, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13432, "processed_events_size": 12024888, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "41920": "2021-02-13T05:49:20Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "41920": "2021-02-13T05:49:20Z" }, "v8worker_events_lost": 0 }, "function_id": 3301735981, "function_name": "curl", "gocb_creds_request_counter": 89, "internal_vb_distribution_stats": { "worker_curl_0": "[342-682]" }, "latency_percentile_stats": { "100": 92400, "50": 1600, "80": 2000, "90": 2300, "95": 2800, "99": 5200 }, "lcb_creds_request_counter": 26954, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.82:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.97:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_curl_0": "[0-341]" }, "172.23.122.82:8096": { "worker_curl_0": "[342-682]" }, "172.23.122.97:8096": { "worker_curl_0": "[683-1023]" } }, "worker_pids": { "worker_curl_0": 41920 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 3, "agg_messages_sent_to_worker": 17906, "agg_queue_memory": 0, "agg_queue_memory_cap": 20971520, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 40278, "dcp_mutation_sent_to_worker": 13432, "dcp_snapshot": 40406, "dcp_stream_req_counter": 341, "dcp_streamreq": 341, "dcp_system_event": 3410, "execution_stats": 825, "failure_stats": 825, "latency_stats": 825, "lcb_exception_stats": 825, "log_level": 1, "num_processed_events": 13432, "processed_events_size": 12024888, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13432, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13432, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 17902, "num_processed_events": 13432, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13432, "processed_events_size": 12024888, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "41950": "2021-02-13T05:49:20Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "41950": "2021-02-13T05:49:20Z" }, "v8worker_events_lost": 0 }, "function_id": 1994061678, "function_name": "n1ql", "gocb_creds_request_counter": 89, "internal_vb_distribution_stats": { "worker_n1ql_0": "[342-682]" }, "latency_percentile_stats": { "100": 83800, "50": 3500, "80": 4900, "90": 6200, "95": 7700, "99": 12800 }, "lcb_creds_request_counter": 26954, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.82:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.97:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_n1ql_0": "[0-341]" }, "172.23.122.82:8096": { "worker_n1ql_0": "[342-682]" }, "172.23.122.97:8096": { "worker_n1ql_0": "[683-1023]" } }, "worker_pids": { "worker_n1ql_0": 41950 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 2, "agg_messages_sent_to_worker": 18340, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 40278, "dcp_mutation_sent_to_worker": 13432, "dcp_snapshot": 40428, "dcp_stream_req_counter": 341, "dcp_streamreq": 341, "dcp_system_event": 3410, "execution_stats": 911, "failure_stats": 911, "latency_stats": 911, "lcb_exception_stats": 911, "log_level": 1, "num_processed_events": 13432, "processed_events_size": 12024888, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13432, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13432, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 18336, "num_processed_events": 13432, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13432, "processed_events_size": 12024888, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "41748": "2021-02-13T05:49:20Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "41748": "2021-02-13T05:49:20Z" }, "v8worker_events_lost": 0 }, "function_id": 881891323, "function_name": "bucket_op", "gocb_creds_request_counter": 89, "internal_vb_distribution_stats": { "worker_bucket_op_0": "[342-682]" }, "latency_percentile_stats": { "100": 23100, "50": 600, "80": 900, "90": 1100, "95": 1400, "99": 2500 }, "lcb_creds_request_counter": 26954, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.82:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.97:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_bucket_op_0": "[0-341]" }, "172.23.122.82:8096": { "worker_bucket_op_0": "[342-682]" }, "172.23.122.97:8096": { "worker_bucket_op_0": "[683-1023]" } }, "worker_pids": { "worker_bucket_op_0": 41748 } } ] [2021-02-12 21:49:21,064] - [eventing_base] [140099341772544] - INFO - Stats for Node 172.23.122.97 is [ { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 1, "agg_messages_sent_to_worker": 18989, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 66943, "dcp_mutation_sent_to_worker": 13389, "dcp_snapshot": 40670, "dcp_stream_req_counter": 4771, "dcp_stream_req_err_counter": 3769, "dcp_streamend": 351, "dcp_streamreq": 1002, "dcp_system_event": 3410, "execution_stats": 911, "failure_stats": 911, "latency_stats": 911, "lcb_exception_stats": 911, "log_level": 1, "num_processed_events": 13389, "processed_events_size": 11986244, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13389, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13389, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 18985, "num_processed_events": 13389, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13389, "processed_events_size": 11986244, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "49851": "2021-02-13T05:49:20Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "49851": "2021-02-13T05:49:20Z" }, "v8worker_events_lost": 0 }, "function_id": 881891323, "function_name": "bucket_op", "gocb_creds_request_counter": 88, "internal_vb_distribution_stats": { "worker_bucket_op_0": "[683-1023]" }, "latency_percentile_stats": { "100": 21500, "50": 700, "80": 900, "90": 1100, "95": 1300, "99": 2300 }, "lcb_creds_request_counter": 26866, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.82:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.97:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_bucket_op_0": "[0-341]" }, "172.23.122.82:8096": { "worker_bucket_op_0": "[342-682]" }, "172.23.122.97:8096": { "worker_bucket_op_0": "[683-1023]" } }, "worker_pids": { "worker_bucket_op_0": 49851 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 2, "agg_messages_sent_to_worker": 18878, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 66943, "dcp_mutation_sent_to_worker": 13389, "dcp_snapshot": 40655, "dcp_stream_req_counter": 11043, "dcp_stream_req_err_counter": 10058, "dcp_streamend": 351, "dcp_streamreq": 985, "dcp_system_event": 3410, "execution_stats": 889, "failure_stats": 889, "latency_stats": 889, "lcb_exception_stats": 889, "log_level": 1, "num_processed_events": 13389, "processed_events_size": 11986244, "thr_count": 1, "thr_map": 1, "timer_events": 13234, "timer_responses_received": 13389, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13389, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13389, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 18874, "num_processed_events": 13389, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13389, "processed_events_size": 11986244, "timer_callback_failure": 0, "timer_callback_success": 13234, "timer_cancel_counter": 0, "timer_create_counter": 13389, "timer_create_failure": 0, "timer_msg_counter": 13234, "timer_responses_sent": 0, "timestamp": { "49884": "2021-02-13T05:49:20Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "49884": "2021-02-13T05:49:20Z" }, "v8worker_events_lost": 0 }, "function_id": 2473559842, "function_name": "timers", "gocb_creds_request_counter": 88, "internal_vb_distribution_stats": { "worker_timers_0": "[683-1023]" }, "latency_percentile_stats": { "100": 23300, "50": 1600, "80": 2100, "90": 2400, "95": 2900, "99": 4700 }, "lcb_creds_request_counter": 26866, "lcb_exception_stats": {}, "metastore_stats": { "metastore_delete_err": 0, "metastore_deletes": 0, "metastore_not_found": 0, "metastore_scan": 0, "metastore_scan_due": 0, "metastore_scan_err": 0, "metastore_set": 0, "metastore_set_err": 0 }, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.82:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.97:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_timers_0": "[0-341]" }, "172.23.122.82:8096": { "worker_timers_0": "[342-682]" }, "172.23.122.97:8096": { "worker_timers_0": "[683-1023]" } }, "worker_pids": { "worker_timers_0": 49884 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "agg_messages_sent_to_worker": 18772, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 40608, "dcp_mutation_sent_to_worker": 13389, "dcp_mutation_suppressed_counter": 13830, "dcp_snapshot": 40529, "dcp_stream_req_counter": 10404, "dcp_stream_req_err_counter": 9419, "dcp_streamend": 351, "dcp_streamreq": 985, "dcp_system_event": 3410, "execution_stats": 868, "failure_stats": 868, "latency_stats": 868, "lcb_exception_stats": 868, "log_level": 1, "num_processed_events": 13389, "processed_events_size": 11994668, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13389, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13389, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 18768, "num_processed_events": 13389, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13389, "processed_events_size": 11994668, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "49917": "2021-02-13T05:49:20Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "49917": "2021-02-13T05:49:20Z" }, "v8worker_events_lost": 0 }, "function_id": 327234242, "function_name": "sbm", "gocb_creds_request_counter": 88, "internal_vb_distribution_stats": { "worker_sbm_0": "[683-1023]" }, "latency_percentile_stats": { "100": 18600, "50": 700, "80": 1000, "90": 1200, "95": 1600, "99": 3000 }, "lcb_creds_request_counter": 26866, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.82:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.97:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_sbm_0": "[0-341]" }, "172.23.122.82:8096": { "worker_sbm_0": "[342-682]" }, "172.23.122.97:8096": { "worker_sbm_0": "[683-1023]" } }, "worker_pids": { "worker_sbm_0": 49917 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 2, "agg_messages_sent_to_worker": 18661, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 66943, "dcp_mutation_sent_to_worker": 13389, "dcp_snapshot": 40637, "dcp_stream_req_counter": 10254, "dcp_stream_req_err_counter": 9269, "dcp_streamend": 351, "dcp_streamreq": 985, "dcp_system_event": 3410, "execution_stats": 846, "failure_stats": 846, "latency_stats": 846, "lcb_exception_stats": 846, "log_level": 1, "num_processed_events": 13389, "processed_events_size": 11986244, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 13389, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 13389, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13389, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13389, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 18657, "num_processed_events": 13389, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13389, "processed_events_size": 11986244, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "49993": "2021-02-13T05:49:20Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "49993": "2021-02-13T05:49:20Z" }, "v8worker_events_lost": 0 }, "function_id": 3301735981, "function_name": "curl", "gocb_creds_request_counter": 88, "internal_vb_distribution_stats": { "worker_curl_0": "[683-1023]" }, "latency_percentile_stats": { "100": 92600, "50": 1500, "80": 1900, "90": 2200, "95": 2600, "99": 4200 }, "lcb_creds_request_counter": 26866, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.82:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.97:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_curl_0": "[0-341]" }, "172.23.122.82:8096": { "worker_curl_0": "[342-682]" }, "172.23.122.97:8096": { "worker_curl_0": "[683-1023]" } }, "worker_pids": { "worker_curl_0": 49993 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 3, "agg_messages_sent_to_worker": 18555, "agg_queue_memory": 0, "agg_queue_memory_cap": 20971520, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 66943, "dcp_mutation_sent_to_worker": 13389, "dcp_snapshot": 40639, "dcp_stream_req_counter": 5805, "dcp_stream_req_err_counter": 4820, "dcp_streamend": 351, "dcp_streamreq": 985, "dcp_system_event": 3410, "execution_stats": 825, "failure_stats": 825, "latency_stats": 825, "lcb_exception_stats": 825, "log_level": 1, "num_processed_events": 13389, "processed_events_size": 11986244, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 13389, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 13389, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 18551, "num_processed_events": 13389, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 9, "on_update_success": 13380, "processed_events_size": 11986244, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "50026": "2021-02-13T05:49:20Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 9, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "50026": "2021-02-13T05:49:20Z" }, "v8worker_events_lost": 0 }, "function_id": 1994061678, "function_name": "n1ql", "gocb_creds_request_counter": 88, "internal_vb_distribution_stats": { "worker_n1ql_0": "[683-1023]" }, "latency_percentile_stats": { "100": 107900, "50": 3400, "80": 4700, "90": 6000, "95": 7600, "99": 13800 }, "lcb_creds_request_counter": 26866, "lcb_exception_stats": { "12009": 9, "209": 9 }, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.117:8096", "start_vb": 0, "vb_count": 342 }, { "host_name": "172.23.122.82:8096", "start_vb": 342, "vb_count": 341 }, { "host_name": "172.23.122.97:8096", "start_vb": 683, "vb_count": 341 } ], "vb_distribution_stats_from_metadata": { "172.23.122.117:8096": { "worker_n1ql_0": "[0-341]" }, "172.23.122.82:8096": { "worker_n1ql_0": "[342-682]" }, "172.23.122.97:8096": { "worker_n1ql_0": "[683-1023]" } }, "worker_pids": { "worker_n1ql_0": 50026 } } ] [2021-02-12 21:49:21,079] - [basetestcase] [140099341772544] - INFO - list of eventing nodes in cluster: [ip:172.23.122.117 port:8091 ssh_username:root, ip:172.23.122.82 port:8091 ssh_username:root, ip:172.23.122.97 port:8091 ssh_username:root] [2021-02-12 21:49:21,081] - [remote_util] [140099341772544] - INFO - SSH Connecting to 172.23.122.117 with username:root, attempt#1 of 5 [2021-02-12 21:49:21,183] - [remote_util] [140099341772544] - INFO - SSH Connected to 172.23.122.117 as root [2021-02-12 21:49:21,442] - [remote_util] [140099341772544] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:49:21,737] - [remote_util] [140099341772544] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:49:21,745] - [rest_client] [140099341772544] - INFO - /diag/eval status on 172.23.122.117:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). [2021-02-12 21:49:21,745] - [remote_util] [140099341772544] - INFO - running command.raw on 172.23.122.117: zgrep "panic" "/opt/couchbase/var/lib/couchbase/logs"/eventing.log* | wc -l [2021-02-12 21:49:21,829] - [remote_util] [140099341772544] - INFO - command executed successfully with root [2021-02-12 21:49:21,830] - [remote_util] [140099341772544] - INFO - running command.raw on 172.23.122.117: ls "/opt/couchbase/var/lib/couchbase/logs"/../crash/| wc -l [2021-02-12 21:49:21,882] - [remote_util] [140099341772544] - INFO - command executed successfully with root [2021-02-12 21:49:21,884] - [remote_util] [140099341772544] - INFO - SSH Connecting to 172.23.122.82 with username:root, attempt#1 of 5 [2021-02-12 21:49:21,982] - [remote_util] [140099341772544] - INFO - SSH Connected to 172.23.122.82 as root [2021-02-12 21:49:22,248] - [remote_util] [140099341772544] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:49:22,544] - [remote_util] [140099341772544] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:49:22,551] - [rest_client] [140099341772544] - INFO - /diag/eval status on 172.23.122.82:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). [2021-02-12 21:49:22,551] - [remote_util] [140099341772544] - INFO - running command.raw on 172.23.122.82: zgrep "panic" "/opt/couchbase/var/lib/couchbase/logs"/eventing.log* | wc -l [2021-02-12 21:49:22,633] - [remote_util] [140099341772544] - INFO - command executed successfully with root [2021-02-12 21:49:22,633] - [remote_util] [140099341772544] - INFO - running command.raw on 172.23.122.82: ls "/opt/couchbase/var/lib/couchbase/logs"/../crash/| wc -l [2021-02-12 21:49:22,686] - [remote_util] [140099341772544] - INFO - command executed successfully with root [2021-02-12 21:49:22,687] - [remote_util] [140099341772544] - INFO - SSH Connecting to 172.23.122.97 with username:root, attempt#1 of 5 [2021-02-12 21:49:22,790] - [remote_util] [140099341772544] - INFO - SSH Connected to 172.23.122.97 as root [2021-02-12 21:49:23,061] - [remote_util] [140099341772544] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:49:23,365] - [remote_util] [140099341772544] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:49:23,373] - [rest_client] [140099341772544] - INFO - /diag/eval status on 172.23.122.97:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). [2021-02-12 21:49:23,373] - [remote_util] [140099341772544] - INFO - running command.raw on 172.23.122.97: zgrep "panic" "/opt/couchbase/var/lib/couchbase/logs"/eventing.log* | wc -l [2021-02-12 21:49:24,233] - [remote_util] [140099341772544] - INFO - command executed successfully with root [2021-02-12 21:49:24,233] - [remote_util] [140099341772544] - INFO - running command.raw on 172.23.122.97: ls "/opt/couchbase/var/lib/couchbase/logs"/../crash/| wc -l [2021-02-12 21:49:24,289] - [remote_util] [140099341772544] - INFO - command executed successfully with root [2021-02-12 21:49:24,571] - [basetestcase] [140099341772544] - INFO - ------- Cluster statistics ------- [2021-02-12 21:49:24,571] - [basetestcase] [140099341772544] - INFO - 172.23.122.117:8091 => {'services': ['eventing'], 'cpu_utilization': 11.62548764629389, 'mem_free': 3002716160, 'mem_total': 4201684992, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-02-12 21:49:24,571] - [basetestcase] [140099341772544] - INFO - 172.23.122.118:8091 => {'services': ['index', 'n1ql'], 'cpu_utilization': 0.8375634517766497, 'mem_free': 3200393216, 'mem_total': 4201689088, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-02-12 21:49:24,571] - [basetestcase] [140099341772544] - INFO - 172.23.122.82:8091 => {'services': ['eventing'], 'cpu_utilization': 10.76723016905071, 'mem_free': 3039051776, 'mem_total': 4201684992, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-02-12 21:49:24,571] - [basetestcase] [140099341772544] - INFO - 172.23.122.90:8091 => {'services': ['kv'], 'cpu_utilization': 5.04266873545384, 'mem_free': 2970976256, 'mem_total': 4201684992, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-02-12 21:49:24,571] - [basetestcase] [140099341772544] - INFO - 172.23.122.97:8091 => {'services': ['eventing'], 'cpu_utilization': 6.927402862985685, 'mem_free': 3126468608, 'mem_total': 4201684992, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-02-12 21:49:24,572] - [basetestcase] [140099341772544] - INFO - 172.23.123.61:8091 => {'services': ['kv'], 'cpu_utilization': 7.004584819154355, 'mem_free': 2870988800, 'mem_total': 4201672704, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-02-12 21:49:24,572] - [basetestcase] [140099341772544] - INFO - --- End of cluster statistics --- [2021-02-12 21:49:24,573] - [remote_util] [140099341772544] - INFO - SSH Connecting to 172.23.123.61 with username:root, attempt#1 of 5 [2021-02-12 21:49:24,673] - [remote_util] [140099341772544] - INFO - SSH Connected to 172.23.123.61 as root [2021-02-12 21:49:24,943] - [remote_util] [140099341772544] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:49:25,242] - [remote_util] [140099341772544] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:49:25,244] - [remote_util] [140099341772544] - INFO - SSH Connecting to 172.23.123.62 with username:root, attempt#1 of 5 [2021-02-12 21:49:25,351] - [remote_util] [140099341772544] - INFO - SSH Connected to 172.23.123.62 as root [2021-02-12 21:49:25,623] - [remote_util] [140099341772544] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:49:25,923] - [remote_util] [140099341772544] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:49:25,925] - [remote_util] [140099341772544] - INFO - SSH Connecting to 172.23.122.90 with username:root, attempt#1 of 5 [2021-02-12 21:49:26,029] - [remote_util] [140099341772544] - INFO - SSH Connected to 172.23.122.90 as root [2021-02-12 21:49:26,317] - [remote_util] [140099341772544] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:49:26,611] - [remote_util] [140099341772544] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:49:26,613] - [remote_util] [140099341772544] - INFO - SSH Connecting to 172.23.122.117 with username:root, attempt#1 of 5 [2021-02-12 21:49:26,712] - [remote_util] [140099341772544] - INFO - SSH Connected to 172.23.122.117 as root [2021-02-12 21:49:26,964] - [remote_util] [140099341772544] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:49:27,264] - [remote_util] [140099341772544] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:49:27,266] - [remote_util] [140099341772544] - INFO - SSH Connecting to 172.23.122.97 with username:root, attempt#1 of 5 [2021-02-12 21:49:27,371] - [remote_util] [140099341772544] - INFO - SSH Connected to 172.23.122.97 as root [2021-02-12 21:49:27,627] - [remote_util] [140099341772544] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:49:27,923] - [remote_util] [140099341772544] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:49:27,925] - [remote_util] [140099341772544] - INFO - SSH Connecting to 172.23.122.82 with username:root, attempt#1 of 5 [2021-02-12 21:49:28,027] - [remote_util] [140099341772544] - INFO - SSH Connected to 172.23.122.82 as root [2021-02-12 21:49:28,288] - [remote_util] [140099341772544] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:49:28,584] - [remote_util] [140099341772544] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:49:28,586] - [remote_util] [140099341772544] - INFO - SSH Connecting to 172.23.122.118 with username:root, attempt#1 of 5 [2021-02-12 21:49:28,687] - [remote_util] [140099341772544] - INFO - SSH Connected to 172.23.122.118 as root [2021-02-12 21:49:28,932] - [remote_util] [140099341772544] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:49:29,223] - [remote_util] [140099341772544] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:49:29,225] - [remote_util] [140099341772544] - INFO - SSH Connecting to 172.23.122.116 with username:root, attempt#1 of 5 [2021-02-12 21:49:29,329] - [remote_util] [140099341772544] - INFO - SSH Connected to 172.23.122.116 as root [2021-02-12 21:49:29,608] - [remote_util] [140099341772544] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:49:29,900] - [remote_util] [140099341772544] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:50:01,040] - [basetestcase] [140099341772544] - INFO - ============== basetestcase cleanup was started for test #5 test_kv_rebalance_out_when_existing_eventing_node_is_processing_mutations ============== [2021-02-12 21:50:01,105] - [bucket_helper] [140099341772544] - INFO - deleting existing buckets ['dst_bucket', 'metadata', 'src_bucket'] on 172.23.123.61 [2021-02-12 21:50:02,446] - [bucket_helper] [140099341772544] - INFO - waiting for bucket deletion to complete.... [2021-02-12 21:50:02,471] - [rest_client] [140099341772544] - INFO - node 172.23.123.61 existing buckets : ['metadata', 'src_bucket'] [2021-02-12 21:50:02,472] - [bucket_helper] [140099341772544] - INFO - deleted bucket : dst_bucket from 172.23.123.61 [2021-02-12 21:50:02,995] - [bucket_helper] [140099341772544] - INFO - waiting for bucket deletion to complete.... [2021-02-12 21:50:03,009] - [rest_client] [140099341772544] - INFO - node 172.23.123.61 existing buckets : ['src_bucket'] [2021-02-12 21:50:03,009] - [bucket_helper] [140099341772544] - INFO - deleted bucket : metadata from 172.23.123.61 [2021-02-12 21:50:04,072] - [bucket_helper] [140099341772544] - INFO - waiting for bucket deletion to complete.... [2021-02-12 21:50:04,074] - [rest_client] [140099341772544] - INFO - node 172.23.123.61 existing buckets : [] [2021-02-12 21:50:04,074] - [bucket_helper] [140099341772544] - INFO - deleted bucket : src_bucket from 172.23.123.61 [2021-02-12 21:50:04,082] - [bucket_helper] [140099341772544] - INFO - Could not find any buckets for node 172.23.123.62, nothing to delete [2021-02-12 21:50:04,089] - [bucket_helper] [140099341772544] - INFO - Could not find any buckets for node 172.23.122.90, nothing to delete [2021-02-12 21:50:04,096] - [bucket_helper] [140099341772544] - INFO - Could not find any buckets for node 172.23.122.117, nothing to delete [2021-02-12 21:50:04,104] - [bucket_helper] [140099341772544] - INFO - Could not find any buckets for node 172.23.122.97, nothing to delete [2021-02-12 21:50:04,114] - [bucket_helper] [140099341772544] - INFO - Could not find any buckets for node 172.23.122.82, nothing to delete [2021-02-12 21:50:04,129] - [bucket_helper] [140099341772544] - INFO - Could not find any buckets for node 172.23.122.118, nothing to delete [2021-02-12 21:50:04,134] - [bucket_helper] [140099341772544] - INFO - Could not find any buckets for node 172.23.122.116, nothing to delete [2021-02-12 21:50:04,141] - [rest_client] [140099341772544] - INFO - -->is_ns_server_running? [2021-02-12 21:50:04,155] - [cluster_helper] [140099341772544] - INFO - rebalancing all nodes in order to remove nodes [2021-02-12 21:50:04,157] - [rest_client] [140099341772544] - INFO - rebalance params : {'knownNodes': 'ns_1@172.23.122.117,ns_1@172.23.122.118,ns_1@172.23.122.82,ns_1@172.23.122.90,ns_1@172.23.122.97,ns_1@172.23.123.61', 'ejectedNodes': 'ns_1@172.23.122.117,ns_1@172.23.122.118,ns_1@172.23.122.82,ns_1@172.23.122.90,ns_1@172.23.122.97', 'user': 'Administrator', 'password': 'password'} [2021-02-12 21:50:04,187] - [rest_client] [140099341772544] - INFO - rebalance operation started [2021-02-12 21:50:04,190] - [rest_client] [140099341772544] - INFO - rebalance percentage : 0.00 % [2021-02-12 21:50:14,203] - [rest_client] [140099341772544] - INFO - rebalance percentage : 41.00 % [2021-02-12 21:50:24,216] - [rest_client] [140099341772544] - INFO - rebalance percentage : 91.00 % [2021-02-12 21:50:44,242] - [rest_client] [140099341772544] - INFO - rebalance progress took 40.05 seconds [2021-02-12 21:50:44,242] - [rest_client] [140099341772544] - INFO - sleep for 10 seconds after rebalance... [2021-02-12 21:50:54,282] - [cluster_helper] [140099341772544] - INFO - removed all the nodes from cluster associated with ip:172.23.123.61 port:8091 ssh_username:root ? [('ns_1@172.23.122.117', 8091), ('ns_1@172.23.122.118', 8091), ('ns_1@172.23.122.82', 8091), ('ns_1@172.23.122.90', 8091), ('ns_1@172.23.122.97', 8091)] [2021-02-12 21:50:54,287] - [cluster_helper] [140099341772544] - INFO - waiting for ns_server @ 172.23.123.61:8091 [2021-02-12 21:50:54,287] - [rest_client] [140099341772544] - INFO - -->is_ns_server_running? [2021-02-12 21:50:54,291] - [cluster_helper] [140099341772544] - INFO - ns_server @ 172.23.123.61:8091 is running [2021-02-12 21:50:54,295] - [cluster_helper] [140099341772544] - INFO - waiting for ns_server @ 172.23.123.62:8091 [2021-02-12 21:50:54,295] - [rest_client] [140099341772544] - INFO - -->is_ns_server_running? [2021-02-12 21:50:54,298] - [cluster_helper] [140099341772544] - INFO - ns_server @ 172.23.123.62:8091 is running [2021-02-12 21:50:54,300] - [cluster_helper] [140099341772544] - INFO - waiting for ns_server @ 172.23.122.90:8091 [2021-02-12 21:50:54,300] - [rest_client] [140099341772544] - INFO - -->is_ns_server_running? [2021-02-12 21:50:54,303] - [cluster_helper] [140099341772544] - INFO - ns_server @ 172.23.122.90:8091 is running [2021-02-12 21:50:54,306] - [cluster_helper] [140099341772544] - INFO - waiting for ns_server @ 172.23.122.117:8091 [2021-02-12 21:50:54,306] - [rest_client] [140099341772544] - INFO - -->is_ns_server_running? [2021-02-12 21:50:54,309] - [cluster_helper] [140099341772544] - INFO - ns_server @ 172.23.122.117:8091 is running [2021-02-12 21:50:54,311] - [cluster_helper] [140099341772544] - INFO - waiting for ns_server @ 172.23.122.97:8091 [2021-02-12 21:50:54,312] - [rest_client] [140099341772544] - INFO - -->is_ns_server_running? [2021-02-12 21:50:54,314] - [cluster_helper] [140099341772544] - INFO - ns_server @ 172.23.122.97:8091 is running [2021-02-12 21:50:54,317] - [cluster_helper] [140099341772544] - INFO - waiting for ns_server @ 172.23.122.82:8091 [2021-02-12 21:50:54,317] - [rest_client] [140099341772544] - INFO - -->is_ns_server_running? [2021-02-12 21:50:54,320] - [cluster_helper] [140099341772544] - INFO - ns_server @ 172.23.122.82:8091 is running [2021-02-12 21:50:54,323] - [cluster_helper] [140099341772544] - INFO - waiting for ns_server @ 172.23.122.118:8091 [2021-02-12 21:50:54,323] - [rest_client] [140099341772544] - INFO - -->is_ns_server_running? [2021-02-12 21:50:54,326] - [cluster_helper] [140099341772544] - INFO - ns_server @ 172.23.122.118:8091 is running [2021-02-12 21:50:54,329] - [cluster_helper] [140099341772544] - INFO - waiting for ns_server @ 172.23.122.116:8091 [2021-02-12 21:50:54,329] - [rest_client] [140099341772544] - INFO - -->is_ns_server_running? [2021-02-12 21:50:54,332] - [cluster_helper] [140099341772544] - INFO - ns_server @ 172.23.122.116:8091 is running [2021-02-12 21:50:54,332] - [basetestcase] [140099341772544] - INFO - ============== basetestcase cleanup was finished for test #5 test_kv_rebalance_out_when_existing_eventing_node_is_processing_mutations ============== [2021-02-12 21:50:54,332] - [basetestcase] [140099341772544] - INFO - closing all ssh connections [2021-02-12 21:50:54,333] - [basetestcase] [140099341772544] - INFO - closing all memcached connections [2021-02-12 21:50:54,335] - [remote_util] [140099341772544] - INFO - SSH Connecting to 172.23.123.61 with username:root, attempt#1 of 5 [2021-02-12 21:50:54,439] - [remote_util] [140099341772544] - INFO - SSH Connected to 172.23.123.61 as root [2021-02-12 21:50:54,696] - [remote_util] [140099341772544] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:50:54,987] - [remote_util] [140099341772544] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:50:54,996] - [remote_util] [140099341772544] - INFO - SSH Connecting to 172.23.123.61 with username:root, attempt#1 of 5 [2021-02-12 21:50:55,093] - [remote_util] [140099341772544] - INFO - SSH Connected to 172.23.123.61 as root [2021-02-12 21:50:55,342] - [remote_util] [140099341772544] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:50:55,631] - [remote_util] [140099341772544] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:50:55,689] - [remote_util] [140099341772544] - INFO - 172.23.123.61 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 21:50:55,740] - [remote_util] [140099341772544] - INFO - Running systemd command on this server [2021-02-12 21:50:55,740] - [remote_util] [140099341772544] - INFO - running command.raw on 172.23.123.61: systemctl stop couchbase-server.service [2021-02-12 21:50:57,592] - [remote_util] [140099341772544] - INFO - command executed successfully with root [2021-02-12 21:50:57,592] - [basetestcase] [140099341772544] - INFO - Couchbase stopped [2021-02-12 21:50:57,592] - [basetestcase] [140099341772544] - INFO - sleep for 5 secs. ... [2021-02-12 21:51:02,598] - [remote_util] [140099341772544] - INFO - running command.raw on 172.23.123.61: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-12 21:51:02,612] - [remote_util] [140099341772544] - INFO - command executed successfully with root [2021-02-12 21:51:02,613] - [remote_util] [140099341772544] - INFO - running command.raw on 172.23.123.61: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-12 21:51:02,671] - [remote_util] [140099341772544] - INFO - command executed successfully with root [2021-02-12 21:51:02,672] - [remote_util] [140099341772544] - INFO - SSH Connecting to 172.23.123.61 with username:root, attempt#1 of 5 [2021-02-12 21:51:02,769] - [remote_util] [140099341772544] - INFO - SSH Connected to 172.23.123.61 as root [2021-02-12 21:51:03,037] - [remote_util] [140099341772544] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:51:03,334] - [remote_util] [140099341772544] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:51:03,353] - [remote_util] [140099341772544] - INFO - 172.23.123.61 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 21:51:03,354] - [remote_util] [140099341772544] - INFO - Checking for process beam.smp on linux [2021-02-12 21:51:03,390] - [remote_util] [140099341772544] - INFO - Starting couchbase server [2021-02-12 21:51:03,484] - [remote_util] [140099341772544] - INFO - Running systemd command on this server [2021-02-12 21:51:03,484] - [remote_util] [140099341772544] - INFO - running command.raw on 172.23.123.61: systemctl start couchbase-server.service [2021-02-12 21:51:03,506] - [remote_util] [140099341772544] - INFO - command executed successfully with root [2021-02-12 21:51:03,507] - [remote_util] [140099341772544] - INFO - 172.23.123.61:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-12 21:51:08,512] - [remote_util] [140099341772544] - INFO - running command.raw on 172.23.123.61: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-12 21:51:08,537] - [remote_util] [140099341772544] - INFO - command executed successfully with root [2021-02-12 21:51:08,537] - [remote_util] [140099341772544] - INFO - Couchbase server status: [' Process: 96233 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-12 21:51:08,537] - [remote_util] [140099341772544] - INFO - Checking for process beam.smp on linux [2021-02-12 21:51:08,604] - [remote_util] [140099341772544] - INFO - process beam.smp is running on 172.23.123.61: with pid 96377 [2021-02-12 21:51:08,604] - [basetestcase] [140099341772544] - INFO - Couchbase started [2021-02-12 21:51:08,605] - [remote_util] [140099341772544] - INFO - SSH Connecting to 172.23.123.62 with username:root, attempt#1 of 5 [2021-02-12 21:51:08,707] - [remote_util] [140099341772544] - INFO - SSH Connected to 172.23.123.62 as root [2021-02-12 21:51:08,976] - [remote_util] [140099341772544] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:51:09,273] - [remote_util] [140099341772544] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:51:09,281] - [remote_util] [140099341772544] - INFO - SSH Connecting to 172.23.123.62 with username:root, attempt#1 of 5 [2021-02-12 21:51:09,379] - [remote_util] [140099341772544] - INFO - SSH Connected to 172.23.123.62 as root [2021-02-12 21:51:09,650] - [remote_util] [140099341772544] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:51:09,948] - [remote_util] [140099341772544] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:51:09,967] - [remote_util] [140099341772544] - INFO - 172.23.123.62 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 21:51:10,021] - [remote_util] [140099341772544] - INFO - Running systemd command on this server [2021-02-12 21:51:10,022] - [remote_util] [140099341772544] - INFO - running command.raw on 172.23.123.62: systemctl stop couchbase-server.service [2021-02-12 21:51:11,698] - [remote_util] [140099341772544] - INFO - command executed successfully with root [2021-02-12 21:51:11,698] - [basetestcase] [140099341772544] - INFO - Couchbase stopped [2021-02-12 21:51:11,698] - [basetestcase] [140099341772544] - INFO - sleep for 5 secs. ... [2021-02-12 21:51:16,703] - [remote_util] [140099341772544] - INFO - running command.raw on 172.23.123.62: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-12 21:51:16,723] - [remote_util] [140099341772544] - INFO - command executed successfully with root [2021-02-12 21:51:16,723] - [remote_util] [140099341772544] - INFO - running command.raw on 172.23.123.62: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-12 21:51:16,775] - [remote_util] [140099341772544] - INFO - command executed successfully with root [2021-02-12 21:51:16,777] - [remote_util] [140099341772544] - INFO - SSH Connecting to 172.23.123.62 with username:root, attempt#1 of 5 [2021-02-12 21:51:16,877] - [remote_util] [140099341772544] - INFO - SSH Connected to 172.23.123.62 as root [2021-02-12 21:51:17,140] - [remote_util] [140099341772544] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:51:17,437] - [remote_util] [140099341772544] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:51:17,495] - [remote_util] [140099341772544] - INFO - 172.23.123.62 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 21:51:17,495] - [remote_util] [140099341772544] - INFO - Checking for process beam.smp on linux [2021-02-12 21:51:17,534] - [remote_util] [140099341772544] - INFO - Starting couchbase server [2021-02-12 21:51:17,627] - [remote_util] [140099341772544] - INFO - Running systemd command on this server [2021-02-12 21:51:17,627] - [remote_util] [140099341772544] - INFO - running command.raw on 172.23.123.62: systemctl start couchbase-server.service [2021-02-12 21:51:17,649] - [remote_util] [140099341772544] - INFO - command executed successfully with root [2021-02-12 21:51:17,649] - [remote_util] [140099341772544] - INFO - 172.23.123.62:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-12 21:51:22,654] - [remote_util] [140099341772544] - INFO - running command.raw on 172.23.123.62: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-12 21:51:22,680] - [remote_util] [140099341772544] - INFO - command executed successfully with root [2021-02-12 21:51:22,680] - [remote_util] [140099341772544] - INFO - Couchbase server status: [' Process: 108865 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-12 21:51:22,680] - [remote_util] [140099341772544] - INFO - Checking for process beam.smp on linux [2021-02-12 21:51:22,753] - [remote_util] [140099341772544] - INFO - process beam.smp is running on 172.23.123.62: with pid 109039 [2021-02-12 21:51:22,753] - [basetestcase] [140099341772544] - INFO - Couchbase started [2021-02-12 21:51:22,755] - [remote_util] [140099341772544] - INFO - SSH Connecting to 172.23.122.90 with username:root, attempt#1 of 5 [2021-02-12 21:51:22,859] - [remote_util] [140099341772544] - INFO - SSH Connected to 172.23.122.90 as root [2021-02-12 21:51:23,146] - [remote_util] [140099341772544] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:51:23,446] - [remote_util] [140099341772544] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:51:23,453] - [remote_util] [140099341772544] - INFO - SSH Connecting to 172.23.122.90 with username:root, attempt#1 of 5 [2021-02-12 21:51:23,550] - [remote_util] [140099341772544] - INFO - SSH Connected to 172.23.122.90 as root [2021-02-12 21:51:23,827] - [remote_util] [140099341772544] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:51:24,123] - [remote_util] [140099341772544] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:51:24,141] - [remote_util] [140099341772544] - INFO - 172.23.122.90 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 21:51:24,193] - [remote_util] [140099341772544] - INFO - Running systemd command on this server [2021-02-12 21:51:24,193] - [remote_util] [140099341772544] - INFO - running command.raw on 172.23.122.90: systemctl stop couchbase-server.service [2021-02-12 21:51:25,847] - [remote_util] [140099341772544] - INFO - command executed successfully with root [2021-02-12 21:51:25,847] - [basetestcase] [140099341772544] - INFO - Couchbase stopped [2021-02-12 21:51:25,847] - [basetestcase] [140099341772544] - INFO - sleep for 5 secs. ... [2021-02-12 21:51:30,852] - [remote_util] [140099341772544] - INFO - running command.raw on 172.23.122.90: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-12 21:51:30,868] - [remote_util] [140099341772544] - INFO - command executed successfully with root [2021-02-12 21:51:30,868] - [remote_util] [140099341772544] - INFO - running command.raw on 172.23.122.90: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-12 21:51:30,922] - [remote_util] [140099341772544] - INFO - command executed successfully with root [2021-02-12 21:51:30,923] - [remote_util] [140099341772544] - INFO - SSH Connecting to 172.23.122.90 with username:root, attempt#1 of 5 [2021-02-12 21:51:31,022] - [remote_util] [140099341772544] - INFO - SSH Connected to 172.23.122.90 as root [2021-02-12 21:51:31,309] - [remote_util] [140099341772544] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:51:31,597] - [remote_util] [140099341772544] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:51:31,614] - [remote_util] [140099341772544] - INFO - 172.23.122.90 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 21:51:31,614] - [remote_util] [140099341772544] - INFO - Checking for process beam.smp on linux [2021-02-12 21:51:31,652] - [remote_util] [140099341772544] - INFO - Starting couchbase server [2021-02-12 21:51:31,741] - [remote_util] [140099341772544] - INFO - Running systemd command on this server [2021-02-12 21:51:31,741] - [remote_util] [140099341772544] - INFO - running command.raw on 172.23.122.90: systemctl start couchbase-server.service [2021-02-12 21:51:31,762] - [remote_util] [140099341772544] - INFO - command executed successfully with root [2021-02-12 21:51:31,762] - [remote_util] [140099341772544] - INFO - 172.23.122.90:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-12 21:51:36,767] - [remote_util] [140099341772544] - INFO - running command.raw on 172.23.122.90: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-12 21:51:36,790] - [remote_util] [140099341772544] - INFO - command executed successfully with root [2021-02-12 21:51:36,790] - [remote_util] [140099341772544] - INFO - Couchbase server status: [' Process: 73913 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-12 21:51:36,790] - [remote_util] [140099341772544] - INFO - Checking for process beam.smp on linux [2021-02-12 21:51:36,858] - [remote_util] [140099341772544] - INFO - process beam.smp is running on 172.23.122.90: with pid 74058 [2021-02-12 21:51:36,858] - [basetestcase] [140099341772544] - INFO - Couchbase started [2021-02-12 21:51:36,859] - [remote_util] [140099341772544] - INFO - SSH Connecting to 172.23.122.117 with username:root, attempt#1 of 5 [2021-02-12 21:51:36,964] - [remote_util] [140099341772544] - INFO - SSH Connected to 172.23.122.117 as root [2021-02-12 21:51:37,221] - [remote_util] [140099341772544] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:51:37,512] - [remote_util] [140099341772544] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:51:37,520] - [remote_util] [140099341772544] - INFO - SSH Connecting to 172.23.122.117 with username:root, attempt#1 of 5 [2021-02-12 21:51:37,617] - [remote_util] [140099341772544] - INFO - SSH Connected to 172.23.122.117 as root [2021-02-12 21:51:37,864] - [remote_util] [140099341772544] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:51:38,154] - [remote_util] [140099341772544] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:51:38,173] - [remote_util] [140099341772544] - INFO - 172.23.122.117 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 21:51:38,224] - [remote_util] [140099341772544] - INFO - Running systemd command on this server [2021-02-12 21:51:38,224] - [remote_util] [140099341772544] - INFO - running command.raw on 172.23.122.117: systemctl stop couchbase-server.service [2021-02-12 21:51:39,972] - [remote_util] [140099341772544] - INFO - command executed successfully with root [2021-02-12 21:51:39,972] - [basetestcase] [140099341772544] - INFO - Couchbase stopped [2021-02-12 21:51:39,973] - [basetestcase] [140099341772544] - INFO - sleep for 5 secs. ... [2021-02-12 21:51:44,978] - [remote_util] [140099341772544] - INFO - running command.raw on 172.23.122.117: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-12 21:51:44,997] - [remote_util] [140099341772544] - INFO - command executed successfully with root [2021-02-12 21:51:44,997] - [remote_util] [140099341772544] - INFO - running command.raw on 172.23.122.117: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-12 21:51:45,050] - [remote_util] [140099341772544] - INFO - command executed successfully with root [2021-02-12 21:51:45,052] - [remote_util] [140099341772544] - INFO - SSH Connecting to 172.23.122.117 with username:root, attempt#1 of 5 [2021-02-12 21:51:45,156] - [remote_util] [140099341772544] - INFO - SSH Connected to 172.23.122.117 as root [2021-02-12 21:51:45,407] - [remote_util] [140099341772544] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:51:45,697] - [remote_util] [140099341772544] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:51:45,715] - [remote_util] [140099341772544] - INFO - 172.23.122.117 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 21:51:45,716] - [remote_util] [140099341772544] - INFO - Checking for process beam.smp on linux [2021-02-12 21:51:45,751] - [remote_util] [140099341772544] - INFO - Starting couchbase server [2021-02-12 21:51:45,842] - [remote_util] [140099341772544] - INFO - Running systemd command on this server [2021-02-12 21:51:45,842] - [remote_util] [140099341772544] - INFO - running command.raw on 172.23.122.117: systemctl start couchbase-server.service [2021-02-12 21:51:45,864] - [remote_util] [140099341772544] - INFO - command executed successfully with root [2021-02-12 21:51:45,864] - [remote_util] [140099341772544] - INFO - 172.23.122.117:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-12 21:51:50,869] - [remote_util] [140099341772544] - INFO - running command.raw on 172.23.122.117: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-12 21:51:50,894] - [remote_util] [140099341772544] - INFO - command executed successfully with root [2021-02-12 21:51:50,894] - [remote_util] [140099341772544] - INFO - Couchbase server status: [' Process: 34226 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-12 21:51:50,894] - [remote_util] [140099341772544] - INFO - Checking for process beam.smp on linux [2021-02-12 21:51:50,962] - [remote_util] [140099341772544] - INFO - process beam.smp is running on 172.23.122.117: with pid 34370 [2021-02-12 21:51:50,962] - [basetestcase] [140099341772544] - INFO - Couchbase started [2021-02-12 21:51:50,964] - [remote_util] [140099341772544] - INFO - SSH Connecting to 172.23.122.97 with username:root, attempt#1 of 5 [2021-02-12 21:51:51,067] - [remote_util] [140099341772544] - INFO - SSH Connected to 172.23.122.97 as root [2021-02-12 21:51:51,330] - [remote_util] [140099341772544] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:51:51,629] - [remote_util] [140099341772544] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:51:51,637] - [remote_util] [140099341772544] - INFO - SSH Connecting to 172.23.122.97 with username:root, attempt#1 of 5 [2021-02-12 21:51:51,736] - [remote_util] [140099341772544] - INFO - SSH Connected to 172.23.122.97 as root [2021-02-12 21:51:51,993] - [remote_util] [140099341772544] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:51:52,293] - [remote_util] [140099341772544] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:51:52,313] - [remote_util] [140099341772544] - INFO - 172.23.122.97 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 21:51:52,368] - [remote_util] [140099341772544] - INFO - Running systemd command on this server [2021-02-12 21:51:52,368] - [remote_util] [140099341772544] - INFO - running command.raw on 172.23.122.97: systemctl stop couchbase-server.service [2021-02-12 21:51:54,054] - [remote_util] [140099341772544] - INFO - command executed successfully with root [2021-02-12 21:51:54,054] - [basetestcase] [140099341772544] - INFO - Couchbase stopped [2021-02-12 21:51:54,054] - [basetestcase] [140099341772544] - INFO - sleep for 5 secs. ... [2021-02-12 21:51:59,059] - [remote_util] [140099341772544] - INFO - running command.raw on 172.23.122.97: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-12 21:51:59,079] - [remote_util] [140099341772544] - INFO - command executed successfully with root [2021-02-12 21:51:59,079] - [remote_util] [140099341772544] - INFO - running command.raw on 172.23.122.97: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-12 21:51:59,133] - [remote_util] [140099341772544] - INFO - command executed successfully with root [2021-02-12 21:51:59,135] - [remote_util] [140099341772544] - INFO - SSH Connecting to 172.23.122.97 with username:root, attempt#1 of 5 [2021-02-12 21:51:59,238] - [remote_util] [140099341772544] - INFO - SSH Connected to 172.23.122.97 as root [2021-02-12 21:51:59,500] - [remote_util] [140099341772544] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:51:59,797] - [remote_util] [140099341772544] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:51:59,816] - [remote_util] [140099341772544] - INFO - 172.23.122.97 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 21:51:59,816] - [remote_util] [140099341772544] - INFO - Checking for process beam.smp on linux [2021-02-12 21:51:59,852] - [remote_util] [140099341772544] - INFO - Starting couchbase server [2021-02-12 21:51:59,943] - [remote_util] [140099341772544] - INFO - Running systemd command on this server [2021-02-12 21:51:59,943] - [remote_util] [140099341772544] - INFO - running command.raw on 172.23.122.97: systemctl start couchbase-server.service [2021-02-12 21:51:59,963] - [remote_util] [140099341772544] - INFO - command executed successfully with root [2021-02-12 21:51:59,963] - [remote_util] [140099341772544] - INFO - 172.23.122.97:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-12 21:52:04,968] - [remote_util] [140099341772544] - INFO - running command.raw on 172.23.122.97: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-12 21:52:04,995] - [remote_util] [140099341772544] - INFO - command executed successfully with root [2021-02-12 21:52:04,995] - [remote_util] [140099341772544] - INFO - Couchbase server status: [' Process: 67420 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-12 21:52:04,996] - [remote_util] [140099341772544] - INFO - Checking for process beam.smp on linux [2021-02-12 21:52:05,065] - [remote_util] [140099341772544] - INFO - process beam.smp is running on 172.23.122.97: with pid 67565 [2021-02-12 21:52:05,065] - [basetestcase] [140099341772544] - INFO - Couchbase started [2021-02-12 21:52:05,067] - [remote_util] [140099341772544] - INFO - SSH Connecting to 172.23.122.82 with username:root, attempt#1 of 5 [2021-02-12 21:52:05,167] - [remote_util] [140099341772544] - INFO - SSH Connected to 172.23.122.82 as root [2021-02-12 21:52:05,431] - [remote_util] [140099341772544] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:52:05,720] - [remote_util] [140099341772544] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:52:05,728] - [remote_util] [140099341772544] - INFO - SSH Connecting to 172.23.122.82 with username:root, attempt#1 of 5 [2021-02-12 21:52:05,824] - [remote_util] [140099341772544] - INFO - SSH Connected to 172.23.122.82 as root [2021-02-12 21:52:06,075] - [remote_util] [140099341772544] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:52:06,365] - [remote_util] [140099341772544] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:52:06,383] - [remote_util] [140099341772544] - INFO - 172.23.122.82 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 21:52:06,431] - [remote_util] [140099341772544] - INFO - Running systemd command on this server [2021-02-12 21:52:06,431] - [remote_util] [140099341772544] - INFO - running command.raw on 172.23.122.82: systemctl stop couchbase-server.service [2021-02-12 21:52:08,429] - [remote_util] [140099341772544] - INFO - command executed successfully with root [2021-02-12 21:52:08,429] - [basetestcase] [140099341772544] - INFO - Couchbase stopped [2021-02-12 21:52:08,429] - [basetestcase] [140099341772544] - INFO - sleep for 5 secs. ... [2021-02-12 21:52:13,435] - [remote_util] [140099341772544] - INFO - running command.raw on 172.23.122.82: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-12 21:52:13,455] - [remote_util] [140099341772544] - INFO - command executed successfully with root [2021-02-12 21:52:13,455] - [remote_util] [140099341772544] - INFO - running command.raw on 172.23.122.82: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-12 21:52:13,508] - [remote_util] [140099341772544] - INFO - command executed successfully with root [2021-02-12 21:52:13,509] - [remote_util] [140099341772544] - INFO - SSH Connecting to 172.23.122.82 with username:root, attempt#1 of 5 [2021-02-12 21:52:13,608] - [remote_util] [140099341772544] - INFO - SSH Connected to 172.23.122.82 as root [2021-02-12 21:52:13,865] - [remote_util] [140099341772544] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:52:14,157] - [remote_util] [140099341772544] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:52:14,214] - [remote_util] [140099341772544] - INFO - 172.23.122.82 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 21:52:14,214] - [remote_util] [140099341772544] - INFO - Checking for process beam.smp on linux [2021-02-12 21:52:14,282] - [remote_util] [140099341772544] - INFO - Starting couchbase server [2021-02-12 21:52:14,376] - [remote_util] [140099341772544] - INFO - Running systemd command on this server [2021-02-12 21:52:14,376] - [remote_util] [140099341772544] - INFO - running command.raw on 172.23.122.82: systemctl start couchbase-server.service [2021-02-12 21:52:14,399] - [remote_util] [140099341772544] - INFO - command executed successfully with root [2021-02-12 21:52:14,399] - [remote_util] [140099341772544] - INFO - 172.23.122.82:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-12 21:52:19,404] - [remote_util] [140099341772544] - INFO - running command.raw on 172.23.122.82: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-12 21:52:19,431] - [remote_util] [140099341772544] - INFO - command executed successfully with root [2021-02-12 21:52:19,432] - [remote_util] [140099341772544] - INFO - Couchbase server status: [' Process: 59268 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-12 21:52:19,432] - [remote_util] [140099341772544] - INFO - Checking for process beam.smp on linux [2021-02-12 21:52:19,503] - [remote_util] [140099341772544] - INFO - process beam.smp is running on 172.23.122.82: with pid 59443 [2021-02-12 21:52:19,503] - [basetestcase] [140099341772544] - INFO - Couchbase started [2021-02-12 21:52:19,505] - [remote_util] [140099341772544] - INFO - SSH Connecting to 172.23.122.118 with username:root, attempt#1 of 5 [2021-02-12 21:52:19,608] - [remote_util] [140099341772544] - INFO - SSH Connected to 172.23.122.118 as root [2021-02-12 21:52:19,857] - [remote_util] [140099341772544] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:52:20,148] - [remote_util] [140099341772544] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:52:20,156] - [remote_util] [140099341772544] - INFO - SSH Connecting to 172.23.122.118 with username:root, attempt#1 of 5 [2021-02-12 21:52:20,252] - [remote_util] [140099341772544] - INFO - SSH Connected to 172.23.122.118 as root [2021-02-12 21:52:20,490] - [remote_util] [140099341772544] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:52:20,780] - [remote_util] [140099341772544] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:52:20,798] - [remote_util] [140099341772544] - INFO - 172.23.122.118 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 21:52:20,847] - [remote_util] [140099341772544] - INFO - Running systemd command on this server [2021-02-12 21:52:20,847] - [remote_util] [140099341772544] - INFO - running command.raw on 172.23.122.118: systemctl stop couchbase-server.service [2021-02-12 21:52:22,627] - [remote_util] [140099341772544] - INFO - command executed successfully with root [2021-02-12 21:52:22,628] - [basetestcase] [140099341772544] - INFO - Couchbase stopped [2021-02-12 21:52:22,628] - [basetestcase] [140099341772544] - INFO - sleep for 5 secs. ... [2021-02-12 21:52:27,633] - [remote_util] [140099341772544] - INFO - running command.raw on 172.23.122.118: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-12 21:52:27,650] - [remote_util] [140099341772544] - INFO - command executed successfully with root [2021-02-12 21:52:27,650] - [remote_util] [140099341772544] - INFO - running command.raw on 172.23.122.118: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-12 21:52:27,703] - [remote_util] [140099341772544] - INFO - command executed successfully with root [2021-02-12 21:52:27,704] - [remote_util] [140099341772544] - INFO - SSH Connecting to 172.23.122.118 with username:root, attempt#1 of 5 [2021-02-12 21:52:27,802] - [remote_util] [140099341772544] - INFO - SSH Connected to 172.23.122.118 as root [2021-02-12 21:52:28,046] - [remote_util] [140099341772544] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:52:28,338] - [remote_util] [140099341772544] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:52:28,355] - [remote_util] [140099341772544] - INFO - 172.23.122.118 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 21:52:28,356] - [remote_util] [140099341772544] - INFO - Checking for process beam.smp on linux [2021-02-12 21:52:28,390] - [remote_util] [140099341772544] - INFO - Starting couchbase server [2021-02-12 21:52:28,480] - [remote_util] [140099341772544] - INFO - Running systemd command on this server [2021-02-12 21:52:28,480] - [remote_util] [140099341772544] - INFO - running command.raw on 172.23.122.118: systemctl start couchbase-server.service [2021-02-12 21:52:28,502] - [remote_util] [140099341772544] - INFO - command executed successfully with root [2021-02-12 21:52:28,502] - [remote_util] [140099341772544] - INFO - 172.23.122.118:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-12 21:52:33,507] - [remote_util] [140099341772544] - INFO - running command.raw on 172.23.122.118: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-12 21:52:33,532] - [remote_util] [140099341772544] - INFO - command executed successfully with root [2021-02-12 21:52:33,532] - [remote_util] [140099341772544] - INFO - Couchbase server status: [' Process: 86225 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-12 21:52:33,533] - [remote_util] [140099341772544] - INFO - Checking for process beam.smp on linux [2021-02-12 21:52:33,599] - [remote_util] [140099341772544] - INFO - process beam.smp is running on 172.23.122.118: with pid 86369 [2021-02-12 21:52:33,600] - [basetestcase] [140099341772544] - INFO - Couchbase started [2021-02-12 21:52:33,601] - [remote_util] [140099341772544] - INFO - SSH Connecting to 172.23.122.116 with username:root, attempt#1 of 5 [2021-02-12 21:52:33,704] - [remote_util] [140099341772544] - INFO - SSH Connected to 172.23.122.116 as root [2021-02-12 21:52:33,979] - [remote_util] [140099341772544] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:52:34,269] - [remote_util] [140099341772544] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:52:34,278] - [remote_util] [140099341772544] - INFO - SSH Connecting to 172.23.122.116 with username:root, attempt#1 of 5 [2021-02-12 21:52:34,374] - [remote_util] [140099341772544] - INFO - SSH Connected to 172.23.122.116 as root [2021-02-12 21:52:34,635] - [remote_util] [140099341772544] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:52:34,919] - [remote_util] [140099341772544] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:52:34,936] - [remote_util] [140099341772544] - INFO - 172.23.122.116 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 21:52:34,982] - [remote_util] [140099341772544] - INFO - Running systemd command on this server [2021-02-12 21:52:34,982] - [remote_util] [140099341772544] - INFO - running command.raw on 172.23.122.116: systemctl stop couchbase-server.service [2021-02-12 21:52:36,655] - [remote_util] [140099341772544] - INFO - command executed successfully with root [2021-02-12 21:52:36,655] - [basetestcase] [140099341772544] - INFO - Couchbase stopped [2021-02-12 21:52:36,656] - [basetestcase] [140099341772544] - INFO - sleep for 5 secs. ... [2021-02-12 21:52:41,661] - [remote_util] [140099341772544] - INFO - running command.raw on 172.23.122.116: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-12 21:52:41,676] - [remote_util] [140099341772544] - INFO - command executed successfully with root [2021-02-12 21:52:41,676] - [remote_util] [140099341772544] - INFO - running command.raw on 172.23.122.116: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-12 21:52:41,729] - [remote_util] [140099341772544] - INFO - command executed successfully with root [2021-02-12 21:52:41,730] - [remote_util] [140099341772544] - INFO - SSH Connecting to 172.23.122.116 with username:root, attempt#1 of 5 [2021-02-12 21:52:41,828] - [remote_util] [140099341772544] - INFO - SSH Connected to 172.23.122.116 as root [2021-02-12 21:52:42,097] - [remote_util] [140099341772544] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:52:42,385] - [remote_util] [140099341772544] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:52:42,402] - [remote_util] [140099341772544] - INFO - 172.23.122.116 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 21:52:42,403] - [remote_util] [140099341772544] - INFO - Checking for process beam.smp on linux [2021-02-12 21:52:42,437] - [remote_util] [140099341772544] - INFO - Starting couchbase server [2021-02-12 21:52:42,525] - [remote_util] [140099341772544] - INFO - Running systemd command on this server [2021-02-12 21:52:42,525] - [remote_util] [140099341772544] - INFO - running command.raw on 172.23.122.116: systemctl start couchbase-server.service [2021-02-12 21:52:42,546] - [remote_util] [140099341772544] - INFO - command executed successfully with root [2021-02-12 21:52:42,547] - [remote_util] [140099341772544] - INFO - 172.23.122.116:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-12 21:52:47,552] - [remote_util] [140099341772544] - INFO - running command.raw on 172.23.122.116: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-12 21:52:47,576] - [remote_util] [140099341772544] - INFO - command executed successfully with root [2021-02-12 21:52:47,576] - [remote_util] [140099341772544] - INFO - Couchbase server status: [' Process: 49555 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-12 21:52:47,576] - [remote_util] [140099341772544] - INFO - Checking for process beam.smp on linux [2021-02-12 21:52:47,644] - [remote_util] [140099341772544] - INFO - process beam.smp is running on 172.23.122.116: with pid 49700 [2021-02-12 21:52:47,644] - [basetestcase] [140099341772544] - INFO - Couchbase started [2021-02-12 21:52:47,644] - [basetestcase] [140099341772544] - INFO - sleep for 10 secs. ... [2021-02-12 21:52:57,673] - [remote_util] [140099341772544] - INFO - SSH Connecting to 172.23.123.61 with username:root, attempt#1 of 5 [2021-02-12 21:52:57,674] - [remote_util] [140099350165248] - INFO - SSH Connecting to 172.23.123.62 with username:root, attempt#1 of 5 [2021-02-12 21:52:57,676] - [remote_util] [140099316594432] - INFO - SSH Connecting to 172.23.122.117 with username:root, attempt#1 of 5 [2021-02-12 21:52:57,678] - [remote_util] [140099324987136] - INFO - SSH Connecting to 172.23.122.90 with username:root, attempt#1 of 5 [2021-02-12 21:52:57,681] - [remote_util] [140099299809024] - INFO - SSH Connecting to 172.23.122.97 with username:root, attempt#1 of 5 [2021-02-12 21:52:57,683] - [remote_util] [140098729400064] - INFO - SSH Connecting to 172.23.122.82 with username:root, attempt#1 of 5 [2021-02-12 21:52:57,685] - [remote_util] [140097311463168] - INFO - SSH Connecting to 172.23.122.116 with username:root, attempt#1 of 5 [2021-02-12 21:52:57,686] - [remote_util] [140098721007360] - INFO - SSH Connecting to 172.23.122.118 with username:root, attempt#1 of 5 [2021-02-12 21:52:57,782] - [remote_util] [140099316594432] - INFO - SSH Connected to 172.23.122.117 as root [2021-02-12 21:52:57,787] - [remote_util] [140099299809024] - INFO - SSH Connected to 172.23.122.97 as root [2021-02-12 21:52:57,787] - [remote_util] [140098729400064] - INFO - SSH Connected to 172.23.122.82 as root [2021-02-12 21:52:57,788] - [remote_util] [140099341772544] - INFO - SSH Connected to 172.23.123.61 as root [2021-02-12 21:52:57,789] - [remote_util] [140098721007360] - INFO - SSH Connected to 172.23.122.118 as root [2021-02-12 21:52:57,790] - [remote_util] [140097311463168] - INFO - SSH Connected to 172.23.122.116 as root [2021-02-12 21:52:57,791] - [remote_util] [140099324987136] - INFO - SSH Connected to 172.23.122.90 as root [2021-02-12 21:52:57,794] - [remote_util] [140099350165248] - INFO - SSH Connected to 172.23.123.62 as root [2021-02-12 21:52:58,056] - [remote_util] [140098721007360] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:52:58,059] - [remote_util] [140099316594432] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:52:58,074] - [remote_util] [140099299809024] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:52:58,084] - [remote_util] [140098729400064] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:52:58,092] - [remote_util] [140097311463168] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:52:58,098] - [remote_util] [140099324987136] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:52:58,125] - [remote_util] [140099341772544] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:52:58,128] - [remote_util] [140099350165248] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 21:52:58,370] - [remote_util] [140099316594432] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:52:58,370] - [remote_util] [140099316594432] - INFO - running command.raw on 172.23.122.117: /opt/couchbase/bin/cbcollect_info 172.23.122.117-20210212-2152-diag.zip [2021-02-12 21:52:58,392] - [remote_util] [140098729400064] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:52:58,393] - [remote_util] [140098729400064] - INFO - running command.raw on 172.23.122.82: /opt/couchbase/bin/cbcollect_info 172.23.122.82-20210212-2152-diag.zip [2021-02-12 21:52:58,402] - [remote_util] [140098721007360] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:52:58,402] - [remote_util] [140098721007360] - INFO - running command.raw on 172.23.122.118: /opt/couchbase/bin/cbcollect_info 172.23.122.118-20210212-2152-diag.zip [2021-02-12 21:52:58,406] - [remote_util] [140099324987136] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:52:58,407] - [remote_util] [140099324987136] - INFO - running command.raw on 172.23.122.90: /opt/couchbase/bin/cbcollect_info 172.23.122.90-20210212-2152-diag.zip [2021-02-12 21:52:58,424] - [remote_util] [140099299809024] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:52:58,424] - [remote_util] [140099299809024] - INFO - running command.raw on 172.23.122.97: /opt/couchbase/bin/cbcollect_info 172.23.122.97-20210212-2152-diag.zip [2021-02-12 21:52:58,432] - [remote_util] [140097311463168] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:52:58,433] - [remote_util] [140097311463168] - INFO - running command.raw on 172.23.122.116: /opt/couchbase/bin/cbcollect_info 172.23.122.116-20210212-2152-diag.zip [2021-02-12 21:52:58,445] - [remote_util] [140099341772544] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:52:58,445] - [remote_util] [140099341772544] - INFO - running command.raw on 172.23.123.61: /opt/couchbase/bin/cbcollect_info 172.23.123.61-20210212-2152-diag.zip [2021-02-12 21:52:58,450] - [remote_util] [140099350165248] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 21:52:58,450] - [remote_util] [140099350165248] - INFO - running command.raw on 172.23.123.62: /opt/couchbase/bin/cbcollect_info 172.23.123.62-20210212-2152-diag.zip [2021-02-12 21:54:24,333] - [remote_util] [140097311463168] - INFO - command executed successfully with root [2021-02-12 21:54:24,556] - [remote_util] [140097311463168] - INFO - running command.raw on 172.23.122.116: rm -f /root/172.23.122.116-20210212-2152-diag.zip [2021-02-12 21:54:24,572] - [remote_util] [140097311463168] - INFO - command executed successfully with root [2021-02-12 21:54:30,074] - [remote_util] [140099316594432] - INFO - command executed successfully with root [2021-02-12 21:54:30,389] - [remote_util] [140099316594432] - INFO - running command.raw on 172.23.122.117: rm -f /root/172.23.122.117-20210212-2152-diag.zip [2021-02-12 21:54:30,406] - [remote_util] [140099316594432] - INFO - command executed successfully with root [2021-02-12 21:54:31,314] - [remote_util] [140098721007360] - INFO - command executed successfully with root [2021-02-12 21:54:31,684] - [remote_util] [140098721007360] - INFO - running command.raw on 172.23.122.118: rm -f /root/172.23.122.118-20210212-2152-diag.zip [2021-02-12 21:54:31,701] - [remote_util] [140098721007360] - INFO - command executed successfully with root [2021-02-12 21:54:36,837] - [remote_util] [140099350165248] - INFO - command executed successfully with root [2021-02-12 21:54:37,178] - [remote_util] [140099341772544] - INFO - command executed successfully with root [2021-02-12 21:54:37,622] - [remote_util] [140099350165248] - INFO - running command.raw on 172.23.123.62: rm -f /root/172.23.123.62-20210212-2152-diag.zip [2021-02-12 21:54:37,646] - [remote_util] [140099350165248] - INFO - command executed successfully with root [2021-02-12 21:54:37,844] - [remote_util] [140099341772544] - INFO - running command.raw on 172.23.123.61: rm -f /root/172.23.123.61-20210212-2152-diag.zip [2021-02-12 21:54:37,864] - [remote_util] [140099341772544] - INFO - command executed successfully with root [2021-02-12 21:54:38,997] - [remote_util] [140098729400064] - INFO - command executed successfully with root [2021-02-12 21:54:39,473] - [remote_util] [140098729400064] - INFO - running command.raw on 172.23.122.82: rm -f /root/172.23.122.82-20210212-2152-diag.zip [2021-02-12 21:54:39,492] - [remote_util] [140098729400064] - INFO - command executed successfully with root [2021-02-12 21:54:40,260] - [remote_util] [140099324987136] - INFO - command executed successfully with root [2021-02-12 21:54:40,783] - [remote_util] [140099324987136] - INFO - running command.raw on 172.23.122.90: rm -f /root/172.23.122.90-20210212-2152-diag.zip [2021-02-12 21:54:40,803] - [remote_util] [140099324987136] - INFO - command executed successfully with root [2021-02-12 21:54:42,013] - [remote_util] [140099299809024] - INFO - command executed successfully with root [2021-02-12 21:54:42,650] - [remote_util] [140099299809024] - INFO - running command.raw on 172.23.122.97: rm -f /root/172.23.122.97-20210212-2152-diag.zip [2021-02-12 21:54:42,672] - [remote_util] [140099299809024] - INFO - command executed successfully with root