[2021-02-27 16:54:28,137] - [eventing_base] [140542730036992] - INFO - Starting Test: test_eventing_rebalance_with_multiple_kv_nodes [2021-02-27 16:54:28,139] - [remote_util] [140542730036992] - INFO - SSH Connecting to 172.23.104.97 with username:root, attempt#1 of 5 [2021-02-27 16:54:28,302] - [remote_util] [140542730036992] - INFO - SSH Connected to 172.23.104.97 as root [2021-02-27 16:54:28,878] - [remote_util] [140542730036992] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-27 16:54:29,319] - [remote_util] [140542730036992] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-27 16:54:29,331] - [rest_client] [140542730036992] - INFO - -->is_ns_server_running? [2021-02-27 16:54:29,360] - [rest_client] [140542730036992] - INFO - Node version in cluster 7.0.0-4554-enterprise [2021-02-27 16:54:29,391] - [rest_client] [140542730036992] - INFO - Node versions in cluster ['7.0.0-4554-enterprise', '7.0.0-4554-enterprise', '7.0.0-4554-enterprise', '7.0.0-4554-enterprise', '7.0.0-4554-enterprise'] [2021-02-27 16:54:29,391] - [basetestcase] [140542730036992] - INFO - ============== basetestcase setup was started for test #2 test_eventing_rebalance_with_multiple_kv_nodes============== [2021-02-27 16:54:29,458] - [bucket_helper] [140542730036992] - INFO - deleting existing buckets ['dst_bucket', 'metadata', 'src_bucket'] on 172.23.104.97 [2021-02-27 16:54:30,712] - [bucket_helper] [140542730036992] - INFO - waiting for bucket deletion to complete.... [2021-02-27 16:54:30,741] - [rest_client] [140542730036992] - INFO - node 172.23.104.97 existing buckets : ['metadata', 'src_bucket'] [2021-02-27 16:54:30,742] - [bucket_helper] [140542730036992] - INFO - deleted bucket : dst_bucket from 172.23.104.97 [2021-02-27 16:54:31,729] - [bucket_helper] [140542730036992] - INFO - waiting for bucket deletion to complete.... [2021-02-27 16:54:31,747] - [rest_client] [140542730036992] - INFO - node 172.23.104.97 existing buckets : ['src_bucket'] [2021-02-27 16:54:31,748] - [bucket_helper] [140542730036992] - INFO - deleted bucket : metadata from 172.23.104.97 [2021-02-27 16:54:33,447] - [bucket_helper] [140542730036992] - INFO - waiting for bucket deletion to complete.... [2021-02-27 16:54:33,452] - [rest_client] [140542730036992] - INFO - node 172.23.104.97 existing buckets : [] [2021-02-27 16:54:33,453] - [bucket_helper] [140542730036992] - INFO - deleted bucket : src_bucket from 172.23.104.97 [2021-02-27 16:54:33,461] - [bucket_helper] [140542730036992] - INFO - Could not find any buckets for node 172.23.121.213, nothing to delete [2021-02-27 16:54:33,467] - [bucket_helper] [140542730036992] - INFO - Could not find any buckets for node 172.23.105.238, nothing to delete [2021-02-27 16:54:33,476] - [bucket_helper] [140542730036992] - INFO - Could not find any buckets for node 172.23.105.247, nothing to delete [2021-02-27 16:54:33,483] - [bucket_helper] [140542730036992] - INFO - Could not find any buckets for node 172.23.106.1, nothing to delete [2021-02-27 16:54:33,490] - [bucket_helper] [140542730036992] - INFO - Could not find any buckets for node 172.23.107.65, nothing to delete [2021-02-27 16:54:33,496] - [bucket_helper] [140542730036992] - INFO - Could not find any buckets for node 172.23.105.240, nothing to delete [2021-02-27 16:54:33,501] - [bucket_helper] [140542730036992] - INFO - Could not find any buckets for node 172.23.105.249, nothing to delete [2021-02-27 16:54:33,513] - [rest_client] [140542730036992] - INFO - -->is_ns_server_running? [2021-02-27 16:54:33,539] - [cluster_helper] [140542730036992] - INFO - rebalancing all nodes in order to remove nodes [2021-02-27 16:54:33,542] - [rest_client] [140542730036992] - INFO - rebalance params : {'knownNodes': 'ns_1@172.23.104.97,ns_1@172.23.105.247,ns_1@172.23.106.1,ns_1@172.23.107.65,ns_1@172.23.121.213', 'ejectedNodes': 'ns_1@172.23.105.247,ns_1@172.23.106.1,ns_1@172.23.107.65,ns_1@172.23.121.213', 'user': 'Administrator', 'password': 'password'} [2021-02-27 16:54:33,602] - [rest_client] [140542730036992] - INFO - rebalance operation started [2021-02-27 16:54:33,605] - [rest_client] [140542730036992] - INFO - rebalance percentage : 0.00 % [2021-02-27 16:54:43,619] - [rest_client] [140542730036992] - INFO - rebalance percentage : 30.00 % [2021-02-27 16:54:53,633] - [rest_client] [140542730036992] - INFO - rebalance percentage : 90.00 % [2021-02-27 16:55:13,653] - [rest_client] [140542730036992] - INFO - rebalance progress took 40.05 seconds [2021-02-27 16:55:13,653] - [rest_client] [140542730036992] - INFO - sleep for 10 seconds after rebalance... [2021-02-27 16:55:23,690] - [cluster_helper] [140542730036992] - INFO - removed all the nodes from cluster associated with ip:172.23.104.97 port:8091 ssh_username:root ? [('ns_1@172.23.105.247', 8091), ('ns_1@172.23.106.1', 8091), ('ns_1@172.23.107.65', 8091), ('ns_1@172.23.121.213', 8091)] [2021-02-27 16:55:23,699] - [cluster_helper] [140542730036992] - INFO - waiting for ns_server @ 172.23.104.97:8091 [2021-02-27 16:55:23,700] - [rest_client] [140542730036992] - INFO - -->is_ns_server_running? [2021-02-27 16:55:23,708] - [cluster_helper] [140542730036992] - INFO - ns_server @ 172.23.104.97:8091 is running [2021-02-27 16:55:23,711] - [cluster_helper] [140542730036992] - INFO - waiting for ns_server @ 172.23.121.213:8091 [2021-02-27 16:55:23,711] - [rest_client] [140542730036992] - INFO - -->is_ns_server_running? [2021-02-27 16:55:23,714] - [cluster_helper] [140542730036992] - INFO - ns_server @ 172.23.121.213:8091 is running [2021-02-27 16:55:23,717] - [cluster_helper] [140542730036992] - INFO - waiting for ns_server @ 172.23.105.238:8091 [2021-02-27 16:55:23,717] - [rest_client] [140542730036992] - INFO - -->is_ns_server_running? [2021-02-27 16:55:23,720] - [cluster_helper] [140542730036992] - INFO - ns_server @ 172.23.105.238:8091 is running [2021-02-27 16:55:23,724] - [cluster_helper] [140542730036992] - INFO - waiting for ns_server @ 172.23.105.247:8091 [2021-02-27 16:55:23,724] - [rest_client] [140542730036992] - INFO - -->is_ns_server_running? [2021-02-27 16:55:23,727] - [cluster_helper] [140542730036992] - INFO - ns_server @ 172.23.105.247:8091 is running [2021-02-27 16:55:23,731] - [cluster_helper] [140542730036992] - INFO - waiting for ns_server @ 172.23.106.1:8091 [2021-02-27 16:55:23,731] - [rest_client] [140542730036992] - INFO - -->is_ns_server_running? [2021-02-27 16:55:23,733] - [cluster_helper] [140542730036992] - INFO - ns_server @ 172.23.106.1:8091 is running [2021-02-27 16:55:23,736] - [cluster_helper] [140542730036992] - INFO - waiting for ns_server @ 172.23.107.65:8091 [2021-02-27 16:55:23,737] - [rest_client] [140542730036992] - INFO - -->is_ns_server_running? [2021-02-27 16:55:23,739] - [cluster_helper] [140542730036992] - INFO - ns_server @ 172.23.107.65:8091 is running [2021-02-27 16:55:23,743] - [cluster_helper] [140542730036992] - INFO - waiting for ns_server @ 172.23.105.240:8091 [2021-02-27 16:55:23,743] - [rest_client] [140542730036992] - INFO - -->is_ns_server_running? [2021-02-27 16:55:23,746] - [cluster_helper] [140542730036992] - INFO - ns_server @ 172.23.105.240:8091 is running [2021-02-27 16:55:23,750] - [cluster_helper] [140542730036992] - INFO - waiting for ns_server @ 172.23.105.249:8091 [2021-02-27 16:55:23,750] - [rest_client] [140542730036992] - INFO - -->is_ns_server_running? [2021-02-27 16:55:23,753] - [cluster_helper] [140542730036992] - INFO - ns_server @ 172.23.105.249:8091 is running [2021-02-27 16:55:23,753] - [basetestcase] [140542730036992] - INFO - Building docker image with java sdk client [2021-02-27 16:55:27,442] - [basetestcase] [140542730036992] - INFO - initializing cluster [2021-02-27 16:55:27,444] - [remote_util] [140542730036992] - INFO - SSH Connecting to 172.23.104.97 with username:root, attempt#1 of 5 [2021-02-27 16:55:27,624] - [remote_util] [140542730036992] - INFO - SSH Connected to 172.23.104.97 as root [2021-02-27 16:55:28,261] - [remote_util] [140542730036992] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-27 16:55:28,695] - [remote_util] [140542730036992] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-27 16:55:28,715] - [remote_util] [140542730036992] - INFO - SSH Connecting to 172.23.104.97 with username:root, attempt#1 of 5 [2021-02-27 16:55:28,879] - [remote_util] [140542730036992] - INFO - SSH Connected to 172.23.104.97 as root [2021-02-27 16:55:29,500] - [remote_util] [140542730036992] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-27 16:55:29,943] - [remote_util] [140542730036992] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-27 16:55:30,026] - [remote_util] [140542730036992] - INFO - 172.23.104.97 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-27 16:55:30,140] - [remote_util] [140542730036992] - INFO - Running systemd command on this server [2021-02-27 16:55:30,140] - [remote_util] [140542730036992] - INFO - running command.raw on 172.23.104.97: systemctl stop couchbase-server.service [2021-02-27 16:55:32,782] - [remote_util] [140542730036992] - INFO - command executed successfully with root [2021-02-27 16:55:32,783] - [basetestcase] [140542730036992] - INFO - Couchbase stopped [2021-02-27 16:55:32,783] - [basetestcase] [140542730036992] - INFO - sleep for 5 secs. ... [2021-02-27 16:55:37,788] - [remote_util] [140542730036992] - INFO - running command.raw on 172.23.104.97: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-27 16:55:37,827] - [remote_util] [140542730036992] - INFO - command executed successfully with root [2021-02-27 16:55:37,827] - [remote_util] [140542730036992] - INFO - running command.raw on 172.23.104.97: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-27 16:55:37,901] - [remote_util] [140542730036992] - INFO - command executed successfully with root [2021-02-27 16:55:37,902] - [remote_util] [140542730036992] - INFO - SSH Connecting to 172.23.104.97 with username:root, attempt#1 of 5 [2021-02-27 16:55:38,069] - [remote_util] [140542730036992] - INFO - SSH Connected to 172.23.104.97 as root [2021-02-27 16:55:38,733] - [remote_util] [140542730036992] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-27 16:55:39,156] - [remote_util] [140542730036992] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-27 16:55:39,243] - [remote_util] [140542730036992] - INFO - 172.23.104.97 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-27 16:55:39,243] - [remote_util] [140542730036992] - INFO - Checking for process beam.smp on linux [2021-02-27 16:55:39,321] - [remote_util] [140542730036992] - INFO - Starting couchbase server [2021-02-27 16:55:39,486] - [remote_util] [140542730036992] - INFO - Running systemd command on this server [2021-02-27 16:55:39,486] - [remote_util] [140542730036992] - INFO - running command.raw on 172.23.104.97: systemctl start couchbase-server.service [2021-02-27 16:55:39,536] - [remote_util] [140542730036992] - INFO - command executed successfully with root [2021-02-27 16:55:39,537] - [remote_util] [140542730036992] - INFO - 172.23.104.97:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-27 16:55:44,540] - [remote_util] [140542730036992] - INFO - running command.raw on 172.23.104.97: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-27 16:55:44,597] - [remote_util] [140542730036992] - INFO - command executed successfully with root [2021-02-27 16:55:44,597] - [remote_util] [140542730036992] - INFO - Couchbase server status: [' Process: 13304 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-27 16:55:44,598] - [remote_util] [140542730036992] - INFO - Checking for process beam.smp on linux [2021-02-27 16:55:44,642] - [remote_util] [140542730036992] - INFO - process beam.smp is running on 172.23.104.97: with pid 13474 [2021-02-27 16:55:44,643] - [basetestcase] [140542730036992] - INFO - Couchbase started [2021-02-27 16:55:44,644] - [remote_util] [140542730036992] - INFO - SSH Connecting to 172.23.121.213 with username:root, attempt#1 of 5 [2021-02-27 16:55:44,745] - [remote_util] [140542730036992] - INFO - SSH Connected to 172.23.121.213 as root [2021-02-27 16:55:45,004] - [remote_util] [140542730036992] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-27 16:55:45,297] - [remote_util] [140542730036992] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-27 16:55:45,307] - [remote_util] [140542730036992] - INFO - SSH Connecting to 172.23.121.213 with username:root, attempt#1 of 5 [2021-02-27 16:55:45,404] - [remote_util] [140542730036992] - INFO - SSH Connected to 172.23.121.213 as root [2021-02-27 16:55:45,659] - [remote_util] [140542730036992] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-27 16:55:45,953] - [remote_util] [140542730036992] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-27 16:55:46,013] - [remote_util] [140542730036992] - INFO - 172.23.121.213 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-27 16:55:46,067] - [remote_util] [140542730036992] - INFO - Running systemd command on this server [2021-02-27 16:55:46,067] - [remote_util] [140542730036992] - INFO - running command.raw on 172.23.121.213: systemctl stop couchbase-server.service [2021-02-27 16:55:47,681] - [remote_util] [140542730036992] - INFO - command executed successfully with root [2021-02-27 16:55:47,681] - [basetestcase] [140542730036992] - INFO - Couchbase stopped [2021-02-27 16:55:47,681] - [basetestcase] [140542730036992] - INFO - sleep for 5 secs. ... [2021-02-27 16:55:52,684] - [remote_util] [140542730036992] - INFO - running command.raw on 172.23.121.213: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-27 16:55:52,705] - [remote_util] [140542730036992] - INFO - command executed successfully with root [2021-02-27 16:55:52,706] - [remote_util] [140542730036992] - INFO - running command.raw on 172.23.121.213: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-27 16:55:52,759] - [remote_util] [140542730036992] - INFO - command executed successfully with root [2021-02-27 16:55:52,761] - [remote_util] [140542730036992] - INFO - SSH Connecting to 172.23.121.213 with username:root, attempt#1 of 5 [2021-02-27 16:55:52,860] - [remote_util] [140542730036992] - INFO - SSH Connected to 172.23.121.213 as root [2021-02-27 16:55:53,130] - [remote_util] [140542730036992] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-27 16:55:53,426] - [remote_util] [140542730036992] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-27 16:55:53,485] - [remote_util] [140542730036992] - INFO - 172.23.121.213 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-27 16:55:53,485] - [remote_util] [140542730036992] - INFO - Checking for process beam.smp on linux [2021-02-27 16:55:53,520] - [remote_util] [140542730036992] - INFO - Starting couchbase server [2021-02-27 16:55:53,614] - [remote_util] [140542730036992] - INFO - Running systemd command on this server [2021-02-27 16:55:53,614] - [remote_util] [140542730036992] - INFO - running command.raw on 172.23.121.213: systemctl start couchbase-server.service [2021-02-27 16:55:53,635] - [remote_util] [140542730036992] - INFO - command executed successfully with root [2021-02-27 16:55:53,635] - [remote_util] [140542730036992] - INFO - 172.23.121.213:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-27 16:55:58,639] - [remote_util] [140542730036992] - INFO - running command.raw on 172.23.121.213: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-27 16:55:58,666] - [remote_util] [140542730036992] - INFO - command executed successfully with root [2021-02-27 16:55:58,666] - [remote_util] [140542730036992] - INFO - Couchbase server status: [' Process: 23289 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-27 16:55:58,666] - [remote_util] [140542730036992] - INFO - Checking for process beam.smp on linux [2021-02-27 16:55:58,735] - [remote_util] [140542730036992] - INFO - process beam.smp is running on 172.23.121.213: with pid 23431 [2021-02-27 16:55:58,736] - [basetestcase] [140542730036992] - INFO - Couchbase started [2021-02-27 16:55:58,738] - [remote_util] [140542730036992] - INFO - SSH Connecting to 172.23.105.238 with username:root, attempt#1 of 5 [2021-02-27 16:55:58,848] - [remote_util] [140542730036992] - INFO - SSH Connected to 172.23.105.238 as root [2021-02-27 16:55:59,138] - [remote_util] [140542730036992] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-27 16:55:59,447] - [remote_util] [140542730036992] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-27 16:55:59,455] - [remote_util] [140542730036992] - INFO - SSH Connecting to 172.23.105.238 with username:root, attempt#1 of 5 [2021-02-27 16:55:59,561] - [remote_util] [140542730036992] - INFO - SSH Connected to 172.23.105.238 as root [2021-02-27 16:55:59,826] - [remote_util] [140542730036992] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-27 16:56:00,128] - [remote_util] [140542730036992] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-27 16:56:00,188] - [remote_util] [140542730036992] - INFO - 172.23.105.238 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-27 16:56:00,245] - [remote_util] [140542730036992] - INFO - Running systemd command on this server [2021-02-27 16:56:00,246] - [remote_util] [140542730036992] - INFO - running command.raw on 172.23.105.238: systemctl stop couchbase-server.service [2021-02-27 16:56:02,092] - [remote_util] [140542730036992] - INFO - command executed successfully with root [2021-02-27 16:56:02,092] - [basetestcase] [140542730036992] - INFO - Couchbase stopped [2021-02-27 16:56:02,092] - [basetestcase] [140542730036992] - INFO - sleep for 5 secs. ... [2021-02-27 16:56:07,095] - [remote_util] [140542730036992] - INFO - running command.raw on 172.23.105.238: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-27 16:56:07,118] - [remote_util] [140542730036992] - INFO - command executed successfully with root [2021-02-27 16:56:07,118] - [remote_util] [140542730036992] - INFO - running command.raw on 172.23.105.238: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-27 16:56:07,171] - [remote_util] [140542730036992] - INFO - command executed successfully with root [2021-02-27 16:56:07,173] - [remote_util] [140542730036992] - INFO - SSH Connecting to 172.23.105.238 with username:root, attempt#1 of 5 [2021-02-27 16:56:07,276] - [remote_util] [140542730036992] - INFO - SSH Connected to 172.23.105.238 as root [2021-02-27 16:56:07,576] - [remote_util] [140542730036992] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-27 16:56:07,909] - [remote_util] [140542730036992] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-27 16:56:07,968] - [remote_util] [140542730036992] - INFO - 172.23.105.238 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-27 16:56:07,968] - [remote_util] [140542730036992] - INFO - Checking for process beam.smp on linux [2021-02-27 16:56:08,045] - [remote_util] [140542730036992] - INFO - Starting couchbase server [2021-02-27 16:56:08,142] - [remote_util] [140542730036992] - INFO - Running systemd command on this server [2021-02-27 16:56:08,143] - [remote_util] [140542730036992] - INFO - running command.raw on 172.23.105.238: systemctl start couchbase-server.service [2021-02-27 16:56:08,166] - [remote_util] [140542730036992] - INFO - command executed successfully with root [2021-02-27 16:56:08,167] - [remote_util] [140542730036992] - INFO - 172.23.105.238:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-27 16:56:13,172] - [remote_util] [140542730036992] - INFO - running command.raw on 172.23.105.238: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-27 16:56:13,204] - [remote_util] [140542730036992] - INFO - command executed successfully with root [2021-02-27 16:56:13,204] - [remote_util] [140542730036992] - INFO - Couchbase server status: [' Process: 116923 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-27 16:56:13,204] - [remote_util] [140542730036992] - INFO - Checking for process beam.smp on linux [2021-02-27 16:56:13,277] - [remote_util] [140542730036992] - INFO - process beam.smp is running on 172.23.105.238: with pid 117066 [2021-02-27 16:56:13,277] - [basetestcase] [140542730036992] - INFO - Couchbase started [2021-02-27 16:56:13,278] - [remote_util] [140542730036992] - INFO - SSH Connecting to 172.23.105.247 with username:root, attempt#1 of 5 [2021-02-27 16:56:13,390] - [remote_util] [140542730036992] - INFO - SSH Connected to 172.23.105.247 as root [2021-02-27 16:56:13,679] - [remote_util] [140542730036992] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-27 16:56:13,984] - [remote_util] [140542730036992] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-27 16:56:13,993] - [remote_util] [140542730036992] - INFO - SSH Connecting to 172.23.105.247 with username:root, attempt#1 of 5 [2021-02-27 16:56:14,094] - [remote_util] [140542730036992] - INFO - SSH Connected to 172.23.105.247 as root [2021-02-27 16:56:14,378] - [remote_util] [140542730036992] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-27 16:56:14,693] - [remote_util] [140542730036992] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-27 16:56:14,715] - [remote_util] [140542730036992] - INFO - 172.23.105.247 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-27 16:56:14,771] - [remote_util] [140542730036992] - INFO - Running systemd command on this server [2021-02-27 16:56:14,771] - [remote_util] [140542730036992] - INFO - running command.raw on 172.23.105.247: systemctl stop couchbase-server.service [2021-02-27 16:56:16,546] - [remote_util] [140542730036992] - INFO - command executed successfully with root [2021-02-27 16:56:16,546] - [basetestcase] [140542730036992] - INFO - Couchbase stopped [2021-02-27 16:56:16,547] - [basetestcase] [140542730036992] - INFO - sleep for 5 secs. ... [2021-02-27 16:56:21,550] - [remote_util] [140542730036992] - INFO - running command.raw on 172.23.105.247: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-27 16:56:21,575] - [remote_util] [140542730036992] - INFO - command executed successfully with root [2021-02-27 16:56:21,575] - [remote_util] [140542730036992] - INFO - running command.raw on 172.23.105.247: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-27 16:56:21,629] - [remote_util] [140542730036992] - INFO - command executed successfully with root [2021-02-27 16:56:21,630] - [remote_util] [140542730036992] - INFO - SSH Connecting to 172.23.105.247 with username:root, attempt#1 of 5 [2021-02-27 16:56:21,736] - [remote_util] [140542730036992] - INFO - SSH Connected to 172.23.105.247 as root [2021-02-27 16:56:22,009] - [remote_util] [140542730036992] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-27 16:56:22,306] - [remote_util] [140542730036992] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-27 16:56:22,365] - [remote_util] [140542730036992] - INFO - 172.23.105.247 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-27 16:56:22,365] - [remote_util] [140542730036992] - INFO - Checking for process beam.smp on linux [2021-02-27 16:56:22,403] - [remote_util] [140542730036992] - INFO - Starting couchbase server [2021-02-27 16:56:22,498] - [remote_util] [140542730036992] - INFO - Running systemd command on this server [2021-02-27 16:56:22,498] - [remote_util] [140542730036992] - INFO - running command.raw on 172.23.105.247: systemctl start couchbase-server.service [2021-02-27 16:56:22,521] - [remote_util] [140542730036992] - INFO - command executed successfully with root [2021-02-27 16:56:22,522] - [remote_util] [140542730036992] - INFO - 172.23.105.247:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-27 16:56:27,525] - [remote_util] [140542730036992] - INFO - running command.raw on 172.23.105.247: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-27 16:56:27,551] - [remote_util] [140542730036992] - INFO - command executed successfully with root [2021-02-27 16:56:27,552] - [remote_util] [140542730036992] - INFO - Couchbase server status: [' Process: 25666 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-27 16:56:27,552] - [remote_util] [140542730036992] - INFO - Checking for process beam.smp on linux [2021-02-27 16:56:27,623] - [remote_util] [140542730036992] - INFO - process beam.smp is running on 172.23.105.247: with pid 25842 [2021-02-27 16:56:27,623] - [basetestcase] [140542730036992] - INFO - Couchbase started [2021-02-27 16:56:27,625] - [remote_util] [140542730036992] - INFO - SSH Connecting to 172.23.106.1 with username:root, attempt#1 of 5 [2021-02-27 16:56:27,731] - [remote_util] [140542730036992] - INFO - SSH Connected to 172.23.106.1 as root [2021-02-27 16:56:27,986] - [remote_util] [140542730036992] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-27 16:56:28,289] - [remote_util] [140542730036992] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-27 16:56:28,299] - [remote_util] [140542730036992] - INFO - SSH Connecting to 172.23.106.1 with username:root, attempt#1 of 5 [2021-02-27 16:56:28,401] - [remote_util] [140542730036992] - INFO - SSH Connected to 172.23.106.1 as root [2021-02-27 16:56:28,653] - [remote_util] [140542730036992] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-27 16:56:28,950] - [remote_util] [140542730036992] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-27 16:56:28,968] - [remote_util] [140542730036992] - INFO - 172.23.106.1 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-27 16:56:29,019] - [remote_util] [140542730036992] - INFO - Running systemd command on this server [2021-02-27 16:56:29,019] - [remote_util] [140542730036992] - INFO - running command.raw on 172.23.106.1: systemctl stop couchbase-server.service [2021-02-27 16:56:30,712] - [remote_util] [140542730036992] - INFO - command executed successfully with root [2021-02-27 16:56:30,712] - [basetestcase] [140542730036992] - INFO - Couchbase stopped [2021-02-27 16:56:30,712] - [basetestcase] [140542730036992] - INFO - sleep for 5 secs. ... [2021-02-27 16:56:35,718] - [remote_util] [140542730036992] - INFO - running command.raw on 172.23.106.1: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-27 16:56:35,739] - [remote_util] [140542730036992] - INFO - command executed successfully with root [2021-02-27 16:56:35,739] - [remote_util] [140542730036992] - INFO - running command.raw on 172.23.106.1: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-27 16:56:35,793] - [remote_util] [140542730036992] - INFO - command executed successfully with root [2021-02-27 16:56:35,794] - [remote_util] [140542730036992] - INFO - SSH Connecting to 172.23.106.1 with username:root, attempt#1 of 5 [2021-02-27 16:56:35,897] - [remote_util] [140542730036992] - INFO - SSH Connected to 172.23.106.1 as root [2021-02-27 16:56:36,146] - [remote_util] [140542730036992] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-27 16:56:36,445] - [remote_util] [140542730036992] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-27 16:56:36,505] - [remote_util] [140542730036992] - INFO - 172.23.106.1 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-27 16:56:36,505] - [remote_util] [140542730036992] - INFO - Checking for process beam.smp on linux [2021-02-27 16:56:36,582] - [remote_util] [140542730036992] - INFO - Starting couchbase server [2021-02-27 16:56:36,679] - [remote_util] [140542730036992] - INFO - Running systemd command on this server [2021-02-27 16:56:36,680] - [remote_util] [140542730036992] - INFO - running command.raw on 172.23.106.1: systemctl start couchbase-server.service [2021-02-27 16:56:36,703] - [remote_util] [140542730036992] - INFO - command executed successfully with root [2021-02-27 16:56:36,703] - [remote_util] [140542730036992] - INFO - 172.23.106.1:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-27 16:56:41,707] - [remote_util] [140542730036992] - INFO - running command.raw on 172.23.106.1: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-27 16:56:41,736] - [remote_util] [140542730036992] - INFO - command executed successfully with root [2021-02-27 16:56:41,736] - [remote_util] [140542730036992] - INFO - Couchbase server status: [' Process: 40997 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-27 16:56:41,736] - [remote_util] [140542730036992] - INFO - Checking for process beam.smp on linux [2021-02-27 16:56:41,805] - [remote_util] [140542730036992] - INFO - process beam.smp is running on 172.23.106.1: with pid 41141 [2021-02-27 16:56:41,805] - [basetestcase] [140542730036992] - INFO - Couchbase started [2021-02-27 16:56:41,807] - [remote_util] [140542730036992] - INFO - SSH Connecting to 172.23.107.65 with username:root, attempt#1 of 5 [2021-02-27 16:56:41,910] - [remote_util] [140542730036992] - INFO - SSH Connected to 172.23.107.65 as root [2021-02-27 16:56:42,193] - [remote_util] [140542730036992] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-27 16:56:42,488] - [remote_util] [140542730036992] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-27 16:56:42,495] - [remote_util] [140542730036992] - INFO - SSH Connecting to 172.23.107.65 with username:root, attempt#1 of 5 [2021-02-27 16:56:42,595] - [remote_util] [140542730036992] - INFO - SSH Connected to 172.23.107.65 as root [2021-02-27 16:56:42,868] - [remote_util] [140542730036992] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-27 16:56:43,162] - [remote_util] [140542730036992] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-27 16:56:43,221] - [remote_util] [140542730036992] - INFO - 172.23.107.65 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-27 16:56:43,274] - [remote_util] [140542730036992] - INFO - Running systemd command on this server [2021-02-27 16:56:43,274] - [remote_util] [140542730036992] - INFO - running command.raw on 172.23.107.65: systemctl stop couchbase-server.service [2021-02-27 16:56:45,065] - [remote_util] [140542730036992] - INFO - command executed successfully with root [2021-02-27 16:56:45,065] - [basetestcase] [140542730036992] - INFO - Couchbase stopped [2021-02-27 16:56:45,065] - [basetestcase] [140542730036992] - INFO - sleep for 5 secs. ... [2021-02-27 16:56:50,071] - [remote_util] [140542730036992] - INFO - running command.raw on 172.23.107.65: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-27 16:56:50,087] - [remote_util] [140542730036992] - INFO - command executed successfully with root [2021-02-27 16:56:50,088] - [remote_util] [140542730036992] - INFO - running command.raw on 172.23.107.65: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-27 16:56:50,141] - [remote_util] [140542730036992] - INFO - command executed successfully with root [2021-02-27 16:56:50,142] - [remote_util] [140542730036992] - INFO - SSH Connecting to 172.23.107.65 with username:root, attempt#1 of 5 [2021-02-27 16:56:50,244] - [remote_util] [140542730036992] - INFO - SSH Connected to 172.23.107.65 as root [2021-02-27 16:56:50,524] - [remote_util] [140542730036992] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-27 16:56:50,818] - [remote_util] [140542730036992] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-27 16:56:50,837] - [remote_util] [140542730036992] - INFO - 172.23.107.65 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-27 16:56:50,837] - [remote_util] [140542730036992] - INFO - Checking for process beam.smp on linux [2021-02-27 16:56:50,876] - [remote_util] [140542730036992] - INFO - Starting couchbase server [2021-02-27 16:56:50,972] - [remote_util] [140542730036992] - INFO - Running systemd command on this server [2021-02-27 16:56:50,972] - [remote_util] [140542730036992] - INFO - running command.raw on 172.23.107.65: systemctl start couchbase-server.service [2021-02-27 16:56:50,998] - [remote_util] [140542730036992] - INFO - command executed successfully with root [2021-02-27 16:56:50,999] - [remote_util] [140542730036992] - INFO - 172.23.107.65:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-27 16:56:56,004] - [remote_util] [140542730036992] - INFO - running command.raw on 172.23.107.65: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-27 16:56:56,029] - [remote_util] [140542730036992] - INFO - command executed successfully with root [2021-02-27 16:56:56,029] - [remote_util] [140542730036992] - INFO - Couchbase server status: [' Process: 8835 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-27 16:56:56,029] - [remote_util] [140542730036992] - INFO - Checking for process beam.smp on linux [2021-02-27 16:56:56,098] - [remote_util] [140542730036992] - INFO - process beam.smp is running on 172.23.107.65: with pid 8979 [2021-02-27 16:56:56,098] - [basetestcase] [140542730036992] - INFO - Couchbase started [2021-02-27 16:56:56,100] - [remote_util] [140542730036992] - INFO - SSH Connecting to 172.23.105.240 with username:root, attempt#1 of 5 [2021-02-27 16:56:56,207] - [remote_util] [140542730036992] - INFO - SSH Connected to 172.23.105.240 as root [2021-02-27 16:56:56,488] - [remote_util] [140542730036992] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-27 16:56:56,793] - [remote_util] [140542730036992] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-27 16:56:56,801] - [remote_util] [140542730036992] - INFO - SSH Connecting to 172.23.105.240 with username:root, attempt#1 of 5 [2021-02-27 16:56:56,903] - [remote_util] [140542730036992] - INFO - SSH Connected to 172.23.105.240 as root [2021-02-27 16:56:57,183] - [remote_util] [140542730036992] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-27 16:56:57,488] - [remote_util] [140542730036992] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-27 16:56:57,547] - [remote_util] [140542730036992] - INFO - 172.23.105.240 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-27 16:56:57,601] - [remote_util] [140542730036992] - INFO - Running systemd command on this server [2021-02-27 16:56:57,602] - [remote_util] [140542730036992] - INFO - running command.raw on 172.23.105.240: systemctl stop couchbase-server.service [2021-02-27 16:56:59,358] - [remote_util] [140542730036992] - INFO - command executed successfully with root [2021-02-27 16:56:59,358] - [basetestcase] [140542730036992] - INFO - Couchbase stopped [2021-02-27 16:56:59,359] - [basetestcase] [140542730036992] - INFO - sleep for 5 secs. ... [2021-02-27 16:57:04,362] - [remote_util] [140542730036992] - INFO - running command.raw on 172.23.105.240: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-27 16:57:04,383] - [remote_util] [140542730036992] - INFO - command executed successfully with root [2021-02-27 16:57:04,383] - [remote_util] [140542730036992] - INFO - running command.raw on 172.23.105.240: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-27 16:57:04,437] - [remote_util] [140542730036992] - INFO - command executed successfully with root [2021-02-27 16:57:04,439] - [remote_util] [140542730036992] - INFO - SSH Connecting to 172.23.105.240 with username:root, attempt#1 of 5 [2021-02-27 16:57:04,541] - [remote_util] [140542730036992] - INFO - SSH Connected to 172.23.105.240 as root [2021-02-27 16:57:04,820] - [remote_util] [140542730036992] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-27 16:57:05,125] - [remote_util] [140542730036992] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-27 16:57:05,184] - [remote_util] [140542730036992] - INFO - 172.23.105.240 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-27 16:57:05,184] - [remote_util] [140542730036992] - INFO - Checking for process beam.smp on linux [2021-02-27 16:57:05,224] - [remote_util] [140542730036992] - INFO - Starting couchbase server [2021-02-27 16:57:05,321] - [remote_util] [140542730036992] - INFO - Running systemd command on this server [2021-02-27 16:57:05,322] - [remote_util] [140542730036992] - INFO - running command.raw on 172.23.105.240: systemctl start couchbase-server.service [2021-02-27 16:57:05,346] - [remote_util] [140542730036992] - INFO - command executed successfully with root [2021-02-27 16:57:05,347] - [remote_util] [140542730036992] - INFO - 172.23.105.240:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-27 16:57:10,351] - [remote_util] [140542730036992] - INFO - running command.raw on 172.23.105.240: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-27 16:57:10,380] - [remote_util] [140542730036992] - INFO - command executed successfully with root [2021-02-27 16:57:10,380] - [remote_util] [140542730036992] - INFO - Couchbase server status: [' Process: 107664 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-27 16:57:10,380] - [remote_util] [140542730036992] - INFO - Checking for process beam.smp on linux [2021-02-27 16:57:10,450] - [remote_util] [140542730036992] - INFO - process beam.smp is running on 172.23.105.240: with pid 107809 [2021-02-27 16:57:10,451] - [basetestcase] [140542730036992] - INFO - Couchbase started [2021-02-27 16:57:10,452] - [remote_util] [140542730036992] - INFO - SSH Connecting to 172.23.105.249 with username:root, attempt#1 of 5 [2021-02-27 16:57:10,562] - [remote_util] [140542730036992] - INFO - SSH Connected to 172.23.105.249 as root [2021-02-27 16:57:10,859] - [remote_util] [140542730036992] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-27 16:57:11,161] - [remote_util] [140542730036992] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-27 16:57:11,170] - [remote_util] [140542730036992] - INFO - SSH Connecting to 172.23.105.249 with username:root, attempt#1 of 5 [2021-02-27 16:57:11,270] - [remote_util] [140542730036992] - INFO - SSH Connected to 172.23.105.249 as root [2021-02-27 16:57:11,556] - [remote_util] [140542730036992] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-27 16:57:11,855] - [remote_util] [140542730036992] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-27 16:57:11,917] - [remote_util] [140542730036992] - INFO - 172.23.105.249 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-27 16:57:11,976] - [remote_util] [140542730036992] - INFO - Running systemd command on this server [2021-02-27 16:57:11,977] - [remote_util] [140542730036992] - INFO - running command.raw on 172.23.105.249: systemctl stop couchbase-server.service [2021-02-27 16:57:13,724] - [remote_util] [140542730036992] - INFO - command executed successfully with root [2021-02-27 16:57:13,724] - [basetestcase] [140542730036992] - INFO - Couchbase stopped [2021-02-27 16:57:13,724] - [basetestcase] [140542730036992] - INFO - sleep for 5 secs. ... [2021-02-27 16:57:18,727] - [remote_util] [140542730036992] - INFO - running command.raw on 172.23.105.249: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-27 16:57:18,742] - [remote_util] [140542730036992] - INFO - command executed successfully with root [2021-02-27 16:57:18,742] - [remote_util] [140542730036992] - INFO - running command.raw on 172.23.105.249: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-27 16:57:18,796] - [remote_util] [140542730036992] - INFO - command executed successfully with root [2021-02-27 16:57:18,798] - [remote_util] [140542730036992] - INFO - SSH Connecting to 172.23.105.249 with username:root, attempt#1 of 5 [2021-02-27 16:57:18,903] - [remote_util] [140542730036992] - INFO - SSH Connected to 172.23.105.249 as root [2021-02-27 16:57:19,190] - [remote_util] [140542730036992] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-27 16:57:19,494] - [remote_util] [140542730036992] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-27 16:57:19,514] - [remote_util] [140542730036992] - INFO - 172.23.105.249 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-27 16:57:19,514] - [remote_util] [140542730036992] - INFO - Checking for process beam.smp on linux [2021-02-27 16:57:19,553] - [remote_util] [140542730036992] - INFO - Starting couchbase server [2021-02-27 16:57:19,652] - [remote_util] [140542730036992] - INFO - Running systemd command on this server [2021-02-27 16:57:19,652] - [remote_util] [140542730036992] - INFO - running command.raw on 172.23.105.249: systemctl start couchbase-server.service [2021-02-27 16:57:19,675] - [remote_util] [140542730036992] - INFO - command executed successfully with root [2021-02-27 16:57:19,676] - [remote_util] [140542730036992] - INFO - 172.23.105.249:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-27 16:57:24,679] - [remote_util] [140542730036992] - INFO - running command.raw on 172.23.105.249: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-27 16:57:24,705] - [remote_util] [140542730036992] - INFO - command executed successfully with root [2021-02-27 16:57:24,705] - [remote_util] [140542730036992] - INFO - Couchbase server status: [' Process: 44282 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-27 16:57:24,705] - [remote_util] [140542730036992] - INFO - Checking for process beam.smp on linux [2021-02-27 16:57:24,776] - [remote_util] [140542730036992] - INFO - process beam.smp is running on 172.23.105.249: with pid 44424 [2021-02-27 16:57:24,776] - [basetestcase] [140542730036992] - INFO - Couchbase started [2021-02-27 16:57:24,777] - [basetestcase] [140542730036992] - INFO - sleep for 10 secs. ... [2021-02-27 16:57:35,331] - [task] [140544298686208] - INFO - server: ip:172.23.104.97 port:8091 ssh_username:root, nodes/self [2021-02-27 16:57:35,336] - [task] [140544298686208] - INFO - {'uptime': '111', 'memoryTotal': 3963314176, 'memoryFree': 3431460864, 'mcdMemoryReserved': 3023, 'mcdMemoryAllocated': 3023, 'status': 'healthy', 'hostname': '172.23.104.97:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4554-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1601, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.104.97', 'ip': '172.23.104.97', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 3779, 'curr_items': 0} [2021-02-27 16:57:35,343] - [rest_client] [140544298686208] - ERROR - GET http://172.23.104.97:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-02-27 16:57:35,344] - [rest_client] [140544298686208] - INFO - pools/default params : memoryQuota=2025 [2021-02-27 16:57:35,355] - [rest_client] [140544298686208] - INFO - --> init_node_services(Administrator,password,172.23.104.97,8091,['kv']) [2021-02-27 16:57:35,356] - [rest_client] [140544298686208] - INFO - /node/controller/setupServices params on 172.23.104.97: 8091:hostname=172.23.104.97&user=Administrator&password=password&services=kv [2021-02-27 16:57:35,391] - [rest_client] [140544298686208] - INFO - --> in init_cluster...Administrator,password,8091 [2021-02-27 16:57:35,392] - [rest_client] [140544298686208] - INFO - settings/web params on 172.23.104.97:8091:port=8091&username=Administrator&password=password [2021-02-27 16:57:35,529] - [rest_client] [140544298686208] - INFO - --> status:True [2021-02-27 16:57:35,531] - [remote_util] [140544298686208] - INFO - SSH Connecting to 172.23.104.97 with username:root, attempt#1 of 5 [2021-02-27 16:57:35,692] - [remote_util] [140544298686208] - INFO - SSH Connected to 172.23.104.97 as root [2021-02-27 16:57:36,342] - [remote_util] [140544298686208] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-27 16:57:36,770] - [remote_util] [140544298686208] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-27 16:57:36,785] - [remote_util] [140544298686208] - INFO - running command.raw on 172.23.104.97: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-02-27 16:57:36,866] - [remote_util] [140544298686208] - INFO - command executed successfully with root [2021-02-27 16:57:36,866] - [remote_util] [140544298686208] - INFO - ['ok'] [2021-02-27 16:57:36,870] - [rest_client] [140544298686208] - INFO - /diag/eval status on 172.23.104.97:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-27 16:57:36,873] - [rest_client] [140544298686208] - INFO - /diag/eval status on 172.23.104.97:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-27 16:57:36,884] - [rest_client] [140544298686208] - INFO - settings/indexes params : storageMode=plasma [2021-02-27 16:57:36,921] - [task] [140544298686208] - INFO - server: ip:172.23.121.213 port:8091 ssh_username:root, nodes/self [2021-02-27 16:57:36,923] - [task] [140544298686208] - INFO - {'uptime': '99', 'memoryTotal': 4201684992, 'memoryFree': 3588628480, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.121.213:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4554-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.121.213', 'ip': '172.23.121.213', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} [2021-02-27 16:57:36,925] - [rest_client] [140544298686208] - ERROR - GET http://172.23.121.213:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-02-27 16:57:36,925] - [rest_client] [140544298686208] - INFO - pools/default params : memoryQuota=2147 [2021-02-27 16:57:36,931] - [rest_client] [140544298686208] - INFO - --> in init_cluster...Administrator,password,8091 [2021-02-27 16:57:36,931] - [rest_client] [140544298686208] - INFO - settings/web params on 172.23.121.213:8091:port=8091&username=Administrator&password=password [2021-02-27 16:57:36,985] - [rest_client] [140544298686208] - INFO - --> status:True [2021-02-27 16:57:36,987] - [remote_util] [140544298686208] - INFO - SSH Connecting to 172.23.121.213 with username:root, attempt#1 of 5 [2021-02-27 16:57:37,085] - [remote_util] [140544298686208] - INFO - SSH Connected to 172.23.121.213 as root [2021-02-27 16:57:37,349] - [remote_util] [140544298686208] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-27 16:57:37,643] - [remote_util] [140544298686208] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-27 16:57:37,650] - [remote_util] [140544298686208] - INFO - running command.raw on 172.23.121.213: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-02-27 16:57:37,706] - [remote_util] [140544298686208] - INFO - command executed successfully with root [2021-02-27 16:57:37,706] - [remote_util] [140544298686208] - INFO - ['ok'] [2021-02-27 16:57:37,708] - [rest_client] [140544298686208] - INFO - /diag/eval status on 172.23.121.213:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-27 16:57:37,710] - [rest_client] [140544298686208] - INFO - /diag/eval status on 172.23.121.213:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-27 16:57:37,714] - [rest_client] [140544298686208] - INFO - settings/indexes params : storageMode=plasma [2021-02-27 16:57:37,732] - [task] [140544298686208] - INFO - server: ip:172.23.105.238 port:8091 ssh_username:root, nodes/self [2021-02-27 16:57:37,735] - [task] [140544298686208] - INFO - {'uptime': '84', 'memoryTotal': 4201840640, 'memoryFree': 3641274368, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.105.238:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4554-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.105.238', 'ip': '172.23.105.238', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} [2021-02-27 16:57:37,737] - [rest_client] [140544298686208] - ERROR - GET http://172.23.105.238:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-02-27 16:57:37,737] - [rest_client] [140544298686208] - INFO - pools/default params : memoryQuota=2147 [2021-02-27 16:57:37,743] - [rest_client] [140544298686208] - INFO - --> in init_cluster...Administrator,password,8091 [2021-02-27 16:57:37,743] - [rest_client] [140544298686208] - INFO - settings/web params on 172.23.105.238:8091:port=8091&username=Administrator&password=password [2021-02-27 16:57:37,803] - [rest_client] [140544298686208] - INFO - --> status:True [2021-02-27 16:57:37,804] - [remote_util] [140544298686208] - INFO - SSH Connecting to 172.23.105.238 with username:root, attempt#1 of 5 [2021-02-27 16:57:37,919] - [remote_util] [140544298686208] - INFO - SSH Connected to 172.23.105.238 as root [2021-02-27 16:57:38,184] - [remote_util] [140544298686208] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-27 16:57:38,488] - [remote_util] [140544298686208] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-27 16:57:38,497] - [remote_util] [140544298686208] - INFO - running command.raw on 172.23.105.238: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-02-27 16:57:38,554] - [remote_util] [140544298686208] - INFO - command executed successfully with root [2021-02-27 16:57:38,554] - [remote_util] [140544298686208] - INFO - ['ok'] [2021-02-27 16:57:38,556] - [rest_client] [140544298686208] - INFO - /diag/eval status on 172.23.105.238:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-27 16:57:38,559] - [rest_client] [140544298686208] - INFO - /diag/eval status on 172.23.105.238:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-27 16:57:38,564] - [rest_client] [140544298686208] - INFO - settings/indexes params : storageMode=plasma [2021-02-27 16:57:38,583] - [task] [140544298686208] - INFO - server: ip:172.23.105.247 port:8091 ssh_username:root, nodes/self [2021-02-27 16:57:38,586] - [task] [140544298686208] - INFO - {'uptime': '70', 'memoryTotal': 4201840640, 'memoryFree': 3678179328, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.105.247:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4554-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.105.247', 'ip': '172.23.105.247', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} [2021-02-27 16:57:38,588] - [rest_client] [140544298686208] - ERROR - GET http://172.23.105.247:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-02-27 16:57:38,588] - [rest_client] [140544298686208] - INFO - pools/default params : memoryQuota=2147 [2021-02-27 16:57:38,594] - [rest_client] [140544298686208] - INFO - --> in init_cluster...Administrator,password,8091 [2021-02-27 16:57:38,594] - [rest_client] [140544298686208] - INFO - settings/web params on 172.23.105.247:8091:port=8091&username=Administrator&password=password [2021-02-27 16:57:38,654] - [rest_client] [140544298686208] - INFO - --> status:True [2021-02-27 16:57:38,656] - [remote_util] [140544298686208] - INFO - SSH Connecting to 172.23.105.247 with username:root, attempt#1 of 5 [2021-02-27 16:57:38,762] - [remote_util] [140544298686208] - INFO - SSH Connected to 172.23.105.247 as root [2021-02-27 16:57:39,044] - [remote_util] [140544298686208] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-27 16:57:39,347] - [remote_util] [140544298686208] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-27 16:57:39,354] - [remote_util] [140544298686208] - INFO - running command.raw on 172.23.105.247: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-02-27 16:57:39,410] - [remote_util] [140544298686208] - INFO - command executed successfully with root [2021-02-27 16:57:39,411] - [remote_util] [140544298686208] - INFO - ['ok'] [2021-02-27 16:57:39,413] - [rest_client] [140544298686208] - INFO - /diag/eval status on 172.23.105.247:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-27 16:57:39,414] - [rest_client] [140544298686208] - INFO - /diag/eval status on 172.23.105.247:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-27 16:57:39,418] - [rest_client] [140544298686208] - INFO - settings/indexes params : storageMode=plasma [2021-02-27 16:57:39,438] - [task] [140544298686208] - INFO - server: ip:172.23.106.1 port:8091 ssh_username:root, nodes/self [2021-02-27 16:57:39,441] - [task] [140544298686208] - INFO - {'uptime': '60', 'memoryTotal': 4201840640, 'memoryFree': 3649544192, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.106.1:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4554-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.106.1', 'ip': '172.23.106.1', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} [2021-02-27 16:57:39,443] - [rest_client] [140544298686208] - ERROR - GET http://172.23.106.1:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-02-27 16:57:39,443] - [rest_client] [140544298686208] - INFO - pools/default params : memoryQuota=2147 [2021-02-27 16:57:39,449] - [rest_client] [140544298686208] - INFO - --> in init_cluster...Administrator,password,8091 [2021-02-27 16:57:39,449] - [rest_client] [140544298686208] - INFO - settings/web params on 172.23.106.1:8091:port=8091&username=Administrator&password=password [2021-02-27 16:57:39,508] - [rest_client] [140544298686208] - INFO - --> status:True [2021-02-27 16:57:39,510] - [remote_util] [140544298686208] - INFO - SSH Connecting to 172.23.106.1 with username:root, attempt#1 of 5 [2021-02-27 16:57:39,614] - [remote_util] [140544298686208] - INFO - SSH Connected to 172.23.106.1 as root [2021-02-27 16:57:39,868] - [remote_util] [140544298686208] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-27 16:57:40,166] - [remote_util] [140544298686208] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-27 16:57:40,174] - [remote_util] [140544298686208] - INFO - running command.raw on 172.23.106.1: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-02-27 16:57:40,231] - [remote_util] [140544298686208] - INFO - command executed successfully with root [2021-02-27 16:57:40,231] - [remote_util] [140544298686208] - INFO - ['ok'] [2021-02-27 16:57:40,234] - [rest_client] [140544298686208] - INFO - /diag/eval status on 172.23.106.1:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-27 16:57:40,236] - [rest_client] [140544298686208] - INFO - /diag/eval status on 172.23.106.1:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-27 16:57:40,240] - [rest_client] [140544298686208] - INFO - settings/indexes params : storageMode=plasma [2021-02-27 16:57:40,260] - [task] [140544298686208] - INFO - server: ip:172.23.107.65 port:8091 ssh_username:root, nodes/self [2021-02-27 16:57:40,263] - [task] [140544298686208] - INFO - {'uptime': '44', 'memoryTotal': 4201840640, 'memoryFree': 3665747968, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.107.65:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4554-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.107.65', 'ip': '172.23.107.65', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} [2021-02-27 16:57:40,265] - [rest_client] [140544298686208] - ERROR - GET http://172.23.107.65:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-02-27 16:57:40,265] - [rest_client] [140544298686208] - INFO - pools/default params : memoryQuota=2147 [2021-02-27 16:57:40,270] - [rest_client] [140544298686208] - INFO - --> in init_cluster...Administrator,password,8091 [2021-02-27 16:57:40,271] - [rest_client] [140544298686208] - INFO - settings/web params on 172.23.107.65:8091:port=8091&username=Administrator&password=password [2021-02-27 16:57:40,329] - [rest_client] [140544298686208] - INFO - --> status:True [2021-02-27 16:57:40,330] - [remote_util] [140544298686208] - INFO - SSH Connecting to 172.23.107.65 with username:root, attempt#1 of 5 [2021-02-27 16:57:40,437] - [remote_util] [140544298686208] - INFO - SSH Connected to 172.23.107.65 as root [2021-02-27 16:57:40,721] - [remote_util] [140544298686208] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-27 16:57:41,016] - [remote_util] [140544298686208] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-27 16:57:41,025] - [remote_util] [140544298686208] - INFO - running command.raw on 172.23.107.65: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-02-27 16:57:41,080] - [remote_util] [140544298686208] - INFO - command executed successfully with root [2021-02-27 16:57:41,080] - [remote_util] [140544298686208] - INFO - ['ok'] [2021-02-27 16:57:41,082] - [rest_client] [140544298686208] - INFO - /diag/eval status on 172.23.107.65:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-27 16:57:41,084] - [rest_client] [140544298686208] - INFO - /diag/eval status on 172.23.107.65:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-27 16:57:41,088] - [rest_client] [140544298686208] - INFO - settings/indexes params : storageMode=plasma [2021-02-27 16:57:41,107] - [task] [140544298686208] - INFO - server: ip:172.23.105.240 port:8091 ssh_username:root, nodes/self [2021-02-27 16:57:41,111] - [task] [140544298686208] - INFO - {'uptime': '29', 'memoryTotal': 4201840640, 'memoryFree': 3657617408, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.105.240:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4554-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.105.240', 'ip': '172.23.105.240', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} [2021-02-27 16:57:41,112] - [rest_client] [140544298686208] - ERROR - GET http://172.23.105.240:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-02-27 16:57:41,112] - [rest_client] [140544298686208] - INFO - pools/default params : memoryQuota=2147 [2021-02-27 16:57:41,118] - [rest_client] [140544298686208] - INFO - --> in init_cluster...Administrator,password,8091 [2021-02-27 16:57:41,118] - [rest_client] [140544298686208] - INFO - settings/web params on 172.23.105.240:8091:port=8091&username=Administrator&password=password [2021-02-27 16:57:41,178] - [rest_client] [140544298686208] - INFO - --> status:True [2021-02-27 16:57:41,180] - [remote_util] [140544298686208] - INFO - SSH Connecting to 172.23.105.240 with username:root, attempt#1 of 5 [2021-02-27 16:57:41,287] - [remote_util] [140544298686208] - INFO - SSH Connected to 172.23.105.240 as root [2021-02-27 16:57:41,573] - [remote_util] [140544298686208] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-27 16:57:41,880] - [remote_util] [140544298686208] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-27 16:57:41,888] - [remote_util] [140544298686208] - INFO - running command.raw on 172.23.105.240: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-02-27 16:57:41,944] - [remote_util] [140544298686208] - INFO - command executed successfully with root [2021-02-27 16:57:41,944] - [remote_util] [140544298686208] - INFO - ['ok'] [2021-02-27 16:57:41,946] - [rest_client] [140544298686208] - INFO - /diag/eval status on 172.23.105.240:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-27 16:57:41,948] - [rest_client] [140544298686208] - INFO - /diag/eval status on 172.23.105.240:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-27 16:57:41,952] - [rest_client] [140544298686208] - INFO - settings/indexes params : storageMode=plasma [2021-02-27 16:57:41,972] - [task] [140544298686208] - INFO - server: ip:172.23.105.249 port:8091 ssh_username:root, nodes/self [2021-02-27 16:57:41,975] - [task] [140544298686208] - INFO - {'uptime': '19', 'memoryTotal': 4201844736, 'memoryFree': 3603959808, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.105.249:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4554-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@cb.local', 'ip': '172.23.105.249', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} [2021-02-27 16:57:41,977] - [rest_client] [140544298686208] - ERROR - GET http://172.23.105.249:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-02-27 16:57:41,977] - [rest_client] [140544298686208] - INFO - pools/default params : memoryQuota=2147 [2021-02-27 16:57:41,983] - [rest_client] [140544298686208] - INFO - --> in init_cluster...Administrator,password,8091 [2021-02-27 16:57:41,983] - [rest_client] [140544298686208] - INFO - settings/web params on 172.23.105.249:8091:port=8091&username=Administrator&password=password [2021-02-27 16:57:42,040] - [rest_client] [140544298686208] - INFO - --> status:True [2021-02-27 16:57:42,041] - [remote_util] [140544298686208] - INFO - SSH Connecting to 172.23.105.249 with username:root, attempt#1 of 5 [2021-02-27 16:57:42,149] - [remote_util] [140544298686208] - INFO - SSH Connected to 172.23.105.249 as root [2021-02-27 16:57:42,444] - [remote_util] [140544298686208] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-27 16:57:42,749] - [remote_util] [140544298686208] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-27 16:57:42,758] - [remote_util] [140544298686208] - INFO - running command.raw on 172.23.105.249: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-02-27 16:57:42,813] - [remote_util] [140544298686208] - INFO - command executed successfully with root [2021-02-27 16:57:42,814] - [remote_util] [140544298686208] - INFO - ['ok'] [2021-02-27 16:57:42,816] - [rest_client] [140544298686208] - INFO - /diag/eval status on 172.23.105.249:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-27 16:57:42,818] - [rest_client] [140544298686208] - INFO - /diag/eval status on 172.23.105.249:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-27 16:57:42,822] - [rest_client] [140544298686208] - INFO - settings/indexes params : storageMode=plasma [2021-02-27 16:57:42,856] - [basetestcase] [140542730036992] - INFO - **** add built-in 'cbadminbucket' user to node 172.23.104.97 **** [2021-02-27 16:57:42,874] - [rest_client] [140542730036992] - ERROR - DELETE http://172.23.104.97:8091/settings/rbac/users/local/cbadminbucket body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:password [2021-02-27 16:57:42,874] - [internal_user] [140542730036992] - INFO - Exception while deleting user. Exception is -b'"User was not found."' [2021-02-27 16:57:43,016] - [basetestcase] [140542730036992] - INFO - sleep for 5 secs. ... [2021-02-27 16:57:48,021] - [basetestcase] [140542730036992] - INFO - **** add 'admin' role to 'cbadminbucket' user **** [2021-02-27 16:57:48,041] - [basetestcase] [140542730036992] - INFO - done initializing cluster [2021-02-27 16:57:48,843] - [task] [140544298686208] - INFO - adding node 172.23.121.213:8091 to cluster [2021-02-27 16:57:48,843] - [rest_client] [140544298686208] - INFO - adding remote node @172.23.121.213:8091 to this cluster @172.23.104.97:8091 [2021-02-27 16:57:58,859] - [rest_client] [140544298686208] - INFO - rebalance progress took 10.02 seconds [2021-02-27 16:57:58,859] - [rest_client] [140544298686208] - INFO - sleep for 10 seconds after rebalance... [2021-02-27 16:58:11,217] - [task] [140544298686208] - INFO - adding node 172.23.105.238:8091 to cluster [2021-02-27 16:58:11,217] - [rest_client] [140544298686208] - INFO - adding remote node @172.23.105.238:8091 to this cluster @172.23.104.97:8091 [2021-02-27 16:58:21,235] - [rest_client] [140544298686208] - INFO - rebalance progress took 10.02 seconds [2021-02-27 16:58:21,235] - [rest_client] [140544298686208] - INFO - sleep for 10 seconds after rebalance... [2021-02-27 16:58:33,761] - [task] [140544298686208] - INFO - adding node 172.23.105.247:8091 to cluster [2021-02-27 16:58:33,761] - [rest_client] [140544298686208] - INFO - adding remote node @172.23.105.247:8091 to this cluster @172.23.104.97:8091 [2021-02-27 16:58:43,780] - [rest_client] [140544298686208] - INFO - rebalance progress took 10.02 seconds [2021-02-27 16:58:43,781] - [rest_client] [140544298686208] - INFO - sleep for 10 seconds after rebalance... [2021-02-27 16:58:56,261] - [task] [140544298686208] - INFO - adding node 172.23.106.1:8091 to cluster [2021-02-27 16:58:56,261] - [rest_client] [140544298686208] - INFO - adding remote node @172.23.106.1:8091 to this cluster @172.23.104.97:8091 [2021-02-27 16:59:06,279] - [rest_client] [140544298686208] - INFO - rebalance progress took 10.02 seconds [2021-02-27 16:59:06,279] - [rest_client] [140544298686208] - INFO - sleep for 10 seconds after rebalance... [2021-02-27 16:59:18,600] - [rest_client] [140544298686208] - INFO - rebalance params : {'knownNodes': 'ns_1@172.23.104.97,ns_1@172.23.105.238,ns_1@172.23.105.247,ns_1@172.23.106.1,ns_1@172.23.121.213', 'ejectedNodes': '', 'user': 'Administrator', 'password': 'password'} [2021-02-27 16:59:18,637] - [rest_client] [140544298686208] - INFO - rebalance operation started [2021-02-27 16:59:18,642] - [rest_client] [140544298686208] - INFO - rebalance percentage : 0.00 % [2021-02-27 16:59:18,642] - [task] [140544298686208] - INFO - Rebalance - status: running, progress: 0.00% [2021-02-27 16:59:28,676] - [task] [140544298686208] - INFO - Rebalance - status: none, progress: 100.00% [2021-02-27 16:59:28,690] - [task] [140544298686208] - INFO - rebalancing was completed with progress: 100% in 10.052551507949829 sec [2021-02-27 16:59:28,692] - [remote_util] [140542730036992] - INFO - SSH Connecting to 172.23.104.97 with username:root, attempt#1 of 5 [2021-02-27 16:59:28,861] - [remote_util] [140542730036992] - INFO - SSH Connected to 172.23.104.97 as root [2021-02-27 16:59:29,475] - [remote_util] [140542730036992] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-27 16:59:29,912] - [remote_util] [140542730036992] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-27 16:59:29,936] - [remote_util] [140542730036992] - INFO - running command.raw on 172.23.104.97: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-02-27 16:59:30,005] - [remote_util] [140542730036992] - INFO - command executed successfully with root [2021-02-27 16:59:30,005] - [remote_util] [140542730036992] - INFO - ['ok'] [2021-02-27 16:59:30,005] - [basetestcase] [140542730036992] - INFO - Enabled diag/eval for non-local hosts from 172.23.104.97 [2021-02-27 16:59:30,005] - [basetestcase] [140542730036992] - INFO - ============== basetestcase setup was finished for test #2 test_eventing_rebalance_with_multiple_kv_nodes ============== [2021-02-27 16:59:30,018] - [remote_util] [140542730036992] - INFO - SSH Connecting to 172.23.104.97 with username:root, attempt#1 of 5 [2021-02-27 16:59:30,192] - [remote_util] [140542730036992] - INFO - SSH Connected to 172.23.104.97 as root [2021-02-27 16:59:30,835] - [remote_util] [140542730036992] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-27 16:59:31,265] - [remote_util] [140542730036992] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-27 16:59:31,267] - [remote_util] [140542730036992] - INFO - SSH Connecting to 172.23.121.213 with username:root, attempt#1 of 5 [2021-02-27 16:59:31,368] - [remote_util] [140542730036992] - INFO - SSH Connected to 172.23.121.213 as root [2021-02-27 16:59:31,640] - [remote_util] [140542730036992] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-27 16:59:31,946] - [remote_util] [140542730036992] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-27 16:59:31,948] - [remote_util] [140542730036992] - INFO - SSH Connecting to 172.23.105.238 with username:root, attempt#1 of 5 [2021-02-27 16:59:32,060] - [remote_util] [140542730036992] - INFO - SSH Connected to 172.23.105.238 as root [2021-02-27 16:59:32,361] - [remote_util] [140542730036992] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-27 16:59:32,679] - [remote_util] [140542730036992] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-27 16:59:32,681] - [remote_util] [140542730036992] - INFO - SSH Connecting to 172.23.105.247 with username:root, attempt#1 of 5 [2021-02-27 16:59:32,793] - [remote_util] [140542730036992] - INFO - SSH Connected to 172.23.105.247 as root [2021-02-27 16:59:33,093] - [remote_util] [140542730036992] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-27 16:59:33,407] - [remote_util] [140542730036992] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-27 16:59:33,409] - [remote_util] [140542730036992] - INFO - SSH Connecting to 172.23.106.1 with username:root, attempt#1 of 5 [2021-02-27 16:59:33,513] - [remote_util] [140542730036992] - INFO - SSH Connected to 172.23.106.1 as root [2021-02-27 16:59:33,770] - [remote_util] [140542730036992] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-27 16:59:34,069] - [remote_util] [140542730036992] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-27 16:59:34,071] - [remote_util] [140542730036992] - INFO - SSH Connecting to 172.23.107.65 with username:root, attempt#1 of 5 [2021-02-27 16:59:34,176] - [remote_util] [140542730036992] - INFO - SSH Connected to 172.23.107.65 as root [2021-02-27 16:59:34,463] - [remote_util] [140542730036992] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-27 16:59:34,763] - [remote_util] [140542730036992] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-27 16:59:34,767] - [remote_util] [140542730036992] - INFO - SSH Connecting to 172.23.105.240 with username:root, attempt#1 of 5 [2021-02-27 16:59:34,877] - [remote_util] [140542730036992] - INFO - SSH Connected to 172.23.105.240 as root [2021-02-27 16:59:35,178] - [remote_util] [140542730036992] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-27 16:59:35,504] - [remote_util] [140542730036992] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-27 16:59:35,507] - [remote_util] [140542730036992] - INFO - SSH Connecting to 172.23.105.249 with username:root, attempt#1 of 5 [2021-02-27 16:59:35,616] - [remote_util] [140542730036992] - INFO - SSH Connected to 172.23.105.249 as root [2021-02-27 16:59:35,932] - [remote_util] [140542730036992] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-27 16:59:36,241] - [remote_util] [140542730036992] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-27 16:59:46,075] - [basetestcase] [140542730036992] - INFO - ------- Cluster statistics ------- [2021-02-27 16:59:46,075] - [basetestcase] [140542730036992] - INFO - 172.23.104.97:8091 => {'services': ['kv'], 'cpu_utilization': 12.49685059208869, 'mem_free': 3386535936, 'mem_total': 3963314176, 'swap_mem_used': 0, 'swap_mem_total': 0} [2021-02-27 16:59:46,075] - [basetestcase] [140542730036992] - INFO - 172.23.105.238:8091 => {'services': ['kv'], 'cpu_utilization': 2.381549260466282, 'mem_free': 3614994432, 'mem_total': 4201840640, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-02-27 16:59:46,075] - [basetestcase] [140542730036992] - INFO - 172.23.105.247:8091 => {'services': ['eventing'], 'cpu_utilization': 2.058749686166206, 'mem_free': 3648696320, 'mem_total': 4201840640, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-02-27 16:59:46,075] - [basetestcase] [140542730036992] - INFO - 172.23.106.1:8091 => {'services': ['index', 'n1ql'], 'cpu_utilization': 5.892722236212541, 'mem_free': 3570651136, 'mem_total': 4201840640, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-02-27 16:59:46,075] - [basetestcase] [140542730036992] - INFO - 172.23.121.213:8091 => {'services': ['kv'], 'cpu_utilization': 1.607232546459066, 'mem_free': 3557343232, 'mem_total': 4201684992, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-02-27 16:59:46,076] - [basetestcase] [140542730036992] - INFO - --- End of cluster statistics --- [2021-02-27 16:59:46,078] - [remote_util] [140542730036992] - INFO - SSH Connecting to 172.23.104.97 with username:root, attempt#1 of 5 [2021-02-27 16:59:46,235] - [remote_util] [140542730036992] - INFO - SSH Connected to 172.23.104.97 as root [2021-02-27 16:59:46,846] - [remote_util] [140542730036992] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-27 16:59:47,260] - [remote_util] [140542730036992] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-27 16:59:50,099] - [basetestcase] [140542730036992] - INFO - list of n1ql nodes in cluster: [ip:172.23.106.1 port:8091 ssh_username:root] [2021-02-27 16:59:50,112] - [basetestcase] [140542730036992] - INFO - list of eventing nodes in cluster: [ip:172.23.105.247 port:8091 ssh_username:root] [2021-02-27 16:59:50,118] - [rest_client] [140542730036992] - INFO - settings/indexes params : storageMode=plasma [2021-02-27 16:59:50,125] - [eventing_base] [140542730036992] - INFO - Setting the min possible memory quota so that adding mode nodes to the cluster wouldn't be a problem. [2021-02-27 16:59:50,125] - [rest_client] [140542730036992] - INFO - pools/default params : memoryQuota=330 [2021-02-27 16:59:50,134] - [rest_client] [140542730036992] - INFO - pools/default params : indexMemoryQuota=256 [2021-02-27 16:59:50,141] - [rest_client] [140542730036992] - INFO - pools/default params : eventingMemoryQuota=512 [2021-02-27 16:59:50,175] - [basetestcase] [140542730036992] - INFO - list of n1ql nodes in cluster: [ip:172.23.106.1 port:8091 ssh_username:root] [2021-02-27 16:59:50,184] - [rest_client] [140542730036992] - INFO - pools/default params : memoryQuota=700 [2021-02-27 16:59:50,191] - [eventing_rebalance_collection] [140542730036992] - INFO - 200 [2021-02-27 16:59:51,117] - [rest_client] [140544298686208] - INFO - http://172.23.104.97:8091/pools/default/buckets with param: name=src_bucket&ramQuotaMB=200&authType=none&replicaNumber=0&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore [2021-02-27 16:59:51,153] - [rest_client] [140544298686208] - INFO - 0.04 seconds to create bucket src_bucket [2021-02-27 16:59:51,154] - [bucket_helper] [140544298686208] - INFO - waiting for memcached bucket : src_bucket in 172.23.104.97 to accept set ops [2021-02-27 16:59:52,165] - [data_helper] [140544298686208] - INFO - creating direct client 172.23.104.97:11210 src_bucket [2021-02-27 16:59:52,341] - [data_helper] [140544298686208] - INFO - creating direct client 172.23.105.238:11210 src_bucket [2021-02-27 16:59:52,474] - [data_helper] [140544298686208] - INFO - creating direct client 172.23.121.213:11210 src_bucket [2021-02-27 16:59:52,704] - [data_helper] [140544298686208] - INFO - creating direct client 172.23.104.97:11210 src_bucket [2021-02-27 16:59:52,874] - [data_helper] [140544298686208] - INFO - creating direct client 172.23.105.238:11210 src_bucket [2021-02-27 16:59:53,013] - [data_helper] [140544298686208] - INFO - creating direct client 172.23.121.213:11210 src_bucket [2021-02-27 16:59:53,305] - [task] [140544298686208] - WARNING - vbucket map not ready after try 0 [2021-02-27 16:59:53,305] - [bucket_helper] [140544298686208] - INFO - waiting for memcached bucket : src_bucket in 172.23.104.97 to accept set ops [2021-02-27 16:59:53,542] - [data_helper] [140544298686208] - INFO - creating direct client 172.23.104.97:11210 src_bucket [2021-02-27 16:59:53,692] - [data_helper] [140544298686208] - INFO - creating direct client 172.23.105.238:11210 src_bucket [2021-02-27 16:59:53,812] - [data_helper] [140544298686208] - INFO - creating direct client 172.23.121.213:11210 src_bucket [2021-02-27 16:59:54,192] - [data_helper] [140544298686208] - INFO - creating direct client 172.23.104.97:11210 src_bucket [2021-02-27 16:59:54,336] - [data_helper] [140544298686208] - INFO - creating direct client 172.23.105.238:11210 src_bucket [2021-02-27 16:59:54,461] - [data_helper] [140544298686208] - INFO - creating direct client 172.23.121.213:11210 src_bucket [2021-02-27 16:59:54,565] - [task] [140544298686208] - WARNING - vbucket map not ready after try 1 [2021-02-27 16:59:54,565] - [bucket_helper] [140544298686208] - INFO - waiting for memcached bucket : src_bucket in 172.23.104.97 to accept set ops [2021-02-27 16:59:54,797] - [data_helper] [140544298686208] - INFO - creating direct client 172.23.104.97:11210 src_bucket [2021-02-27 16:59:54,945] - [data_helper] [140544298686208] - INFO - creating direct client 172.23.105.238:11210 src_bucket [2021-02-27 16:59:55,259] - [data_helper] [140544298686208] - INFO - creating direct client 172.23.121.213:11210 src_bucket [2021-02-27 16:59:55,467] - [data_helper] [140544298686208] - INFO - creating direct client 172.23.104.97:11210 src_bucket [2021-02-27 16:59:55,614] - [data_helper] [140544298686208] - INFO - creating direct client 172.23.105.238:11210 src_bucket [2021-02-27 16:59:55,735] - [data_helper] [140544298686208] - INFO - creating direct client 172.23.121.213:11210 src_bucket [2021-02-27 16:59:55,842] - [task] [140544298686208] - WARNING - vbucket map not ready after try 2 [2021-02-27 16:59:55,842] - [bucket_helper] [140544298686208] - INFO - waiting for memcached bucket : src_bucket in 172.23.104.97 to accept set ops [2021-02-27 16:59:56,259] - [data_helper] [140544298686208] - INFO - creating direct client 172.23.104.97:11210 src_bucket [2021-02-27 16:59:56,411] - [data_helper] [140544298686208] - INFO - creating direct client 172.23.105.238:11210 src_bucket [2021-02-27 16:59:56,557] - [data_helper] [140544298686208] - INFO - creating direct client 172.23.121.213:11210 src_bucket [2021-02-27 16:59:56,769] - [data_helper] [140544298686208] - INFO - creating direct client 172.23.104.97:11210 src_bucket [2021-02-27 16:59:56,928] - [data_helper] [140544298686208] - INFO - creating direct client 172.23.105.238:11210 src_bucket [2021-02-27 16:59:57,058] - [data_helper] [140544298686208] - INFO - creating direct client 172.23.121.213:11210 src_bucket [2021-02-27 16:59:57,376] - [task] [140544298686208] - WARNING - vbucket map not ready after try 3 [2021-02-27 16:59:57,376] - [bucket_helper] [140544298686208] - INFO - waiting for memcached bucket : src_bucket in 172.23.104.97 to accept set ops [2021-02-27 16:59:57,626] - [data_helper] [140544298686208] - INFO - creating direct client 172.23.104.97:11210 src_bucket [2021-02-27 16:59:57,784] - [data_helper] [140544298686208] - INFO - creating direct client 172.23.105.238:11210 src_bucket [2021-02-27 16:59:57,912] - [data_helper] [140544298686208] - INFO - creating direct client 172.23.121.213:11210 src_bucket [2021-02-27 16:59:58,129] - [data_helper] [140544298686208] - INFO - creating direct client 172.23.104.97:11210 src_bucket [2021-02-27 16:59:58,417] - [data_helper] [140544298686208] - INFO - creating direct client 172.23.105.238:11210 src_bucket [2021-02-27 16:59:58,470] - [data_helper] [140544298686208] - INFO - creating direct client 172.23.121.213:11210 src_bucket [2021-02-27 16:59:58,507] - [task] [140544298686208] - WARNING - vbucket map not ready after try 4 [2021-02-27 16:59:58,507] - [bucket_helper] [140544298686208] - INFO - waiting for memcached bucket : src_bucket in 172.23.104.97 to accept set ops [2021-02-27 16:59:58,629] - [data_helper] [140544298686208] - INFO - creating direct client 172.23.104.97:11210 src_bucket [2021-02-27 16:59:58,705] - [data_helper] [140544298686208] - INFO - creating direct client 172.23.105.238:11210 src_bucket [2021-02-27 16:59:58,761] - [data_helper] [140544298686208] - INFO - creating direct client 172.23.121.213:11210 src_bucket [2021-02-27 16:59:59,040] - [data_helper] [140544298686208] - INFO - creating direct client 172.23.104.97:11210 src_bucket [2021-02-27 16:59:59,119] - [data_helper] [140544298686208] - INFO - creating direct client 172.23.105.238:11210 src_bucket [2021-02-27 16:59:59,187] - [data_helper] [140544298686208] - INFO - creating direct client 172.23.121.213:11210 src_bucket [2021-02-27 16:59:59,232] - [task] [140544298686208] - WARNING - vbucket map not ready after try 5 [2021-02-27 17:00:00,283] - [rest_client] [140544298686208] - INFO - http://172.23.104.97:8091/pools/default/buckets with param: name=dst_bucket&ramQuotaMB=200&authType=none&replicaNumber=0&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore [2021-02-27 17:00:00,320] - [rest_client] [140544298686208] - INFO - 0.04 seconds to create bucket dst_bucket [2021-02-27 17:00:00,321] - [bucket_helper] [140544298686208] - INFO - waiting for memcached bucket : dst_bucket in 172.23.104.97 to accept set ops [2021-02-27 17:00:01,260] - [data_helper] [140544298686208] - INFO - creating direct client 172.23.104.97:11210 dst_bucket [2021-02-27 17:00:01,382] - [data_helper] [140544298686208] - INFO - creating direct client 172.23.105.238:11210 dst_bucket [2021-02-27 17:00:01,477] - [data_helper] [140544298686208] - INFO - creating direct client 172.23.121.213:11210 dst_bucket [2021-02-27 17:00:01,636] - [data_helper] [140544298686208] - INFO - creating direct client 172.23.104.97:11210 dst_bucket [2021-02-27 17:00:01,766] - [data_helper] [140544298686208] - INFO - creating direct client 172.23.105.238:11210 dst_bucket [2021-02-27 17:00:01,854] - [data_helper] [140544298686208] - INFO - creating direct client 172.23.121.213:11210 dst_bucket [2021-02-27 17:00:02,133] - [task] [140544298686208] - WARNING - vbucket map not ready after try 0 [2021-02-27 17:00:02,133] - [bucket_helper] [140544298686208] - INFO - waiting for memcached bucket : dst_bucket in 172.23.104.97 to accept set ops [2021-02-27 17:00:02,367] - [data_helper] [140544298686208] - INFO - creating direct client 172.23.104.97:11210 dst_bucket [2021-02-27 17:00:02,497] - [data_helper] [140544298686208] - INFO - creating direct client 172.23.105.238:11210 dst_bucket [2021-02-27 17:00:02,590] - [data_helper] [140544298686208] - INFO - creating direct client 172.23.121.213:11210 dst_bucket [2021-02-27 17:00:02,747] - [data_helper] [140544298686208] - INFO - creating direct client 172.23.104.97:11210 dst_bucket [2021-02-27 17:00:02,872] - [data_helper] [140544298686208] - INFO - creating direct client 172.23.105.238:11210 dst_bucket [2021-02-27 17:00:03,142] - [data_helper] [140544298686208] - INFO - creating direct client 172.23.121.213:11210 dst_bucket [2021-02-27 17:00:03,218] - [task] [140544298686208] - WARNING - vbucket map not ready after try 1 [2021-02-27 17:00:03,218] - [bucket_helper] [140544298686208] - INFO - waiting for memcached bucket : dst_bucket in 172.23.104.97 to accept set ops [2021-02-27 17:00:03,412] - [data_helper] [140544298686208] - INFO - creating direct client 172.23.104.97:11210 dst_bucket [2021-02-27 17:00:03,545] - [data_helper] [140544298686208] - INFO - creating direct client 172.23.105.238:11210 dst_bucket [2021-02-27 17:00:03,642] - [data_helper] [140544298686208] - INFO - creating direct client 172.23.121.213:11210 dst_bucket [2021-02-27 17:00:03,964] - [data_helper] [140544298686208] - INFO - creating direct client 172.23.104.97:11210 dst_bucket [2021-02-27 17:00:04,090] - [data_helper] [140544298686208] - INFO - creating direct client 172.23.105.238:11210 dst_bucket [2021-02-27 17:00:04,170] - [data_helper] [140544298686208] - INFO - creating direct client 172.23.121.213:11210 dst_bucket [2021-02-27 17:00:04,243] - [task] [140544298686208] - WARNING - vbucket map not ready after try 2 [2021-02-27 17:00:04,244] - [bucket_helper] [140544298686208] - INFO - waiting for memcached bucket : dst_bucket in 172.23.104.97 to accept set ops [2021-02-27 17:00:04,445] - [data_helper] [140544298686208] - INFO - creating direct client 172.23.104.97:11210 dst_bucket [2021-02-27 17:00:04,559] - [data_helper] [140544298686208] - INFO - creating direct client 172.23.105.238:11210 dst_bucket [2021-02-27 17:00:04,839] - [data_helper] [140544298686208] - INFO - creating direct client 172.23.121.213:11210 dst_bucket [2021-02-27 17:00:04,986] - [data_helper] [140544298686208] - INFO - creating direct client 172.23.104.97:11210 dst_bucket [2021-02-27 17:00:05,107] - [data_helper] [140544298686208] - INFO - creating direct client 172.23.105.238:11210 dst_bucket [2021-02-27 17:00:05,191] - [data_helper] [140544298686208] - INFO - creating direct client 172.23.121.213:11210 dst_bucket [2021-02-27 17:00:05,265] - [task] [140544298686208] - WARNING - vbucket map not ready after try 3 [2021-02-27 17:00:05,265] - [bucket_helper] [140544298686208] - INFO - waiting for memcached bucket : dst_bucket in 172.23.104.97 to accept set ops [2021-02-27 17:00:05,682] - [data_helper] [140544298686208] - INFO - creating direct client 172.23.104.97:11210 dst_bucket [2021-02-27 17:00:05,796] - [data_helper] [140544298686208] - INFO - creating direct client 172.23.105.238:11210 dst_bucket [2021-02-27 17:00:05,891] - [data_helper] [140544298686208] - INFO - creating direct client 172.23.121.213:11210 dst_bucket [2021-02-27 17:00:06,041] - [data_helper] [140544298686208] - INFO - creating direct client 172.23.104.97:11210 dst_bucket [2021-02-27 17:00:06,155] - [data_helper] [140544298686208] - INFO - creating direct client 172.23.105.238:11210 dst_bucket [2021-02-27 17:00:06,443] - [data_helper] [140544298686208] - INFO - creating direct client 172.23.121.213:11210 dst_bucket [2021-02-27 17:00:06,518] - [task] [140544298686208] - WARNING - vbucket map not ready after try 4 [2021-02-27 17:00:06,518] - [bucket_helper] [140544298686208] - INFO - waiting for memcached bucket : dst_bucket in 172.23.104.97 to accept set ops [2021-02-27 17:00:06,706] - [data_helper] [140544298686208] - INFO - creating direct client 172.23.104.97:11210 dst_bucket [2021-02-27 17:00:06,823] - [data_helper] [140544298686208] - INFO - creating direct client 172.23.105.238:11210 dst_bucket [2021-02-27 17:00:06,910] - [data_helper] [140544298686208] - INFO - creating direct client 172.23.121.213:11210 dst_bucket [2021-02-27 17:00:07,066] - [data_helper] [140544298686208] - INFO - creating direct client 172.23.104.97:11210 dst_bucket [2021-02-27 17:00:07,354] - [data_helper] [140544298686208] - INFO - creating direct client 172.23.105.238:11210 dst_bucket [2021-02-27 17:00:07,404] - [data_helper] [140544298686208] - INFO - creating direct client 172.23.121.213:11210 dst_bucket [2021-02-27 17:00:07,440] - [task] [140544298686208] - WARNING - vbucket map not ready after try 5 [2021-02-27 17:00:08,486] - [rest_client] [140544298686208] - INFO - http://172.23.104.97:8091/pools/default/buckets with param: name=metadata&ramQuotaMB=200&authType=none&replicaNumber=0&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore [2021-02-27 17:00:08,524] - [rest_client] [140544298686208] - INFO - 0.04 seconds to create bucket metadata [2021-02-27 17:00:08,524] - [bucket_helper] [140544298686208] - INFO - waiting for memcached bucket : metadata in 172.23.104.97 to accept set ops [2021-02-27 17:00:09,425] - [data_helper] [140544298686208] - INFO - creating direct client 172.23.104.97:11210 metadata [2021-02-27 17:00:09,670] - [data_helper] [140544298686208] - INFO - creating direct client 172.23.105.238:11210 metadata [2021-02-27 17:00:09,823] - [data_helper] [140544298686208] - INFO - creating direct client 172.23.121.213:11210 metadata [2021-02-27 17:00:10,230] - [data_helper] [140544298686208] - INFO - creating direct client 172.23.104.97:11210 metadata [2021-02-27 17:00:10,413] - [data_helper] [140544298686208] - INFO - creating direct client 172.23.105.238:11210 metadata [2021-02-27 17:00:10,545] - [data_helper] [140544298686208] - INFO - creating direct client 172.23.121.213:11210 metadata [2021-02-27 17:00:10,664] - [task] [140544298686208] - WARNING - vbucket map not ready after try 0 [2021-02-27 17:00:10,664] - [bucket_helper] [140544298686208] - INFO - waiting for memcached bucket : metadata in 172.23.104.97 to accept set ops [2021-02-27 17:00:11,120] - [data_helper] [140544298686208] - INFO - creating direct client 172.23.104.97:11210 metadata [2021-02-27 17:00:11,308] - [data_helper] [140544298686208] - INFO - creating direct client 172.23.105.238:11210 metadata [2021-02-27 17:00:11,453] - [data_helper] [140544298686208] - INFO - creating direct client 172.23.121.213:11210 metadata [2021-02-27 17:00:11,722] - [data_helper] [140544298686208] - INFO - creating direct client 172.23.104.97:11210 metadata [2021-02-27 17:00:11,898] - [data_helper] [140544298686208] - INFO - creating direct client 172.23.105.238:11210 metadata [2021-02-27 17:00:12,037] - [data_helper] [140544298686208] - INFO - creating direct client 172.23.121.213:11210 metadata [2021-02-27 17:00:12,168] - [task] [140544298686208] - WARNING - vbucket map not ready after try 1 [2021-02-27 17:00:12,168] - [bucket_helper] [140544298686208] - INFO - waiting for memcached bucket : metadata in 172.23.104.97 to accept set ops [2021-02-27 17:00:12,632] - [data_helper] [140544298686208] - INFO - creating direct client 172.23.104.97:11210 metadata [2021-02-27 17:00:12,796] - [data_helper] [140544298686208] - INFO - creating direct client 172.23.105.238:11210 metadata [2021-02-27 17:00:12,946] - [data_helper] [140544298686208] - INFO - creating direct client 172.23.121.213:11210 metadata [2021-02-27 17:00:13,194] - [data_helper] [140544298686208] - INFO - creating direct client 172.23.104.97:11210 metadata [2021-02-27 17:00:13,364] - [data_helper] [140544298686208] - INFO - creating direct client 172.23.105.238:11210 metadata [2021-02-27 17:00:13,698] - [data_helper] [140544298686208] - INFO - creating direct client 172.23.121.213:11210 metadata [2021-02-27 17:00:13,824] - [task] [140544298686208] - WARNING - vbucket map not ready after try 2 [2021-02-27 17:00:13,824] - [bucket_helper] [140544298686208] - INFO - waiting for memcached bucket : metadata in 172.23.104.97 to accept set ops [2021-02-27 17:00:14,085] - [data_helper] [140544298686208] - INFO - creating direct client 172.23.104.97:11210 metadata [2021-02-27 17:00:14,259] - [data_helper] [140544298686208] - INFO - creating direct client 172.23.105.238:11210 metadata [2021-02-27 17:00:14,407] - [data_helper] [140544298686208] - INFO - creating direct client 172.23.121.213:11210 metadata [2021-02-27 17:00:14,778] - [data_helper] [140544298686208] - INFO - creating direct client 172.23.104.97:11210 metadata [2021-02-27 17:00:14,868] - [data_helper] [140544298686208] - INFO - creating direct client 172.23.105.238:11210 metadata [2021-02-27 17:00:14,923] - [data_helper] [140544298686208] - INFO - creating direct client 172.23.121.213:11210 metadata [2021-02-27 17:00:14,961] - [task] [140544298686208] - WARNING - vbucket map not ready after try 3 [2021-02-27 17:00:14,961] - [bucket_helper] [140544298686208] - INFO - waiting for memcached bucket : metadata in 172.23.104.97 to accept set ops [2021-02-27 17:00:15,091] - [data_helper] [140544298686208] - INFO - creating direct client 172.23.104.97:11210 metadata [2021-02-27 17:00:15,191] - [data_helper] [140544298686208] - INFO - creating direct client 172.23.105.238:11210 metadata [2021-02-27 17:00:15,449] - [data_helper] [140544298686208] - INFO - creating direct client 172.23.121.213:11210 metadata [2021-02-27 17:00:15,549] - [data_helper] [140544298686208] - INFO - creating direct client 172.23.104.97:11210 metadata [2021-02-27 17:00:15,631] - [data_helper] [140544298686208] - INFO - creating direct client 172.23.105.238:11210 metadata [2021-02-27 17:00:15,681] - [data_helper] [140544298686208] - INFO - creating direct client 172.23.121.213:11210 metadata [2021-02-27 17:00:15,724] - [task] [140544298686208] - WARNING - vbucket map not ready after try 4 [2021-02-27 17:00:15,724] - [bucket_helper] [140544298686208] - INFO - waiting for memcached bucket : metadata in 172.23.104.97 to accept set ops [2021-02-27 17:00:16,041] - [data_helper] [140544298686208] - INFO - creating direct client 172.23.104.97:11210 metadata [2021-02-27 17:00:16,119] - [data_helper] [140544298686208] - INFO - creating direct client 172.23.105.238:11210 metadata [2021-02-27 17:00:16,173] - [data_helper] [140544298686208] - INFO - creating direct client 172.23.121.213:11210 metadata [2021-02-27 17:00:16,272] - [data_helper] [140544298686208] - INFO - creating direct client 172.23.104.97:11210 metadata [2021-02-27 17:00:16,356] - [data_helper] [140544298686208] - INFO - creating direct client 172.23.105.238:11210 metadata [2021-02-27 17:00:16,414] - [data_helper] [140544298686208] - INFO - creating direct client 172.23.121.213:11210 metadata [2021-02-27 17:00:16,648] - [task] [140544298686208] - WARNING - vbucket map not ready after try 5 [2021-02-27 17:00:16,739] - [rest_client] [140542730036992] - INFO - http://172.23.105.247:8091/pools/default/buckets/dst_bucket/scopes with params: name=scope_0 [2021-02-27 17:00:16,739] - [rest_client] [140542730036992] - INFO - Scope created dst_bucket->scope_0 {'uid': '1'} [2021-02-27 17:00:16,773] - [rest_client] [140542730036992] - INFO - http://172.23.105.247:8091/pools/default/buckets/dst_bucket/scopes with params: name=scope_1 [2021-02-27 17:00:16,773] - [rest_client] [140542730036992] - INFO - Scope created dst_bucket->scope_1 {'uid': '2'} [2021-02-27 17:00:16,809] - [rest_client] [140542730036992] - INFO - http://172.23.105.247:8091/pools/default/buckets/dst_bucket/scopes with params: name=scope_2 [2021-02-27 17:00:16,809] - [rest_client] [140542730036992] - INFO - Scope created dst_bucket->scope_2 {'uid': '3'} [2021-02-27 17:00:16,845] - [rest_client] [140542730036992] - INFO - http://172.23.105.247:8091/pools/default/buckets/dst_bucket/scopes with params: name=scope_3 [2021-02-27 17:00:16,845] - [rest_client] [140542730036992] - INFO - Scope created dst_bucket->scope_3 {'uid': '4'} [2021-02-27 17:00:16,879] - [rest_client] [140542730036992] - INFO - http://172.23.105.247:8091/pools/default/buckets/dst_bucket/scopes with params: name=scope_4 [2021-02-27 17:00:16,880] - [rest_client] [140542730036992] - INFO - Scope created dst_bucket->scope_4 {'uid': '5'} [2021-02-27 17:00:16,911] - [rest_client] [140542730036992] - INFO - http://172.23.105.247:8091/pools/default/buckets/src_bucket/scopes with params: name=scope_0 [2021-02-27 17:00:16,912] - [rest_client] [140542730036992] - INFO - Scope created src_bucket->scope_0 {'uid': '1'} [2021-02-27 17:00:16,954] - [rest_client] [140542730036992] - INFO - http://172.23.105.247:8091/pools/default/buckets/src_bucket/scopes with params: name=scope_1 [2021-02-27 17:00:16,954] - [rest_client] [140542730036992] - INFO - Scope created src_bucket->scope_1 {'uid': '2'} [2021-02-27 17:00:16,996] - [rest_client] [140542730036992] - INFO - http://172.23.105.247:8091/pools/default/buckets/src_bucket/scopes with params: name=scope_2 [2021-02-27 17:00:16,996] - [rest_client] [140542730036992] - INFO - Scope created src_bucket->scope_2 {'uid': '3'} [2021-02-27 17:00:17,033] - [rest_client] [140542730036992] - INFO - http://172.23.105.247:8091/pools/default/buckets/src_bucket/scopes with params: name=scope_3 [2021-02-27 17:00:17,033] - [rest_client] [140542730036992] - INFO - Scope created src_bucket->scope_3 {'uid': '4'} [2021-02-27 17:00:17,065] - [rest_client] [140542730036992] - INFO - http://172.23.105.247:8091/pools/default/buckets/src_bucket/scopes with params: name=scope_4 [2021-02-27 17:00:17,065] - [rest_client] [140542730036992] - INFO - Scope created src_bucket->scope_4 {'uid': '5'} [2021-02-27 17:00:17,105] - [rest_client] [140542730036992] - INFO - http://172.23.105.247:8091/pools/default/buckets/dst_bucket/scopes/scope_1/collections with params: name=coll_0 [2021-02-27 17:00:17,106] - [rest_client] [140542730036992] - INFO - Collection created dst_bucket->scope_1->coll_0 manifest:{'uid': '6'} [2021-02-27 17:00:17,142] - [rest_client] [140542730036992] - INFO - http://172.23.105.247:8091/pools/default/buckets/dst_bucket/scopes/scope_1/collections with params: name=coll_1 [2021-02-27 17:00:17,142] - [rest_client] [140542730036992] - INFO - Collection created dst_bucket->scope_1->coll_1 manifest:{'uid': '7'} [2021-02-27 17:00:17,175] - [rest_client] [140542730036992] - INFO - http://172.23.105.247:8091/pools/default/buckets/dst_bucket/scopes/scope_1/collections with params: name=coll_2 [2021-02-27 17:00:17,175] - [rest_client] [140542730036992] - INFO - Collection created dst_bucket->scope_1->coll_2 manifest:{'uid': '8'} [2021-02-27 17:00:17,212] - [rest_client] [140542730036992] - INFO - http://172.23.105.247:8091/pools/default/buckets/dst_bucket/scopes/scope_1/collections with params: name=coll_3 [2021-02-27 17:00:17,212] - [rest_client] [140542730036992] - INFO - Collection created dst_bucket->scope_1->coll_3 manifest:{'uid': '9'} [2021-02-27 17:00:17,248] - [rest_client] [140542730036992] - INFO - http://172.23.105.247:8091/pools/default/buckets/dst_bucket/scopes/scope_1/collections with params: name=coll_4 [2021-02-27 17:00:17,248] - [rest_client] [140542730036992] - INFO - Collection created dst_bucket->scope_1->coll_4 manifest:{'uid': 'a'} [2021-02-27 17:00:17,286] - [rest_client] [140542730036992] - INFO - http://172.23.105.247:8091/pools/default/buckets/src_bucket/scopes/scope_1/collections with params: name=coll_0 [2021-02-27 17:00:17,286] - [rest_client] [140542730036992] - INFO - Collection created src_bucket->scope_1->coll_0 manifest:{'uid': '6'} [2021-02-27 17:00:17,327] - [rest_client] [140542730036992] - INFO - http://172.23.105.247:8091/pools/default/buckets/src_bucket/scopes/scope_1/collections with params: name=coll_1 [2021-02-27 17:00:17,327] - [rest_client] [140542730036992] - INFO - Collection created src_bucket->scope_1->coll_1 manifest:{'uid': '7'} [2021-02-27 17:00:17,362] - [rest_client] [140542730036992] - INFO - http://172.23.105.247:8091/pools/default/buckets/src_bucket/scopes/scope_1/collections with params: name=coll_2 [2021-02-27 17:00:17,362] - [rest_client] [140542730036992] - INFO - Collection created src_bucket->scope_1->coll_2 manifest:{'uid': '8'} [2021-02-27 17:00:17,398] - [rest_client] [140542730036992] - INFO - http://172.23.105.247:8091/pools/default/buckets/src_bucket/scopes/scope_1/collections with params: name=coll_3 [2021-02-27 17:00:17,398] - [rest_client] [140542730036992] - INFO - Collection created src_bucket->scope_1->coll_3 manifest:{'uid': '9'} [2021-02-27 17:00:17,434] - [rest_client] [140542730036992] - INFO - http://172.23.105.247:8091/pools/default/buckets/src_bucket/scopes/scope_1/collections with params: name=coll_4 [2021-02-27 17:00:17,435] - [rest_client] [140542730036992] - INFO - Collection created src_bucket->scope_1->coll_4 manifest:{'uid': 'a'} [2021-02-27 17:00:18,630] - [eventing_base] [140542730036992] - INFO - saving function bucket_op [2021-02-27 17:00:19,819] - [eventing_base] [140542730036992] - INFO - saving function timers [2021-02-27 17:00:21,071] - [eventing_base] [140542730036992] - INFO - saving function sbm [2021-02-27 17:00:22,280] - [eventing_base] [140542730036992] - INFO - saving function curl [2021-02-27 17:00:23,504] - [eventing_base] [140542730036992] - INFO - saving function n1ql [2021-02-27 17:00:24,762] - [basetestcase] [140542730036992] - INFO - sleep for 5 secs. Waiting for bucket_op to deployed... ... [2021-02-27 17:00:29,791] - [basetestcase] [140542730036992] - INFO - sleep for 5 secs. Waiting for bucket_op to deployed... ... [2021-02-27 17:00:34,823] - [basetestcase] [140542730036992] - INFO - sleep for 5 secs. Waiting for bucket_op to deployed... ... [2021-02-27 17:00:39,864] - [basetestcase] [140542730036992] - INFO - sleep for 5 secs. Waiting for bucket_op to deployed... ... [2021-02-27 17:00:46,174] - [basetestcase] [140542730036992] - INFO - sleep for 5 secs. Waiting for timers to deployed... ... [2021-02-27 17:00:51,198] - [basetestcase] [140542730036992] - INFO - sleep for 5 secs. Waiting for timers to deployed... ... [2021-02-27 17:00:56,255] - [basetestcase] [140542730036992] - INFO - sleep for 5 secs. Waiting for timers to deployed... ... [2021-02-27 17:01:01,282] - [basetestcase] [140542730036992] - INFO - sleep for 5 secs. Waiting for timers to deployed... ... [2021-02-27 17:01:07,638] - [basetestcase] [140542730036992] - INFO - sleep for 5 secs. Waiting for sbm to deployed... ... [2021-02-27 17:01:12,665] - [basetestcase] [140542730036992] - INFO - sleep for 5 secs. Waiting for sbm to deployed... ... [2021-02-27 17:01:17,725] - [basetestcase] [140542730036992] - INFO - sleep for 5 secs. Waiting for sbm to deployed... ... [2021-02-27 17:01:22,753] - [basetestcase] [140542730036992] - INFO - sleep for 5 secs. Waiting for sbm to deployed... ... [2021-02-27 17:01:28,983] - [basetestcase] [140542730036992] - INFO - sleep for 5 secs. Waiting for curl to deployed... ... [2021-02-27 17:01:34,006] - [basetestcase] [140542730036992] - INFO - sleep for 5 secs. Waiting for curl to deployed... ... [2021-02-27 17:01:39,057] - [basetestcase] [140542730036992] - INFO - sleep for 5 secs. Waiting for curl to deployed... ... [2021-02-27 17:01:44,084] - [basetestcase] [140542730036992] - INFO - sleep for 5 secs. Waiting for curl to deployed... ... [2021-02-27 17:01:50,326] - [basetestcase] [140542730036992] - INFO - sleep for 5 secs. Waiting for n1ql to deployed... ... [2021-02-27 17:01:55,353] - [basetestcase] [140542730036992] - INFO - sleep for 5 secs. Waiting for n1ql to deployed... ... [2021-02-27 17:02:00,387] - [basetestcase] [140542730036992] - INFO - sleep for 5 secs. Waiting for n1ql to deployed... ... [2021-02-27 17:02:05,412] - [basetestcase] [140542730036992] - INFO - sleep for 5 secs. Waiting for n1ql to deployed... ... [2021-02-27 17:02:10,763] - [task] [140544298686208] - INFO - java -jar java_sdk_client/collections/target/javaclient/javaclient.jar -i 172.23.104.97 -u Administrator -p password -b src_bucket -s _default -c _default -n 40320 -pc 100 -pu 0 -pd 0 -l uniform -dsn 1 -dpx doc_ -dt Person -de 0 -ds 500 -ac False -st 0 -en 40319 -o False -sd False [2021-02-27 17:02:13,455] - [rest_client] [140542730036992] - INFO - rebalance reached >100% in 3.0047976970672607 seconds [2021-02-27 17:02:21,409] - [task] [140544298686208] - INFO - java -jar java_sdk_client/collections/target/javaclient/javaclient.jar -i 172.23.104.97 -u Administrator -p password -b src_bucket -s scope_1 -c coll_1 -n 40320 -pc 100 -pu 0 -pd 0 -l uniform -dsn 1 -dpx doc_ -dt Person -de 0 -ds 500 -ac False -st 0 -en 40319 -o False -sd False [2021-02-27 17:02:32,076] - [task] [140544298686208] - INFO - adding node 172.23.107.65:8091 to cluster [2021-02-27 17:02:32,077] - [rest_client] [140544298686208] - INFO - adding remote node @172.23.107.65:8091 to this cluster @172.23.104.97:8091 [2021-02-27 17:02:42,097] - [rest_client] [140544298686208] - INFO - rebalance progress took 10.02 seconds [2021-02-27 17:02:42,098] - [rest_client] [140544298686208] - INFO - sleep for 10 seconds after rebalance... [2021-02-27 17:02:54,774] - [task] [140544298686208] - INFO - adding node 172.23.105.240:8091 to cluster [2021-02-27 17:02:54,775] - [rest_client] [140544298686208] - INFO - adding remote node @172.23.105.240:8091 to this cluster @172.23.104.97:8091 [2021-02-27 17:03:04,793] - [rest_client] [140544298686208] - INFO - rebalance progress took 10.02 seconds [2021-02-27 17:03:04,793] - [rest_client] [140544298686208] - INFO - sleep for 10 seconds after rebalance... [2021-02-27 17:03:18,225] - [rest_client] [140544298686208] - INFO - rebalance params : {'knownNodes': 'ns_1@172.23.104.97,ns_1@172.23.105.238,ns_1@172.23.105.240,ns_1@172.23.105.247,ns_1@172.23.106.1,ns_1@172.23.107.65,ns_1@172.23.121.213', 'ejectedNodes': '', 'user': 'Administrator', 'password': 'password'} [2021-02-27 17:03:18,265] - [rest_client] [140544298686208] - INFO - rebalance operation started [2021-02-27 17:03:18,270] - [rest_client] [140544298686208] - INFO - rebalance percentage : 0.00 % [2021-02-27 17:03:18,270] - [task] [140544298686208] - INFO - Rebalance - status: running, progress: 0.00% [2021-02-27 17:03:28,328] - [rest_client] [140544298686208] - INFO - rebalance percentage : 18.00 % [2021-02-27 17:03:28,328] - [task] [140544298686208] - INFO - Rebalance - status: running, progress: 18.00% [2021-02-27 17:03:38,415] - [rest_client] [140544298686208] - INFO - rebalance percentage : 44.00 % [2021-02-27 17:03:38,415] - [task] [140544298686208] - INFO - Rebalance - status: running, progress: 44.00% [2021-02-27 17:03:48,479] - [rest_client] [140544298686208] - INFO - rebalance percentage : 70.00 % [2021-02-27 17:03:48,479] - [task] [140544298686208] - INFO - Rebalance - status: running, progress: 70.00% [2021-02-27 17:03:58,523] - [rest_client] [140544298686208] - INFO - rebalance percentage : 78.00 % [2021-02-27 17:03:58,523] - [task] [140544298686208] - INFO - Rebalance - status: running, progress: 78.00% [2021-02-27 17:04:08,567] - [rest_client] [140544298686208] - INFO - rebalance percentage : 79.00 % [2021-02-27 17:04:08,567] - [task] [140544298686208] - INFO - Rebalance - status: running, progress: 79.00% [2021-02-27 17:04:18,607] - [rest_client] [140544298686208] - INFO - rebalance percentage : 81.00 % [2021-02-27 17:04:18,608] - [task] [140544298686208] - INFO - Rebalance - status: running, progress: 81.00% [2021-02-27 17:04:28,647] - [rest_client] [140544298686208] - INFO - rebalance percentage : 83.00 % [2021-02-27 17:04:28,647] - [task] [140544298686208] - INFO - Rebalance - status: running, progress: 83.00% [2021-02-27 17:04:38,686] - [rest_client] [140544298686208] - INFO - rebalance percentage : 84.00 % [2021-02-27 17:04:38,686] - [task] [140544298686208] - INFO - Rebalance - status: running, progress: 84.00% [2021-02-27 17:04:48,730] - [rest_client] [140544298686208] - INFO - rebalance percentage : 85.00 % [2021-02-27 17:04:48,730] - [task] [140544298686208] - INFO - Rebalance - status: running, progress: 85.00% [2021-02-27 17:04:58,777] - [rest_client] [140544298686208] - INFO - rebalance percentage : 86.00 % [2021-02-27 17:04:58,777] - [task] [140544298686208] - INFO - Rebalance - status: running, progress: 86.00% [2021-02-27 17:05:08,831] - [rest_client] [140544298686208] - INFO - rebalance percentage : 87.00 % [2021-02-27 17:05:08,832] - [task] [140544298686208] - INFO - Rebalance - status: running, progress: 87.00% [2021-02-27 17:05:18,878] - [rest_client] [140544298686208] - INFO - rebalance percentage : 87.00 % [2021-02-27 17:05:18,879] - [task] [140544298686208] - INFO - Rebalance - status: running, progress: 87.00% [2021-02-27 17:05:28,934] - [rest_client] [140544298686208] - INFO - rebalance percentage : 90.00 % [2021-02-27 17:05:28,934] - [task] [140544298686208] - INFO - Rebalance - status: running, progress: 90.00% [2021-02-27 17:05:38,980] - [rest_client] [140544298686208] - INFO - rebalance percentage : 91.00 % [2021-02-27 17:05:38,980] - [task] [140544298686208] - INFO - Rebalance - status: running, progress: 91.00% [2021-02-27 17:05:49,029] - [rest_client] [140544298686208] - INFO - rebalance percentage : 92.00 % [2021-02-27 17:05:49,029] - [task] [140544298686208] - INFO - Rebalance - status: running, progress: 92.00% [2021-02-27 17:05:59,075] - [task] [140544298686208] - INFO - Rebalance - status: none, progress: 100.00% [2021-02-27 17:05:59,099] - [task] [140544298686208] - INFO - rebalancing was completed with progress: 100% in 160.83309984207153 sec [2021-02-27 17:05:59,116] - [basetestcase] [140542730036992] - INFO - list of eventing nodes in cluster: [ip:172.23.105.247 port:8091 ssh_username:root] [2021-02-27 17:05:59,122] - [tuq_helper] [140542730036992] - INFO - RUN QUERY create primary index on dst_bucket.scope_1.coll_0 [2021-02-27 17:05:59,127] - [rest_client] [140542730036992] - INFO - query params : statement=create+primary+index+on+dst_bucket.scope_1.coll_0 [2021-02-27 17:06:02,368] - [tuq_helper] [140542730036992] - INFO - TOTAL ELAPSED TIME: 3.23816301s [2021-02-27 17:06:02,407] - [basetestcase] [140542730036992] - INFO - list of n1ql nodes in cluster: [ip:172.23.106.1 port:8091 ssh_username:root] [2021-02-27 17:06:02,413] - [rest_client] [140542730036992] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_0 [2021-02-27 17:06:02,426] - [eventing_base] [140542730036992] - INFO - Final docs count... Current : 40320 Expected : 40320 [2021-02-27 17:06:02,440] - [basetestcase] [140542730036992] - INFO - list of eventing nodes in cluster: [ip:172.23.105.247 port:8091 ssh_username:root] [2021-02-27 17:06:02,446] - [tuq_helper] [140542730036992] - INFO - RUN QUERY create primary index on dst_bucket.scope_1.coll_1 [2021-02-27 17:06:02,451] - [rest_client] [140542730036992] - INFO - query params : statement=create+primary+index+on+dst_bucket.scope_1.coll_1 [2021-02-27 17:06:05,991] - [tuq_helper] [140542730036992] - INFO - TOTAL ELAPSED TIME: 3.538465734s [2021-02-27 17:06:06,028] - [basetestcase] [140542730036992] - INFO - list of n1ql nodes in cluster: [ip:172.23.106.1 port:8091 ssh_username:root] [2021-02-27 17:06:06,032] - [rest_client] [140542730036992] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_1 [2021-02-27 17:06:06,046] - [eventing_base] [140542730036992] - INFO - Final docs count... Current : 40320 Expected : 40320 [2021-02-27 17:06:06,061] - [basetestcase] [140542730036992] - INFO - list of eventing nodes in cluster: [ip:172.23.105.247 port:8091 ssh_username:root] [2021-02-27 17:06:06,067] - [tuq_helper] [140542730036992] - INFO - RUN QUERY create primary index on dst_bucket.scope_1.coll_3 [2021-02-27 17:06:06,071] - [rest_client] [140542730036992] - INFO - query params : statement=create+primary+index+on+dst_bucket.scope_1.coll_3 [2021-02-27 17:06:09,497] - [tuq_helper] [140542730036992] - INFO - TOTAL ELAPSED TIME: 3.424427991s [2021-02-27 17:06:09,541] - [basetestcase] [140542730036992] - INFO - list of n1ql nodes in cluster: [ip:172.23.106.1 port:8091 ssh_username:root] [2021-02-27 17:06:09,549] - [rest_client] [140542730036992] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_3 [2021-02-27 17:06:09,561] - [eventing_base] [140542730036992] - INFO - Final docs count... Current : 40320 Expected : 40320 [2021-02-27 17:06:09,582] - [basetestcase] [140542730036992] - INFO - list of eventing nodes in cluster: [ip:172.23.105.247 port:8091 ssh_username:root] [2021-02-27 17:06:09,587] - [tuq_helper] [140542730036992] - INFO - RUN QUERY create primary index on dst_bucket.scope_1.coll_4 [2021-02-27 17:06:09,592] - [rest_client] [140542730036992] - INFO - query params : statement=create+primary+index+on+dst_bucket.scope_1.coll_4 [2021-02-27 17:06:12,863] - [tuq_helper] [140542730036992] - INFO - TOTAL ELAPSED TIME: 3.269686447s [2021-02-27 17:06:12,910] - [basetestcase] [140542730036992] - INFO - list of n1ql nodes in cluster: [ip:172.23.106.1 port:8091 ssh_username:root] [2021-02-27 17:06:12,915] - [rest_client] [140542730036992] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-27 17:06:12,928] - [basetestcase] [140542730036992] - INFO - sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_4 to complete bucket operations... Current : 40316 Expected : 40320 ... [2021-02-27 17:06:43,032] - [basetestcase] [140542730036992] - INFO - list of n1ql nodes in cluster: [ip:172.23.106.1 port:8091 ssh_username:root] [2021-02-27 17:06:43,038] - [rest_client] [140542730036992] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-27 17:06:43,104] - [basetestcase] [140542730036992] - INFO - list of n1ql nodes in cluster: [ip:172.23.106.1 port:8091 ssh_username:root] [2021-02-27 17:06:43,109] - [rest_client] [140542730036992] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-27 17:06:43,121] - [basetestcase] [140542730036992] - INFO - sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_4 to complete bucket operations... Current : 40316 Expected : 40320 ... [2021-02-27 17:07:13,232] - [basetestcase] [140542730036992] - INFO - list of n1ql nodes in cluster: [ip:172.23.106.1 port:8091 ssh_username:root] [2021-02-27 17:07:13,241] - [rest_client] [140542730036992] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-27 17:07:13,317] - [basetestcase] [140542730036992] - INFO - list of n1ql nodes in cluster: [ip:172.23.106.1 port:8091 ssh_username:root] [2021-02-27 17:07:13,325] - [rest_client] [140542730036992] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-27 17:07:13,337] - [basetestcase] [140542730036992] - INFO - sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_4 to complete bucket operations... Current : 40316 Expected : 40320 ... [2021-02-27 17:07:43,438] - [basetestcase] [140542730036992] - INFO - list of n1ql nodes in cluster: [ip:172.23.106.1 port:8091 ssh_username:root] [2021-02-27 17:07:43,444] - [rest_client] [140542730036992] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-27 17:07:43,511] - [basetestcase] [140542730036992] - INFO - list of n1ql nodes in cluster: [ip:172.23.106.1 port:8091 ssh_username:root] [2021-02-27 17:07:43,517] - [rest_client] [140542730036992] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-27 17:07:43,528] - [basetestcase] [140542730036992] - INFO - sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_4 to complete bucket operations... Current : 40316 Expected : 40320 ... [2021-02-27 17:08:13,596] - [basetestcase] [140542730036992] - INFO - list of n1ql nodes in cluster: [ip:172.23.106.1 port:8091 ssh_username:root] [2021-02-27 17:08:13,602] - [rest_client] [140542730036992] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-27 17:08:13,633] - [basetestcase] [140542730036992] - INFO - list of n1ql nodes in cluster: [ip:172.23.106.1 port:8091 ssh_username:root] [2021-02-27 17:08:13,638] - [rest_client] [140542730036992] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-27 17:08:13,641] - [basetestcase] [140542730036992] - INFO - sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_4 to complete bucket operations... Current : 40316 Expected : 40320 ... [2021-02-27 17:08:43,714] - [basetestcase] [140542730036992] - INFO - list of n1ql nodes in cluster: [ip:172.23.106.1 port:8091 ssh_username:root] [2021-02-27 17:08:43,720] - [rest_client] [140542730036992] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-27 17:08:43,752] - [basetestcase] [140542730036992] - INFO - list of n1ql nodes in cluster: [ip:172.23.106.1 port:8091 ssh_username:root] [2021-02-27 17:08:43,757] - [rest_client] [140542730036992] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-27 17:08:43,761] - [basetestcase] [140542730036992] - INFO - sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_4 to complete bucket operations... Current : 40316 Expected : 40320 ... [2021-02-27 17:09:13,832] - [basetestcase] [140542730036992] - INFO - list of n1ql nodes in cluster: [ip:172.23.106.1 port:8091 ssh_username:root] [2021-02-27 17:09:13,839] - [rest_client] [140542730036992] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-27 17:09:13,867] - [basetestcase] [140542730036992] - INFO - list of n1ql nodes in cluster: [ip:172.23.106.1 port:8091 ssh_username:root] [2021-02-27 17:09:13,873] - [rest_client] [140542730036992] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-27 17:09:13,877] - [basetestcase] [140542730036992] - INFO - sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_4 to complete bucket operations... Current : 40316 Expected : 40320 ... [2021-02-27 17:09:43,960] - [basetestcase] [140542730036992] - INFO - list of n1ql nodes in cluster: [ip:172.23.106.1 port:8091 ssh_username:root] [2021-02-27 17:09:43,965] - [rest_client] [140542730036992] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-27 17:09:44,020] - [basetestcase] [140542730036992] - INFO - list of n1ql nodes in cluster: [ip:172.23.106.1 port:8091 ssh_username:root] [2021-02-27 17:09:44,025] - [rest_client] [140542730036992] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-27 17:09:44,031] - [basetestcase] [140542730036992] - INFO - sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_4 to complete bucket operations... Current : 40316 Expected : 40320 ... [2021-02-27 17:10:14,111] - [basetestcase] [140542730036992] - INFO - list of n1ql nodes in cluster: [ip:172.23.106.1 port:8091 ssh_username:root] [2021-02-27 17:10:14,117] - [rest_client] [140542730036992] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-27 17:10:14,149] - [basetestcase] [140542730036992] - INFO - list of n1ql nodes in cluster: [ip:172.23.106.1 port:8091 ssh_username:root] [2021-02-27 17:10:14,154] - [rest_client] [140542730036992] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-27 17:10:14,159] - [basetestcase] [140542730036992] - INFO - sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_4 to complete bucket operations... Current : 40316 Expected : 40320 ... [2021-02-27 17:10:44,238] - [basetestcase] [140542730036992] - INFO - list of n1ql nodes in cluster: [ip:172.23.106.1 port:8091 ssh_username:root] [2021-02-27 17:10:44,245] - [rest_client] [140542730036992] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-27 17:10:44,274] - [basetestcase] [140542730036992] - INFO - list of n1ql nodes in cluster: [ip:172.23.106.1 port:8091 ssh_username:root] [2021-02-27 17:10:44,281] - [rest_client] [140542730036992] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-27 17:10:44,284] - [basetestcase] [140542730036992] - INFO - sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_4 to complete bucket operations... Current : 40316 Expected : 40320 ... [2021-02-27 17:11:14,355] - [basetestcase] [140542730036992] - INFO - list of n1ql nodes in cluster: [ip:172.23.106.1 port:8091 ssh_username:root] [2021-02-27 17:11:14,361] - [rest_client] [140542730036992] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-27 17:11:14,389] - [basetestcase] [140542730036992] - INFO - list of n1ql nodes in cluster: [ip:172.23.106.1 port:8091 ssh_username:root] [2021-02-27 17:11:14,395] - [rest_client] [140542730036992] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-27 17:11:14,398] - [basetestcase] [140542730036992] - INFO - sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_4 to complete bucket operations... Current : 40316 Expected : 40320 ... [2021-02-27 17:11:44,474] - [basetestcase] [140542730036992] - INFO - list of n1ql nodes in cluster: [ip:172.23.106.1 port:8091 ssh_username:root] [2021-02-27 17:11:44,481] - [rest_client] [140542730036992] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-27 17:11:44,520] - [eventing_base] [140542730036992] - INFO - Compacting bucket : dst_bucket [2021-02-27 17:11:44,549] - [rest_client] [140542730036992] - INFO - bucket compaction successful [2021-02-27 17:11:44,549] - [eventing_base] [140542730036992] - INFO - Compacting bucket : metadata [2021-02-27 17:11:44,565] - [rest_client] [140542730036992] - INFO - bucket compaction successful [2021-02-27 17:11:44,565] - [eventing_base] [140542730036992] - INFO - Compacting bucket : src_bucket [2021-02-27 17:11:44,588] - [rest_client] [140542730036992] - INFO - bucket compaction successful [2021-02-27 17:11:44,624] - [basetestcase] [140542730036992] - INFO - list of n1ql nodes in cluster: [ip:172.23.106.1 port:8091 ssh_username:root] [2021-02-27 17:11:44,630] - [rest_client] [140542730036992] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-27 17:11:44,640] - [basetestcase] [140542730036992] - INFO - sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_4 to complete bucket operations... Current : 40316 Expected : 40320 ... [2021-02-27 17:12:14,707] - [basetestcase] [140542730036992] - INFO - list of n1ql nodes in cluster: [ip:172.23.106.1 port:8091 ssh_username:root] [2021-02-27 17:12:14,714] - [rest_client] [140542730036992] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-27 17:12:14,747] - [basetestcase] [140542730036992] - INFO - list of n1ql nodes in cluster: [ip:172.23.106.1 port:8091 ssh_username:root] [2021-02-27 17:12:14,753] - [rest_client] [140542730036992] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-27 17:12:14,757] - [basetestcase] [140542730036992] - INFO - sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_4 to complete bucket operations... Current : 40316 Expected : 40320 ... [2021-02-27 17:12:44,819] - [basetestcase] [140542730036992] - INFO - list of n1ql nodes in cluster: [ip:172.23.106.1 port:8091 ssh_username:root] [2021-02-27 17:12:44,827] - [rest_client] [140542730036992] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-27 17:12:44,858] - [basetestcase] [140542730036992] - INFO - list of n1ql nodes in cluster: [ip:172.23.106.1 port:8091 ssh_username:root] [2021-02-27 17:12:44,863] - [rest_client] [140542730036992] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-27 17:12:44,866] - [basetestcase] [140542730036992] - INFO - sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_4 to complete bucket operations... Current : 40316 Expected : 40320 ... [2021-02-27 17:13:14,936] - [basetestcase] [140542730036992] - INFO - list of n1ql nodes in cluster: [ip:172.23.106.1 port:8091 ssh_username:root] [2021-02-27 17:13:14,946] - [rest_client] [140542730036992] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-27 17:13:14,976] - [basetestcase] [140542730036992] - INFO - list of n1ql nodes in cluster: [ip:172.23.106.1 port:8091 ssh_username:root] [2021-02-27 17:13:14,982] - [rest_client] [140542730036992] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-27 17:13:14,987] - [basetestcase] [140542730036992] - INFO - sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_4 to complete bucket operations... Current : 40316 Expected : 40320 ... [2021-02-27 17:13:45,059] - [basetestcase] [140542730036992] - INFO - list of n1ql nodes in cluster: [ip:172.23.106.1 port:8091 ssh_username:root] [2021-02-27 17:13:45,066] - [rest_client] [140542730036992] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-27 17:13:45,096] - [basetestcase] [140542730036992] - INFO - list of n1ql nodes in cluster: [ip:172.23.106.1 port:8091 ssh_username:root] [2021-02-27 17:13:45,102] - [rest_client] [140542730036992] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-27 17:13:45,106] - [basetestcase] [140542730036992] - INFO - sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_4 to complete bucket operations... Current : 40316 Expected : 40320 ... [2021-02-27 17:14:15,171] - [basetestcase] [140542730036992] - INFO - list of n1ql nodes in cluster: [ip:172.23.106.1 port:8091 ssh_username:root] [2021-02-27 17:14:15,178] - [rest_client] [140542730036992] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-27 17:14:15,207] - [basetestcase] [140542730036992] - INFO - list of n1ql nodes in cluster: [ip:172.23.106.1 port:8091 ssh_username:root] [2021-02-27 17:14:15,214] - [rest_client] [140542730036992] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-27 17:14:15,217] - [basetestcase] [140542730036992] - INFO - sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_4 to complete bucket operations... Current : 40316 Expected : 40320 ... [2021-02-27 17:14:45,300] - [basetestcase] [140542730036992] - INFO - list of n1ql nodes in cluster: [ip:172.23.106.1 port:8091 ssh_username:root] [2021-02-27 17:14:45,309] - [rest_client] [140542730036992] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-27 17:14:45,339] - [basetestcase] [140542730036992] - INFO - list of n1ql nodes in cluster: [ip:172.23.106.1 port:8091 ssh_username:root] [2021-02-27 17:14:45,343] - [rest_client] [140542730036992] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-27 17:14:45,347] - [basetestcase] [140542730036992] - INFO - sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_4 to complete bucket operations... Current : 40316 Expected : 40320 ... [2021-02-27 17:15:15,427] - [basetestcase] [140542730036992] - INFO - list of n1ql nodes in cluster: [ip:172.23.106.1 port:8091 ssh_username:root] [2021-02-27 17:15:15,432] - [rest_client] [140542730036992] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-27 17:15:15,464] - [basetestcase] [140542730036992] - INFO - list of n1ql nodes in cluster: [ip:172.23.106.1 port:8091 ssh_username:root] [2021-02-27 17:15:15,471] - [rest_client] [140542730036992] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-27 17:15:15,475] - [basetestcase] [140542730036992] - INFO - sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_4 to complete bucket operations... Current : 40316 Expected : 40320 ... [2021-02-27 17:15:45,545] - [basetestcase] [140542730036992] - INFO - list of n1ql nodes in cluster: [ip:172.23.106.1 port:8091 ssh_username:root] [2021-02-27 17:15:45,552] - [rest_client] [140542730036992] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-27 17:15:45,579] - [basetestcase] [140542730036992] - INFO - list of n1ql nodes in cluster: [ip:172.23.106.1 port:8091 ssh_username:root] [2021-02-27 17:15:45,585] - [rest_client] [140542730036992] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-27 17:15:45,590] - [basetestcase] [140542730036992] - INFO - sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_4 to complete bucket operations... Current : 40316 Expected : 40320 ... [2021-02-27 17:16:15,655] - [basetestcase] [140542730036992] - INFO - list of n1ql nodes in cluster: [ip:172.23.106.1 port:8091 ssh_username:root] [2021-02-27 17:16:15,661] - [rest_client] [140542730036992] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-27 17:16:15,688] - [basetestcase] [140542730036992] - INFO - list of n1ql nodes in cluster: [ip:172.23.106.1 port:8091 ssh_username:root] [2021-02-27 17:16:15,694] - [rest_client] [140542730036992] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-27 17:16:15,713] - [basetestcase] [140542730036992] - INFO - list of eventing nodes in cluster: [ip:172.23.105.247 port:8091 ssh_username:root] [2021-02-27 17:16:15,766] - [eventing_base] [140542730036992] - INFO - Stats for Node 172.23.105.247 is [ { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 4, "agg_messages_sent_to_worker": 47920, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 201600, "dcp_mutation_sent_to_worker": 40320, "dcp_snapshot": 122190, "dcp_stream_req_counter": 2453, "dcp_stream_req_err_counter": 22, "dcp_streamend": 1163, "dcp_streamreq": 2431, "dcp_system_event": 10240, "execution_stats": 876, "failure_stats": 876, "latency_stats": 876, "lcb_exception_stats": 876, "log_level": 1, "num_processed_events": 40320, "processed_events_size": 36102328, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 40320, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 40320, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 40320, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 40320, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 47916, "no_op_counter": 0, "num_processed_events": 40320, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 40320, "processed_events_size": 36102328, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "28457": "2021-02-28T01:16:15Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "28457": "2021-02-28T01:16:15Z" }, "v8worker_events_lost": 0 }, "function_id": 953325219, "function_name": "curl", "gocb_creds_request_counter": 100, "internal_vb_distribution_stats": { "worker_curl_0": "[0-1023]" }, "latency_percentile_stats": { "100": 36000, "50": 1700, "80": 2300, "90": 2900, "95": 3600, "99": 7100 }, "lcb_creds_request_counter": 80739, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.105.247:8096", "start_vb": 0, "vb_count": 1024 } ], "vb_distribution_stats_from_metadata": { "172.23.105.247:8096": { "worker_curl_0": "[0-1023]" } }, "worker_pids": { "worker_curl_0": 28457 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 3, "agg_messages_sent_to_worker": 47799, "agg_queue_memory": 0, "agg_queue_memory_cap": 20971520, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 201600, "dcp_mutation_sent_to_worker": 40320, "dcp_snapshot": 114853, "dcp_stream_req_counter": 2461, "dcp_stream_req_err_counter": 21, "dcp_streamend": 1148, "dcp_streamreq": 2440, "dcp_system_event": 10240, "execution_stats": 855, "failure_stats": 855, "latency_stats": 855, "lcb_exception_stats": 855, "log_level": 1, "num_processed_events": 40320, "processed_events_size": 36102328, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 40320, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 40320, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 47795, "no_op_counter": 0, "num_processed_events": 40320, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 4, "on_update_success": 40316, "processed_events_size": 36102328, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "28499": "2021-02-28T01:16:15Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 4, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "28499": "2021-02-28T01:16:15Z" }, "v8worker_events_lost": 0 }, "function_id": 852536931, "function_name": "n1ql", "gocb_creds_request_counter": 100, "internal_vb_distribution_stats": { "worker_n1ql_0": "[0-1023]" }, "latency_percentile_stats": { "100": 191300, "50": 3100, "80": 4400, "90": 6100, "95": 8100, "99": 15400 }, "lcb_creds_request_counter": 80739, "lcb_exception_stats": { "12009": 4, "209": 4 }, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.105.247:8096", "start_vb": 0, "vb_count": 1024 } ], "vb_distribution_stats_from_metadata": { "172.23.105.247:8096": { "worker_n1ql_0": "[0-1023]" } }, "worker_pids": { "worker_n1ql_0": 28499 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 3, "agg_messages_sent_to_worker": 48240, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 201600, "dcp_mutation_sent_to_worker": 40320, "dcp_snapshot": 122279, "dcp_stream_req_counter": 2698, "dcp_stream_req_err_counter": 319, "dcp_streamend": 1160, "dcp_streamreq": 2379, "dcp_system_event": 10240, "execution_stats": 940, "failure_stats": 940, "latency_stats": 940, "lcb_exception_stats": 940, "log_level": 1, "num_processed_events": 40320, "processed_events_size": 36102328, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 40320, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 40320, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 48236, "no_op_counter": 0, "num_processed_events": 40320, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 40320, "processed_events_size": 36102328, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "28238": "2021-02-28T01:16:15Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "28238": "2021-02-28T01:16:15Z" }, "v8worker_events_lost": 0 }, "function_id": 553897986, "function_name": "bucket_op", "gocb_creds_request_counter": 100, "internal_vb_distribution_stats": { "worker_bucket_op_0": "[0-1023]" }, "latency_percentile_stats": { "100": 28000, "50": 800, "80": 1200, "90": 1600, "95": 2100, "99": 4100 }, "lcb_creds_request_counter": 80739, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.105.247:8096", "start_vb": 0, "vb_count": 1024 } ], "vb_distribution_stats_from_metadata": { "172.23.105.247:8096": { "worker_bucket_op_0": "[0-1023]" } }, "worker_pids": { "worker_bucket_op_0": 28238 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 6, "agg_messages_sent_to_worker": 51254, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 215856, "dcp_mutation_sent_to_worker": 43294, "dcp_snapshot": 122427, "dcp_stream_req_counter": 3241, "dcp_stream_req_err_counter": 852, "dcp_streamend": 1165, "dcp_streamreq": 2389, "dcp_system_event": 10240, "execution_stats": 919, "failure_stats": 919, "latency_stats": 919, "lcb_exception_stats": 919, "log_level": 1, "num_processed_events": 43294, "processed_events_size": 38764968, "thr_count": 1, "thr_map": 1, "timer_events": 40320, "timer_responses_received": 40320, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 43294, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 43294, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 2974, "lcb_retry_failure": 0, "messages_parsed": 51250, "no_op_counter": 0, "num_processed_events": 43294, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 40320, "processed_events_size": 38764968, "timer_callback_failure": 0, "timer_callback_success": 40320, "timer_cancel_counter": 0, "timer_create_counter": 40320, "timer_create_failure": 0, "timer_msg_counter": 40320, "timer_responses_sent": 0, "timestamp": { "28280": "2021-02-28T01:16:15Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "28280": "2021-02-28T01:16:15Z" }, "v8worker_events_lost": 0 }, "function_id": 1701896260, "function_name": "timers", "gocb_creds_request_counter": 100, "internal_vb_distribution_stats": { "worker_timers_0": "[0-1023]" }, "latency_percentile_stats": { "100": 211100, "50": 1700, "80": 2500, "90": 3200, "95": 4500, "99": 10100 }, "lcb_creds_request_counter": 80739, "lcb_exception_stats": {}, "metastore_stats": { "metastore_delete_err": 0, "metastore_deletes": 0, "metastore_not_found": 0, "metastore_scan": 0, "metastore_scan_due": 0, "metastore_scan_err": 0, "metastore_set": 0, "metastore_set_err": 0 }, "planner_stats": [ { "host_name": "172.23.105.247:8096", "start_vb": 0, "vb_count": 1024 } ], "vb_distribution_stats_from_metadata": { "172.23.105.247:8096": { "worker_timers_0": "[0-1023]" } }, "worker_pids": { "worker_timers_0": 28280 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 1, "agg_messages_sent_to_worker": 48054, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 150080, "dcp_mutation_sent_to_worker": 40320, "dcp_mutation_suppressed_counter": 69440, "dcp_snapshot": 122167, "dcp_stream_req_counter": 2432, "dcp_stream_req_err_counter": 23, "dcp_streamend": 1186, "dcp_streamreq": 2409, "dcp_system_event": 10240, "execution_stats": 898, "failure_stats": 898, "latency_stats": 898, "lcb_exception_stats": 898, "log_level": 1, "num_processed_events": 40320, "processed_events_size": 36106772, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 40320, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 40320, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 48050, "no_op_counter": 0, "num_processed_events": 40320, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 40320, "processed_events_size": 36106772, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "28372": "2021-02-28T01:16:15Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "28372": "2021-02-28T01:16:15Z" }, "v8worker_events_lost": 0 }, "function_id": 362645722, "function_name": "sbm", "gocb_creds_request_counter": 100, "internal_vb_distribution_stats": { "worker_sbm_0": "[0-1023]" }, "latency_percentile_stats": { "100": 31700, "50": 900, "80": 1300, "90": 1700, "95": 2200, "99": 4300 }, "lcb_creds_request_counter": 80739, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.105.247:8096", "start_vb": 0, "vb_count": 1024 } ], "vb_distribution_stats_from_metadata": { "172.23.105.247:8096": { "worker_sbm_0": "[0-1023]" } }, "worker_pids": { "worker_sbm_0": 28372 } } ] [2021-02-27 17:16:21,565] - [eventing_base] [140542730036992] - INFO - Stats for Node 172.23.105.247 is [ { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 3, "agg_messages_sent_to_worker": 48240, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 201600, "dcp_mutation_sent_to_worker": 40320, "dcp_snapshot": 122279, "dcp_stream_req_counter": 2698, "dcp_stream_req_err_counter": 319, "dcp_streamend": 1160, "dcp_streamreq": 2379, "dcp_system_event": 10240, "execution_stats": 940, "failure_stats": 940, "latency_stats": 940, "lcb_exception_stats": 940, "log_level": 1, "num_processed_events": 40320, "processed_events_size": 36102328, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 40320, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 40320, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 48236, "no_op_counter": 0, "num_processed_events": 40320, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 40320, "processed_events_size": 36102328, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "28238": "2021-02-28T01:16:15Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "28238": "2021-02-28T01:16:15Z" }, "v8worker_events_lost": 0 }, "function_id": 553897986, "function_name": "bucket_op", "gocb_creds_request_counter": 100, "internal_vb_distribution_stats": { "worker_bucket_op_0": "[0-1023]" }, "latency_percentile_stats": { "100": 28000, "50": 800, "80": 1200, "90": 1600, "95": 2100, "99": 4100 }, "lcb_creds_request_counter": 80739, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.105.247:8096", "start_vb": 0, "vb_count": 1024 } ], "vb_distribution_stats_from_metadata": { "172.23.105.247:8096": { "worker_bucket_op_0": "[0-1023]" } }, "worker_pids": { "worker_bucket_op_0": 28238 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 6, "agg_messages_sent_to_worker": 51254, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 215856, "dcp_mutation_sent_to_worker": 43294, "dcp_snapshot": 122427, "dcp_stream_req_counter": 3241, "dcp_stream_req_err_counter": 852, "dcp_streamend": 1165, "dcp_streamreq": 2389, "dcp_system_event": 10240, "execution_stats": 919, "failure_stats": 919, "latency_stats": 919, "lcb_exception_stats": 919, "log_level": 1, "num_processed_events": 43294, "processed_events_size": 38764968, "thr_count": 1, "thr_map": 1, "timer_events": 40320, "timer_responses_received": 40320, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 43294, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 43294, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 2974, "lcb_retry_failure": 0, "messages_parsed": 51250, "no_op_counter": 0, "num_processed_events": 43294, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 40320, "processed_events_size": 38764968, "timer_callback_failure": 0, "timer_callback_success": 40320, "timer_cancel_counter": 0, "timer_create_counter": 40320, "timer_create_failure": 0, "timer_msg_counter": 40320, "timer_responses_sent": 0, "timestamp": { "28280": "2021-02-28T01:16:15Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "28280": "2021-02-28T01:16:15Z" }, "v8worker_events_lost": 0 }, "function_id": 1701896260, "function_name": "timers", "gocb_creds_request_counter": 100, "internal_vb_distribution_stats": { "worker_timers_0": "[0-1023]" }, "latency_percentile_stats": { "100": 211100, "50": 1700, "80": 2500, "90": 3200, "95": 4500, "99": 10100 }, "lcb_creds_request_counter": 80739, "lcb_exception_stats": {}, "metastore_stats": { "metastore_delete_err": 0, "metastore_deletes": 0, "metastore_not_found": 0, "metastore_scan": 0, "metastore_scan_due": 0, "metastore_scan_err": 0, "metastore_set": 0, "metastore_set_err": 0 }, "planner_stats": [ { "host_name": "172.23.105.247:8096", "start_vb": 0, "vb_count": 1024 } ], "vb_distribution_stats_from_metadata": { "172.23.105.247:8096": { "worker_timers_0": "[0-1023]" } }, "worker_pids": { "worker_timers_0": 28280 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 1, "agg_messages_sent_to_worker": 48054, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 150080, "dcp_mutation_sent_to_worker": 40320, "dcp_mutation_suppressed_counter": 69440, "dcp_snapshot": 122167, "dcp_stream_req_counter": 2432, "dcp_stream_req_err_counter": 23, "dcp_streamend": 1186, "dcp_streamreq": 2409, "dcp_system_event": 10240, "execution_stats": 898, "failure_stats": 898, "latency_stats": 898, "lcb_exception_stats": 898, "log_level": 1, "num_processed_events": 40320, "processed_events_size": 36106772, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 40320, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 40320, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 48050, "no_op_counter": 0, "num_processed_events": 40320, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 40320, "processed_events_size": 36106772, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "28372": "2021-02-28T01:16:15Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "28372": "2021-02-28T01:16:15Z" }, "v8worker_events_lost": 0 }, "function_id": 362645722, "function_name": "sbm", "gocb_creds_request_counter": 100, "internal_vb_distribution_stats": { "worker_sbm_0": "[0-1023]" }, "latency_percentile_stats": { "100": 31700, "50": 900, "80": 1300, "90": 1700, "95": 2200, "99": 4300 }, "lcb_creds_request_counter": 80739, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.105.247:8096", "start_vb": 0, "vb_count": 1024 } ], "vb_distribution_stats_from_metadata": { "172.23.105.247:8096": { "worker_sbm_0": "[0-1023]" } }, "worker_pids": { "worker_sbm_0": 28372 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 4, "agg_messages_sent_to_worker": 47920, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 201600, "dcp_mutation_sent_to_worker": 40320, "dcp_snapshot": 122190, "dcp_stream_req_counter": 2453, "dcp_stream_req_err_counter": 22, "dcp_streamend": 1163, "dcp_streamreq": 2431, "dcp_system_event": 10240, "execution_stats": 876, "failure_stats": 876, "latency_stats": 876, "lcb_exception_stats": 876, "log_level": 1, "num_processed_events": 40320, "processed_events_size": 36102328, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 40320, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 40320, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 40320, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 40320, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 47916, "no_op_counter": 0, "num_processed_events": 40320, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 40320, "processed_events_size": 36102328, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "28457": "2021-02-28T01:16:15Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "28457": "2021-02-28T01:16:15Z" }, "v8worker_events_lost": 0 }, "function_id": 953325219, "function_name": "curl", "gocb_creds_request_counter": 100, "internal_vb_distribution_stats": { "worker_curl_0": "[0-1023]" }, "latency_percentile_stats": { "100": 36000, "50": 1700, "80": 2300, "90": 2900, "95": 3600, "99": 7100 }, "lcb_creds_request_counter": 80739, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.105.247:8096", "start_vb": 0, "vb_count": 1024 } ], "vb_distribution_stats_from_metadata": { "172.23.105.247:8096": { "worker_curl_0": "[0-1023]" } }, "worker_pids": { "worker_curl_0": 28457 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 3, "agg_messages_sent_to_worker": 47799, "agg_queue_memory": 0, "agg_queue_memory_cap": 20971520, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 201600, "dcp_mutation_sent_to_worker": 40320, "dcp_snapshot": 114853, "dcp_stream_req_counter": 2461, "dcp_stream_req_err_counter": 21, "dcp_streamend": 1148, "dcp_streamreq": 2440, "dcp_system_event": 10240, "execution_stats": 855, "failure_stats": 855, "latency_stats": 855, "lcb_exception_stats": 855, "log_level": 1, "num_processed_events": 40320, "processed_events_size": 36102328, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 40320, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 40320, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 47795, "no_op_counter": 0, "num_processed_events": 40320, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 4, "on_update_success": 40316, "processed_events_size": 36102328, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "28499": "2021-02-28T01:16:15Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 4, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "28499": "2021-02-28T01:16:15Z" }, "v8worker_events_lost": 0 }, "function_id": 852536931, "function_name": "n1ql", "gocb_creds_request_counter": 100, "internal_vb_distribution_stats": { "worker_n1ql_0": "[0-1023]" }, "latency_percentile_stats": { "100": 191300, "50": 3100, "80": 4400, "90": 6100, "95": 8100, "99": 15400 }, "lcb_creds_request_counter": 80739, "lcb_exception_stats": { "12009": 4, "209": 4 }, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.105.247:8096", "start_vb": 0, "vb_count": 1024 } ], "vb_distribution_stats_from_metadata": { "172.23.105.247:8096": { "worker_n1ql_0": "[0-1023]" } }, "worker_pids": { "worker_n1ql_0": 28499 } } ] [2021-02-27 17:16:22,583] - [eventing_base] [140542730036992] - INFO - Execution stats update_success: 40320 update_failure: 0 delete_success: 0 delete_failure: 0 [2021-02-27 17:16:22,625] - [basetestcase] [140542730036992] - INFO - list of eventing nodes in cluster: [ip:172.23.105.247 port:8091 ssh_username:root] [2021-02-27 17:16:22,672] - [eventing_base] [140542730036992] - INFO - Stats for Node 172.23.105.247 is [ { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 1, "agg_messages_sent_to_worker": 48084, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 150080, "dcp_mutation_sent_to_worker": 40320, "dcp_mutation_suppressed_counter": 69440, "dcp_snapshot": 122167, "dcp_stream_req_counter": 2432, "dcp_stream_req_err_counter": 23, "dcp_streamend": 1186, "dcp_streamreq": 2409, "dcp_system_event": 10240, "execution_stats": 904, "failure_stats": 904, "latency_stats": 904, "lcb_exception_stats": 904, "log_level": 1, "num_processed_events": 40320, "processed_events_size": 36106772, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 40320, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 40320, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 48080, "no_op_counter": 0, "num_processed_events": 40320, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 40320, "processed_events_size": 36106772, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "28372": "2021-02-28T01:16:21Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "28372": "2021-02-28T01:16:21Z" }, "v8worker_events_lost": 0 }, "function_id": 362645722, "function_name": "sbm", "gocb_creds_request_counter": 100, "internal_vb_distribution_stats": { "worker_sbm_0": "[0-1023]" }, "latency_percentile_stats": { "100": 31700, "50": 900, "80": 1300, "90": 1700, "95": 2200, "99": 4300 }, "lcb_creds_request_counter": 80739, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.105.247:8096", "start_vb": 0, "vb_count": 1024 } ], "vb_distribution_stats_from_metadata": { "172.23.105.247:8096": { "worker_sbm_0": "[0-1023]" } }, "worker_pids": { "worker_sbm_0": 28372 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 4, "agg_messages_sent_to_worker": 47955, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 201600, "dcp_mutation_sent_to_worker": 40320, "dcp_snapshot": 122190, "dcp_stream_req_counter": 2453, "dcp_stream_req_err_counter": 22, "dcp_streamend": 1163, "dcp_streamreq": 2431, "dcp_system_event": 10240, "execution_stats": 883, "failure_stats": 883, "latency_stats": 883, "lcb_exception_stats": 883, "log_level": 1, "num_processed_events": 40320, "processed_events_size": 36102328, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 40320, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 40320, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 40320, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 40320, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 47951, "no_op_counter": 0, "num_processed_events": 40320, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 40320, "processed_events_size": 36102328, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "28457": "2021-02-28T01:16:22Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "28457": "2021-02-28T01:16:22Z" }, "v8worker_events_lost": 0 }, "function_id": 953325219, "function_name": "curl", "gocb_creds_request_counter": 100, "internal_vb_distribution_stats": { "worker_curl_0": "[0-1023]" }, "latency_percentile_stats": { "100": 36000, "50": 1700, "80": 2300, "90": 2900, "95": 3600, "99": 7100 }, "lcb_creds_request_counter": 80739, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.105.247:8096", "start_vb": 0, "vb_count": 1024 } ], "vb_distribution_stats_from_metadata": { "172.23.105.247:8096": { "worker_curl_0": "[0-1023]" } }, "worker_pids": { "worker_curl_0": 28457 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 3, "agg_messages_sent_to_worker": 47834, "agg_queue_memory": 0, "agg_queue_memory_cap": 20971520, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 201600, "dcp_mutation_sent_to_worker": 40320, "dcp_snapshot": 114853, "dcp_stream_req_counter": 2461, "dcp_stream_req_err_counter": 21, "dcp_streamend": 1148, "dcp_streamreq": 2440, "dcp_system_event": 10240, "execution_stats": 862, "failure_stats": 862, "latency_stats": 862, "lcb_exception_stats": 862, "log_level": 1, "num_processed_events": 40320, "processed_events_size": 36102328, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 40320, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 40320, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 47830, "no_op_counter": 0, "num_processed_events": 40320, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 4, "on_update_success": 40316, "processed_events_size": 36102328, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "28499": "2021-02-28T01:16:22Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 4, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "28499": "2021-02-28T01:16:22Z" }, "v8worker_events_lost": 0 }, "function_id": 852536931, "function_name": "n1ql", "gocb_creds_request_counter": 100, "internal_vb_distribution_stats": { "worker_n1ql_0": "[0-1023]" }, "latency_percentile_stats": { "100": 191300, "50": 3100, "80": 4400, "90": 6100, "95": 8100, "99": 15400 }, "lcb_creds_request_counter": 80739, "lcb_exception_stats": { "12009": 4, "209": 4 }, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.105.247:8096", "start_vb": 0, "vb_count": 1024 } ], "vb_distribution_stats_from_metadata": { "172.23.105.247:8096": { "worker_n1ql_0": "[0-1023]" } }, "worker_pids": { "worker_n1ql_0": 28499 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 3, "agg_messages_sent_to_worker": 48275, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 201600, "dcp_mutation_sent_to_worker": 40320, "dcp_snapshot": 122279, "dcp_stream_req_counter": 2698, "dcp_stream_req_err_counter": 319, "dcp_streamend": 1160, "dcp_streamreq": 2379, "dcp_system_event": 10240, "execution_stats": 947, "failure_stats": 947, "latency_stats": 947, "lcb_exception_stats": 947, "log_level": 1, "num_processed_events": 40320, "processed_events_size": 36102328, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 40320, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 40320, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 48271, "no_op_counter": 0, "num_processed_events": 40320, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 40320, "processed_events_size": 36102328, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "28238": "2021-02-28T01:16:22Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "28238": "2021-02-28T01:16:22Z" }, "v8worker_events_lost": 0 }, "function_id": 553897986, "function_name": "bucket_op", "gocb_creds_request_counter": 100, "internal_vb_distribution_stats": { "worker_bucket_op_0": "[0-1023]" }, "latency_percentile_stats": { "100": 28000, "50": 800, "80": 1200, "90": 1600, "95": 2100, "99": 4100 }, "lcb_creds_request_counter": 80739, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.105.247:8096", "start_vb": 0, "vb_count": 1024 } ], "vb_distribution_stats_from_metadata": { "172.23.105.247:8096": { "worker_bucket_op_0": "[0-1023]" } }, "worker_pids": { "worker_bucket_op_0": 28238 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 6, "agg_messages_sent_to_worker": 51289, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 215856, "dcp_mutation_sent_to_worker": 43294, "dcp_snapshot": 122427, "dcp_stream_req_counter": 3241, "dcp_stream_req_err_counter": 852, "dcp_streamend": 1165, "dcp_streamreq": 2389, "dcp_system_event": 10240, "execution_stats": 926, "failure_stats": 926, "latency_stats": 926, "lcb_exception_stats": 926, "log_level": 1, "num_processed_events": 43294, "processed_events_size": 38764968, "thr_count": 1, "thr_map": 1, "timer_events": 40320, "timer_responses_received": 40320, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 43294, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 43294, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 2974, "lcb_retry_failure": 0, "messages_parsed": 51285, "no_op_counter": 0, "num_processed_events": 43294, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 40320, "processed_events_size": 38764968, "timer_callback_failure": 0, "timer_callback_success": 40320, "timer_cancel_counter": 0, "timer_create_counter": 40320, "timer_create_failure": 0, "timer_msg_counter": 40320, "timer_responses_sent": 0, "timestamp": { "28280": "2021-02-28T01:16:22Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "28280": "2021-02-28T01:16:22Z" }, "v8worker_events_lost": 0 }, "function_id": 1701896260, "function_name": "timers", "gocb_creds_request_counter": 100, "internal_vb_distribution_stats": { "worker_timers_0": "[0-1023]" }, "latency_percentile_stats": { "100": 211100, "50": 1700, "80": 2500, "90": 3200, "95": 4500, "99": 10100 }, "lcb_creds_request_counter": 80739, "lcb_exception_stats": {}, "metastore_stats": { "metastore_delete_err": 0, "metastore_deletes": 0, "metastore_not_found": 0, "metastore_scan": 0, "metastore_scan_due": 0, "metastore_scan_err": 0, "metastore_set": 0, "metastore_set_err": 0 }, "planner_stats": [ { "host_name": "172.23.105.247:8096", "start_vb": 0, "vb_count": 1024 } ], "vb_distribution_stats_from_metadata": { "172.23.105.247:8096": { "worker_timers_0": "[0-1023]" } }, "worker_pids": { "worker_timers_0": 28280 } } ] [2021-02-27 17:16:22,708] - [basetestcase] [140542730036992] - INFO - list of eventing nodes in cluster: [ip:172.23.105.247 port:8091 ssh_username:root] [2021-02-27 17:16:22,709] - [remote_util] [140542730036992] - INFO - SSH Connecting to 172.23.105.247 with username:root, attempt#1 of 5 [2021-02-27 17:16:22,821] - [remote_util] [140542730036992] - INFO - SSH Connected to 172.23.105.247 as root [2021-02-27 17:16:23,130] - [remote_util] [140542730036992] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-27 17:16:23,491] - [remote_util] [140542730036992] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-27 17:16:23,499] - [rest_client] [140542730036992] - INFO - /diag/eval status on 172.23.105.247:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). [2021-02-27 17:16:23,499] - [remote_util] [140542730036992] - INFO - running command.raw on 172.23.105.247: zgrep "panic" "/opt/couchbase/var/lib/couchbase/logs"/eventing.log* | wc -l [2021-02-27 17:16:24,438] - [remote_util] [140542730036992] - INFO - command executed successfully with root [2021-02-27 17:16:24,438] - [remote_util] [140542730036992] - INFO - running command.raw on 172.23.105.247: ls "/opt/couchbase/var/lib/couchbase/logs"/../crash/| wc -l [2021-02-27 17:16:24,454] - [remote_util] [140542730036992] - INFO - command executed successfully with root [2021-02-27 17:16:24,578] - [basetestcase] [140542730036992] - INFO - ------- Cluster statistics ------- [2021-02-27 17:16:24,579] - [basetestcase] [140542730036992] - INFO - 172.23.104.97:8091 => {'services': ['kv'], 'cpu_utilization': 9.095554420030659, 'mem_free': 3069485056, 'mem_total': 3963314176, 'swap_mem_used': 0, 'swap_mem_total': 0} [2021-02-27 17:16:24,579] - [basetestcase] [140542730036992] - INFO - 172.23.105.238:8091 => {'services': ['kv'], 'cpu_utilization': 3.865717192268566, 'mem_free': 3210051584, 'mem_total': 4201840640, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-02-27 17:16:24,579] - [basetestcase] [140542730036992] - INFO - 172.23.105.240:8091 => {'services': ['kv'], 'cpu_utilization': 4.223918575063613, 'mem_free': 3221856256, 'mem_total': 4201840640, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-02-27 17:16:24,579] - [basetestcase] [140542730036992] - INFO - 172.23.105.247:8091 => {'services': ['eventing'], 'cpu_utilization': 7.2764540097361, 'mem_free': 3080564736, 'mem_total': 4201840640, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-02-27 17:16:24,579] - [basetestcase] [140542730036992] - INFO - 172.23.106.1:8091 => {'services': ['index', 'n1ql'], 'cpu_utilization': 0.9190707173857544, 'mem_free': 3283861504, 'mem_total': 4201840640, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-02-27 17:16:24,579] - [basetestcase] [140542730036992] - INFO - 172.23.107.65:8091 => {'services': ['kv'], 'cpu_utilization': 2.31023102310231, 'mem_free': 3239239680, 'mem_total': 4201840640, 'swap_mem_used': 57409536, 'swap_mem_total': 3758092288} [2021-02-27 17:16:24,579] - [basetestcase] [140542730036992] - INFO - 172.23.121.213:8091 => {'services': ['kv'], 'cpu_utilization': 3.094875697615424, 'mem_free': 3163951104, 'mem_total': 4201684992, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-02-27 17:16:24,579] - [basetestcase] [140542730036992] - INFO - --- End of cluster statistics --- [2021-02-27 17:16:24,581] - [remote_util] [140542730036992] - INFO - SSH Connecting to 172.23.104.97 with username:root, attempt#1 of 5 [2021-02-27 17:16:24,747] - [remote_util] [140542730036992] - INFO - SSH Connected to 172.23.104.97 as root [2021-02-27 17:16:25,399] - [remote_util] [140542730036992] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-27 17:16:25,832] - [remote_util] [140542730036992] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-27 17:16:25,835] - [remote_util] [140542730036992] - INFO - SSH Connecting to 172.23.121.213 with username:root, attempt#1 of 5 [2021-02-27 17:16:25,938] - [remote_util] [140542730036992] - INFO - SSH Connected to 172.23.121.213 as root [2021-02-27 17:16:26,227] - [remote_util] [140542730036992] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-27 17:16:26,531] - [remote_util] [140542730036992] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-27 17:16:26,533] - [remote_util] [140542730036992] - INFO - SSH Connecting to 172.23.105.238 with username:root, attempt#1 of 5 [2021-02-27 17:16:26,643] - [remote_util] [140542730036992] - INFO - SSH Connected to 172.23.105.238 as root [2021-02-27 17:16:26,935] - [remote_util] [140542730036992] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-27 17:16:27,297] - [remote_util] [140542730036992] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-27 17:16:27,299] - [remote_util] [140542730036992] - INFO - SSH Connecting to 172.23.105.247 with username:root, attempt#1 of 5 [2021-02-27 17:16:27,404] - [remote_util] [140542730036992] - INFO - SSH Connected to 172.23.105.247 as root [2021-02-27 17:16:27,703] - [remote_util] [140542730036992] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-27 17:16:28,023] - [remote_util] [140542730036992] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-27 17:16:28,025] - [remote_util] [140542730036992] - INFO - SSH Connecting to 172.23.106.1 with username:root, attempt#1 of 5 [2021-02-27 17:16:28,130] - [remote_util] [140542730036992] - INFO - SSH Connected to 172.23.106.1 as root [2021-02-27 17:16:28,400] - [remote_util] [140542730036992] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-27 17:16:28,707] - [remote_util] [140542730036992] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-27 17:16:28,709] - [remote_util] [140542730036992] - INFO - SSH Connecting to 172.23.107.65 with username:root, attempt#1 of 5 [2021-02-27 17:16:28,811] - [remote_util] [140542730036992] - INFO - SSH Connected to 172.23.107.65 as root [2021-02-27 17:16:29,095] - [remote_util] [140542730036992] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-27 17:16:29,391] - [remote_util] [140542730036992] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-27 17:16:29,394] - [remote_util] [140542730036992] - INFO - SSH Connecting to 172.23.105.240 with username:root, attempt#1 of 5 [2021-02-27 17:16:29,509] - [remote_util] [140542730036992] - INFO - SSH Connected to 172.23.105.240 as root [2021-02-27 17:16:29,810] - [remote_util] [140542730036992] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-27 17:16:30,134] - [remote_util] [140542730036992] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-27 17:16:30,136] - [remote_util] [140542730036992] - INFO - SSH Connecting to 172.23.105.249 with username:root, attempt#1 of 5 [2021-02-27 17:16:30,244] - [remote_util] [140542730036992] - INFO - SSH Connected to 172.23.105.249 as root [2021-02-27 17:16:30,567] - [remote_util] [140542730036992] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-27 17:16:30,880] - [remote_util] [140542730036992] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-27 17:16:48,927] - [basetestcase] [140542730036992] - INFO - ============== basetestcase cleanup was started for test #2 test_eventing_rebalance_with_multiple_kv_nodes ============== [2021-02-27 17:16:49,295] - [bucket_helper] [140542730036992] - INFO - deleting existing buckets ['dst_bucket', 'metadata', 'src_bucket'] on 172.23.104.97 [2021-02-27 17:16:50,638] - [bucket_helper] [140542730036992] - INFO - waiting for bucket deletion to complete.... [2021-02-27 17:16:50,681] - [rest_client] [140542730036992] - INFO - node 172.23.104.97 existing buckets : ['metadata', 'src_bucket'] [2021-02-27 17:16:50,682] - [bucket_helper] [140542730036992] - INFO - deleted bucket : dst_bucket from 172.23.104.97 [2021-02-27 17:16:51,198] - [bucket_helper] [140542730036992] - INFO - waiting for bucket deletion to complete.... [2021-02-27 17:16:51,221] - [rest_client] [140542730036992] - INFO - node 172.23.104.97 existing buckets : ['src_bucket'] [2021-02-27 17:16:51,222] - [bucket_helper] [140542730036992] - INFO - deleted bucket : metadata from 172.23.104.97 [2021-02-27 17:16:52,524] - [bucket_helper] [140542730036992] - INFO - waiting for bucket deletion to complete.... [2021-02-27 17:16:52,530] - [rest_client] [140542730036992] - INFO - node 172.23.104.97 existing buckets : [] [2021-02-27 17:16:52,530] - [bucket_helper] [140542730036992] - INFO - deleted bucket : src_bucket from 172.23.104.97 [2021-02-27 17:16:52,539] - [bucket_helper] [140542730036992] - INFO - Could not find any buckets for node 172.23.121.213, nothing to delete [2021-02-27 17:16:52,547] - [bucket_helper] [140542730036992] - INFO - Could not find any buckets for node 172.23.105.238, nothing to delete [2021-02-27 17:16:52,560] - [bucket_helper] [140542730036992] - INFO - Could not find any buckets for node 172.23.105.247, nothing to delete [2021-02-27 17:16:52,566] - [bucket_helper] [140542730036992] - INFO - Could not find any buckets for node 172.23.106.1, nothing to delete [2021-02-27 17:16:52,573] - [bucket_helper] [140542730036992] - INFO - Could not find any buckets for node 172.23.107.65, nothing to delete [2021-02-27 17:16:52,582] - [bucket_helper] [140542730036992] - INFO - Could not find any buckets for node 172.23.105.240, nothing to delete [2021-02-27 17:16:52,587] - [bucket_helper] [140542730036992] - INFO - Could not find any buckets for node 172.23.105.249, nothing to delete [2021-02-27 17:16:52,600] - [rest_client] [140542730036992] - INFO - -->is_ns_server_running? [2021-02-27 17:16:52,627] - [cluster_helper] [140542730036992] - INFO - rebalancing all nodes in order to remove nodes [2021-02-27 17:16:52,631] - [rest_client] [140542730036992] - INFO - rebalance params : {'knownNodes': 'ns_1@172.23.104.97,ns_1@172.23.105.238,ns_1@172.23.105.240,ns_1@172.23.105.247,ns_1@172.23.106.1,ns_1@172.23.107.65,ns_1@172.23.121.213', 'ejectedNodes': 'ns_1@172.23.105.238,ns_1@172.23.105.240,ns_1@172.23.105.247,ns_1@172.23.106.1,ns_1@172.23.107.65,ns_1@172.23.121.213', 'user': 'Administrator', 'password': 'password'} [2021-02-27 17:16:52,664] - [rest_client] [140542730036992] - INFO - rebalance operation started [2021-02-27 17:16:52,668] - [rest_client] [140542730036992] - INFO - rebalance percentage : 0.00 % [2021-02-27 17:17:02,682] - [rest_client] [140542730036992] - INFO - rebalance percentage : 78.00 % [2021-02-27 17:17:12,697] - [rest_client] [140542730036992] - INFO - rebalance percentage : 92.00 % [2021-02-27 17:17:32,725] - [rest_client] [140542730036992] - INFO - rebalance progress took 40.06 seconds [2021-02-27 17:17:32,726] - [rest_client] [140542730036992] - INFO - sleep for 10 seconds after rebalance... [2021-02-27 17:17:42,769] - [cluster_helper] [140542730036992] - INFO - removed all the nodes from cluster associated with ip:172.23.104.97 port:8091 ssh_username:root ? [('ns_1@172.23.105.238', 8091), ('ns_1@172.23.105.240', 8091), ('ns_1@172.23.105.247', 8091), ('ns_1@172.23.106.1', 8091), ('ns_1@172.23.107.65', 8091), ('ns_1@172.23.121.213', 8091)] [2021-02-27 17:17:42,781] - [cluster_helper] [140542730036992] - INFO - waiting for ns_server @ 172.23.104.97:8091 [2021-02-27 17:17:42,781] - [rest_client] [140542730036992] - INFO - -->is_ns_server_running? [2021-02-27 17:17:42,789] - [cluster_helper] [140542730036992] - INFO - ns_server @ 172.23.104.97:8091 is running [2021-02-27 17:17:42,792] - [cluster_helper] [140542730036992] - INFO - waiting for ns_server @ 172.23.121.213:8091 [2021-02-27 17:17:42,793] - [rest_client] [140542730036992] - INFO - -->is_ns_server_running? [2021-02-27 17:17:42,795] - [cluster_helper] [140542730036992] - INFO - ns_server @ 172.23.121.213:8091 is running [2021-02-27 17:17:42,799] - [cluster_helper] [140542730036992] - INFO - waiting for ns_server @ 172.23.105.238:8091 [2021-02-27 17:17:42,799] - [rest_client] [140542730036992] - INFO - -->is_ns_server_running? [2021-02-27 17:17:42,802] - [cluster_helper] [140542730036992] - INFO - ns_server @ 172.23.105.238:8091 is running [2021-02-27 17:17:42,805] - [cluster_helper] [140542730036992] - INFO - waiting for ns_server @ 172.23.105.247:8091 [2021-02-27 17:17:42,805] - [rest_client] [140542730036992] - INFO - -->is_ns_server_running? [2021-02-27 17:17:42,809] - [cluster_helper] [140542730036992] - INFO - ns_server @ 172.23.105.247:8091 is running [2021-02-27 17:17:42,812] - [cluster_helper] [140542730036992] - INFO - waiting for ns_server @ 172.23.106.1:8091 [2021-02-27 17:17:42,812] - [rest_client] [140542730036992] - INFO - -->is_ns_server_running? [2021-02-27 17:17:42,815] - [cluster_helper] [140542730036992] - INFO - ns_server @ 172.23.106.1:8091 is running [2021-02-27 17:17:42,818] - [cluster_helper] [140542730036992] - INFO - waiting for ns_server @ 172.23.107.65:8091 [2021-02-27 17:17:42,818] - [rest_client] [140542730036992] - INFO - -->is_ns_server_running? [2021-02-27 17:17:42,821] - [cluster_helper] [140542730036992] - INFO - ns_server @ 172.23.107.65:8091 is running [2021-02-27 17:17:42,825] - [cluster_helper] [140542730036992] - INFO - waiting for ns_server @ 172.23.105.240:8091 [2021-02-27 17:17:42,825] - [rest_client] [140542730036992] - INFO - -->is_ns_server_running? [2021-02-27 17:17:42,828] - [cluster_helper] [140542730036992] - INFO - ns_server @ 172.23.105.240:8091 is running [2021-02-27 17:17:42,832] - [cluster_helper] [140542730036992] - INFO - waiting for ns_server @ 172.23.105.249:8091 [2021-02-27 17:17:42,832] - [rest_client] [140542730036992] - INFO - -->is_ns_server_running? [2021-02-27 17:17:42,835] - [cluster_helper] [140542730036992] - INFO - ns_server @ 172.23.105.249:8091 is running [2021-02-27 17:17:42,835] - [basetestcase] [140542730036992] - INFO - ============== basetestcase cleanup was finished for test #2 test_eventing_rebalance_with_multiple_kv_nodes ============== [2021-02-27 17:17:42,835] - [basetestcase] [140542730036992] - INFO - closing all ssh connections [2021-02-27 17:17:42,836] - [basetestcase] [140542730036992] - INFO - closing all memcached connections [2021-02-27 17:17:42,837] - [remote_util] [140542730036992] - INFO - SSH Connecting to 172.23.104.97 with username:root, attempt#1 of 5 [2021-02-27 17:17:43,002] - [remote_util] [140542730036992] - INFO - SSH Connected to 172.23.104.97 as root [2021-02-27 17:17:43,664] - [remote_util] [140542730036992] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-27 17:17:44,083] - [remote_util] [140542730036992] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-27 17:17:44,101] - [remote_util] [140542730036992] - INFO - SSH Connecting to 172.23.104.97 with username:root, attempt#1 of 5 [2021-02-27 17:17:44,263] - [remote_util] [140542730036992] - INFO - SSH Connected to 172.23.104.97 as root [2021-02-27 17:17:44,858] - [remote_util] [140542730036992] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-27 17:17:45,288] - [remote_util] [140542730036992] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-27 17:17:45,330] - [remote_util] [140542730036992] - INFO - 172.23.104.97 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-27 17:17:45,443] - [remote_util] [140542730036992] - INFO - Running systemd command on this server [2021-02-27 17:17:45,443] - [remote_util] [140542730036992] - INFO - running command.raw on 172.23.104.97: systemctl stop couchbase-server.service [2021-02-27 17:17:47,932] - [remote_util] [140542730036992] - INFO - command executed successfully with root [2021-02-27 17:17:47,932] - [basetestcase] [140542730036992] - INFO - Couchbase stopped [2021-02-27 17:17:47,933] - [basetestcase] [140542730036992] - INFO - sleep for 5 secs. ... [2021-02-27 17:17:52,939] - [remote_util] [140542730036992] - INFO - running command.raw on 172.23.104.97: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-27 17:17:52,974] - [remote_util] [140542730036992] - INFO - command executed successfully with root [2021-02-27 17:17:52,975] - [remote_util] [140542730036992] - INFO - running command.raw on 172.23.104.97: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-27 17:17:53,058] - [remote_util] [140542730036992] - INFO - command executed successfully with root [2021-02-27 17:17:53,060] - [remote_util] [140542730036992] - INFO - SSH Connecting to 172.23.104.97 with username:root, attempt#1 of 5 [2021-02-27 17:17:53,226] - [remote_util] [140542730036992] - INFO - SSH Connected to 172.23.104.97 as root [2021-02-27 17:17:53,848] - [remote_util] [140542730036992] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-27 17:17:54,270] - [remote_util] [140542730036992] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-27 17:17:54,348] - [remote_util] [140542730036992] - INFO - 172.23.104.97 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-27 17:17:54,349] - [remote_util] [140542730036992] - INFO - Checking for process beam.smp on linux [2021-02-27 17:17:54,440] - [remote_util] [140542730036992] - INFO - Starting couchbase server [2021-02-27 17:17:54,594] - [remote_util] [140542730036992] - INFO - Running systemd command on this server [2021-02-27 17:17:54,594] - [remote_util] [140542730036992] - INFO - running command.raw on 172.23.104.97: systemctl start couchbase-server.service [2021-02-27 17:17:54,641] - [remote_util] [140542730036992] - INFO - command executed successfully with root [2021-02-27 17:17:54,641] - [remote_util] [140542730036992] - INFO - 172.23.104.97:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-27 17:17:59,647] - [remote_util] [140542730036992] - INFO - running command.raw on 172.23.104.97: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-27 17:17:59,696] - [remote_util] [140542730036992] - INFO - command executed successfully with root [2021-02-27 17:17:59,697] - [remote_util] [140542730036992] - INFO - Couchbase server status: [' Process: 18159 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-27 17:17:59,697] - [remote_util] [140542730036992] - INFO - Checking for process beam.smp on linux [2021-02-27 17:17:59,790] - [remote_util] [140542730036992] - INFO - process beam.smp is running on 172.23.104.97: with pid 18338 [2021-02-27 17:17:59,790] - [basetestcase] [140542730036992] - INFO - Couchbase started [2021-02-27 17:17:59,791] - [remote_util] [140542730036992] - INFO - SSH Connecting to 172.23.121.213 with username:root, attempt#1 of 5 [2021-02-27 17:17:59,906] - [remote_util] [140542730036992] - INFO - SSH Connected to 172.23.121.213 as root [2021-02-27 17:18:00,184] - [remote_util] [140542730036992] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-27 17:18:00,489] - [remote_util] [140542730036992] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-27 17:18:00,499] - [remote_util] [140542730036992] - INFO - SSH Connecting to 172.23.121.213 with username:root, attempt#1 of 5 [2021-02-27 17:18:00,603] - [remote_util] [140542730036992] - INFO - SSH Connected to 172.23.121.213 as root [2021-02-27 17:18:00,882] - [remote_util] [140542730036992] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-27 17:18:01,188] - [remote_util] [140542730036992] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-27 17:18:01,209] - [remote_util] [140542730036992] - INFO - 172.23.121.213 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-27 17:18:01,267] - [remote_util] [140542730036992] - INFO - Running systemd command on this server [2021-02-27 17:18:01,267] - [remote_util] [140542730036992] - INFO - running command.raw on 172.23.121.213: systemctl stop couchbase-server.service [2021-02-27 17:18:02,914] - [remote_util] [140542730036992] - INFO - command executed successfully with root [2021-02-27 17:18:02,914] - [basetestcase] [140542730036992] - INFO - Couchbase stopped [2021-02-27 17:18:02,915] - [basetestcase] [140542730036992] - INFO - sleep for 5 secs. ... [2021-02-27 17:18:07,920] - [remote_util] [140542730036992] - INFO - running command.raw on 172.23.121.213: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-27 17:18:07,935] - [remote_util] [140542730036992] - INFO - command executed successfully with root [2021-02-27 17:18:07,936] - [remote_util] [140542730036992] - INFO - running command.raw on 172.23.121.213: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-27 17:18:07,990] - [remote_util] [140542730036992] - INFO - command executed successfully with root [2021-02-27 17:18:07,993] - [remote_util] [140542730036992] - INFO - SSH Connecting to 172.23.121.213 with username:root, attempt#1 of 5 [2021-02-27 17:18:08,100] - [remote_util] [140542730036992] - INFO - SSH Connected to 172.23.121.213 as root [2021-02-27 17:18:08,371] - [remote_util] [140542730036992] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-27 17:18:08,689] - [remote_util] [140542730036992] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-27 17:18:08,713] - [remote_util] [140542730036992] - INFO - 172.23.121.213 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-27 17:18:08,713] - [remote_util] [140542730036992] - INFO - Checking for process beam.smp on linux [2021-02-27 17:18:08,756] - [remote_util] [140542730036992] - INFO - Starting couchbase server [2021-02-27 17:18:08,862] - [remote_util] [140542730036992] - INFO - Running systemd command on this server [2021-02-27 17:18:08,862] - [remote_util] [140542730036992] - INFO - running command.raw on 172.23.121.213: systemctl start couchbase-server.service [2021-02-27 17:18:08,888] - [remote_util] [140542730036992] - INFO - command executed successfully with root [2021-02-27 17:18:08,889] - [remote_util] [140542730036992] - INFO - 172.23.121.213:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-27 17:18:13,891] - [remote_util] [140542730036992] - INFO - running command.raw on 172.23.121.213: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-27 17:18:13,915] - [remote_util] [140542730036992] - INFO - command executed successfully with root [2021-02-27 17:18:13,915] - [remote_util] [140542730036992] - INFO - Couchbase server status: [' Process: 29208 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-27 17:18:13,915] - [remote_util] [140542730036992] - INFO - Checking for process beam.smp on linux [2021-02-27 17:18:13,984] - [remote_util] [140542730036992] - INFO - process beam.smp is running on 172.23.121.213: with pid 29351 [2021-02-27 17:18:13,985] - [basetestcase] [140542730036992] - INFO - Couchbase started [2021-02-27 17:18:13,986] - [remote_util] [140542730036992] - INFO - SSH Connecting to 172.23.105.238 with username:root, attempt#1 of 5 [2021-02-27 17:18:14,093] - [remote_util] [140542730036992] - INFO - SSH Connected to 172.23.105.238 as root [2021-02-27 17:18:14,363] - [remote_util] [140542730036992] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-27 17:18:14,671] - [remote_util] [140542730036992] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-27 17:18:14,679] - [remote_util] [140542730036992] - INFO - SSH Connecting to 172.23.105.238 with username:root, attempt#1 of 5 [2021-02-27 17:18:14,783] - [remote_util] [140542730036992] - INFO - SSH Connected to 172.23.105.238 as root [2021-02-27 17:18:15,061] - [remote_util] [140542730036992] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-27 17:18:15,373] - [remote_util] [140542730036992] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-27 17:18:15,395] - [remote_util] [140542730036992] - INFO - 172.23.105.238 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-27 17:18:15,454] - [remote_util] [140542730036992] - INFO - Running systemd command on this server [2021-02-27 17:18:15,454] - [remote_util] [140542730036992] - INFO - running command.raw on 172.23.105.238: systemctl stop couchbase-server.service [2021-02-27 17:18:17,239] - [remote_util] [140542730036992] - INFO - command executed successfully with root [2021-02-27 17:18:17,239] - [basetestcase] [140542730036992] - INFO - Couchbase stopped [2021-02-27 17:18:17,239] - [basetestcase] [140542730036992] - INFO - sleep for 5 secs. ... [2021-02-27 17:18:22,245] - [remote_util] [140542730036992] - INFO - running command.raw on 172.23.105.238: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-27 17:18:22,263] - [remote_util] [140542730036992] - INFO - command executed successfully with root [2021-02-27 17:18:22,263] - [remote_util] [140542730036992] - INFO - running command.raw on 172.23.105.238: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-27 17:18:22,319] - [remote_util] [140542730036992] - INFO - command executed successfully with root [2021-02-27 17:18:22,321] - [remote_util] [140542730036992] - INFO - SSH Connecting to 172.23.105.238 with username:root, attempt#1 of 5 [2021-02-27 17:18:22,427] - [remote_util] [140542730036992] - INFO - SSH Connected to 172.23.105.238 as root [2021-02-27 17:18:22,707] - [remote_util] [140542730036992] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-27 17:18:23,014] - [remote_util] [140542730036992] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-27 17:18:23,073] - [remote_util] [140542730036992] - INFO - 172.23.105.238 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-27 17:18:23,074] - [remote_util] [140542730036992] - INFO - Checking for process beam.smp on linux [2021-02-27 17:18:23,115] - [remote_util] [140542730036992] - INFO - Starting couchbase server [2021-02-27 17:18:23,213] - [remote_util] [140542730036992] - INFO - Running systemd command on this server [2021-02-27 17:18:23,213] - [remote_util] [140542730036992] - INFO - running command.raw on 172.23.105.238: systemctl start couchbase-server.service [2021-02-27 17:18:23,239] - [remote_util] [140542730036992] - INFO - command executed successfully with root [2021-02-27 17:18:23,239] - [remote_util] [140542730036992] - INFO - 172.23.105.238:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-27 17:18:28,244] - [remote_util] [140542730036992] - INFO - running command.raw on 172.23.105.238: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-27 17:18:28,271] - [remote_util] [140542730036992] - INFO - command executed successfully with root [2021-02-27 17:18:28,271] - [remote_util] [140542730036992] - INFO - Couchbase server status: [' Process: 122660 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-27 17:18:28,271] - [remote_util] [140542730036992] - INFO - Checking for process beam.smp on linux [2021-02-27 17:18:28,343] - [remote_util] [140542730036992] - INFO - process beam.smp is running on 172.23.105.238: with pid 122804 [2021-02-27 17:18:28,344] - [basetestcase] [140542730036992] - INFO - Couchbase started [2021-02-27 17:18:28,346] - [remote_util] [140542730036992] - INFO - SSH Connecting to 172.23.105.247 with username:root, attempt#1 of 5 [2021-02-27 17:18:28,455] - [remote_util] [140542730036992] - INFO - SSH Connected to 172.23.105.247 as root [2021-02-27 17:18:28,750] - [remote_util] [140542730036992] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-27 17:18:29,061] - [remote_util] [140542730036992] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-27 17:18:29,070] - [remote_util] [140542730036992] - INFO - SSH Connecting to 172.23.105.247 with username:root, attempt#1 of 5 [2021-02-27 17:18:29,174] - [remote_util] [140542730036992] - INFO - SSH Connected to 172.23.105.247 as root [2021-02-27 17:18:29,457] - [remote_util] [140542730036992] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-27 17:18:29,768] - [remote_util] [140542730036992] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-27 17:18:29,829] - [remote_util] [140542730036992] - INFO - 172.23.105.247 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-27 17:18:29,936] - [remote_util] [140542730036992] - INFO - Running systemd command on this server [2021-02-27 17:18:29,936] - [remote_util] [140542730036992] - INFO - running command.raw on 172.23.105.247: systemctl stop couchbase-server.service [2021-02-27 17:18:31,797] - [remote_util] [140542730036992] - INFO - command executed successfully with root [2021-02-27 17:18:31,798] - [basetestcase] [140542730036992] - INFO - Couchbase stopped [2021-02-27 17:18:31,798] - [basetestcase] [140542730036992] - INFO - sleep for 5 secs. ... [2021-02-27 17:18:36,803] - [remote_util] [140542730036992] - INFO - running command.raw on 172.23.105.247: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-27 17:18:36,833] - [remote_util] [140542730036992] - INFO - command executed successfully with root [2021-02-27 17:18:36,833] - [remote_util] [140542730036992] - INFO - running command.raw on 172.23.105.247: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-27 17:18:36,887] - [remote_util] [140542730036992] - INFO - command executed successfully with root [2021-02-27 17:18:36,889] - [remote_util] [140542730036992] - INFO - SSH Connecting to 172.23.105.247 with username:root, attempt#1 of 5 [2021-02-27 17:18:37,001] - [remote_util] [140542730036992] - INFO - SSH Connected to 172.23.105.247 as root [2021-02-27 17:18:37,283] - [remote_util] [140542730036992] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-27 17:18:37,589] - [remote_util] [140542730036992] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-27 17:18:37,648] - [remote_util] [140542730036992] - INFO - 172.23.105.247 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-27 17:18:37,649] - [remote_util] [140542730036992] - INFO - Checking for process beam.smp on linux [2021-02-27 17:18:37,686] - [remote_util] [140542730036992] - INFO - Starting couchbase server [2021-02-27 17:18:37,786] - [remote_util] [140542730036992] - INFO - Running systemd command on this server [2021-02-27 17:18:37,786] - [remote_util] [140542730036992] - INFO - running command.raw on 172.23.105.247: systemctl start couchbase-server.service [2021-02-27 17:18:37,814] - [remote_util] [140542730036992] - INFO - command executed successfully with root [2021-02-27 17:18:37,814] - [remote_util] [140542730036992] - INFO - 172.23.105.247:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-27 17:18:42,818] - [remote_util] [140542730036992] - INFO - running command.raw on 172.23.105.247: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-27 17:18:42,844] - [remote_util] [140542730036992] - INFO - command executed successfully with root [2021-02-27 17:18:42,844] - [remote_util] [140542730036992] - INFO - Couchbase server status: [' Process: 72392 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-27 17:18:42,845] - [remote_util] [140542730036992] - INFO - Checking for process beam.smp on linux [2021-02-27 17:18:42,913] - [remote_util] [140542730036992] - INFO - process beam.smp is running on 172.23.105.247: with pid 72536 [2021-02-27 17:18:42,913] - [basetestcase] [140542730036992] - INFO - Couchbase started [2021-02-27 17:18:42,915] - [remote_util] [140542730036992] - INFO - SSH Connecting to 172.23.106.1 with username:root, attempt#1 of 5 [2021-02-27 17:18:43,020] - [remote_util] [140542730036992] - INFO - SSH Connected to 172.23.106.1 as root [2021-02-27 17:18:43,289] - [remote_util] [140542730036992] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-27 17:18:43,597] - [remote_util] [140542730036992] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-27 17:18:43,604] - [remote_util] [140542730036992] - INFO - SSH Connecting to 172.23.106.1 with username:root, attempt#1 of 5 [2021-02-27 17:18:43,705] - [remote_util] [140542730036992] - INFO - SSH Connected to 172.23.106.1 as root [2021-02-27 17:18:43,954] - [remote_util] [140542730036992] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-27 17:18:44,249] - [remote_util] [140542730036992] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-27 17:18:44,309] - [remote_util] [140542730036992] - INFO - 172.23.106.1 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-27 17:18:44,367] - [remote_util] [140542730036992] - INFO - Running systemd command on this server [2021-02-27 17:18:44,367] - [remote_util] [140542730036992] - INFO - running command.raw on 172.23.106.1: systemctl stop couchbase-server.service [2021-02-27 17:18:46,108] - [remote_util] [140542730036992] - INFO - command executed successfully with root [2021-02-27 17:18:46,108] - [basetestcase] [140542730036992] - INFO - Couchbase stopped [2021-02-27 17:18:46,108] - [basetestcase] [140542730036992] - INFO - sleep for 5 secs. ... [2021-02-27 17:18:51,113] - [remote_util] [140542730036992] - INFO - running command.raw on 172.23.106.1: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-27 17:18:51,132] - [remote_util] [140542730036992] - INFO - command executed successfully with root [2021-02-27 17:18:51,133] - [remote_util] [140542730036992] - INFO - running command.raw on 172.23.106.1: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-27 17:18:51,187] - [remote_util] [140542730036992] - INFO - command executed successfully with root [2021-02-27 17:18:51,189] - [remote_util] [140542730036992] - INFO - SSH Connecting to 172.23.106.1 with username:root, attempt#1 of 5 [2021-02-27 17:18:51,293] - [remote_util] [140542730036992] - INFO - SSH Connected to 172.23.106.1 as root [2021-02-27 17:18:51,547] - [remote_util] [140542730036992] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-27 17:18:51,850] - [remote_util] [140542730036992] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-27 17:18:51,870] - [remote_util] [140542730036992] - INFO - 172.23.106.1 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-27 17:18:51,870] - [remote_util] [140542730036992] - INFO - Checking for process beam.smp on linux [2021-02-27 17:18:51,906] - [remote_util] [140542730036992] - INFO - Starting couchbase server [2021-02-27 17:18:52,007] - [remote_util] [140542730036992] - INFO - Running systemd command on this server [2021-02-27 17:18:52,007] - [remote_util] [140542730036992] - INFO - running command.raw on 172.23.106.1: systemctl start couchbase-server.service [2021-02-27 17:18:52,031] - [remote_util] [140542730036992] - INFO - command executed successfully with root [2021-02-27 17:18:52,031] - [remote_util] [140542730036992] - INFO - 172.23.106.1:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-27 17:18:57,032] - [remote_util] [140542730036992] - INFO - running command.raw on 172.23.106.1: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-27 17:18:57,056] - [remote_util] [140542730036992] - INFO - command executed successfully with root [2021-02-27 17:18:57,057] - [remote_util] [140542730036992] - INFO - Couchbase server status: [' Process: 47144 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-27 17:18:57,057] - [remote_util] [140542730036992] - INFO - Checking for process beam.smp on linux [2021-02-27 17:18:57,127] - [remote_util] [140542730036992] - INFO - process beam.smp is running on 172.23.106.1: with pid 47287 [2021-02-27 17:18:57,127] - [basetestcase] [140542730036992] - INFO - Couchbase started [2021-02-27 17:18:57,129] - [remote_util] [140542730036992] - INFO - SSH Connecting to 172.23.107.65 with username:root, attempt#1 of 5 [2021-02-27 17:18:57,235] - [remote_util] [140542730036992] - INFO - SSH Connected to 172.23.107.65 as root [2021-02-27 17:18:57,518] - [remote_util] [140542730036992] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-27 17:18:57,816] - [remote_util] [140542730036992] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-27 17:18:57,825] - [remote_util] [140542730036992] - INFO - SSH Connecting to 172.23.107.65 with username:root, attempt#1 of 5 [2021-02-27 17:18:57,924] - [remote_util] [140542730036992] - INFO - SSH Connected to 172.23.107.65 as root [2021-02-27 17:18:58,204] - [remote_util] [140542730036992] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-27 17:18:58,504] - [remote_util] [140542730036992] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-27 17:18:58,524] - [remote_util] [140542730036992] - INFO - 172.23.107.65 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-27 17:18:58,579] - [remote_util] [140542730036992] - INFO - Running systemd command on this server [2021-02-27 17:18:58,579] - [remote_util] [140542730036992] - INFO - running command.raw on 172.23.107.65: systemctl stop couchbase-server.service [2021-02-27 17:19:00,306] - [remote_util] [140542730036992] - INFO - command executed successfully with root [2021-02-27 17:19:00,306] - [basetestcase] [140542730036992] - INFO - Couchbase stopped [2021-02-27 17:19:00,306] - [basetestcase] [140542730036992] - INFO - sleep for 5 secs. ... [2021-02-27 17:19:05,309] - [remote_util] [140542730036992] - INFO - running command.raw on 172.23.107.65: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-27 17:19:05,324] - [remote_util] [140542730036992] - INFO - command executed successfully with root [2021-02-27 17:19:05,324] - [remote_util] [140542730036992] - INFO - running command.raw on 172.23.107.65: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-27 17:19:05,378] - [remote_util] [140542730036992] - INFO - command executed successfully with root [2021-02-27 17:19:05,379] - [remote_util] [140542730036992] - INFO - SSH Connecting to 172.23.107.65 with username:root, attempt#1 of 5 [2021-02-27 17:19:05,480] - [remote_util] [140542730036992] - INFO - SSH Connected to 172.23.107.65 as root [2021-02-27 17:19:05,764] - [remote_util] [140542730036992] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-27 17:19:06,059] - [remote_util] [140542730036992] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-27 17:19:06,079] - [remote_util] [140542730036992] - INFO - 172.23.107.65 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-27 17:19:06,079] - [remote_util] [140542730036992] - INFO - Checking for process beam.smp on linux [2021-02-27 17:19:06,117] - [remote_util] [140542730036992] - INFO - Starting couchbase server [2021-02-27 17:19:06,213] - [remote_util] [140542730036992] - INFO - Running systemd command on this server [2021-02-27 17:19:06,213] - [remote_util] [140542730036992] - INFO - running command.raw on 172.23.107.65: systemctl start couchbase-server.service [2021-02-27 17:19:06,238] - [remote_util] [140542730036992] - INFO - command executed successfully with root [2021-02-27 17:19:06,238] - [remote_util] [140542730036992] - INFO - 172.23.107.65:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-27 17:19:11,243] - [remote_util] [140542730036992] - INFO - running command.raw on 172.23.107.65: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-27 17:19:11,271] - [remote_util] [140542730036992] - INFO - command executed successfully with root [2021-02-27 17:19:11,271] - [remote_util] [140542730036992] - INFO - Couchbase server status: [' Process: 14767 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-27 17:19:11,271] - [remote_util] [140542730036992] - INFO - Checking for process beam.smp on linux [2021-02-27 17:19:11,341] - [remote_util] [140542730036992] - INFO - process beam.smp is running on 172.23.107.65: with pid 14911 [2021-02-27 17:19:11,342] - [basetestcase] [140542730036992] - INFO - Couchbase started [2021-02-27 17:19:11,344] - [remote_util] [140542730036992] - INFO - SSH Connecting to 172.23.105.240 with username:root, attempt#1 of 5 [2021-02-27 17:19:11,448] - [remote_util] [140542730036992] - INFO - SSH Connected to 172.23.105.240 as root [2021-02-27 17:19:11,721] - [remote_util] [140542730036992] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-27 17:19:12,022] - [remote_util] [140542730036992] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-27 17:19:12,031] - [remote_util] [140542730036992] - INFO - SSH Connecting to 172.23.105.240 with username:root, attempt#1 of 5 [2021-02-27 17:19:12,129] - [remote_util] [140542730036992] - INFO - SSH Connected to 172.23.105.240 as root [2021-02-27 17:19:12,402] - [remote_util] [140542730036992] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-27 17:19:12,710] - [remote_util] [140542730036992] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-27 17:19:12,771] - [remote_util] [140542730036992] - INFO - 172.23.105.240 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-27 17:19:12,828] - [remote_util] [140542730036992] - INFO - Running systemd command on this server [2021-02-27 17:19:12,828] - [remote_util] [140542730036992] - INFO - running command.raw on 172.23.105.240: systemctl stop couchbase-server.service [2021-02-27 17:19:14,546] - [remote_util] [140542730036992] - INFO - command executed successfully with root [2021-02-27 17:19:14,546] - [basetestcase] [140542730036992] - INFO - Couchbase stopped [2021-02-27 17:19:14,547] - [basetestcase] [140542730036992] - INFO - sleep for 5 secs. ... [2021-02-27 17:19:19,552] - [remote_util] [140542730036992] - INFO - running command.raw on 172.23.105.240: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-27 17:19:19,572] - [remote_util] [140542730036992] - INFO - command executed successfully with root [2021-02-27 17:19:19,573] - [remote_util] [140542730036992] - INFO - running command.raw on 172.23.105.240: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-27 17:19:19,628] - [remote_util] [140542730036992] - INFO - command executed successfully with root [2021-02-27 17:19:19,630] - [remote_util] [140542730036992] - INFO - SSH Connecting to 172.23.105.240 with username:root, attempt#1 of 5 [2021-02-27 17:19:19,735] - [remote_util] [140542730036992] - INFO - SSH Connected to 172.23.105.240 as root [2021-02-27 17:19:20,010] - [remote_util] [140542730036992] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-27 17:19:20,319] - [remote_util] [140542730036992] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-27 17:19:20,380] - [remote_util] [140542730036992] - INFO - 172.23.105.240 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-27 17:19:20,380] - [remote_util] [140542730036992] - INFO - Checking for process beam.smp on linux [2021-02-27 17:19:20,421] - [remote_util] [140542730036992] - INFO - Starting couchbase server [2021-02-27 17:19:20,521] - [remote_util] [140542730036992] - INFO - Running systemd command on this server [2021-02-27 17:19:20,521] - [remote_util] [140542730036992] - INFO - running command.raw on 172.23.105.240: systemctl start couchbase-server.service [2021-02-27 17:19:20,545] - [remote_util] [140542730036992] - INFO - command executed successfully with root [2021-02-27 17:19:20,545] - [remote_util] [140542730036992] - INFO - 172.23.105.240:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-27 17:19:25,547] - [remote_util] [140542730036992] - INFO - running command.raw on 172.23.105.240: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-27 17:19:25,579] - [remote_util] [140542730036992] - INFO - command executed successfully with root [2021-02-27 17:19:25,579] - [remote_util] [140542730036992] - INFO - Couchbase server status: [' Process: 113605 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-27 17:19:25,579] - [remote_util] [140542730036992] - INFO - Checking for process beam.smp on linux [2021-02-27 17:19:25,651] - [remote_util] [140542730036992] - INFO - process beam.smp is running on 172.23.105.240: with pid 113750 [2021-02-27 17:19:25,651] - [basetestcase] [140542730036992] - INFO - Couchbase started [2021-02-27 17:19:25,653] - [remote_util] [140542730036992] - INFO - SSH Connecting to 172.23.105.249 with username:root, attempt#1 of 5 [2021-02-27 17:19:25,760] - [remote_util] [140542730036992] - INFO - SSH Connected to 172.23.105.249 as root [2021-02-27 17:19:26,054] - [remote_util] [140542730036992] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-27 17:19:26,354] - [remote_util] [140542730036992] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-27 17:19:26,363] - [remote_util] [140542730036992] - INFO - SSH Connecting to 172.23.105.249 with username:root, attempt#1 of 5 [2021-02-27 17:19:26,464] - [remote_util] [140542730036992] - INFO - SSH Connected to 172.23.105.249 as root [2021-02-27 17:19:26,744] - [remote_util] [140542730036992] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-27 17:19:27,043] - [remote_util] [140542730036992] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-27 17:19:27,064] - [remote_util] [140542730036992] - INFO - 172.23.105.249 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-27 17:19:27,118] - [remote_util] [140542730036992] - INFO - Running systemd command on this server [2021-02-27 17:19:27,118] - [remote_util] [140542730036992] - INFO - running command.raw on 172.23.105.249: systemctl stop couchbase-server.service [2021-02-27 17:19:28,696] - [remote_util] [140542730036992] - INFO - command executed successfully with root [2021-02-27 17:19:28,697] - [basetestcase] [140542730036992] - INFO - Couchbase stopped [2021-02-27 17:19:28,697] - [basetestcase] [140542730036992] - INFO - sleep for 5 secs. ... [2021-02-27 17:19:33,702] - [remote_util] [140542730036992] - INFO - running command.raw on 172.23.105.249: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-27 17:19:33,719] - [remote_util] [140542730036992] - INFO - command executed successfully with root [2021-02-27 17:19:33,719] - [remote_util] [140542730036992] - INFO - running command.raw on 172.23.105.249: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-27 17:19:33,773] - [remote_util] [140542730036992] - INFO - command executed successfully with root [2021-02-27 17:19:33,775] - [remote_util] [140542730036992] - INFO - SSH Connecting to 172.23.105.249 with username:root, attempt#1 of 5 [2021-02-27 17:19:33,880] - [remote_util] [140542730036992] - INFO - SSH Connected to 172.23.105.249 as root [2021-02-27 17:19:34,174] - [remote_util] [140542730036992] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-27 17:19:34,472] - [remote_util] [140542730036992] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-27 17:19:34,533] - [remote_util] [140542730036992] - INFO - 172.23.105.249 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-27 17:19:34,533] - [remote_util] [140542730036992] - INFO - Checking for process beam.smp on linux [2021-02-27 17:19:34,574] - [remote_util] [140542730036992] - INFO - Starting couchbase server [2021-02-27 17:19:34,670] - [remote_util] [140542730036992] - INFO - Running systemd command on this server [2021-02-27 17:19:34,670] - [remote_util] [140542730036992] - INFO - running command.raw on 172.23.105.249: systemctl start couchbase-server.service [2021-02-27 17:19:34,695] - [remote_util] [140542730036992] - INFO - command executed successfully with root [2021-02-27 17:19:34,695] - [remote_util] [140542730036992] - INFO - 172.23.105.249:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-27 17:19:39,700] - [remote_util] [140542730036992] - INFO - running command.raw on 172.23.105.249: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-27 17:19:39,725] - [remote_util] [140542730036992] - INFO - command executed successfully with root [2021-02-27 17:19:39,725] - [remote_util] [140542730036992] - INFO - Couchbase server status: [' Process: 49780 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-27 17:19:39,725] - [remote_util] [140542730036992] - INFO - Checking for process beam.smp on linux [2021-02-27 17:19:39,795] - [remote_util] [140542730036992] - INFO - process beam.smp is running on 172.23.105.249: with pid 49922 [2021-02-27 17:19:39,795] - [basetestcase] [140542730036992] - INFO - Couchbase started [2021-02-27 17:19:39,795] - [basetestcase] [140542730036992] - INFO - sleep for 10 secs. ... [2021-02-27 17:19:49,840] - [remote_util] [140544332257024] - INFO - SSH Connecting to 172.23.121.213 with username:root, attempt#1 of 5 [2021-02-27 17:19:49,841] - [remote_util] [140542730036992] - INFO - SSH Connecting to 172.23.104.97 with username:root, attempt#1 of 5 [2021-02-27 17:19:49,843] - [remote_util] [140544315471616] - INFO - SSH Connecting to 172.23.105.238 with username:root, attempt#1 of 5 [2021-02-27 17:19:49,847] - [remote_util] [140543778600704] - INFO - SSH Connecting to 172.23.106.1 with username:root, attempt#1 of 5 [2021-02-27 17:19:49,849] - [remote_util] [140544307078912] - INFO - SSH Connecting to 172.23.105.247 with username:root, attempt#1 of 5 [2021-02-27 17:19:49,850] - [remote_util] [140543770208000] - INFO - SSH Connecting to 172.23.107.65 with username:root, attempt#1 of 5 [2021-02-27 17:19:49,852] - [remote_util] [140543275300608] - INFO - SSH Connecting to 172.23.105.240 with username:root, attempt#1 of 5 [2021-02-27 17:19:49,854] - [remote_util] [140542738429696] - INFO - SSH Connecting to 172.23.105.249 with username:root, attempt#1 of 5 [2021-02-27 17:19:49,947] - [remote_util] [140544332257024] - INFO - SSH Connected to 172.23.121.213 as root [2021-02-27 17:19:49,956] - [remote_util] [140543778600704] - INFO - SSH Connected to 172.23.106.1 as root [2021-02-27 17:19:49,957] - [remote_util] [140544315471616] - INFO - SSH Connected to 172.23.105.238 as root [2021-02-27 17:19:49,957] - [remote_util] [140543770208000] - INFO - SSH Connected to 172.23.107.65 as root [2021-02-27 17:19:49,958] - [remote_util] [140544307078912] - INFO - SSH Connected to 172.23.105.247 as root [2021-02-27 17:19:49,964] - [remote_util] [140542738429696] - INFO - SSH Connected to 172.23.105.249 as root [2021-02-27 17:19:49,968] - [remote_util] [140543275300608] - INFO - SSH Connected to 172.23.105.240 as root [2021-02-27 17:19:50,010] - [remote_util] [140542730036992] - INFO - SSH Connected to 172.23.104.97 as root [2021-02-27 17:19:50,220] - [remote_util] [140543778600704] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-27 17:19:50,255] - [remote_util] [140544332257024] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-27 17:19:50,270] - [remote_util] [140544315471616] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-27 17:19:50,277] - [remote_util] [140544307078912] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-27 17:19:50,279] - [remote_util] [140543770208000] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-27 17:19:50,293] - [remote_util] [140542738429696] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-27 17:19:50,293] - [remote_util] [140543275300608] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-27 17:19:50,572] - [remote_util] [140543778600704] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-27 17:19:50,572] - [remote_util] [140543778600704] - INFO - running command.raw on 172.23.106.1: /opt/couchbase/bin/cbcollect_info 172.23.106.1-20210227-1719-diag.zip [2021-02-27 17:19:50,585] - [remote_util] [140544332257024] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-27 17:19:50,585] - [remote_util] [140544332257024] - INFO - running command.raw on 172.23.121.213: /opt/couchbase/bin/cbcollect_info 172.23.121.213-20210227-1719-diag.zip [2021-02-27 17:19:50,588] - [remote_util] [140543770208000] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-27 17:19:50,588] - [remote_util] [140543770208000] - INFO - running command.raw on 172.23.107.65: /opt/couchbase/bin/cbcollect_info 172.23.107.65-20210227-1719-diag.zip [2021-02-27 17:19:50,594] - [remote_util] [140544315471616] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-27 17:19:50,594] - [remote_util] [140544315471616] - INFO - running command.raw on 172.23.105.238: /opt/couchbase/bin/cbcollect_info 172.23.105.238-20210227-1719-diag.zip [2021-02-27 17:19:50,596] - [remote_util] [140544307078912] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-27 17:19:50,597] - [remote_util] [140544307078912] - INFO - running command.raw on 172.23.105.247: /opt/couchbase/bin/cbcollect_info 172.23.105.247-20210227-1719-diag.zip [2021-02-27 17:19:50,601] - [remote_util] [140542738429696] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-27 17:19:50,601] - [remote_util] [140542738429696] - INFO - running command.raw on 172.23.105.249: /opt/couchbase/bin/cbcollect_info 172.23.105.249-20210227-1719-diag.zip [2021-02-27 17:19:50,608] - [remote_util] [140543275300608] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-27 17:19:50,609] - [remote_util] [140543275300608] - INFO - running command.raw on 172.23.105.240: /opt/couchbase/bin/cbcollect_info 172.23.105.240-20210227-1719-diag.zip [2021-02-27 17:19:50,670] - [remote_util] [140542730036992] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-27 17:19:51,151] - [remote_util] [140542730036992] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-27 17:19:51,152] - [remote_util] [140542730036992] - INFO - running command.raw on 172.23.104.97: /opt/couchbase/bin/cbcollect_info 172.23.104.97-20210227-1719-diag.zip [2021-02-27 17:21:15,428] - [remote_util] [140543770208000] - INFO - command executed successfully with root [2021-02-27 17:21:15,841] - [remote_util] [140543770208000] - INFO - running command.raw on 172.23.107.65: rm -f /root/172.23.107.65-20210227-1719-diag.zip [2021-02-27 17:21:15,859] - [remote_util] [140543770208000] - INFO - command executed successfully with root [2021-02-27 17:21:19,369] - [remote_util] [140542738429696] - INFO - command executed successfully with root [2021-02-27 17:21:19,771] - [remote_util] [140542738429696] - INFO - running command.raw on 172.23.105.249: rm -f /root/172.23.105.249-20210227-1719-diag.zip [2021-02-27 17:21:19,791] - [remote_util] [140542738429696] - INFO - command executed successfully with root [2021-02-27 17:21:21,829] - [remote_util] [140543778600704] - INFO - command executed successfully with root [2021-02-27 17:21:21,881] - [remote_util] [140544332257024] - INFO - command executed successfully with root [2021-02-27 17:21:22,546] - [remote_util] [140543778600704] - INFO - running command.raw on 172.23.106.1: rm -f /root/172.23.106.1-20210227-1719-diag.zip [2021-02-27 17:21:22,603] - [remote_util] [140543778600704] - INFO - command executed successfully with root [2021-02-27 17:21:22,632] - [remote_util] [140544332257024] - INFO - running command.raw on 172.23.121.213: rm -f /root/172.23.121.213-20210227-1719-diag.zip [2021-02-27 17:21:22,652] - [remote_util] [140544332257024] - INFO - command executed successfully with root [2021-02-27 17:21:24,076] - [remote_util] [140544307078912] - INFO - command executed successfully with root [2021-02-27 17:21:24,580] - [remote_util] [140544307078912] - INFO - running command.raw on 172.23.105.247: rm -f /root/172.23.105.247-20210227-1719-diag.zip [2021-02-27 17:21:24,602] - [remote_util] [140544307078912] - INFO - command executed successfully with root [2021-02-27 17:21:24,760] - [remote_util] [140544315471616] - INFO - command executed successfully with root [2021-02-27 17:21:25,222] - [remote_util] [140544315471616] - INFO - running command.raw on 172.23.105.238: rm -f /root/172.23.105.238-20210227-1719-diag.zip [2021-02-27 17:21:25,242] - [remote_util] [140544315471616] - INFO - command executed successfully with root [2021-02-27 17:21:26,250] - [remote_util] [140543275300608] - INFO - command executed successfully with root [2021-02-27 17:21:26,717] - [remote_util] [140543275300608] - INFO - running command.raw on 172.23.105.240: rm -f /root/172.23.105.240-20210227-1719-diag.zip [2021-02-27 17:21:26,738] - [remote_util] [140543275300608] - INFO - command executed successfully with root [2021-02-27 17:22:00,931] - [remote_util] [140542730036992] - INFO - command executed successfully with root [2021-02-27 17:22:01,565] - [remote_util] [140542730036992] - INFO - running command.raw on 172.23.104.97: rm -f /root/172.23.104.97-20210227-1719-diag.zip [2021-02-27 17:22:01,607] - [remote_util] [140542730036992] - INFO - command executed successfully with root