[2021-02-22 22:42:20,414] - [eventing_base] [140689256060672] - INFO - Starting Test: test_eventing_rebalance_swap_delete_recreate_collections [2021-02-22 22:42:20,417] - [remote_util] [140689256060672] - INFO - SSH Connecting to 172.23.123.65 with username:root, attempt#1 of 5 [2021-02-22 22:42:20,526] - [remote_util] [140689256060672] - INFO - SSH Connected to 172.23.123.65 as root [2021-02-22 22:42:20,822] - [remote_util] [140689256060672] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 22:42:21,136] - [remote_util] [140689256060672] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 22:42:21,142] - [rest_client] [140689256060672] - INFO - -->is_ns_server_running? [2021-02-22 22:42:21,156] - [rest_client] [140689256060672] - INFO - Node version in cluster 7.0.0-4514-enterprise [2021-02-22 22:42:21,162] - [rest_client] [140689256060672] - ERROR - GET http://172.23.123.65:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-02-22 22:42:21,163] - [rest_client] [140689256060672] - INFO - Node versions in cluster [] [2021-02-22 22:42:21,164] - [basetestcase] [140689256060672] - INFO - ============== basetestcase setup was started for test #14 test_eventing_rebalance_swap_delete_recreate_collections============== [2021-02-22 22:42:21,169] - [rest_client] [140689256060672] - ERROR - GET http://172.23.123.65:8091/pools/default/ body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-02-22 22:42:21,185] - [bucket_helper] [140689256060672] - INFO - Could not find any buckets for node 172.23.123.65, nothing to delete [2021-02-22 22:42:21,196] - [bucket_helper] [140689256060672] - INFO - Could not find any buckets for node 172.23.122.182, nothing to delete [2021-02-22 22:42:21,209] - [bucket_helper] [140689256060672] - INFO - Could not find any buckets for node 172.23.122.179, nothing to delete [2021-02-22 22:42:21,221] - [bucket_helper] [140689256060672] - INFO - Could not find any buckets for node 172.23.122.153, nothing to delete [2021-02-22 22:42:21,233] - [bucket_helper] [140689256060672] - INFO - Could not find any buckets for node 172.23.122.139, nothing to delete [2021-02-22 22:42:21,246] - [bucket_helper] [140689256060672] - INFO - Could not find any buckets for node 172.23.122.140, nothing to delete [2021-02-22 22:42:21,258] - [bucket_helper] [140689256060672] - INFO - Could not find any buckets for node 172.23.122.138, nothing to delete [2021-02-22 22:42:21,269] - [bucket_helper] [140689256060672] - INFO - Could not find any buckets for node 172.23.122.141, nothing to delete [2021-02-22 22:42:21,274] - [rest_client] [140689256060672] - INFO - -->is_ns_server_running? [2021-02-22 22:42:21,286] - [cluster_helper] [140689256060672] - INFO - waiting for ns_server @ 172.23.123.65:8091 [2021-02-22 22:42:21,287] - [rest_client] [140689256060672] - INFO - -->is_ns_server_running? [2021-02-22 22:42:21,291] - [cluster_helper] [140689256060672] - INFO - ns_server @ 172.23.123.65:8091 is running [2021-02-22 22:42:21,294] - [cluster_helper] [140689256060672] - INFO - waiting for ns_server @ 172.23.122.182:8091 [2021-02-22 22:42:21,295] - [rest_client] [140689256060672] - INFO - -->is_ns_server_running? [2021-02-22 22:42:21,299] - [cluster_helper] [140689256060672] - INFO - ns_server @ 172.23.122.182:8091 is running [2021-02-22 22:42:21,303] - [cluster_helper] [140689256060672] - INFO - waiting for ns_server @ 172.23.122.179:8091 [2021-02-22 22:42:21,303] - [rest_client] [140689256060672] - INFO - -->is_ns_server_running? [2021-02-22 22:42:21,306] - [cluster_helper] [140689256060672] - INFO - ns_server @ 172.23.122.179:8091 is running [2021-02-22 22:42:21,311] - [cluster_helper] [140689256060672] - INFO - waiting for ns_server @ 172.23.122.153:8091 [2021-02-22 22:42:21,311] - [rest_client] [140689256060672] - INFO - -->is_ns_server_running? [2021-02-22 22:42:21,315] - [cluster_helper] [140689256060672] - INFO - ns_server @ 172.23.122.153:8091 is running [2021-02-22 22:42:21,319] - [cluster_helper] [140689256060672] - INFO - waiting for ns_server @ 172.23.122.139:8091 [2021-02-22 22:42:21,319] - [rest_client] [140689256060672] - INFO - -->is_ns_server_running? [2021-02-22 22:42:21,323] - [cluster_helper] [140689256060672] - INFO - ns_server @ 172.23.122.139:8091 is running [2021-02-22 22:42:21,327] - [cluster_helper] [140689256060672] - INFO - waiting for ns_server @ 172.23.122.140:8091 [2021-02-22 22:42:21,327] - [rest_client] [140689256060672] - INFO - -->is_ns_server_running? [2021-02-22 22:42:21,331] - [cluster_helper] [140689256060672] - INFO - ns_server @ 172.23.122.140:8091 is running [2021-02-22 22:42:21,335] - [cluster_helper] [140689256060672] - INFO - waiting for ns_server @ 172.23.122.138:8091 [2021-02-22 22:42:21,335] - [rest_client] [140689256060672] - INFO - -->is_ns_server_running? [2021-02-22 22:42:21,339] - [cluster_helper] [140689256060672] - INFO - ns_server @ 172.23.122.138:8091 is running [2021-02-22 22:42:21,343] - [cluster_helper] [140689256060672] - INFO - waiting for ns_server @ 172.23.122.141:8091 [2021-02-22 22:42:21,344] - [rest_client] [140689256060672] - INFO - -->is_ns_server_running? [2021-02-22 22:42:21,347] - [cluster_helper] [140689256060672] - INFO - ns_server @ 172.23.122.141:8091 is running [2021-02-22 22:42:21,348] - [basetestcase] [140689256060672] - INFO - Building docker image with java sdk client [2021-02-22 22:42:29,761] - [basetestcase] [140689256060672] - INFO - initializing cluster [2021-02-22 22:42:29,767] - [remote_util] [140689256060672] - INFO - SSH Connecting to 172.23.123.65 with username:root, attempt#1 of 5 [2021-02-22 22:42:29,878] - [remote_util] [140689256060672] - INFO - SSH Connected to 172.23.123.65 as root [2021-02-22 22:42:30,187] - [remote_util] [140689256060672] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 22:42:30,508] - [remote_util] [140689256060672] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 22:42:30,523] - [remote_util] [140689256060672] - INFO - SSH Connecting to 172.23.123.65 with username:root, attempt#1 of 5 [2021-02-22 22:42:30,629] - [remote_util] [140689256060672] - INFO - SSH Connected to 172.23.123.65 as root [2021-02-22 22:42:30,926] - [remote_util] [140689256060672] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 22:42:31,239] - [remote_util] [140689256060672] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 22:42:31,301] - [remote_util] [140689256060672] - INFO - 172.23.123.65 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-22 22:42:31,409] - [remote_util] [140689256060672] - INFO - Running systemd command on this server [2021-02-22 22:42:31,409] - [remote_util] [140689256060672] - INFO - running command.raw on 172.23.123.65: systemctl stop couchbase-server.service [2021-02-22 22:42:33,112] - [remote_util] [140689256060672] - INFO - command executed successfully with root [2021-02-22 22:42:33,112] - [basetestcase] [140689256060672] - INFO - Couchbase stopped [2021-02-22 22:42:33,113] - [basetestcase] [140689256060672] - INFO - sleep for 5 secs. ... [2021-02-22 22:42:38,118] - [remote_util] [140689256060672] - INFO - running command.raw on 172.23.123.65: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-22 22:42:38,135] - [remote_util] [140689256060672] - INFO - command executed successfully with root [2021-02-22 22:42:38,135] - [remote_util] [140689256060672] - INFO - running command.raw on 172.23.123.65: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-22 22:42:38,191] - [remote_util] [140689256060672] - INFO - command executed successfully with root [2021-02-22 22:42:38,194] - [remote_util] [140689256060672] - INFO - SSH Connecting to 172.23.123.65 with username:root, attempt#1 of 5 [2021-02-22 22:42:38,295] - [remote_util] [140689256060672] - INFO - SSH Connected to 172.23.123.65 as root [2021-02-22 22:42:38,574] - [remote_util] [140689256060672] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 22:42:38,874] - [remote_util] [140689256060672] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 22:42:38,936] - [remote_util] [140689256060672] - INFO - 172.23.123.65 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-22 22:42:38,936] - [remote_util] [140689256060672] - INFO - Checking for process beam.smp on linux [2021-02-22 22:42:39,018] - [remote_util] [140689256060672] - INFO - Starting couchbase server [2021-02-22 22:42:39,118] - [remote_util] [140689256060672] - INFO - Running systemd command on this server [2021-02-22 22:42:39,118] - [remote_util] [140689256060672] - INFO - running command.raw on 172.23.123.65: systemctl start couchbase-server.service [2021-02-22 22:42:39,140] - [remote_util] [140689256060672] - INFO - command executed successfully with root [2021-02-22 22:42:39,141] - [remote_util] [140689256060672] - INFO - 172.23.123.65:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-22 22:42:44,146] - [remote_util] [140689256060672] - INFO - running command.raw on 172.23.123.65: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-22 22:42:44,172] - [remote_util] [140689256060672] - INFO - command executed successfully with root [2021-02-22 22:42:44,172] - [remote_util] [140689256060672] - INFO - Couchbase server status: [' Process: 14073 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-22 22:42:44,173] - [remote_util] [140689256060672] - INFO - Checking for process beam.smp on linux [2021-02-22 22:42:44,245] - [remote_util] [140689256060672] - INFO - process beam.smp is running on 172.23.123.65: with pid 14216 [2021-02-22 22:42:44,245] - [basetestcase] [140689256060672] - INFO - Couchbase started [2021-02-22 22:42:44,249] - [remote_util] [140689256060672] - INFO - SSH Connecting to 172.23.122.182 with username:root, attempt#1 of 5 [2021-02-22 22:42:44,352] - [remote_util] [140689256060672] - INFO - SSH Connected to 172.23.122.182 as root [2021-02-22 22:42:44,610] - [remote_util] [140689256060672] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 22:42:44,912] - [remote_util] [140689256060672] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 22:42:44,928] - [remote_util] [140689256060672] - INFO - SSH Connecting to 172.23.122.182 with username:root, attempt#1 of 5 [2021-02-22 22:42:45,028] - [remote_util] [140689256060672] - INFO - SSH Connected to 172.23.122.182 as root [2021-02-22 22:42:45,284] - [remote_util] [140689256060672] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 22:42:45,630] - [remote_util] [140689256060672] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 22:42:45,693] - [remote_util] [140689256060672] - INFO - 172.23.122.182 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-22 22:42:45,793] - [remote_util] [140689256060672] - INFO - Running systemd command on this server [2021-02-22 22:42:45,793] - [remote_util] [140689256060672] - INFO - running command.raw on 172.23.122.182: systemctl stop couchbase-server.service [2021-02-22 22:42:47,480] - [remote_util] [140689256060672] - INFO - command executed successfully with root [2021-02-22 22:42:47,481] - [basetestcase] [140689256060672] - INFO - Couchbase stopped [2021-02-22 22:42:47,482] - [basetestcase] [140689256060672] - INFO - sleep for 5 secs. ... [2021-02-22 22:42:52,488] - [remote_util] [140689256060672] - INFO - running command.raw on 172.23.122.182: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-22 22:42:52,504] - [remote_util] [140689256060672] - INFO - command executed successfully with root [2021-02-22 22:42:52,505] - [remote_util] [140689256060672] - INFO - running command.raw on 172.23.122.182: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-22 22:42:52,558] - [remote_util] [140689256060672] - INFO - command executed successfully with root [2021-02-22 22:42:52,562] - [remote_util] [140689256060672] - INFO - SSH Connecting to 172.23.122.182 with username:root, attempt#1 of 5 [2021-02-22 22:42:52,663] - [remote_util] [140689256060672] - INFO - SSH Connected to 172.23.122.182 as root [2021-02-22 22:42:52,921] - [remote_util] [140689256060672] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 22:42:53,223] - [remote_util] [140689256060672] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 22:42:53,284] - [remote_util] [140689256060672] - INFO - 172.23.122.182 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-22 22:42:53,285] - [remote_util] [140689256060672] - INFO - Checking for process beam.smp on linux [2021-02-22 22:42:53,363] - [remote_util] [140689256060672] - INFO - Starting couchbase server [2021-02-22 22:42:53,465] - [remote_util] [140689256060672] - INFO - Running systemd command on this server [2021-02-22 22:42:53,466] - [remote_util] [140689256060672] - INFO - running command.raw on 172.23.122.182: systemctl start couchbase-server.service [2021-02-22 22:42:53,487] - [remote_util] [140689256060672] - INFO - command executed successfully with root [2021-02-22 22:42:53,487] - [remote_util] [140689256060672] - INFO - 172.23.122.182:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-22 22:42:58,493] - [remote_util] [140689256060672] - INFO - running command.raw on 172.23.122.182: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-22 22:42:58,520] - [remote_util] [140689256060672] - INFO - command executed successfully with root [2021-02-22 22:42:58,521] - [remote_util] [140689256060672] - INFO - Couchbase server status: [' Process: 30826 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-22 22:42:58,521] - [remote_util] [140689256060672] - INFO - Checking for process beam.smp on linux [2021-02-22 22:42:58,591] - [remote_util] [140689256060672] - INFO - process beam.smp is running on 172.23.122.182: with pid 30971 [2021-02-22 22:42:58,591] - [basetestcase] [140689256060672] - INFO - Couchbase started [2021-02-22 22:42:58,595] - [remote_util] [140689256060672] - INFO - SSH Connecting to 172.23.122.179 with username:root, attempt#1 of 5 [2021-02-22 22:42:58,702] - [remote_util] [140689256060672] - INFO - SSH Connected to 172.23.122.179 as root [2021-02-22 22:42:58,982] - [remote_util] [140689256060672] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 22:42:59,286] - [remote_util] [140689256060672] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 22:42:59,299] - [remote_util] [140689256060672] - INFO - SSH Connecting to 172.23.122.179 with username:root, attempt#1 of 5 [2021-02-22 22:42:59,400] - [remote_util] [140689256060672] - INFO - SSH Connected to 172.23.122.179 as root [2021-02-22 22:42:59,672] - [remote_util] [140689256060672] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 22:42:59,983] - [remote_util] [140689256060672] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 22:43:00,047] - [remote_util] [140689256060672] - INFO - 172.23.122.179 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-22 22:43:00,231] - [remote_util] [140689256060672] - INFO - Running systemd command on this server [2021-02-22 22:43:00,231] - [remote_util] [140689256060672] - INFO - running command.raw on 172.23.122.179: systemctl stop couchbase-server.service [2021-02-22 22:43:01,905] - [remote_util] [140689256060672] - INFO - command executed successfully with root [2021-02-22 22:43:01,906] - [basetestcase] [140689256060672] - INFO - Couchbase stopped [2021-02-22 22:43:01,906] - [basetestcase] [140689256060672] - INFO - sleep for 5 secs. ... [2021-02-22 22:43:06,911] - [remote_util] [140689256060672] - INFO - running command.raw on 172.23.122.179: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-22 22:43:06,928] - [remote_util] [140689256060672] - INFO - command executed successfully with root [2021-02-22 22:43:06,928] - [remote_util] [140689256060672] - INFO - running command.raw on 172.23.122.179: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-22 22:43:06,983] - [remote_util] [140689256060672] - INFO - command executed successfully with root [2021-02-22 22:43:06,987] - [remote_util] [140689256060672] - INFO - SSH Connecting to 172.23.122.179 with username:root, attempt#1 of 5 [2021-02-22 22:43:07,091] - [remote_util] [140689256060672] - INFO - SSH Connected to 172.23.122.179 as root [2021-02-22 22:43:07,366] - [remote_util] [140689256060672] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 22:43:07,672] - [remote_util] [140689256060672] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 22:43:07,733] - [remote_util] [140689256060672] - INFO - 172.23.122.179 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-22 22:43:07,734] - [remote_util] [140689256060672] - INFO - Checking for process beam.smp on linux [2021-02-22 22:43:07,811] - [remote_util] [140689256060672] - INFO - Starting couchbase server [2021-02-22 22:43:07,915] - [remote_util] [140689256060672] - INFO - Running systemd command on this server [2021-02-22 22:43:07,915] - [remote_util] [140689256060672] - INFO - running command.raw on 172.23.122.179: systemctl start couchbase-server.service [2021-02-22 22:43:07,939] - [remote_util] [140689256060672] - INFO - command executed successfully with root [2021-02-22 22:43:07,939] - [remote_util] [140689256060672] - INFO - 172.23.122.179:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-22 22:43:12,945] - [remote_util] [140689256060672] - INFO - running command.raw on 172.23.122.179: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-22 22:43:12,973] - [remote_util] [140689256060672] - INFO - command executed successfully with root [2021-02-22 22:43:12,973] - [remote_util] [140689256060672] - INFO - Couchbase server status: [' Process: 62160 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-22 22:43:12,974] - [remote_util] [140689256060672] - INFO - Checking for process beam.smp on linux [2021-02-22 22:43:13,042] - [remote_util] [140689256060672] - INFO - process beam.smp is running on 172.23.122.179: with pid 62304 [2021-02-22 22:43:13,042] - [basetestcase] [140689256060672] - INFO - Couchbase started [2021-02-22 22:43:13,046] - [remote_util] [140689256060672] - INFO - SSH Connecting to 172.23.122.153 with username:root, attempt#1 of 5 [2021-02-22 22:43:13,149] - [remote_util] [140689256060672] - INFO - SSH Connected to 172.23.122.153 as root [2021-02-22 22:43:13,427] - [remote_util] [140689256060672] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 22:43:13,721] - [remote_util] [140689256060672] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 22:43:13,733] - [remote_util] [140689256060672] - INFO - SSH Connecting to 172.23.122.153 with username:root, attempt#1 of 5 [2021-02-22 22:43:13,832] - [remote_util] [140689256060672] - INFO - SSH Connected to 172.23.122.153 as root [2021-02-22 22:43:14,102] - [remote_util] [140689256060672] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 22:43:14,399] - [remote_util] [140689256060672] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 22:43:14,460] - [remote_util] [140689256060672] - INFO - 172.23.122.153 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-22 22:43:14,561] - [remote_util] [140689256060672] - INFO - Running systemd command on this server [2021-02-22 22:43:14,561] - [remote_util] [140689256060672] - INFO - running command.raw on 172.23.122.153: systemctl stop couchbase-server.service [2021-02-22 22:43:16,388] - [remote_util] [140689256060672] - INFO - command executed successfully with root [2021-02-22 22:43:16,389] - [basetestcase] [140689256060672] - INFO - Couchbase stopped [2021-02-22 22:43:16,390] - [basetestcase] [140689256060672] - INFO - sleep for 5 secs. ... [2021-02-22 22:43:21,395] - [remote_util] [140689256060672] - INFO - running command.raw on 172.23.122.153: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-22 22:43:21,410] - [remote_util] [140689256060672] - INFO - command executed successfully with root [2021-02-22 22:43:21,410] - [remote_util] [140689256060672] - INFO - running command.raw on 172.23.122.153: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-22 22:43:21,464] - [remote_util] [140689256060672] - INFO - command executed successfully with root [2021-02-22 22:43:21,467] - [remote_util] [140689256060672] - INFO - SSH Connecting to 172.23.122.153 with username:root, attempt#1 of 5 [2021-02-22 22:43:21,564] - [remote_util] [140689256060672] - INFO - SSH Connected to 172.23.122.153 as root [2021-02-22 22:43:21,849] - [remote_util] [140689256060672] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 22:43:22,148] - [remote_util] [140689256060672] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 22:43:22,209] - [remote_util] [140689256060672] - INFO - 172.23.122.153 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-22 22:43:22,209] - [remote_util] [140689256060672] - INFO - Checking for process beam.smp on linux [2021-02-22 22:43:22,284] - [remote_util] [140689256060672] - INFO - Starting couchbase server [2021-02-22 22:43:22,385] - [remote_util] [140689256060672] - INFO - Running systemd command on this server [2021-02-22 22:43:22,386] - [remote_util] [140689256060672] - INFO - running command.raw on 172.23.122.153: systemctl start couchbase-server.service [2021-02-22 22:43:22,408] - [remote_util] [140689256060672] - INFO - command executed successfully with root [2021-02-22 22:43:22,408] - [remote_util] [140689256060672] - INFO - 172.23.122.153:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-22 22:43:27,411] - [remote_util] [140689256060672] - INFO - running command.raw on 172.23.122.153: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-22 22:43:27,437] - [remote_util] [140689256060672] - INFO - command executed successfully with root [2021-02-22 22:43:27,438] - [remote_util] [140689256060672] - INFO - Couchbase server status: [' Process: 112146 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-22 22:43:27,438] - [remote_util] [140689256060672] - INFO - Checking for process beam.smp on linux [2021-02-22 22:43:27,507] - [remote_util] [140689256060672] - INFO - process beam.smp is running on 172.23.122.153: with pid 112292 [2021-02-22 22:43:27,507] - [basetestcase] [140689256060672] - INFO - Couchbase started [2021-02-22 22:43:27,511] - [remote_util] [140689256060672] - INFO - SSH Connecting to 172.23.122.139 with username:root, attempt#1 of 5 [2021-02-22 22:43:27,615] - [remote_util] [140689256060672] - INFO - SSH Connected to 172.23.122.139 as root [2021-02-22 22:43:27,891] - [remote_util] [140689256060672] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 22:43:28,199] - [remote_util] [140689256060672] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 22:43:28,211] - [remote_util] [140689256060672] - INFO - SSH Connecting to 172.23.122.139 with username:root, attempt#1 of 5 [2021-02-22 22:43:28,313] - [remote_util] [140689256060672] - INFO - SSH Connected to 172.23.122.139 as root [2021-02-22 22:43:28,582] - [remote_util] [140689256060672] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 22:43:28,885] - [remote_util] [140689256060672] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 22:43:28,947] - [remote_util] [140689256060672] - INFO - 172.23.122.139 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-22 22:43:29,051] - [remote_util] [140689256060672] - INFO - Running systemd command on this server [2021-02-22 22:43:29,051] - [remote_util] [140689256060672] - INFO - running command.raw on 172.23.122.139: systemctl stop couchbase-server.service [2021-02-22 22:43:30,814] - [remote_util] [140689256060672] - INFO - command executed successfully with root [2021-02-22 22:43:30,815] - [basetestcase] [140689256060672] - INFO - Couchbase stopped [2021-02-22 22:43:30,815] - [basetestcase] [140689256060672] - INFO - sleep for 5 secs. ... [2021-02-22 22:43:35,821] - [remote_util] [140689256060672] - INFO - running command.raw on 172.23.122.139: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-22 22:43:35,839] - [remote_util] [140689256060672] - INFO - command executed successfully with root [2021-02-22 22:43:35,839] - [remote_util] [140689256060672] - INFO - running command.raw on 172.23.122.139: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-22 22:43:35,893] - [remote_util] [140689256060672] - INFO - command executed successfully with root [2021-02-22 22:43:35,897] - [remote_util] [140689256060672] - INFO - SSH Connecting to 172.23.122.139 with username:root, attempt#1 of 5 [2021-02-22 22:43:36,001] - [remote_util] [140689256060672] - INFO - SSH Connected to 172.23.122.139 as root [2021-02-22 22:43:36,281] - [remote_util] [140689256060672] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 22:43:36,588] - [remote_util] [140689256060672] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 22:43:36,650] - [remote_util] [140689256060672] - INFO - 172.23.122.139 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-22 22:43:36,650] - [remote_util] [140689256060672] - INFO - Checking for process beam.smp on linux [2021-02-22 22:43:36,730] - [remote_util] [140689256060672] - INFO - Starting couchbase server [2021-02-22 22:43:36,841] - [remote_util] [140689256060672] - INFO - Running systemd command on this server [2021-02-22 22:43:36,841] - [remote_util] [140689256060672] - INFO - running command.raw on 172.23.122.139: systemctl start couchbase-server.service [2021-02-22 22:43:36,863] - [remote_util] [140689256060672] - INFO - command executed successfully with root [2021-02-22 22:43:36,863] - [remote_util] [140689256060672] - INFO - 172.23.122.139:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-22 22:43:41,869] - [remote_util] [140689256060672] - INFO - running command.raw on 172.23.122.139: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-22 22:43:41,896] - [remote_util] [140689256060672] - INFO - command executed successfully with root [2021-02-22 22:43:41,897] - [remote_util] [140689256060672] - INFO - Couchbase server status: [' Process: 66276 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-22 22:43:41,898] - [remote_util] [140689256060672] - INFO - Checking for process beam.smp on linux [2021-02-22 22:43:41,966] - [remote_util] [140689256060672] - INFO - process beam.smp is running on 172.23.122.139: with pid 66465 [2021-02-22 22:43:41,967] - [basetestcase] [140689256060672] - INFO - Couchbase started [2021-02-22 22:43:41,971] - [remote_util] [140689256060672] - INFO - SSH Connecting to 172.23.122.140 with username:root, attempt#1 of 5 [2021-02-22 22:43:42,075] - [remote_util] [140689256060672] - INFO - SSH Connected to 172.23.122.140 as root [2021-02-22 22:43:42,375] - [remote_util] [140689256060672] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 22:43:42,716] - [remote_util] [140689256060672] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 22:43:42,729] - [remote_util] [140689256060672] - INFO - SSH Connecting to 172.23.122.140 with username:root, attempt#1 of 5 [2021-02-22 22:43:42,828] - [remote_util] [140689256060672] - INFO - SSH Connected to 172.23.122.140 as root [2021-02-22 22:43:43,108] - [remote_util] [140689256060672] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 22:43:43,406] - [remote_util] [140689256060672] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 22:43:43,469] - [remote_util] [140689256060672] - INFO - 172.23.122.140 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-22 22:43:43,569] - [remote_util] [140689256060672] - INFO - Running systemd command on this server [2021-02-22 22:43:43,570] - [remote_util] [140689256060672] - INFO - running command.raw on 172.23.122.140: systemctl stop couchbase-server.service [2021-02-22 22:43:45,217] - [remote_util] [140689256060672] - INFO - command executed successfully with root [2021-02-22 22:43:45,218] - [basetestcase] [140689256060672] - INFO - Couchbase stopped [2021-02-22 22:43:45,218] - [basetestcase] [140689256060672] - INFO - sleep for 5 secs. ... [2021-02-22 22:43:50,219] - [remote_util] [140689256060672] - INFO - running command.raw on 172.23.122.140: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-22 22:43:50,236] - [remote_util] [140689256060672] - INFO - command executed successfully with root [2021-02-22 22:43:50,237] - [remote_util] [140689256060672] - INFO - running command.raw on 172.23.122.140: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-22 22:43:50,290] - [remote_util] [140689256060672] - INFO - command executed successfully with root [2021-02-22 22:43:50,293] - [remote_util] [140689256060672] - INFO - SSH Connecting to 172.23.122.140 with username:root, attempt#1 of 5 [2021-02-22 22:43:50,394] - [remote_util] [140689256060672] - INFO - SSH Connected to 172.23.122.140 as root [2021-02-22 22:43:50,681] - [remote_util] [140689256060672] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 22:43:50,980] - [remote_util] [140689256060672] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 22:43:51,042] - [remote_util] [140689256060672] - INFO - 172.23.122.140 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-22 22:43:51,042] - [remote_util] [140689256060672] - INFO - Checking for process beam.smp on linux [2021-02-22 22:43:51,122] - [remote_util] [140689256060672] - INFO - Starting couchbase server [2021-02-22 22:43:51,225] - [remote_util] [140689256060672] - INFO - Running systemd command on this server [2021-02-22 22:43:51,225] - [remote_util] [140689256060672] - INFO - running command.raw on 172.23.122.140: systemctl start couchbase-server.service [2021-02-22 22:43:51,247] - [remote_util] [140689256060672] - INFO - command executed successfully with root [2021-02-22 22:43:51,248] - [remote_util] [140689256060672] - INFO - 172.23.122.140:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-22 22:43:56,253] - [remote_util] [140689256060672] - INFO - running command.raw on 172.23.122.140: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-22 22:43:56,279] - [remote_util] [140689256060672] - INFO - command executed successfully with root [2021-02-22 22:43:56,279] - [remote_util] [140689256060672] - INFO - Couchbase server status: [' Process: 130098 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-22 22:43:56,279] - [remote_util] [140689256060672] - INFO - Checking for process beam.smp on linux [2021-02-22 22:43:56,347] - [remote_util] [140689256060672] - INFO - process beam.smp is running on 172.23.122.140: with pid 130244 [2021-02-22 22:43:56,347] - [basetestcase] [140689256060672] - INFO - Couchbase started [2021-02-22 22:43:56,351] - [remote_util] [140689256060672] - INFO - SSH Connecting to 172.23.122.138 with username:root, attempt#1 of 5 [2021-02-22 22:43:56,458] - [remote_util] [140689256060672] - INFO - SSH Connected to 172.23.122.138 as root [2021-02-22 22:43:56,748] - [remote_util] [140689256060672] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 22:43:57,053] - [remote_util] [140689256060672] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 22:43:57,067] - [remote_util] [140689256060672] - INFO - SSH Connecting to 172.23.122.138 with username:root, attempt#1 of 5 [2021-02-22 22:43:57,167] - [remote_util] [140689256060672] - INFO - SSH Connected to 172.23.122.138 as root [2021-02-22 22:43:57,449] - [remote_util] [140689256060672] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 22:43:57,750] - [remote_util] [140689256060672] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 22:43:57,812] - [remote_util] [140689256060672] - INFO - 172.23.122.138 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-22 22:43:57,914] - [remote_util] [140689256060672] - INFO - Running systemd command on this server [2021-02-22 22:43:57,914] - [remote_util] [140689256060672] - INFO - running command.raw on 172.23.122.138: systemctl stop couchbase-server.service [2021-02-22 22:43:59,560] - [remote_util] [140689256060672] - INFO - command executed successfully with root [2021-02-22 22:43:59,561] - [basetestcase] [140689256060672] - INFO - Couchbase stopped [2021-02-22 22:43:59,561] - [basetestcase] [140689256060672] - INFO - sleep for 5 secs. ... [2021-02-22 22:44:04,567] - [remote_util] [140689256060672] - INFO - running command.raw on 172.23.122.138: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-22 22:44:04,584] - [remote_util] [140689256060672] - INFO - command executed successfully with root [2021-02-22 22:44:04,584] - [remote_util] [140689256060672] - INFO - running command.raw on 172.23.122.138: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-22 22:44:04,638] - [remote_util] [140689256060672] - INFO - command executed successfully with root [2021-02-22 22:44:04,642] - [remote_util] [140689256060672] - INFO - SSH Connecting to 172.23.122.138 with username:root, attempt#1 of 5 [2021-02-22 22:44:04,754] - [remote_util] [140689256060672] - INFO - SSH Connected to 172.23.122.138 as root [2021-02-22 22:44:05,040] - [remote_util] [140689256060672] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 22:44:05,343] - [remote_util] [140689256060672] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 22:44:05,406] - [remote_util] [140689256060672] - INFO - 172.23.122.138 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-22 22:44:05,406] - [remote_util] [140689256060672] - INFO - Checking for process beam.smp on linux [2021-02-22 22:44:05,487] - [remote_util] [140689256060672] - INFO - Starting couchbase server [2021-02-22 22:44:05,591] - [remote_util] [140689256060672] - INFO - Running systemd command on this server [2021-02-22 22:44:05,591] - [remote_util] [140689256060672] - INFO - running command.raw on 172.23.122.138: systemctl start couchbase-server.service [2021-02-22 22:44:05,614] - [remote_util] [140689256060672] - INFO - command executed successfully with root [2021-02-22 22:44:05,615] - [remote_util] [140689256060672] - INFO - 172.23.122.138:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-22 22:44:10,620] - [remote_util] [140689256060672] - INFO - running command.raw on 172.23.122.138: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-22 22:44:10,649] - [remote_util] [140689256060672] - INFO - command executed successfully with root [2021-02-22 22:44:10,650] - [remote_util] [140689256060672] - INFO - Couchbase server status: [' Process: 86011 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-22 22:44:10,650] - [remote_util] [140689256060672] - INFO - Checking for process beam.smp on linux [2021-02-22 22:44:10,721] - [remote_util] [140689256060672] - INFO - process beam.smp is running on 172.23.122.138: with pid 86189 [2021-02-22 22:44:10,722] - [basetestcase] [140689256060672] - INFO - Couchbase started [2021-02-22 22:44:10,725] - [remote_util] [140689256060672] - INFO - SSH Connecting to 172.23.122.141 with username:root, attempt#1 of 5 [2021-02-22 22:44:10,832] - [remote_util] [140689256060672] - INFO - SSH Connected to 172.23.122.141 as root [2021-02-22 22:44:11,102] - [remote_util] [140689256060672] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 22:44:11,401] - [remote_util] [140689256060672] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 22:44:11,414] - [remote_util] [140689256060672] - INFO - SSH Connecting to 172.23.122.141 with username:root, attempt#1 of 5 [2021-02-22 22:44:11,511] - [remote_util] [140689256060672] - INFO - SSH Connected to 172.23.122.141 as root [2021-02-22 22:44:11,772] - [remote_util] [140689256060672] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 22:44:12,073] - [remote_util] [140689256060672] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 22:44:12,134] - [remote_util] [140689256060672] - INFO - 172.23.122.141 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-22 22:44:12,233] - [remote_util] [140689256060672] - INFO - Running systemd command on this server [2021-02-22 22:44:12,233] - [remote_util] [140689256060672] - INFO - running command.raw on 172.23.122.141: systemctl stop couchbase-server.service [2021-02-22 22:44:13,966] - [remote_util] [140689256060672] - INFO - command executed successfully with root [2021-02-22 22:44:13,966] - [basetestcase] [140689256060672] - INFO - Couchbase stopped [2021-02-22 22:44:13,967] - [basetestcase] [140689256060672] - INFO - sleep for 5 secs. ... [2021-02-22 22:44:18,972] - [remote_util] [140689256060672] - INFO - running command.raw on 172.23.122.141: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-22 22:44:18,990] - [remote_util] [140689256060672] - INFO - command executed successfully with root [2021-02-22 22:44:18,990] - [remote_util] [140689256060672] - INFO - running command.raw on 172.23.122.141: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-22 22:44:19,044] - [remote_util] [140689256060672] - INFO - command executed successfully with root [2021-02-22 22:44:19,047] - [remote_util] [140689256060672] - INFO - SSH Connecting to 172.23.122.141 with username:root, attempt#1 of 5 [2021-02-22 22:44:19,147] - [remote_util] [140689256060672] - INFO - SSH Connected to 172.23.122.141 as root [2021-02-22 22:44:19,414] - [remote_util] [140689256060672] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 22:44:19,713] - [remote_util] [140689256060672] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 22:44:19,772] - [remote_util] [140689256060672] - INFO - 172.23.122.141 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-22 22:44:19,773] - [remote_util] [140689256060672] - INFO - Checking for process beam.smp on linux [2021-02-22 22:44:19,849] - [remote_util] [140689256060672] - INFO - Starting couchbase server [2021-02-22 22:44:19,988] - [remote_util] [140689256060672] - INFO - Running systemd command on this server [2021-02-22 22:44:19,989] - [remote_util] [140689256060672] - INFO - running command.raw on 172.23.122.141: systemctl start couchbase-server.service [2021-02-22 22:44:20,010] - [remote_util] [140689256060672] - INFO - command executed successfully with root [2021-02-22 22:44:20,011] - [remote_util] [140689256060672] - INFO - 172.23.122.141:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-22 22:44:25,012] - [remote_util] [140689256060672] - INFO - running command.raw on 172.23.122.141: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-22 22:44:25,037] - [remote_util] [140689256060672] - INFO - command executed successfully with root [2021-02-22 22:44:25,037] - [remote_util] [140689256060672] - INFO - Couchbase server status: [' Process: 53878 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-22 22:44:25,038] - [remote_util] [140689256060672] - INFO - Checking for process beam.smp on linux [2021-02-22 22:44:25,103] - [remote_util] [140689256060672] - INFO - process beam.smp is running on 172.23.122.141: with pid 54024 [2021-02-22 22:44:25,103] - [basetestcase] [140689256060672] - INFO - Couchbase started [2021-02-22 22:44:25,105] - [basetestcase] [140689256060672] - INFO - sleep for 10 secs. ... [2021-02-22 22:44:35,570] - [task] [140688398202624] - INFO - server: ip:172.23.123.65 port:8091 ssh_username:root, nodes/self [2021-02-22 22:44:35,575] - [task] [140688398202624] - INFO - {'uptime': '110', 'memoryTotal': 4201672704, 'memoryFree': 3547799552, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.123.65:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4514-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.123.65', 'ip': '172.23.123.65', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} [2021-02-22 22:44:35,578] - [rest_client] [140688398202624] - ERROR - GET http://172.23.123.65:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-02-22 22:44:35,579] - [rest_client] [140688398202624] - INFO - pools/default params : memoryQuota=2147 [2021-02-22 22:44:35,586] - [rest_client] [140688398202624] - INFO - --> init_node_services(Administrator,password,172.23.123.65,8091,['kv']) [2021-02-22 22:44:35,586] - [rest_client] [140688398202624] - INFO - /node/controller/setupServices params on 172.23.123.65: 8091:hostname=172.23.123.65&user=Administrator&password=password&services=kv [2021-02-22 22:44:35,618] - [rest_client] [140688398202624] - INFO - --> in init_cluster...Administrator,password,8091 [2021-02-22 22:44:35,618] - [rest_client] [140688398202624] - INFO - settings/web params on 172.23.123.65:8091:port=8091&username=Administrator&password=password [2021-02-22 22:44:35,677] - [rest_client] [140688398202624] - INFO - --> status:True [2021-02-22 22:44:35,681] - [remote_util] [140688398202624] - INFO - SSH Connecting to 172.23.123.65 with username:root, attempt#1 of 5 [2021-02-22 22:44:35,790] - [remote_util] [140688398202624] - INFO - SSH Connected to 172.23.123.65 as root [2021-02-22 22:44:36,091] - [remote_util] [140688398202624] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 22:44:36,404] - [remote_util] [140688398202624] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 22:44:36,415] - [remote_util] [140688398202624] - INFO - running command.raw on 172.23.123.65: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-02-22 22:44:36,471] - [remote_util] [140688398202624] - INFO - command executed successfully with root [2021-02-22 22:44:36,472] - [remote_util] [140688398202624] - INFO - ['ok'] [2021-02-22 22:44:36,475] - [rest_client] [140688398202624] - INFO - /diag/eval status on 172.23.123.65:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-22 22:44:36,478] - [rest_client] [140688398202624] - INFO - /diag/eval status on 172.23.123.65:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-22 22:44:36,483] - [rest_client] [140688398202624] - INFO - settings/indexes params : storageMode=plasma [2021-02-22 22:44:36,510] - [task] [140688398202624] - INFO - server: ip:172.23.122.182 port:8091 ssh_username:root, nodes/self [2021-02-22 22:44:36,515] - [task] [140688398202624] - INFO - {'uptime': '100', 'memoryTotal': 4201684992, 'memoryFree': 3571433472, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.122.182:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4514-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.122.182', 'ip': '172.23.122.182', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} [2021-02-22 22:44:36,518] - [rest_client] [140688398202624] - ERROR - GET http://172.23.122.182:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-02-22 22:44:36,519] - [rest_client] [140688398202624] - INFO - pools/default params : memoryQuota=2147 [2021-02-22 22:44:36,533] - [rest_client] [140688398202624] - INFO - --> in init_cluster...Administrator,password,8091 [2021-02-22 22:44:36,534] - [rest_client] [140688398202624] - INFO - settings/web params on 172.23.122.182:8091:port=8091&username=Administrator&password=password [2021-02-22 22:44:36,589] - [rest_client] [140688398202624] - INFO - --> status:True [2021-02-22 22:44:36,593] - [remote_util] [140688398202624] - INFO - SSH Connecting to 172.23.122.182 with username:root, attempt#1 of 5 [2021-02-22 22:44:36,699] - [remote_util] [140688398202624] - INFO - SSH Connected to 172.23.122.182 as root [2021-02-22 22:44:36,961] - [remote_util] [140688398202624] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 22:44:37,310] - [remote_util] [140688398202624] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 22:44:37,322] - [remote_util] [140688398202624] - INFO - running command.raw on 172.23.122.182: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-02-22 22:44:37,376] - [remote_util] [140688398202624] - INFO - command executed successfully with root [2021-02-22 22:44:37,377] - [remote_util] [140688398202624] - INFO - ['ok'] [2021-02-22 22:44:37,380] - [rest_client] [140688398202624] - INFO - /diag/eval status on 172.23.122.182:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-22 22:44:37,383] - [rest_client] [140688398202624] - INFO - /diag/eval status on 172.23.122.182:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-22 22:44:37,388] - [rest_client] [140688398202624] - INFO - settings/indexes params : storageMode=plasma [2021-02-22 22:44:37,410] - [task] [140688398202624] - INFO - server: ip:172.23.122.179 port:8091 ssh_username:root, nodes/self [2021-02-22 22:44:37,414] - [task] [140688398202624] - INFO - {'uptime': '85', 'memoryTotal': 4201684992, 'memoryFree': 3607879680, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.122.179:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4514-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.122.179', 'ip': '172.23.122.179', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} [2021-02-22 22:44:37,417] - [rest_client] [140688398202624] - ERROR - GET http://172.23.122.179:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-02-22 22:44:37,418] - [rest_client] [140688398202624] - INFO - pools/default params : memoryQuota=2147 [2021-02-22 22:44:37,425] - [rest_client] [140688398202624] - INFO - --> in init_cluster...Administrator,password,8091 [2021-02-22 22:44:37,425] - [rest_client] [140688398202624] - INFO - settings/web params on 172.23.122.179:8091:port=8091&username=Administrator&password=password [2021-02-22 22:44:37,479] - [rest_client] [140688398202624] - INFO - --> status:True [2021-02-22 22:44:37,482] - [remote_util] [140688398202624] - INFO - SSH Connecting to 172.23.122.179 with username:root, attempt#1 of 5 [2021-02-22 22:44:37,592] - [remote_util] [140688398202624] - INFO - SSH Connected to 172.23.122.179 as root [2021-02-22 22:44:37,878] - [remote_util] [140688398202624] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 22:44:38,185] - [remote_util] [140688398202624] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 22:44:38,196] - [remote_util] [140688398202624] - INFO - running command.raw on 172.23.122.179: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-02-22 22:44:38,251] - [remote_util] [140688398202624] - INFO - command executed successfully with root [2021-02-22 22:44:38,251] - [remote_util] [140688398202624] - INFO - ['ok'] [2021-02-22 22:44:38,255] - [rest_client] [140688398202624] - INFO - /diag/eval status on 172.23.122.179:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-22 22:44:38,257] - [rest_client] [140688398202624] - INFO - /diag/eval status on 172.23.122.179:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-22 22:44:38,264] - [rest_client] [140688398202624] - INFO - settings/indexes params : storageMode=plasma [2021-02-22 22:44:38,283] - [task] [140688398202624] - INFO - server: ip:172.23.122.153 port:8091 ssh_username:root, nodes/self [2021-02-22 22:44:38,288] - [task] [140688398202624] - INFO - {'uptime': '69', 'memoryTotal': 4201684992, 'memoryFree': 3619991552, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.122.153:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4514-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.122.153', 'ip': '172.23.122.153', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} [2021-02-22 22:44:38,290] - [rest_client] [140688398202624] - ERROR - GET http://172.23.122.153:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-02-22 22:44:38,291] - [rest_client] [140688398202624] - INFO - pools/default params : memoryQuota=2147 [2021-02-22 22:44:38,297] - [rest_client] [140688398202624] - INFO - --> in init_cluster...Administrator,password,8091 [2021-02-22 22:44:38,298] - [rest_client] [140688398202624] - INFO - settings/web params on 172.23.122.153:8091:port=8091&username=Administrator&password=password [2021-02-22 22:44:38,352] - [rest_client] [140688398202624] - INFO - --> status:True [2021-02-22 22:44:38,355] - [remote_util] [140688398202624] - INFO - SSH Connecting to 172.23.122.153 with username:root, attempt#1 of 5 [2021-02-22 22:44:38,460] - [remote_util] [140688398202624] - INFO - SSH Connected to 172.23.122.153 as root [2021-02-22 22:44:38,740] - [remote_util] [140688398202624] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 22:44:39,042] - [remote_util] [140688398202624] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 22:44:39,053] - [remote_util] [140688398202624] - INFO - running command.raw on 172.23.122.153: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-02-22 22:44:39,110] - [remote_util] [140688398202624] - INFO - command executed successfully with root [2021-02-22 22:44:39,111] - [remote_util] [140688398202624] - INFO - ['ok'] [2021-02-22 22:44:39,115] - [rest_client] [140688398202624] - INFO - /diag/eval status on 172.23.122.153:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-22 22:44:39,119] - [rest_client] [140688398202624] - INFO - /diag/eval status on 172.23.122.153:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-22 22:44:39,124] - [rest_client] [140688398202624] - INFO - settings/indexes params : storageMode=plasma [2021-02-22 22:44:39,146] - [task] [140688398202624] - INFO - server: ip:172.23.122.139 port:8091 ssh_username:root, nodes/self [2021-02-22 22:44:39,150] - [task] [140688398202624] - INFO - {'uptime': '59', 'memoryTotal': 4201684992, 'memoryFree': 3594244096, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.122.139:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4514-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.122.139', 'ip': '172.23.122.139', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} [2021-02-22 22:44:39,153] - [rest_client] [140688398202624] - ERROR - GET http://172.23.122.139:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-02-22 22:44:39,154] - [rest_client] [140688398202624] - INFO - pools/default params : memoryQuota=2147 [2021-02-22 22:44:39,161] - [rest_client] [140688398202624] - INFO - --> in init_cluster...Administrator,password,8091 [2021-02-22 22:44:39,161] - [rest_client] [140688398202624] - INFO - settings/web params on 172.23.122.139:8091:port=8091&username=Administrator&password=password [2021-02-22 22:44:39,222] - [rest_client] [140688398202624] - INFO - --> status:True [2021-02-22 22:44:39,225] - [remote_util] [140688398202624] - INFO - SSH Connecting to 172.23.122.139 with username:root, attempt#1 of 5 [2021-02-22 22:44:39,331] - [remote_util] [140688398202624] - INFO - SSH Connected to 172.23.122.139 as root [2021-02-22 22:44:39,610] - [remote_util] [140688398202624] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 22:44:39,911] - [remote_util] [140688398202624] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 22:44:39,923] - [remote_util] [140688398202624] - INFO - running command.raw on 172.23.122.139: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-02-22 22:44:39,976] - [remote_util] [140688398202624] - INFO - command executed successfully with root [2021-02-22 22:44:39,977] - [remote_util] [140688398202624] - INFO - ['ok'] [2021-02-22 22:44:39,981] - [rest_client] [140688398202624] - INFO - /diag/eval status on 172.23.122.139:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-22 22:44:39,987] - [rest_client] [140688398202624] - INFO - /diag/eval status on 172.23.122.139:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-22 22:44:39,992] - [rest_client] [140688398202624] - INFO - settings/indexes params : storageMode=plasma [2021-02-22 22:44:40,014] - [task] [140688398202624] - INFO - server: ip:172.23.122.140 port:8091 ssh_username:root, nodes/self [2021-02-22 22:44:40,018] - [task] [140688398202624] - INFO - {'uptime': '44', 'memoryTotal': 4201684992, 'memoryFree': 3592495104, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.122.140:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4514-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.122.140', 'ip': '172.23.122.140', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} [2021-02-22 22:44:40,021] - [rest_client] [140688398202624] - ERROR - GET http://172.23.122.140:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-02-22 22:44:40,022] - [rest_client] [140688398202624] - INFO - pools/default params : memoryQuota=2147 [2021-02-22 22:44:40,028] - [rest_client] [140688398202624] - INFO - --> in init_cluster...Administrator,password,8091 [2021-02-22 22:44:40,029] - [rest_client] [140688398202624] - INFO - settings/web params on 172.23.122.140:8091:port=8091&username=Administrator&password=password [2021-02-22 22:44:40,084] - [rest_client] [140688398202624] - INFO - --> status:True [2021-02-22 22:44:40,087] - [remote_util] [140688398202624] - INFO - SSH Connecting to 172.23.122.140 with username:root, attempt#1 of 5 [2021-02-22 22:44:40,191] - [remote_util] [140688398202624] - INFO - SSH Connected to 172.23.122.140 as root [2021-02-22 22:44:40,476] - [remote_util] [140688398202624] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 22:44:40,779] - [remote_util] [140688398202624] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 22:44:40,791] - [remote_util] [140688398202624] - INFO - running command.raw on 172.23.122.140: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-02-22 22:44:40,844] - [remote_util] [140688398202624] - INFO - command executed successfully with root [2021-02-22 22:44:40,844] - [remote_util] [140688398202624] - INFO - ['ok'] [2021-02-22 22:44:40,848] - [rest_client] [140688398202624] - INFO - /diag/eval status on 172.23.122.140:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-22 22:44:40,851] - [rest_client] [140688398202624] - INFO - /diag/eval status on 172.23.122.140:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-22 22:44:40,856] - [rest_client] [140688398202624] - INFO - settings/indexes params : storageMode=plasma [2021-02-22 22:44:40,876] - [task] [140688398202624] - INFO - server: ip:172.23.122.138 port:8091 ssh_username:root, nodes/self [2021-02-22 22:44:40,880] - [task] [140688398202624] - INFO - {'uptime': '29', 'memoryTotal': 4201684992, 'memoryFree': 3592097792, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.122.138:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4514-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.122.138', 'ip': '172.23.122.138', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} [2021-02-22 22:44:40,883] - [rest_client] [140688398202624] - ERROR - GET http://172.23.122.138:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-02-22 22:44:40,884] - [rest_client] [140688398202624] - INFO - pools/default params : memoryQuota=2147 [2021-02-22 22:44:40,890] - [rest_client] [140688398202624] - INFO - --> in init_cluster...Administrator,password,8091 [2021-02-22 22:44:40,891] - [rest_client] [140688398202624] - INFO - settings/web params on 172.23.122.138:8091:port=8091&username=Administrator&password=password [2021-02-22 22:44:40,946] - [rest_client] [140688398202624] - INFO - --> status:True [2021-02-22 22:44:40,949] - [remote_util] [140688398202624] - INFO - SSH Connecting to 172.23.122.138 with username:root, attempt#1 of 5 [2021-02-22 22:44:41,053] - [remote_util] [140688398202624] - INFO - SSH Connected to 172.23.122.138 as root [2021-02-22 22:44:41,346] - [remote_util] [140688398202624] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 22:44:41,661] - [remote_util] [140688398202624] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 22:44:41,674] - [remote_util] [140688398202624] - INFO - running command.raw on 172.23.122.138: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-02-22 22:44:41,728] - [remote_util] [140688398202624] - INFO - command executed successfully with root [2021-02-22 22:44:41,728] - [remote_util] [140688398202624] - INFO - ['ok'] [2021-02-22 22:44:41,732] - [rest_client] [140688398202624] - INFO - /diag/eval status on 172.23.122.138:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-22 22:44:41,734] - [rest_client] [140688398202624] - INFO - /diag/eval status on 172.23.122.138:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-22 22:44:41,739] - [rest_client] [140688398202624] - INFO - settings/indexes params : storageMode=plasma [2021-02-22 22:44:41,762] - [task] [140688398202624] - INFO - server: ip:172.23.122.141 port:8091 ssh_username:root, nodes/self [2021-02-22 22:44:41,765] - [task] [140688398202624] - INFO - {'uptime': '19', 'memoryTotal': 4201689088, 'memoryFree': 3616051200, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.122.141:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4514-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@cb.local', 'ip': '172.23.122.141', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} [2021-02-22 22:44:41,768] - [rest_client] [140688398202624] - ERROR - GET http://172.23.122.141:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-02-22 22:44:41,769] - [rest_client] [140688398202624] - INFO - pools/default params : memoryQuota=2147 [2021-02-22 22:44:41,776] - [rest_client] [140688398202624] - INFO - --> in init_cluster...Administrator,password,8091 [2021-02-22 22:44:41,777] - [rest_client] [140688398202624] - INFO - settings/web params on 172.23.122.141:8091:port=8091&username=Administrator&password=password [2021-02-22 22:44:41,832] - [rest_client] [140688398202624] - INFO - --> status:True [2021-02-22 22:44:41,835] - [remote_util] [140688398202624] - INFO - SSH Connecting to 172.23.122.141 with username:root, attempt#1 of 5 [2021-02-22 22:44:41,936] - [remote_util] [140688398202624] - INFO - SSH Connected to 172.23.122.141 as root [2021-02-22 22:44:42,205] - [remote_util] [140688398202624] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 22:44:42,544] - [remote_util] [140688398202624] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 22:44:42,554] - [remote_util] [140688398202624] - INFO - running command.raw on 172.23.122.141: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-02-22 22:44:42,610] - [remote_util] [140688398202624] - INFO - command executed successfully with root [2021-02-22 22:44:42,610] - [remote_util] [140688398202624] - INFO - ['ok'] [2021-02-22 22:44:42,614] - [rest_client] [140688398202624] - INFO - /diag/eval status on 172.23.122.141:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-22 22:44:42,617] - [rest_client] [140688398202624] - INFO - /diag/eval status on 172.23.122.141:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-22 22:44:42,621] - [rest_client] [140688398202624] - INFO - settings/indexes params : storageMode=plasma [2021-02-22 22:44:42,648] - [basetestcase] [140689256060672] - INFO - **** add built-in 'cbadminbucket' user to node 172.23.123.65 **** [2021-02-22 22:44:42,664] - [rest_client] [140689256060672] - ERROR - DELETE http://172.23.123.65:8091/settings/rbac/users/local/cbadminbucket body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:password [2021-02-22 22:44:42,665] - [internal_user] [140689256060672] - INFO - Exception while deleting user. Exception is -b'"User was not found."' [2021-02-22 22:44:42,729] - [basetestcase] [140689256060672] - INFO - sleep for 5 secs. ... [2021-02-22 22:44:47,734] - [basetestcase] [140689256060672] - INFO - **** add 'admin' role to 'cbadminbucket' user **** [2021-02-22 22:44:47,749] - [basetestcase] [140689256060672] - INFO - done initializing cluster [2021-02-22 22:44:48,645] - [task] [140688398202624] - INFO - adding node 172.23.122.182:8091 to cluster [2021-02-22 22:44:48,645] - [rest_client] [140688398202624] - INFO - adding remote node @172.23.122.182:8091 to this cluster @172.23.123.65:8091 [2021-02-22 22:44:58,663] - [rest_client] [140688398202624] - INFO - rebalance progress took 10.02 seconds [2021-02-22 22:44:58,664] - [rest_client] [140688398202624] - INFO - sleep for 10 seconds after rebalance... [2021-02-22 22:45:10,869] - [task] [140688398202624] - INFO - adding node 172.23.122.179:8091 to cluster [2021-02-22 22:45:10,869] - [rest_client] [140688398202624] - INFO - adding remote node @172.23.122.179:8091 to this cluster @172.23.123.65:8091 [2021-02-22 22:45:20,886] - [rest_client] [140688398202624] - INFO - rebalance progress took 10.02 seconds [2021-02-22 22:45:20,886] - [rest_client] [140688398202624] - INFO - sleep for 10 seconds after rebalance... [2021-02-22 22:45:33,054] - [task] [140688398202624] - INFO - adding node 172.23.122.153:8091 to cluster [2021-02-22 22:45:33,055] - [rest_client] [140688398202624] - INFO - adding remote node @172.23.122.153:8091 to this cluster @172.23.123.65:8091 [2021-02-22 22:45:43,072] - [rest_client] [140688398202624] - INFO - rebalance progress took 10.02 seconds [2021-02-22 22:45:43,072] - [rest_client] [140688398202624] - INFO - sleep for 10 seconds after rebalance... [2021-02-22 22:45:55,638] - [task] [140688398202624] - INFO - adding node 172.23.122.139:8091 to cluster [2021-02-22 22:45:55,639] - [rest_client] [140688398202624] - INFO - adding remote node @172.23.122.139:8091 to this cluster @172.23.123.65:8091 [2021-02-22 22:46:05,656] - [rest_client] [140688398202624] - INFO - rebalance progress took 10.02 seconds [2021-02-22 22:46:05,656] - [rest_client] [140688398202624] - INFO - sleep for 10 seconds after rebalance... [2021-02-22 22:46:18,249] - [rest_client] [140688398202624] - INFO - rebalance params : {'knownNodes': 'ns_1@172.23.122.139,ns_1@172.23.122.153,ns_1@172.23.122.179,ns_1@172.23.122.182,ns_1@172.23.123.65', 'ejectedNodes': '', 'user': 'Administrator', 'password': 'password'} [2021-02-22 22:46:18,278] - [rest_client] [140688398202624] - INFO - rebalance operation started [2021-02-22 22:46:18,282] - [rest_client] [140688398202624] - INFO - rebalance percentage : 0.00 % [2021-02-22 22:46:18,282] - [task] [140688398202624] - INFO - Rebalance - status: running, progress: 0.00% [2021-02-22 22:46:28,304] - [task] [140688398202624] - INFO - Rebalance - status: none, progress: 100.00% [2021-02-22 22:46:28,312] - [task] [140688398202624] - INFO - rebalancing was completed with progress: 100% in 10.033611536026001 sec [2021-02-22 22:46:28,316] - [remote_util] [140689256060672] - INFO - SSH Connecting to 172.23.123.65 with username:root, attempt#1 of 5 [2021-02-22 22:46:28,421] - [remote_util] [140689256060672] - INFO - SSH Connected to 172.23.123.65 as root [2021-02-22 22:46:28,715] - [remote_util] [140689256060672] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 22:46:29,026] - [remote_util] [140689256060672] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 22:46:29,036] - [remote_util] [140689256060672] - INFO - running command.raw on 172.23.123.65: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-02-22 22:46:29,090] - [remote_util] [140689256060672] - INFO - command executed successfully with root [2021-02-22 22:46:29,091] - [remote_util] [140689256060672] - INFO - ['ok'] [2021-02-22 22:46:29,091] - [basetestcase] [140689256060672] - INFO - Enabled diag/eval for non-local hosts from 172.23.123.65 [2021-02-22 22:46:29,091] - [basetestcase] [140689256060672] - INFO - ============== basetestcase setup was finished for test #14 test_eventing_rebalance_swap_delete_recreate_collections ============== [2021-02-22 22:46:29,104] - [remote_util] [140689256060672] - INFO - SSH Connecting to 172.23.123.65 with username:root, attempt#1 of 5 [2021-02-22 22:46:29,208] - [remote_util] [140689256060672] - INFO - SSH Connected to 172.23.123.65 as root [2021-02-22 22:46:29,495] - [remote_util] [140689256060672] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 22:46:29,810] - [remote_util] [140689256060672] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 22:46:29,814] - [remote_util] [140689256060672] - INFO - SSH Connecting to 172.23.122.182 with username:root, attempt#1 of 5 [2021-02-22 22:46:29,912] - [remote_util] [140689256060672] - INFO - SSH Connected to 172.23.122.182 as root [2021-02-22 22:46:30,161] - [remote_util] [140689256060672] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 22:46:30,455] - [remote_util] [140689256060672] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 22:46:30,459] - [remote_util] [140689256060672] - INFO - SSH Connecting to 172.23.122.179 with username:root, attempt#1 of 5 [2021-02-22 22:46:30,561] - [remote_util] [140689256060672] - INFO - SSH Connected to 172.23.122.179 as root [2021-02-22 22:46:30,836] - [remote_util] [140689256060672] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 22:46:31,136] - [remote_util] [140689256060672] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 22:46:31,140] - [remote_util] [140689256060672] - INFO - SSH Connecting to 172.23.122.153 with username:root, attempt#1 of 5 [2021-02-22 22:46:31,245] - [remote_util] [140689256060672] - INFO - SSH Connected to 172.23.122.153 as root [2021-02-22 22:46:31,517] - [remote_util] [140689256060672] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 22:46:31,856] - [remote_util] [140689256060672] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 22:46:31,861] - [remote_util] [140689256060672] - INFO - SSH Connecting to 172.23.122.139 with username:root, attempt#1 of 5 [2021-02-22 22:46:31,964] - [remote_util] [140689256060672] - INFO - SSH Connected to 172.23.122.139 as root [2021-02-22 22:46:32,237] - [remote_util] [140689256060672] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 22:46:32,542] - [remote_util] [140689256060672] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 22:46:32,546] - [remote_util] [140689256060672] - INFO - SSH Connecting to 172.23.122.140 with username:root, attempt#1 of 5 [2021-02-22 22:46:32,649] - [remote_util] [140689256060672] - INFO - SSH Connected to 172.23.122.140 as root [2021-02-22 22:46:32,928] - [remote_util] [140689256060672] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 22:46:33,269] - [remote_util] [140689256060672] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 22:46:33,273] - [remote_util] [140689256060672] - INFO - SSH Connecting to 172.23.122.138 with username:root, attempt#1 of 5 [2021-02-22 22:46:33,377] - [remote_util] [140689256060672] - INFO - SSH Connected to 172.23.122.138 as root [2021-02-22 22:46:33,667] - [remote_util] [140689256060672] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 22:46:33,970] - [remote_util] [140689256060672] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 22:46:33,976] - [remote_util] [140689256060672] - INFO - SSH Connecting to 172.23.122.141 with username:root, attempt#1 of 5 [2021-02-22 22:46:34,076] - [remote_util] [140689256060672] - INFO - SSH Connected to 172.23.122.141 as root [2021-02-22 22:46:34,343] - [remote_util] [140689256060672] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 22:46:34,641] - [remote_util] [140689256060672] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 22:48:21,530] - [basetestcase] [140689256060672] - INFO - ------- Cluster statistics ------- [2021-02-22 22:48:21,531] - [basetestcase] [140689256060672] - INFO - 172.23.122.139:8091 => {'services': ['n1ql'], 'cpu_utilization': 0.5022601707684581, 'mem_free': 3541811200, 'mem_total': 4201684992, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-02-22 22:48:21,531] - [basetestcase] [140689256060672] - INFO - 172.23.122.153:8091 => {'services': ['index'], 'cpu_utilization': 0.6770310932798396, 'mem_free': 3565051904, 'mem_total': 4201684992, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-02-22 22:48:21,532] - [basetestcase] [140689256060672] - INFO - 172.23.122.179:8091 => {'services': ['eventing'], 'cpu_utilization': 0.7014028056112225, 'mem_free': 3568857088, 'mem_total': 4201684992, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-02-22 22:48:21,532] - [basetestcase] [140689256060672] - INFO - 172.23.122.182:8091 => {'services': ['eventing'], 'cpu_utilization': 0.6526104417670683, 'mem_free': 3535351808, 'mem_total': 4201684992, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-02-22 22:48:21,532] - [basetestcase] [140689256060672] - INFO - 172.23.123.65:8091 => {'services': ['kv'], 'cpu_utilization': 0.7532011046949536, 'mem_free': 3507343360, 'mem_total': 4201672704, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-02-22 22:48:21,533] - [basetestcase] [140689256060672] - INFO - --- End of cluster statistics --- [2021-02-22 22:48:21,537] - [remote_util] [140689256060672] - INFO - SSH Connecting to 172.23.123.65 with username:root, attempt#1 of 5 [2021-02-22 22:48:21,644] - [remote_util] [140689256060672] - INFO - SSH Connected to 172.23.123.65 as root [2021-02-22 22:48:21,944] - [remote_util] [140689256060672] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 22:48:22,260] - [remote_util] [140689256060672] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 22:48:28,579] - [basetestcase] [140689256060672] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.139 port:8091 ssh_username:root] [2021-02-22 22:48:28,588] - [basetestcase] [140689256060672] - INFO - list of eventing nodes in cluster: [ip:172.23.122.179 port:8091 ssh_username:root, ip:172.23.122.182 port:8091 ssh_username:root] [2021-02-22 22:48:28,595] - [rest_client] [140689256060672] - INFO - settings/indexes params : storageMode=plasma [2021-02-22 22:48:28,604] - [eventing_base] [140689256060672] - INFO - Setting the min possible memory quota so that adding mode nodes to the cluster wouldn't be a problem. [2021-02-22 22:48:28,604] - [rest_client] [140689256060672] - INFO - pools/default params : memoryQuota=330 [2021-02-22 22:48:28,611] - [rest_client] [140689256060672] - INFO - pools/default params : indexMemoryQuota=256 [2021-02-22 22:48:28,617] - [rest_client] [140689256060672] - INFO - pools/default params : eventingMemoryQuota=512 [2021-02-22 22:48:28,638] - [basetestcase] [140689256060672] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.139 port:8091 ssh_username:root] [2021-02-22 22:48:28,643] - [rest_client] [140689256060672] - INFO - pools/default params : memoryQuota=700 [2021-02-22 22:48:28,651] - [eventing_rebalance_collection] [140689256060672] - INFO - 200 [2021-02-22 22:48:28,902] - [rest_client] [140688398202624] - INFO - http://172.23.123.65:8091/pools/default/buckets with param: name=src_bucket&ramQuotaMB=200&authType=none&replicaNumber=0&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore [2021-02-22 22:48:28,930] - [rest_client] [140688398202624] - INFO - 0.03 seconds to create bucket src_bucket [2021-02-22 22:48:28,931] - [bucket_helper] [140688398202624] - INFO - waiting for memcached bucket : src_bucket in 172.23.123.65 to accept set ops [2021-02-22 22:48:29,611] - [data_helper] [140688398202624] - INFO - creating direct client 172.23.123.65:11210 src_bucket [2021-02-22 22:48:30,063] - [data_helper] [140688398202624] - INFO - creating direct client 172.23.123.65:11210 src_bucket [2021-02-22 22:48:30,151] - [task] [140688398202624] - WARNING - vbucket map not ready after try 0 [2021-02-22 22:48:30,152] - [bucket_helper] [140688398202624] - INFO - waiting for memcached bucket : src_bucket in 172.23.123.65 to accept set ops [2021-02-22 22:48:30,301] - [data_helper] [140688398202624] - INFO - creating direct client 172.23.123.65:11210 src_bucket [2021-02-22 22:48:30,457] - [data_helper] [140688398202624] - INFO - creating direct client 172.23.123.65:11210 src_bucket [2021-02-22 22:48:30,879] - [task] [140688398202624] - WARNING - vbucket map not ready after try 1 [2021-02-22 22:48:30,879] - [bucket_helper] [140688398202624] - INFO - waiting for memcached bucket : src_bucket in 172.23.123.65 to accept set ops [2021-02-22 22:48:31,031] - [data_helper] [140688398202624] - INFO - creating direct client 172.23.123.65:11210 src_bucket [2021-02-22 22:48:31,186] - [data_helper] [140688398202624] - INFO - creating direct client 172.23.123.65:11210 src_bucket [2021-02-22 22:48:31,271] - [task] [140688398202624] - WARNING - vbucket map not ready after try 2 [2021-02-22 22:48:31,272] - [bucket_helper] [140688398202624] - INFO - waiting for memcached bucket : src_bucket in 172.23.123.65 to accept set ops [2021-02-22 22:48:31,424] - [data_helper] [140688398202624] - INFO - creating direct client 172.23.123.65:11210 src_bucket [2021-02-22 22:48:31,910] - [data_helper] [140688398202624] - INFO - creating direct client 172.23.123.65:11210 src_bucket [2021-02-22 22:48:32,007] - [task] [140688398202624] - WARNING - vbucket map not ready after try 3 [2021-02-22 22:48:32,007] - [bucket_helper] [140688398202624] - INFO - waiting for memcached bucket : src_bucket in 172.23.123.65 to accept set ops [2021-02-22 22:48:32,150] - [data_helper] [140688398202624] - INFO - creating direct client 172.23.123.65:11210 src_bucket [2021-02-22 22:48:32,298] - [data_helper] [140688398202624] - INFO - creating direct client 172.23.123.65:11210 src_bucket [2021-02-22 22:48:32,386] - [task] [140688398202624] - WARNING - vbucket map not ready after try 4 [2021-02-22 22:48:32,387] - [bucket_helper] [140688398202624] - INFO - waiting for memcached bucket : src_bucket in 172.23.123.65 to accept set ops [2021-02-22 22:48:32,859] - [data_helper] [140688398202624] - INFO - creating direct client 172.23.123.65:11210 src_bucket [2021-02-22 22:48:33,027] - [data_helper] [140688398202624] - INFO - creating direct client 172.23.123.65:11210 src_bucket [2021-02-22 22:48:33,117] - [task] [140688398202624] - WARNING - vbucket map not ready after try 5 [2021-02-22 22:48:34,148] - [rest_client] [140688398202624] - INFO - http://172.23.123.65:8091/pools/default/buckets with param: name=dst_bucket&ramQuotaMB=200&authType=none&replicaNumber=0&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore [2021-02-22 22:48:34,178] - [rest_client] [140688398202624] - INFO - 0.03 seconds to create bucket dst_bucket [2021-02-22 22:48:34,178] - [bucket_helper] [140688398202624] - INFO - waiting for memcached bucket : dst_bucket in 172.23.123.65 to accept set ops [2021-02-22 22:48:34,880] - [data_helper] [140688398202624] - INFO - creating direct client 172.23.123.65:11210 dst_bucket [2021-02-22 22:48:35,041] - [data_helper] [140688398202624] - INFO - creating direct client 172.23.123.65:11210 dst_bucket [2021-02-22 22:48:35,473] - [task] [140688398202624] - WARNING - vbucket map not ready after try 0 [2021-02-22 22:48:35,474] - [bucket_helper] [140688398202624] - INFO - waiting for memcached bucket : dst_bucket in 172.23.123.65 to accept set ops [2021-02-22 22:48:35,612] - [data_helper] [140688398202624] - INFO - creating direct client 172.23.123.65:11210 dst_bucket [2021-02-22 22:48:35,746] - [data_helper] [140688398202624] - INFO - creating direct client 172.23.123.65:11210 dst_bucket [2021-02-22 22:48:35,833] - [task] [140688398202624] - WARNING - vbucket map not ready after try 1 [2021-02-22 22:48:35,833] - [bucket_helper] [140688398202624] - INFO - waiting for memcached bucket : dst_bucket in 172.23.123.65 to accept set ops [2021-02-22 22:48:35,965] - [data_helper] [140688398202624] - INFO - creating direct client 172.23.123.65:11210 dst_bucket [2021-02-22 22:48:36,422] - [data_helper] [140688398202624] - INFO - creating direct client 172.23.123.65:11210 dst_bucket [2021-02-22 22:48:36,504] - [task] [140688398202624] - WARNING - vbucket map not ready after try 2 [2021-02-22 22:48:36,505] - [bucket_helper] [140688398202624] - INFO - waiting for memcached bucket : dst_bucket in 172.23.123.65 to accept set ops [2021-02-22 22:48:36,639] - [data_helper] [140688398202624] - INFO - creating direct client 172.23.123.65:11210 dst_bucket [2021-02-22 22:48:36,776] - [data_helper] [140688398202624] - INFO - creating direct client 172.23.123.65:11210 dst_bucket [2021-02-22 22:48:36,862] - [task] [140688398202624] - WARNING - vbucket map not ready after try 3 [2021-02-22 22:48:36,862] - [bucket_helper] [140688398202624] - INFO - waiting for memcached bucket : dst_bucket in 172.23.123.65 to accept set ops [2021-02-22 22:48:37,306] - [data_helper] [140688398202624] - INFO - creating direct client 172.23.123.65:11210 dst_bucket [2021-02-22 22:48:37,453] - [data_helper] [140688398202624] - INFO - creating direct client 172.23.123.65:11210 dst_bucket [2021-02-22 22:48:37,538] - [task] [140688398202624] - WARNING - vbucket map not ready after try 4 [2021-02-22 22:48:37,539] - [bucket_helper] [140688398202624] - INFO - waiting for memcached bucket : dst_bucket in 172.23.123.65 to accept set ops [2021-02-22 22:48:37,679] - [data_helper] [140688398202624] - INFO - creating direct client 172.23.123.65:11210 dst_bucket [2021-02-22 22:48:37,820] - [data_helper] [140688398202624] - INFO - creating direct client 172.23.123.65:11210 dst_bucket [2021-02-22 22:48:38,226] - [task] [140688398202624] - WARNING - vbucket map not ready after try 5 [2021-02-22 22:48:39,258] - [rest_client] [140688398202624] - INFO - http://172.23.123.65:8091/pools/default/buckets with param: name=metadata&ramQuotaMB=200&authType=none&replicaNumber=0&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore [2021-02-22 22:48:39,288] - [rest_client] [140688398202624] - INFO - 0.03 seconds to create bucket metadata [2021-02-22 22:48:39,289] - [bucket_helper] [140688398202624] - INFO - waiting for memcached bucket : metadata in 172.23.123.65 to accept set ops [2021-02-22 22:48:39,947] - [data_helper] [140688398202624] - INFO - creating direct client 172.23.123.65:11210 metadata [2021-02-22 22:48:40,084] - [data_helper] [140688398202624] - INFO - creating direct client 172.23.123.65:11210 metadata [2021-02-22 22:48:40,203] - [task] [140688398202624] - WARNING - vbucket map not ready after try 0 [2021-02-22 22:48:40,203] - [bucket_helper] [140688398202624] - INFO - waiting for memcached bucket : metadata in 172.23.123.65 to accept set ops [2021-02-22 22:48:40,342] - [data_helper] [140688398202624] - INFO - creating direct client 172.23.123.65:11210 metadata [2021-02-22 22:48:40,801] - [data_helper] [140688398202624] - INFO - creating direct client 172.23.123.65:11210 metadata [2021-02-22 22:48:40,887] - [task] [140688398202624] - WARNING - vbucket map not ready after try 1 [2021-02-22 22:48:40,887] - [bucket_helper] [140688398202624] - INFO - waiting for memcached bucket : metadata in 172.23.123.65 to accept set ops [2021-02-22 22:48:41,044] - [data_helper] [140688398202624] - INFO - creating direct client 172.23.123.65:11210 metadata [2021-02-22 22:48:41,193] - [data_helper] [140688398202624] - INFO - creating direct client 172.23.123.65:11210 metadata [2021-02-22 22:48:41,281] - [task] [140688398202624] - WARNING - vbucket map not ready after try 2 [2021-02-22 22:48:41,282] - [bucket_helper] [140688398202624] - INFO - waiting for memcached bucket : metadata in 172.23.123.65 to accept set ops [2021-02-22 22:48:41,734] - [data_helper] [140688398202624] - INFO - creating direct client 172.23.123.65:11210 metadata [2021-02-22 22:48:41,878] - [data_helper] [140688398202624] - INFO - creating direct client 172.23.123.65:11210 metadata [2021-02-22 22:48:41,967] - [task] [140688398202624] - WARNING - vbucket map not ready after try 3 [2021-02-22 22:48:41,967] - [bucket_helper] [140688398202624] - INFO - waiting for memcached bucket : metadata in 172.23.123.65 to accept set ops [2021-02-22 22:48:42,116] - [data_helper] [140688398202624] - INFO - creating direct client 172.23.123.65:11210 metadata [2021-02-22 22:48:42,598] - [data_helper] [140688398202624] - INFO - creating direct client 172.23.123.65:11210 metadata [2021-02-22 22:48:42,684] - [task] [140688398202624] - WARNING - vbucket map not ready after try 4 [2021-02-22 22:48:42,684] - [bucket_helper] [140688398202624] - INFO - waiting for memcached bucket : metadata in 172.23.123.65 to accept set ops [2021-02-22 22:48:42,836] - [data_helper] [140688398202624] - INFO - creating direct client 172.23.123.65:11210 metadata [2021-02-22 22:48:42,990] - [data_helper] [140688398202624] - INFO - creating direct client 172.23.123.65:11210 metadata [2021-02-22 22:48:43,078] - [task] [140688398202624] - WARNING - vbucket map not ready after try 5 [2021-02-22 22:48:43,497] - [rest_client] [140689256060672] - INFO - http://172.23.122.179:8091/pools/default/buckets/dst_bucket/scopes with params: name=scope_0 [2021-02-22 22:48:43,498] - [rest_client] [140689256060672] - INFO - Scope created dst_bucket->scope_0 {'uid': '1'} [2021-02-22 22:48:43,525] - [rest_client] [140689256060672] - INFO - http://172.23.122.179:8091/pools/default/buckets/dst_bucket/scopes with params: name=scope_1 [2021-02-22 22:48:43,525] - [rest_client] [140689256060672] - INFO - Scope created dst_bucket->scope_1 {'uid': '2'} [2021-02-22 22:48:43,555] - [rest_client] [140689256060672] - INFO - http://172.23.122.179:8091/pools/default/buckets/dst_bucket/scopes with params: name=scope_2 [2021-02-22 22:48:43,555] - [rest_client] [140689256060672] - INFO - Scope created dst_bucket->scope_2 {'uid': '3'} [2021-02-22 22:48:43,585] - [rest_client] [140689256060672] - INFO - http://172.23.122.179:8091/pools/default/buckets/dst_bucket/scopes with params: name=scope_3 [2021-02-22 22:48:43,585] - [rest_client] [140689256060672] - INFO - Scope created dst_bucket->scope_3 {'uid': '4'} [2021-02-22 22:48:43,615] - [rest_client] [140689256060672] - INFO - http://172.23.122.179:8091/pools/default/buckets/dst_bucket/scopes with params: name=scope_4 [2021-02-22 22:48:43,615] - [rest_client] [140689256060672] - INFO - Scope created dst_bucket->scope_4 {'uid': '5'} [2021-02-22 22:48:43,645] - [rest_client] [140689256060672] - INFO - http://172.23.122.179:8091/pools/default/buckets/src_bucket/scopes with params: name=scope_0 [2021-02-22 22:48:43,645] - [rest_client] [140689256060672] - INFO - Scope created src_bucket->scope_0 {'uid': '1'} [2021-02-22 22:48:43,677] - [rest_client] [140689256060672] - INFO - http://172.23.122.179:8091/pools/default/buckets/src_bucket/scopes with params: name=scope_1 [2021-02-22 22:48:43,677] - [rest_client] [140689256060672] - INFO - Scope created src_bucket->scope_1 {'uid': '2'} [2021-02-22 22:48:43,708] - [rest_client] [140689256060672] - INFO - http://172.23.122.179:8091/pools/default/buckets/src_bucket/scopes with params: name=scope_2 [2021-02-22 22:48:43,708] - [rest_client] [140689256060672] - INFO - Scope created src_bucket->scope_2 {'uid': '3'} [2021-02-22 22:48:43,736] - [rest_client] [140689256060672] - INFO - http://172.23.122.179:8091/pools/default/buckets/src_bucket/scopes with params: name=scope_3 [2021-02-22 22:48:43,737] - [rest_client] [140689256060672] - INFO - Scope created src_bucket->scope_3 {'uid': '4'} [2021-02-22 22:48:43,765] - [rest_client] [140689256060672] - INFO - http://172.23.122.179:8091/pools/default/buckets/src_bucket/scopes with params: name=scope_4 [2021-02-22 22:48:43,765] - [rest_client] [140689256060672] - INFO - Scope created src_bucket->scope_4 {'uid': '5'} [2021-02-22 22:48:43,802] - [rest_client] [140689256060672] - INFO - http://172.23.122.179:8091/pools/default/buckets/dst_bucket/scopes/scope_1/collections with params: name=coll_0 [2021-02-22 22:48:43,802] - [rest_client] [140689256060672] - INFO - Collection created dst_bucket->scope_1->coll_0 manifest:{'uid': '6'} [2021-02-22 22:48:43,836] - [rest_client] [140689256060672] - INFO - http://172.23.122.179:8091/pools/default/buckets/dst_bucket/scopes/scope_1/collections with params: name=coll_1 [2021-02-22 22:48:43,836] - [rest_client] [140689256060672] - INFO - Collection created dst_bucket->scope_1->coll_1 manifest:{'uid': '7'} [2021-02-22 22:48:43,866] - [rest_client] [140689256060672] - INFO - http://172.23.122.179:8091/pools/default/buckets/dst_bucket/scopes/scope_1/collections with params: name=coll_2 [2021-02-22 22:48:43,866] - [rest_client] [140689256060672] - INFO - Collection created dst_bucket->scope_1->coll_2 manifest:{'uid': '8'} [2021-02-22 22:48:43,896] - [rest_client] [140689256060672] - INFO - http://172.23.122.179:8091/pools/default/buckets/dst_bucket/scopes/scope_1/collections with params: name=coll_3 [2021-02-22 22:48:43,896] - [rest_client] [140689256060672] - INFO - Collection created dst_bucket->scope_1->coll_3 manifest:{'uid': '9'} [2021-02-22 22:48:43,926] - [rest_client] [140689256060672] - INFO - http://172.23.122.179:8091/pools/default/buckets/dst_bucket/scopes/scope_1/collections with params: name=coll_4 [2021-02-22 22:48:43,926] - [rest_client] [140689256060672] - INFO - Collection created dst_bucket->scope_1->coll_4 manifest:{'uid': 'a'} [2021-02-22 22:48:43,958] - [rest_client] [140689256060672] - INFO - http://172.23.122.179:8091/pools/default/buckets/src_bucket/scopes/scope_1/collections with params: name=coll_0 [2021-02-22 22:48:43,958] - [rest_client] [140689256060672] - INFO - Collection created src_bucket->scope_1->coll_0 manifest:{'uid': '6'} [2021-02-22 22:48:43,991] - [rest_client] [140689256060672] - INFO - http://172.23.122.179:8091/pools/default/buckets/src_bucket/scopes/scope_1/collections with params: name=coll_1 [2021-02-22 22:48:43,992] - [rest_client] [140689256060672] - INFO - Collection created src_bucket->scope_1->coll_1 manifest:{'uid': '7'} [2021-02-22 22:48:44,021] - [rest_client] [140689256060672] - INFO - http://172.23.122.179:8091/pools/default/buckets/src_bucket/scopes/scope_1/collections with params: name=coll_2 [2021-02-22 22:48:44,021] - [rest_client] [140689256060672] - INFO - Collection created src_bucket->scope_1->coll_2 manifest:{'uid': '8'} [2021-02-22 22:48:44,052] - [rest_client] [140689256060672] - INFO - http://172.23.122.179:8091/pools/default/buckets/src_bucket/scopes/scope_1/collections with params: name=coll_3 [2021-02-22 22:48:44,052] - [rest_client] [140689256060672] - INFO - Collection created src_bucket->scope_1->coll_3 manifest:{'uid': '9'} [2021-02-22 22:48:44,083] - [rest_client] [140689256060672] - INFO - http://172.23.122.179:8091/pools/default/buckets/src_bucket/scopes/scope_1/collections with params: name=coll_4 [2021-02-22 22:48:44,083] - [rest_client] [140689256060672] - INFO - Collection created src_bucket->scope_1->coll_4 manifest:{'uid': 'a'} [2021-02-22 22:48:45,362] - [eventing_base] [140689256060672] - INFO - saving function bucket_op [2021-02-22 22:48:46,512] - [eventing_base] [140689256060672] - INFO - saving function timers [2021-02-22 22:48:47,743] - [eventing_base] [140689256060672] - INFO - saving function sbm [2021-02-22 22:48:48,932] - [eventing_base] [140689256060672] - INFO - saving function curl [2021-02-22 22:48:50,119] - [eventing_base] [140689256060672] - INFO - saving function n1ql [2021-02-22 22:48:51,357] - [basetestcase] [140689256060672] - INFO - sleep for 5 secs. Waiting for bucket_op to deployed... ... [2021-02-22 22:48:56,385] - [basetestcase] [140689256060672] - INFO - sleep for 5 secs. Waiting for bucket_op to deployed... ... [2021-02-22 22:49:01,414] - [basetestcase] [140689256060672] - INFO - sleep for 5 secs. Waiting for bucket_op to deployed... ... [2021-02-22 22:49:06,455] - [basetestcase] [140689256060672] - INFO - sleep for 5 secs. Waiting for bucket_op to deployed... ... [2021-02-22 22:49:12,719] - [basetestcase] [140689256060672] - INFO - sleep for 5 secs. Waiting for timers to deployed... ... [2021-02-22 22:49:17,756] - [basetestcase] [140689256060672] - INFO - sleep for 5 secs. Waiting for timers to deployed... ... [2021-02-22 22:49:22,794] - [basetestcase] [140689256060672] - INFO - sleep for 5 secs. Waiting for timers to deployed... ... [2021-02-22 22:49:27,826] - [basetestcase] [140689256060672] - INFO - sleep for 5 secs. Waiting for timers to deployed... ... [2021-02-22 22:49:34,159] - [basetestcase] [140689256060672] - INFO - sleep for 5 secs. Waiting for sbm to deployed... ... [2021-02-22 22:49:39,193] - [basetestcase] [140689256060672] - INFO - sleep for 5 secs. Waiting for sbm to deployed... ... [2021-02-22 22:49:44,237] - [basetestcase] [140689256060672] - INFO - sleep for 5 secs. Waiting for sbm to deployed... ... [2021-02-22 22:49:49,269] - [basetestcase] [140689256060672] - INFO - sleep for 5 secs. Waiting for sbm to deployed... ... [2021-02-22 22:49:55,482] - [basetestcase] [140689256060672] - INFO - sleep for 5 secs. Waiting for curl to deployed... ... [2021-02-22 22:50:00,514] - [basetestcase] [140689256060672] - INFO - sleep for 5 secs. Waiting for curl to deployed... ... [2021-02-22 22:50:05,552] - [basetestcase] [140689256060672] - INFO - sleep for 5 secs. Waiting for curl to deployed... ... [2021-02-22 22:50:10,580] - [basetestcase] [140689256060672] - INFO - sleep for 5 secs. Waiting for curl to deployed... ... [2021-02-22 22:50:16,809] - [basetestcase] [140689256060672] - INFO - sleep for 5 secs. Waiting for n1ql to deployed... ... [2021-02-22 22:50:21,840] - [basetestcase] [140689256060672] - INFO - sleep for 5 secs. Waiting for n1ql to deployed... ... [2021-02-22 22:50:26,870] - [basetestcase] [140689256060672] - INFO - sleep for 5 secs. Waiting for n1ql to deployed... ... [2021-02-22 22:50:31,902] - [basetestcase] [140689256060672] - INFO - sleep for 5 secs. Waiting for n1ql to deployed... ... [2021-02-22 22:50:36,953] - [basetestcase] [140689256060672] - INFO - list of eventing nodes in cluster: [ip:172.23.122.179 port:8091 ssh_username:root, ip:172.23.122.182 port:8091 ssh_username:root] [2021-02-22 22:50:37,203] - [task] [140688398202624] - INFO - java -jar java_sdk_client/collections/target/javaclient/javaclient.jar -i 172.23.123.65 -u Administrator -p password -b src_bucket -s _default -c _default -n 40320 -pc 100 -pu 0 -pd 0 -l uniform -dsn 1 -dpx doc_ -dt Person -de 0 -ds 500 -ac False -st 0 -en 40319 -o False -sd False [2021-02-22 22:50:40,055] - [rest_client] [140689256060672] - INFO - rebalance reached >100% in 3.0064961910247803 seconds [2021-02-22 22:51:06,224] - [task] [140688398202624] - INFO - java -jar java_sdk_client/collections/target/javaclient/javaclient.jar -i 172.23.123.65 -u Administrator -p password -b src_bucket -s scope_1 -c coll_1 -n 40320 -pc 100 -pu 0 -pd 0 -l uniform -dsn 1 -dpx doc_ -dt Person -de 0 -ds 500 -ac False -st 0 -en 40319 -o False -sd False [2021-02-22 22:51:53,648] - [task] [140688398202624] - INFO - adding node 172.23.122.140:8091 to cluster [2021-02-22 22:51:53,649] - [rest_client] [140688398202624] - INFO - adding remote node @172.23.122.140:8091 to this cluster @172.23.123.65:8091 [2021-02-22 22:52:03,670] - [rest_client] [140688398202624] - INFO - rebalance progress took 10.02 seconds [2021-02-22 22:52:03,671] - [rest_client] [140688398202624] - INFO - sleep for 10 seconds after rebalance... [2021-02-22 22:52:16,033] - [rest_client] [140688398202624] - INFO - rebalance params : {'knownNodes': 'ns_1@172.23.122.139,ns_1@172.23.122.140,ns_1@172.23.122.153,ns_1@172.23.122.179,ns_1@172.23.122.182,ns_1@172.23.123.65', 'ejectedNodes': 'ns_1@172.23.122.179,ns_1@172.23.122.182', 'user': 'Administrator', 'password': 'password'} [2021-02-22 22:52:16,062] - [rest_client] [140688398202624] - INFO - rebalance operation started [2021-02-22 22:52:16,066] - [rest_client] [140688398202624] - INFO - rebalance percentage : 0.00 % [2021-02-22 22:52:16,066] - [task] [140688398202624] - INFO - Rebalance - status: running, progress: 0.00% [2021-02-22 22:52:26,100] - [rest_client] [140688398202624] - INFO - rebalance percentage : 33.00 % [2021-02-22 22:52:26,100] - [task] [140688398202624] - INFO - Rebalance - status: running, progress: 33.00% [2021-02-22 22:52:36,126] - [rest_client] [140688398202624] - INFO - rebalance percentage : 33.00 % [2021-02-22 22:52:36,126] - [task] [140688398202624] - INFO - Rebalance - status: running, progress: 33.00% [2021-02-22 22:52:46,152] - [rest_client] [140688398202624] - INFO - rebalance percentage : 33.00 % [2021-02-22 22:52:46,153] - [task] [140688398202624] - INFO - Rebalance - status: running, progress: 33.00% [2021-02-22 22:52:56,179] - [rest_client] [140688398202624] - INFO - rebalance percentage : 83.00 % [2021-02-22 22:52:56,179] - [task] [140688398202624] - INFO - Rebalance - status: running, progress: 83.00% [2021-02-22 22:53:06,206] - [rest_client] [140688398202624] - INFO - rebalance percentage : 83.00 % [2021-02-22 22:53:06,207] - [task] [140688398202624] - INFO - Rebalance - status: running, progress: 83.00% [2021-02-22 22:53:16,232] - [rest_client] [140688398202624] - INFO - rebalance percentage : 83.00 % [2021-02-22 22:53:16,233] - [task] [140688398202624] - INFO - Rebalance - status: running, progress: 83.00% [2021-02-22 22:53:26,259] - [rest_client] [140688398202624] - INFO - rebalance percentage : 83.00 % [2021-02-22 22:53:26,260] - [task] [140688398202624] - INFO - Rebalance - status: running, progress: 83.00% [2021-02-22 22:53:36,287] - [rest_client] [140688398202624] - INFO - rebalance percentage : 83.00 % [2021-02-22 22:53:36,287] - [task] [140688398202624] - INFO - Rebalance - status: running, progress: 83.00% [2021-02-22 22:53:46,316] - [rest_client] [140688398202624] - INFO - rebalance percentage : 83.00 % [2021-02-22 22:53:46,316] - [task] [140688398202624] - INFO - Rebalance - status: running, progress: 83.00% [2021-02-22 22:53:56,341] - [task] [140688398202624] - INFO - Rebalance - status: none, progress: 100.00% [2021-02-22 22:53:56,355] - [rest_client] [140688398202624] - ERROR - GET http://172.23.122.179:8091/nodes/self body: headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"Node is unknown to this cluster."' auth: Administrator:password [2021-02-22 22:53:56,356] - [rest_client] [140688398202624] - ERROR - Error Node is unknown to this cluster. was gotten, 5 seconds sleep before retry [2021-02-22 22:54:01,366] - [rest_client] [140688398202624] - ERROR - GET http://172.23.122.179:8091/nodes/self body: headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"Node is unknown to this cluster."' auth: Administrator:password [2021-02-22 22:54:01,367] - [rest_client] [140688398202624] - ERROR - Error Node is unknown to this cluster. was gotten, 5 seconds sleep before retry [2021-02-22 22:54:06,376] - [rest_client] [140688398202624] - ERROR - GET http://172.23.122.179:8091/nodes/self body: headers: {'Content-Type': 'application/json', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"Node is unknown to this cluster."' auth: Administrator:password [2021-02-22 22:54:06,377] - [rest_client] [140688398202624] - ERROR - Error Node is unknown to this cluster. was gotten, 5 seconds sleep before retry [2021-02-22 22:54:11,382] - [rest_client] [140688398202624] - ERROR - node 172.23.122.179:8091 is in a broken state! [2021-02-22 22:54:11,383] - [task] [140688398202624] - ERROR - unable to reach the host @ 172.23.122.179 [2021-02-22 22:54:11,396] - [task] [140688398202624] - ERROR - node 172.23.122.179:8091 was not cleaned after removing from cluster [2021-02-22 22:54:11,397] - [task] [140688398202624] - INFO - rebalancing was completed with progress: 100% in 115.33419632911682 sec [2021-02-22 22:54:11,412] - [basetestcase] [140689256060672] - INFO - list of eventing nodes in cluster: [ip:172.23.122.140 port:8091 ssh_username:root] [2021-02-22 22:54:11,418] - [tuq_helper] [140689256060672] - INFO - RUN QUERY create primary index on dst_bucket.scope_1.coll_3 [2021-02-22 22:54:11,423] - [rest_client] [140689256060672] - INFO - query params : statement=create+primary+index+on+dst_bucket.scope_1.coll_3 [2021-02-22 22:54:13,555] - [tuq_helper] [140689256060672] - INFO - TOTAL ELAPSED TIME: 2.128509382s [2021-02-22 22:54:13,573] - [basetestcase] [140689256060672] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.139 port:8091 ssh_username:root] [2021-02-22 22:54:13,579] - [rest_client] [140689256060672] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_3 [2021-02-22 22:54:13,591] - [eventing_base] [140689256060672] - INFO - Final docs count... Current : 40320 Expected : 40320 [2021-02-22 22:54:13,600] - [basetestcase] [140689256060672] - INFO - list of eventing nodes in cluster: [ip:172.23.122.140 port:8091 ssh_username:root] [2021-02-22 22:54:13,605] - [tuq_helper] [140689256060672] - INFO - RUN QUERY create primary index on dst_bucket.scope_1.coll_4 [2021-02-22 22:54:13,610] - [rest_client] [140689256060672] - INFO - query params : statement=create+primary+index+on+dst_bucket.scope_1.coll_4 [2021-02-22 22:54:16,548] - [tuq_helper] [140689256060672] - INFO - TOTAL ELAPSED TIME: 2.934871744s [2021-02-22 22:54:16,568] - [basetestcase] [140689256060672] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.139 port:8091 ssh_username:root] [2021-02-22 22:54:16,573] - [rest_client] [140689256060672] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-22 22:54:16,582] - [eventing_base] [140689256060672] - INFO - Final docs count... Current : 40320 Expected : 40320 [2021-02-22 22:54:16,593] - [basetestcase] [140689256060672] - INFO - list of eventing nodes in cluster: [ip:172.23.122.140 port:8091 ssh_username:root] [2021-02-22 22:54:16,598] - [tuq_helper] [140689256060672] - INFO - RUN QUERY create primary index on src_bucket.scope_1.coll_1 [2021-02-22 22:54:16,603] - [rest_client] [140689256060672] - INFO - query params : statement=create+primary+index+on+src_bucket.scope_1.coll_1 [2021-02-22 22:54:19,449] - [tuq_helper] [140689256060672] - INFO - TOTAL ELAPSED TIME: 2.843405148s [2021-02-22 22:54:19,470] - [basetestcase] [140689256060672] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.139 port:8091 ssh_username:root] [2021-02-22 22:54:19,476] - [rest_client] [140689256060672] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.scope_1.coll_1 [2021-02-22 22:54:19,489] - [basetestcase] [140689256060672] - INFO - sleep for 30 secs. Waiting for handler code src_bucket.scope_1.coll_1 to complete bucket operations... Current : 116253 Expected : 80640 ... [2021-02-22 22:54:49,535] - [basetestcase] [140689256060672] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.139 port:8091 ssh_username:root] [2021-02-22 22:54:49,541] - [rest_client] [140689256060672] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.scope_1.coll_1 [2021-02-22 22:54:49,561] - [basetestcase] [140689256060672] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.139 port:8091 ssh_username:root] [2021-02-22 22:54:49,566] - [rest_client] [140689256060672] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.scope_1.coll_1 [2021-02-22 22:54:49,570] - [basetestcase] [140689256060672] - INFO - sleep for 30 secs. Waiting for handler code src_bucket.scope_1.coll_1 to complete bucket operations... Current : 116253 Expected : 80640 ... [2021-02-22 22:55:19,625] - [basetestcase] [140689256060672] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.139 port:8091 ssh_username:root] [2021-02-22 22:55:19,631] - [rest_client] [140689256060672] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.scope_1.coll_1 [2021-02-22 22:55:19,649] - [basetestcase] [140689256060672] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.139 port:8091 ssh_username:root] [2021-02-22 22:55:19,655] - [rest_client] [140689256060672] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.scope_1.coll_1 [2021-02-22 22:55:19,659] - [basetestcase] [140689256060672] - INFO - sleep for 30 secs. Waiting for handler code src_bucket.scope_1.coll_1 to complete bucket operations... Current : 116253 Expected : 80640 ... [2021-02-22 22:55:49,681] - [basetestcase] [140689256060672] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.139 port:8091 ssh_username:root] [2021-02-22 22:55:49,687] - [rest_client] [140689256060672] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.scope_1.coll_1 [2021-02-22 22:55:49,706] - [basetestcase] [140689256060672] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.139 port:8091 ssh_username:root] [2021-02-22 22:55:49,712] - [rest_client] [140689256060672] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.scope_1.coll_1 [2021-02-22 22:55:49,716] - [basetestcase] [140689256060672] - INFO - sleep for 30 secs. Waiting for handler code src_bucket.scope_1.coll_1 to complete bucket operations... Current : 116253 Expected : 80640 ... [2021-02-22 22:56:19,751] - [basetestcase] [140689256060672] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.139 port:8091 ssh_username:root] [2021-02-22 22:56:19,758] - [rest_client] [140689256060672] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.scope_1.coll_1 [2021-02-22 22:56:19,777] - [basetestcase] [140689256060672] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.139 port:8091 ssh_username:root] [2021-02-22 22:56:19,783] - [rest_client] [140689256060672] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.scope_1.coll_1 [2021-02-22 22:56:19,786] - [basetestcase] [140689256060672] - INFO - sleep for 30 secs. Waiting for handler code src_bucket.scope_1.coll_1 to complete bucket operations... Current : 116253 Expected : 80640 ... [2021-02-22 22:56:49,835] - [basetestcase] [140689256060672] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.139 port:8091 ssh_username:root] [2021-02-22 22:56:49,841] - [rest_client] [140689256060672] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.scope_1.coll_1 [2021-02-22 22:56:49,860] - [basetestcase] [140689256060672] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.139 port:8091 ssh_username:root] [2021-02-22 22:56:49,865] - [rest_client] [140689256060672] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.scope_1.coll_1 [2021-02-22 22:56:49,869] - [basetestcase] [140689256060672] - INFO - sleep for 30 secs. Waiting for handler code src_bucket.scope_1.coll_1 to complete bucket operations... Current : 116253 Expected : 80640 ... [2021-02-22 22:57:19,910] - [basetestcase] [140689256060672] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.139 port:8091 ssh_username:root] [2021-02-22 22:57:19,917] - [rest_client] [140689256060672] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.scope_1.coll_1 [2021-02-22 22:57:19,935] - [basetestcase] [140689256060672] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.139 port:8091 ssh_username:root] [2021-02-22 22:57:19,940] - [rest_client] [140689256060672] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.scope_1.coll_1 [2021-02-22 22:57:19,945] - [basetestcase] [140689256060672] - INFO - sleep for 30 secs. Waiting for handler code src_bucket.scope_1.coll_1 to complete bucket operations... Current : 116253 Expected : 80640 ... [2021-02-22 22:57:49,971] - [basetestcase] [140689256060672] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.139 port:8091 ssh_username:root] [2021-02-22 22:57:49,977] - [rest_client] [140689256060672] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.scope_1.coll_1 [2021-02-22 22:57:49,995] - [basetestcase] [140689256060672] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.139 port:8091 ssh_username:root] [2021-02-22 22:57:50,001] - [rest_client] [140689256060672] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.scope_1.coll_1 [2021-02-22 22:57:50,005] - [basetestcase] [140689256060672] - INFO - sleep for 30 secs. Waiting for handler code src_bucket.scope_1.coll_1 to complete bucket operations... Current : 116253 Expected : 80640 ... [2021-02-22 22:58:20,052] - [basetestcase] [140689256060672] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.139 port:8091 ssh_username:root] [2021-02-22 22:58:20,059] - [rest_client] [140689256060672] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.scope_1.coll_1 [2021-02-22 22:58:20,079] - [basetestcase] [140689256060672] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.139 port:8091 ssh_username:root] [2021-02-22 22:58:20,084] - [rest_client] [140689256060672] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.scope_1.coll_1 [2021-02-22 22:58:20,089] - [basetestcase] [140689256060672] - INFO - sleep for 30 secs. Waiting for handler code src_bucket.scope_1.coll_1 to complete bucket operations... Current : 116253 Expected : 80640 ... [2021-02-22 22:58:50,136] - [basetestcase] [140689256060672] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.139 port:8091 ssh_username:root] [2021-02-22 22:58:50,143] - [rest_client] [140689256060672] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.scope_1.coll_1 [2021-02-22 22:58:50,162] - [basetestcase] [140689256060672] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.139 port:8091 ssh_username:root] [2021-02-22 22:58:50,167] - [rest_client] [140689256060672] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.scope_1.coll_1 [2021-02-22 22:58:50,172] - [basetestcase] [140689256060672] - INFO - sleep for 30 secs. Waiting for handler code src_bucket.scope_1.coll_1 to complete bucket operations... Current : 116253 Expected : 80640 ... [2021-02-22 22:59:20,204] - [basetestcase] [140689256060672] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.139 port:8091 ssh_username:root] [2021-02-22 22:59:20,214] - [rest_client] [140689256060672] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.scope_1.coll_1 [2021-02-22 22:59:20,237] - [basetestcase] [140689256060672] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.139 port:8091 ssh_username:root] [2021-02-22 22:59:20,244] - [rest_client] [140689256060672] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.scope_1.coll_1 [2021-02-22 22:59:20,249] - [basetestcase] [140689256060672] - INFO - sleep for 30 secs. Waiting for handler code src_bucket.scope_1.coll_1 to complete bucket operations... Current : 116253 Expected : 80640 ... [2021-02-22 22:59:50,296] - [basetestcase] [140689256060672] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.139 port:8091 ssh_username:root] [2021-02-22 22:59:50,302] - [rest_client] [140689256060672] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.scope_1.coll_1 [2021-02-22 22:59:50,318] - [eventing_base] [140689256060672] - INFO - Compacting bucket : dst_bucket [2021-02-22 22:59:50,325] - [rest_client] [140689256060672] - INFO - bucket compaction successful [2021-02-22 22:59:50,325] - [eventing_base] [140689256060672] - INFO - Compacting bucket : metadata [2021-02-22 22:59:50,332] - [rest_client] [140689256060672] - INFO - bucket compaction successful [2021-02-22 22:59:50,333] - [eventing_base] [140689256060672] - INFO - Compacting bucket : src_bucket [2021-02-22 22:59:50,337] - [rest_client] [140689256060672] - INFO - bucket compaction successful [2021-02-22 22:59:50,350] - [basetestcase] [140689256060672] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.139 port:8091 ssh_username:root] [2021-02-22 22:59:50,356] - [rest_client] [140689256060672] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.scope_1.coll_1 [2021-02-22 22:59:50,360] - [basetestcase] [140689256060672] - INFO - sleep for 30 secs. Waiting for handler code src_bucket.scope_1.coll_1 to complete bucket operations... Current : 116253 Expected : 80640 ... [2021-02-22 23:00:20,384] - [basetestcase] [140689256060672] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.139 port:8091 ssh_username:root] [2021-02-22 23:00:20,391] - [rest_client] [140689256060672] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.scope_1.coll_1 [2021-02-22 23:00:20,409] - [basetestcase] [140689256060672] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.139 port:8091 ssh_username:root] [2021-02-22 23:00:20,416] - [rest_client] [140689256060672] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.scope_1.coll_1 [2021-02-22 23:00:20,420] - [basetestcase] [140689256060672] - INFO - sleep for 30 secs. Waiting for handler code src_bucket.scope_1.coll_1 to complete bucket operations... Current : 116253 Expected : 80640 ... [2021-02-22 23:00:50,468] - [basetestcase] [140689256060672] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.139 port:8091 ssh_username:root] [2021-02-22 23:00:50,474] - [rest_client] [140689256060672] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.scope_1.coll_1 [2021-02-22 23:00:50,494] - [basetestcase] [140689256060672] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.139 port:8091 ssh_username:root] [2021-02-22 23:00:50,499] - [rest_client] [140689256060672] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.scope_1.coll_1 [2021-02-22 23:00:50,504] - [basetestcase] [140689256060672] - INFO - sleep for 30 secs. Waiting for handler code src_bucket.scope_1.coll_1 to complete bucket operations... Current : 116253 Expected : 80640 ... [2021-02-22 23:01:20,547] - [basetestcase] [140689256060672] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.139 port:8091 ssh_username:root] [2021-02-22 23:01:20,555] - [rest_client] [140689256060672] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.scope_1.coll_1 [2021-02-22 23:01:20,578] - [basetestcase] [140689256060672] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.139 port:8091 ssh_username:root] [2021-02-22 23:01:20,586] - [rest_client] [140689256060672] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.scope_1.coll_1 [2021-02-22 23:01:20,592] - [basetestcase] [140689256060672] - INFO - sleep for 30 secs. Waiting for handler code src_bucket.scope_1.coll_1 to complete bucket operations... Current : 116253 Expected : 80640 ... [2021-02-22 23:01:50,623] - [basetestcase] [140689256060672] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.139 port:8091 ssh_username:root] [2021-02-22 23:01:50,630] - [rest_client] [140689256060672] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.scope_1.coll_1 [2021-02-22 23:01:50,649] - [basetestcase] [140689256060672] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.139 port:8091 ssh_username:root] [2021-02-22 23:01:50,655] - [rest_client] [140689256060672] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.scope_1.coll_1 [2021-02-22 23:01:50,660] - [basetestcase] [140689256060672] - INFO - sleep for 30 secs. Waiting for handler code src_bucket.scope_1.coll_1 to complete bucket operations... Current : 116253 Expected : 80640 ... [2021-02-22 23:02:20,677] - [basetestcase] [140689256060672] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.139 port:8091 ssh_username:root] [2021-02-22 23:02:20,683] - [rest_client] [140689256060672] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.scope_1.coll_1 [2021-02-22 23:02:20,702] - [basetestcase] [140689256060672] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.139 port:8091 ssh_username:root] [2021-02-22 23:02:20,707] - [rest_client] [140689256060672] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.scope_1.coll_1 [2021-02-22 23:02:20,711] - [basetestcase] [140689256060672] - INFO - sleep for 30 secs. Waiting for handler code src_bucket.scope_1.coll_1 to complete bucket operations... Current : 116253 Expected : 80640 ... [2021-02-22 23:02:50,731] - [basetestcase] [140689256060672] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.139 port:8091 ssh_username:root] [2021-02-22 23:02:50,738] - [rest_client] [140689256060672] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.scope_1.coll_1 [2021-02-22 23:02:50,756] - [basetestcase] [140689256060672] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.139 port:8091 ssh_username:root] [2021-02-22 23:02:50,762] - [rest_client] [140689256060672] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.scope_1.coll_1 [2021-02-22 23:02:50,766] - [basetestcase] [140689256060672] - INFO - sleep for 30 secs. Waiting for handler code src_bucket.scope_1.coll_1 to complete bucket operations... Current : 116253 Expected : 80640 ... [2021-02-22 23:03:20,807] - [basetestcase] [140689256060672] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.139 port:8091 ssh_username:root] [2021-02-22 23:03:20,814] - [rest_client] [140689256060672] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.scope_1.coll_1 [2021-02-22 23:03:20,833] - [basetestcase] [140689256060672] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.139 port:8091 ssh_username:root] [2021-02-22 23:03:20,839] - [rest_client] [140689256060672] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.scope_1.coll_1 [2021-02-22 23:03:20,843] - [basetestcase] [140689256060672] - INFO - sleep for 30 secs. Waiting for handler code src_bucket.scope_1.coll_1 to complete bucket operations... Current : 116253 Expected : 80640 ... [2021-02-22 23:03:50,889] - [basetestcase] [140689256060672] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.139 port:8091 ssh_username:root] [2021-02-22 23:03:50,896] - [rest_client] [140689256060672] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.scope_1.coll_1 [2021-02-22 23:03:50,916] - [basetestcase] [140689256060672] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.139 port:8091 ssh_username:root] [2021-02-22 23:03:50,922] - [rest_client] [140689256060672] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.scope_1.coll_1 [2021-02-22 23:03:50,926] - [basetestcase] [140689256060672] - INFO - sleep for 30 secs. Waiting for handler code src_bucket.scope_1.coll_1 to complete bucket operations... Current : 116253 Expected : 80640 ... [2021-02-22 23:04:20,973] - [basetestcase] [140689256060672] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.139 port:8091 ssh_username:root] [2021-02-22 23:04:20,980] - [rest_client] [140689256060672] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.scope_1.coll_1 [2021-02-22 23:04:20,999] - [basetestcase] [140689256060672] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.139 port:8091 ssh_username:root] [2021-02-22 23:04:21,005] - [rest_client] [140689256060672] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.scope_1.coll_1 [2021-02-22 23:04:21,017] - [basetestcase] [140689256060672] - INFO - list of eventing nodes in cluster: [ip:172.23.122.140 port:8091 ssh_username:root] [2021-02-22 23:04:21,067] - [eventing_base] [140689256060672] - INFO - Stats for Node 172.23.122.140 is [ { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 18, "agg_messages_sent_to_worker": 44757, "agg_queue_memory": 40109456, "agg_queue_memory_cap": 53477376, "agg_queue_size": 39586, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 156409, "dcp_mutation_sent_to_worker": 40156, "dcp_snapshot": 1024, "dcp_stream_req_counter": 1024, "dcp_streamreq": 1024, "dcp_system_event": 8600, "execution_stats": 713, "failure_stats": 713, "latency_stats": 713, "lcb_exception_stats": 713, "log_level": 1, "num_processed_events": 568, "processed_events_size": 507492, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 40109456, "agg_queue_size": 39586, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 570, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 40156, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 44753, "no_op_counter": 0, "num_processed_events": 568, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 568, "on_update_success": 0, "processed_events_size": 507492, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "3710": "2021-02-23T07:04:20Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 568, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "3710": "2021-02-23T07:04:20Z" }, "v8worker_events_lost": 0 }, "function_id": 1325846238, "function_name": "bucket_op", "gocb_creds_request_counter": 23, "internal_vb_distribution_stats": { "worker_bucket_op_0": "[0-1023]" }, "latency_percentile_stats": { "100": 2507100, "50": 2500400, "80": 2500400, "90": 2500400, "95": 2500500, "99": 2501000 }, "lcb_creds_request_counter": 24, "lcb_exception_stats": { "211": 568 }, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.140:8096", "start_vb": 0, "vb_count": 1024 } ], "vb_distribution_stats_from_metadata": { "172.23.122.140:8096": { "worker_bucket_op_0": "[0-1023]" } }, "worker_pids": { "worker_bucket_op_0": 3710 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 5, "agg_messages_sent_to_worker": 4999, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 116253, "dcp_snapshot": 1024, "dcp_stream_req_counter": 1024, "dcp_streamreq": 1024, "execution_stats": 697, "failure_stats": 697, "latency_stats": 697, "lcb_exception_stats": 697, "log_level": 1, "num_processed_events": 479, "processed_events_size": 53436, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 0, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 0, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 4995, "no_op_counter": 479, "num_processed_events": 479, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 0, "processed_events_size": 53436, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "3739": "2021-02-23T07:04:20Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "3739": "2021-02-23T07:04:20Z" }, "v8worker_events_lost": 0 }, "function_id": 657260674, "function_name": "curl", "gocb_creds_request_counter": 23, "internal_vb_distribution_stats": { "worker_curl_0": "[0-1023]" }, "latency_percentile_stats": { "100": 0, "50": 0, "80": 0, "90": 0, "95": 0, "99": 0 }, "lcb_creds_request_counter": 24, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.140:8096", "start_vb": 0, "vb_count": 1024 } ], "vb_distribution_stats_from_metadata": { "172.23.122.140:8096": { "worker_curl_0": "[0-1023]" } }, "worker_pids": { "worker_curl_0": 3739 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 4, "agg_messages_sent_to_worker": 4913, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 116253, "dcp_snapshot": 1024, "dcp_stream_req_counter": 1024, "dcp_streamreq": 1024, "execution_stats": 680, "failure_stats": 680, "latency_stats": 680, "lcb_exception_stats": 680, "log_level": 1, "num_processed_events": 479, "processed_events_size": 53436, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 0, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 0, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 4909, "no_op_counter": 479, "num_processed_events": 479, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 0, "processed_events_size": 53436, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "3769": "2021-02-23T07:04:20Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "3769": "2021-02-23T07:04:20Z" }, "v8worker_events_lost": 0 }, "function_id": 699295493, "function_name": "n1ql", "gocb_creds_request_counter": 23, "internal_vb_distribution_stats": { "worker_n1ql_0": "[0-1023]" }, "latency_percentile_stats": { "100": 0, "50": 0, "80": 0, "90": 0, "95": 0, "99": 0 }, "lcb_creds_request_counter": 24, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.140:8096", "start_vb": 0, "vb_count": 1024 } ], "vb_distribution_stats_from_metadata": { "172.23.122.140:8096": { "worker_n1ql_0": "[0-1023]" } }, "worker_pids": { "worker_n1ql_0": 3769 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "agg_messages_sent_to_worker": 4353, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 1140, "dcp_mutation_suppressed_counter": 1140, "dcp_snapshot": 567, "dcp_stream_req_counter": 1024, "dcp_streamreq": 1024, "execution_stats": 664, "failure_stats": 664, "latency_stats": 664, "lcb_exception_stats": 664, "log_level": 1, "num_processed_events": 0, "processed_events_size": 0, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 0, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 0, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 4349, "no_op_counter": 0, "num_processed_events": 0, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 0, "processed_events_size": 0, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "3796": "2021-02-23T07:04:20Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "3796": "2021-02-23T07:04:20Z" }, "v8worker_events_lost": 0 }, "function_id": 2459091863, "function_name": "sbm", "gocb_creds_request_counter": 23, "internal_vb_distribution_stats": { "worker_sbm_0": "[0-1023]" }, "latency_percentile_stats": { "100": 0, "50": 0, "80": 0, "90": 0, "95": 0, "99": 0 }, "lcb_creds_request_counter": 24, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.140:8096", "start_vb": 0, "vb_count": 1024 } ], "vb_distribution_stats_from_metadata": { "172.23.122.140:8096": { "worker_sbm_0": "[0-1023]" } }, "worker_pids": { "worker_sbm_0": 3796 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 3, "agg_messages_sent_to_worker": 18753, "agg_queue_memory": 13431859, "agg_queue_memory_cap": 20971520, "agg_queue_size": 13251, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 130734, "dcp_mutation_sent_to_worker": 14481, "dcp_snapshot": 1024, "dcp_stream_req_counter": 1024, "dcp_streamreq": 1024, "execution_stats": 648, "failure_stats": 648, "latency_stats": 648, "lcb_exception_stats": 648, "log_level": 1, "num_processed_events": 1232, "processed_events_size": 1103060, "thr_count": 1, "thr_map": 1, "timer_events": 516, "timer_responses_received": 1232, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 13431859, "agg_queue_size": 13251, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 1232, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 14481, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 18749, "no_op_counter": 0, "num_processed_events": 1232, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 1232, "processed_events_size": 1103060, "timer_callback_failure": 514, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 1232, "timer_create_failure": 0, "timer_msg_counter": 516, "timer_responses_sent": 0, "timestamp": { "3854": "2021-02-23T07:04:20Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 514, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "3854": "2021-02-23T07:04:20Z" }, "v8worker_events_lost": 0 }, "function_id": 1125753195, "function_name": "timers", "gocb_creds_request_counter": 23, "internal_vb_distribution_stats": { "worker_timers_0": "[0-1023]" }, "latency_percentile_stats": { "100": 2514900, "50": 1500, "80": 2500300, "90": 2500300, "95": 2500400, "99": 2500500 }, "lcb_creds_request_counter": 24, "lcb_exception_stats": { "211": 514 }, "metastore_stats": { "metastore_delete_err": 0, "metastore_deletes": 0, "metastore_not_found": 0, "metastore_scan": 0, "metastore_scan_due": 0, "metastore_scan_err": 0, "metastore_set": 0, "metastore_set_err": 0 }, "planner_stats": [ { "host_name": "172.23.122.140:8096", "start_vb": 0, "vb_count": 1024 } ], "vb_distribution_stats_from_metadata": { "172.23.122.140:8096": { "worker_timers_0": "[0-1023]" } }, "worker_pids": { "worker_timers_0": 3854 } } ] [2021-02-22 23:04:26,112] - [eventing_base] [140689256060672] - INFO - Stats for Node 172.23.122.140 is [ { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 18, "agg_messages_sent_to_worker": 44757, "agg_queue_memory": 40109456, "agg_queue_memory_cap": 53477376, "agg_queue_size": 39586, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 156409, "dcp_mutation_sent_to_worker": 40156, "dcp_snapshot": 1024, "dcp_stream_req_counter": 1024, "dcp_streamreq": 1024, "dcp_system_event": 8600, "execution_stats": 713, "failure_stats": 713, "latency_stats": 713, "lcb_exception_stats": 713, "log_level": 1, "num_processed_events": 568, "processed_events_size": 507492, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 40109456, "agg_queue_size": 39586, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 570, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 40156, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 44753, "no_op_counter": 0, "num_processed_events": 568, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 568, "on_update_success": 0, "processed_events_size": 507492, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "3710": "2021-02-23T07:04:20Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 568, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "3710": "2021-02-23T07:04:20Z" }, "v8worker_events_lost": 0 }, "function_id": 1325846238, "function_name": "bucket_op", "gocb_creds_request_counter": 23, "internal_vb_distribution_stats": { "worker_bucket_op_0": "[0-1023]" }, "latency_percentile_stats": { "100": 2507100, "50": 2500400, "80": 2500400, "90": 2500400, "95": 2500500, "99": 2501000 }, "lcb_creds_request_counter": 24, "lcb_exception_stats": { "211": 568 }, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.140:8096", "start_vb": 0, "vb_count": 1024 } ], "vb_distribution_stats_from_metadata": { "172.23.122.140:8096": { "worker_bucket_op_0": "[0-1023]" } }, "worker_pids": { "worker_bucket_op_0": 3710 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 5, "agg_messages_sent_to_worker": 4999, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 116253, "dcp_snapshot": 1024, "dcp_stream_req_counter": 1024, "dcp_streamreq": 1024, "execution_stats": 697, "failure_stats": 697, "latency_stats": 697, "lcb_exception_stats": 697, "log_level": 1, "num_processed_events": 479, "processed_events_size": 53436, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 0, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 0, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 4995, "no_op_counter": 479, "num_processed_events": 479, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 0, "processed_events_size": 53436, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "3739": "2021-02-23T07:04:20Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "3739": "2021-02-23T07:04:20Z" }, "v8worker_events_lost": 0 }, "function_id": 657260674, "function_name": "curl", "gocb_creds_request_counter": 23, "internal_vb_distribution_stats": { "worker_curl_0": "[0-1023]" }, "latency_percentile_stats": { "100": 0, "50": 0, "80": 0, "90": 0, "95": 0, "99": 0 }, "lcb_creds_request_counter": 24, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.140:8096", "start_vb": 0, "vb_count": 1024 } ], "vb_distribution_stats_from_metadata": { "172.23.122.140:8096": { "worker_curl_0": "[0-1023]" } }, "worker_pids": { "worker_curl_0": 3739 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 4, "agg_messages_sent_to_worker": 4913, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 116253, "dcp_snapshot": 1024, "dcp_stream_req_counter": 1024, "dcp_streamreq": 1024, "execution_stats": 680, "failure_stats": 680, "latency_stats": 680, "lcb_exception_stats": 680, "log_level": 1, "num_processed_events": 479, "processed_events_size": 53436, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 0, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 0, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 4909, "no_op_counter": 479, "num_processed_events": 479, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 0, "processed_events_size": 53436, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "3769": "2021-02-23T07:04:20Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "3769": "2021-02-23T07:04:20Z" }, "v8worker_events_lost": 0 }, "function_id": 699295493, "function_name": "n1ql", "gocb_creds_request_counter": 23, "internal_vb_distribution_stats": { "worker_n1ql_0": "[0-1023]" }, "latency_percentile_stats": { "100": 0, "50": 0, "80": 0, "90": 0, "95": 0, "99": 0 }, "lcb_creds_request_counter": 24, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.140:8096", "start_vb": 0, "vb_count": 1024 } ], "vb_distribution_stats_from_metadata": { "172.23.122.140:8096": { "worker_n1ql_0": "[0-1023]" } }, "worker_pids": { "worker_n1ql_0": 3769 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "agg_messages_sent_to_worker": 4353, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 1140, "dcp_mutation_suppressed_counter": 1140, "dcp_snapshot": 567, "dcp_stream_req_counter": 1024, "dcp_streamreq": 1024, "execution_stats": 664, "failure_stats": 664, "latency_stats": 664, "lcb_exception_stats": 664, "log_level": 1, "num_processed_events": 0, "processed_events_size": 0, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 0, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 0, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 4349, "no_op_counter": 0, "num_processed_events": 0, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 0, "processed_events_size": 0, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "3796": "2021-02-23T07:04:20Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "3796": "2021-02-23T07:04:20Z" }, "v8worker_events_lost": 0 }, "function_id": 2459091863, "function_name": "sbm", "gocb_creds_request_counter": 23, "internal_vb_distribution_stats": { "worker_sbm_0": "[0-1023]" }, "latency_percentile_stats": { "100": 0, "50": 0, "80": 0, "90": 0, "95": 0, "99": 0 }, "lcb_creds_request_counter": 24, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.140:8096", "start_vb": 0, "vb_count": 1024 } ], "vb_distribution_stats_from_metadata": { "172.23.122.140:8096": { "worker_sbm_0": "[0-1023]" } }, "worker_pids": { "worker_sbm_0": 3796 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 3, "agg_messages_sent_to_worker": 18753, "agg_queue_memory": 13431859, "agg_queue_memory_cap": 20971520, "agg_queue_size": 13251, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 130734, "dcp_mutation_sent_to_worker": 14481, "dcp_snapshot": 1024, "dcp_stream_req_counter": 1024, "dcp_streamreq": 1024, "execution_stats": 648, "failure_stats": 648, "latency_stats": 648, "lcb_exception_stats": 648, "log_level": 1, "num_processed_events": 1232, "processed_events_size": 1103060, "thr_count": 1, "thr_map": 1, "timer_events": 516, "timer_responses_received": 1232, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 13431859, "agg_queue_size": 13251, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 1232, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 14481, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 18749, "no_op_counter": 0, "num_processed_events": 1232, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 1232, "processed_events_size": 1103060, "timer_callback_failure": 514, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 1232, "timer_create_failure": 0, "timer_msg_counter": 516, "timer_responses_sent": 0, "timestamp": { "3854": "2021-02-23T07:04:20Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 514, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "3854": "2021-02-23T07:04:20Z" }, "v8worker_events_lost": 0 }, "function_id": 1125753195, "function_name": "timers", "gocb_creds_request_counter": 23, "internal_vb_distribution_stats": { "worker_timers_0": "[0-1023]" }, "latency_percentile_stats": { "100": 2514900, "50": 1500, "80": 2500300, "90": 2500300, "95": 2500400, "99": 2500500 }, "lcb_creds_request_counter": 24, "lcb_exception_stats": { "211": 514 }, "metastore_stats": { "metastore_delete_err": 0, "metastore_deletes": 0, "metastore_not_found": 0, "metastore_scan": 0, "metastore_scan_due": 0, "metastore_scan_err": 0, "metastore_set": 0, "metastore_set_err": 0 }, "planner_stats": [ { "host_name": "172.23.122.140:8096", "start_vb": 0, "vb_count": 1024 } ], "vb_distribution_stats_from_metadata": { "172.23.122.140:8096": { "worker_timers_0": "[0-1023]" } }, "worker_pids": { "worker_timers_0": 3854 } } ] [2021-02-22 23:04:28,554] - [eventing_base] [140689256060672] - INFO - Execution stats update_success: 0 update_failure: 568 delete_success: 0 delete_failure: 0 [2021-02-22 23:04:28,571] - [basetestcase] [140689256060672] - INFO - list of eventing nodes in cluster: [ip:172.23.122.140 port:8091 ssh_username:root] [2021-02-22 23:04:28,620] - [eventing_base] [140689256060672] - INFO - Stats for Node 172.23.122.140 is [ { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 18, "agg_messages_sent_to_worker": 44797, "agg_queue_memory": 40102434, "agg_queue_memory_cap": 53477376, "agg_queue_size": 39579, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 156409, "dcp_mutation_sent_to_worker": 40156, "dcp_snapshot": 1024, "dcp_stream_req_counter": 1024, "dcp_streamreq": 1024, "dcp_system_event": 8600, "execution_stats": 721, "failure_stats": 721, "latency_stats": 721, "lcb_exception_stats": 721, "log_level": 1, "num_processed_events": 575, "processed_events_size": 513712, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 40102434, "agg_queue_size": 39579, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 577, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 40156, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 44793, "no_op_counter": 0, "num_processed_events": 575, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 575, "on_update_success": 0, "processed_events_size": 513712, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "3710": "2021-02-23T07:04:28Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 575, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "3710": "2021-02-23T07:04:28Z" }, "v8worker_events_lost": 0 }, "function_id": 1325846238, "function_name": "bucket_op", "gocb_creds_request_counter": 23, "internal_vb_distribution_stats": { "worker_bucket_op_0": "[0-1023]" }, "latency_percentile_stats": { "100": 2507100, "50": 2500400, "80": 2500400, "90": 2500400, "95": 2500500, "99": 2501000 }, "lcb_creds_request_counter": 24, "lcb_exception_stats": { "211": 575 }, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.140:8096", "start_vb": 0, "vb_count": 1024 } ], "vb_distribution_stats_from_metadata": { "172.23.122.140:8096": { "worker_bucket_op_0": "[0-1023]" } }, "worker_pids": { "worker_bucket_op_0": 3710 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 5, "agg_messages_sent_to_worker": 5034, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 116253, "dcp_snapshot": 1024, "dcp_stream_req_counter": 1024, "dcp_streamreq": 1024, "execution_stats": 704, "failure_stats": 704, "latency_stats": 704, "lcb_exception_stats": 704, "log_level": 1, "num_processed_events": 479, "processed_events_size": 53436, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 0, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 0, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 5030, "no_op_counter": 479, "num_processed_events": 479, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 0, "processed_events_size": 53436, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "3739": "2021-02-23T07:04:27Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "3739": "2021-02-23T07:04:27Z" }, "v8worker_events_lost": 0 }, "function_id": 657260674, "function_name": "curl", "gocb_creds_request_counter": 23, "internal_vb_distribution_stats": { "worker_curl_0": "[0-1023]" }, "latency_percentile_stats": { "100": 0, "50": 0, "80": 0, "90": 0, "95": 0, "99": 0 }, "lcb_creds_request_counter": 24, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.140:8096", "start_vb": 0, "vb_count": 1024 } ], "vb_distribution_stats_from_metadata": { "172.23.122.140:8096": { "worker_curl_0": "[0-1023]" } }, "worker_pids": { "worker_curl_0": 3739 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 4, "agg_messages_sent_to_worker": 4953, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 116253, "dcp_snapshot": 1024, "dcp_stream_req_counter": 1024, "dcp_streamreq": 1024, "execution_stats": 688, "failure_stats": 688, "latency_stats": 688, "lcb_exception_stats": 688, "log_level": 1, "num_processed_events": 479, "processed_events_size": 53436, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 0, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 0, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 4949, "no_op_counter": 479, "num_processed_events": 479, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 0, "processed_events_size": 53436, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "3769": "2021-02-23T07:04:28Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "3769": "2021-02-23T07:04:28Z" }, "v8worker_events_lost": 0 }, "function_id": 699295493, "function_name": "n1ql", "gocb_creds_request_counter": 23, "internal_vb_distribution_stats": { "worker_n1ql_0": "[0-1023]" }, "latency_percentile_stats": { "100": 0, "50": 0, "80": 0, "90": 0, "95": 0, "99": 0 }, "lcb_creds_request_counter": 24, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.140:8096", "start_vb": 0, "vb_count": 1024 } ], "vb_distribution_stats_from_metadata": { "172.23.122.140:8096": { "worker_n1ql_0": "[0-1023]" } }, "worker_pids": { "worker_n1ql_0": 3769 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "agg_messages_sent_to_worker": 4388, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 1140, "dcp_mutation_suppressed_counter": 1140, "dcp_snapshot": 567, "dcp_stream_req_counter": 1024, "dcp_streamreq": 1024, "execution_stats": 671, "failure_stats": 671, "latency_stats": 671, "lcb_exception_stats": 671, "log_level": 1, "num_processed_events": 0, "processed_events_size": 0, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 0, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 0, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 4384, "no_op_counter": 0, "num_processed_events": 0, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 0, "processed_events_size": 0, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "3796": "2021-02-23T07:04:27Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "3796": "2021-02-23T07:04:27Z" }, "v8worker_events_lost": 0 }, "function_id": 2459091863, "function_name": "sbm", "gocb_creds_request_counter": 23, "internal_vb_distribution_stats": { "worker_sbm_0": "[0-1023]" }, "latency_percentile_stats": { "100": 0, "50": 0, "80": 0, "90": 0, "95": 0, "99": 0 }, "lcb_creds_request_counter": 24, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.122.140:8096", "start_vb": 0, "vb_count": 1024 } ], "vb_distribution_stats_from_metadata": { "172.23.122.140:8096": { "worker_sbm_0": "[0-1023]" } }, "worker_pids": { "worker_sbm_0": 3796 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 3, "agg_messages_sent_to_worker": 18788, "agg_queue_memory": 13431859, "agg_queue_memory_cap": 20971520, "agg_queue_size": 13251, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_mutation": 130734, "dcp_mutation_sent_to_worker": 14481, "dcp_snapshot": 1024, "dcp_stream_req_counter": 1024, "dcp_streamreq": 1024, "execution_stats": 655, "failure_stats": 655, "latency_stats": 655, "lcb_exception_stats": 655, "log_level": 1, "num_processed_events": 1232, "processed_events_size": 1103060, "thr_count": 1, "thr_map": 1, "timer_events": 522, "timer_responses_received": 1232, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 13431859, "agg_queue_size": 13251, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 0, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 1232, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 14481, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 18784, "no_op_counter": 0, "num_processed_events": 1232, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 1232, "processed_events_size": 1103060, "timer_callback_failure": 520, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 1232, "timer_create_failure": 0, "timer_msg_counter": 522, "timer_responses_sent": 0, "timestamp": { "3854": "2021-02-23T07:04:27Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 520, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "3854": "2021-02-23T07:04:27Z" }, "v8worker_events_lost": 0 }, "function_id": 1125753195, "function_name": "timers", "gocb_creds_request_counter": 23, "internal_vb_distribution_stats": { "worker_timers_0": "[0-1023]" }, "latency_percentile_stats": { "100": 2514900, "50": 1500, "80": 2500300, "90": 2500300, "95": 2500400, "99": 2500500 }, "lcb_creds_request_counter": 24, "lcb_exception_stats": { "211": 520 }, "metastore_stats": { "metastore_delete_err": 0, "metastore_deletes": 0, "metastore_not_found": 0, "metastore_scan": 0, "metastore_scan_due": 0, "metastore_scan_err": 0, "metastore_set": 0, "metastore_set_err": 0 }, "planner_stats": [ { "host_name": "172.23.122.140:8096", "start_vb": 0, "vb_count": 1024 } ], "vb_distribution_stats_from_metadata": { "172.23.122.140:8096": { "worker_timers_0": "[0-1023]" } }, "worker_pids": { "worker_timers_0": 3854 } } ] [2021-02-22 23:04:28,637] - [basetestcase] [140689256060672] - INFO - list of eventing nodes in cluster: [ip:172.23.122.140 port:8091 ssh_username:root] [2021-02-22 23:04:28,641] - [remote_util] [140689256060672] - INFO - SSH Connecting to 172.23.122.140 with username:root, attempt#1 of 5 [2021-02-22 23:04:28,752] - [remote_util] [140689256060672] - INFO - SSH Connected to 172.23.122.140 as root [2021-02-22 23:04:29,052] - [remote_util] [140689256060672] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 23:04:29,368] - [remote_util] [140689256060672] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 23:04:29,382] - [rest_client] [140689256060672] - INFO - /diag/eval status on 172.23.122.140:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). [2021-02-22 23:04:29,383] - [remote_util] [140689256060672] - INFO - running command.raw on 172.23.122.140: zgrep "panic" "/opt/couchbase/var/lib/couchbase/logs"/eventing.log* | wc -l [2021-02-22 23:04:30,441] - [remote_util] [140689256060672] - INFO - command executed successfully with root [2021-02-22 23:04:30,442] - [remote_util] [140689256060672] - INFO - running command.raw on 172.23.122.140: ls "/opt/couchbase/var/lib/couchbase/logs"/../crash/| wc -l [2021-02-22 23:04:30,458] - [remote_util] [140689256060672] - INFO - command executed successfully with root [2021-02-22 23:04:30,532] - [basetestcase] [140689256060672] - INFO - ------- Cluster statistics ------- [2021-02-22 23:04:30,532] - [basetestcase] [140689256060672] - INFO - 172.23.122.139:8091 => {'services': ['n1ql'], 'cpu_utilization': 1.004268139593271, 'mem_free': 3486932992, 'mem_total': 4201684992, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-02-22 23:04:30,533] - [basetestcase] [140689256060672] - INFO - 172.23.122.140:8091 => {'services': ['eventing'], 'cpu_utilization': 3.256168913762401, 'mem_free': 3264065536, 'mem_total': 4201684992, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-02-22 23:04:30,533] - [basetestcase] [140689256060672] - INFO - 172.23.122.153:8091 => {'services': ['index'], 'cpu_utilization': 1.015486164001016, 'mem_free': 3333222400, 'mem_total': 4201684992, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-02-22 23:04:30,533] - [basetestcase] [140689256060672] - INFO - 172.23.123.65:8091 => {'services': ['kv'], 'cpu_utilization': 5.629477993858751, 'mem_free': 2779459584, 'mem_total': 4201672704, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-02-22 23:04:30,533] - [basetestcase] [140689256060672] - INFO - --- End of cluster statistics --- [2021-02-22 23:04:30,537] - [remote_util] [140689256060672] - INFO - SSH Connecting to 172.23.123.65 with username:root, attempt#1 of 5 [2021-02-22 23:04:30,648] - [remote_util] [140689256060672] - INFO - SSH Connected to 172.23.123.65 as root [2021-02-22 23:04:30,953] - [remote_util] [140689256060672] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 23:04:31,268] - [remote_util] [140689256060672] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 23:04:31,274] - [remote_util] [140689256060672] - INFO - SSH Connecting to 172.23.122.182 with username:root, attempt#1 of 5 [2021-02-22 23:04:31,385] - [remote_util] [140689256060672] - INFO - SSH Connected to 172.23.122.182 as root [2021-02-22 23:04:31,664] - [remote_util] [140689256060672] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 23:04:32,013] - [remote_util] [140689256060672] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 23:04:32,017] - [remote_util] [140689256060672] - INFO - SSH Connecting to 172.23.122.179 with username:root, attempt#1 of 5 [2021-02-22 23:04:32,127] - [remote_util] [140689256060672] - INFO - SSH Connected to 172.23.122.179 as root [2021-02-22 23:04:32,414] - [remote_util] [140689256060672] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 23:04:32,727] - [remote_util] [140689256060672] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 23:04:32,730] - [remote_util] [140689256060672] - INFO - SSH Connecting to 172.23.122.153 with username:root, attempt#1 of 5 [2021-02-22 23:04:32,837] - [remote_util] [140689256060672] - INFO - SSH Connected to 172.23.122.153 as root [2021-02-22 23:04:33,115] - [remote_util] [140689256060672] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 23:04:33,421] - [remote_util] [140689256060672] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 23:04:33,427] - [remote_util] [140689256060672] - INFO - SSH Connecting to 172.23.122.139 with username:root, attempt#1 of 5 [2021-02-22 23:04:33,534] - [remote_util] [140689256060672] - INFO - SSH Connected to 172.23.122.139 as root [2021-02-22 23:04:33,823] - [remote_util] [140689256060672] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 23:04:34,141] - [remote_util] [140689256060672] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 23:04:34,145] - [remote_util] [140689256060672] - INFO - SSH Connecting to 172.23.122.140 with username:root, attempt#1 of 5 [2021-02-22 23:04:34,251] - [remote_util] [140689256060672] - INFO - SSH Connected to 172.23.122.140 as root [2021-02-22 23:04:34,551] - [remote_util] [140689256060672] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 23:04:34,857] - [remote_util] [140689256060672] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 23:04:34,861] - [remote_util] [140689256060672] - INFO - SSH Connecting to 172.23.122.138 with username:root, attempt#1 of 5 [2021-02-22 23:04:34,973] - [remote_util] [140689256060672] - INFO - SSH Connected to 172.23.122.138 as root [2021-02-22 23:04:35,277] - [remote_util] [140689256060672] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 23:04:35,586] - [remote_util] [140689256060672] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 23:04:35,590] - [remote_util] [140689256060672] - INFO - SSH Connecting to 172.23.122.141 with username:root, attempt#1 of 5 [2021-02-22 23:04:35,698] - [remote_util] [140689256060672] - INFO - SSH Connected to 172.23.122.141 as root [2021-02-22 23:04:35,977] - [remote_util] [140689256060672] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 23:04:36,281] - [remote_util] [140689256060672] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 23:06:26,800] - [basetestcase] [140689256060672] - INFO - ============== basetestcase cleanup was started for test #14 test_eventing_rebalance_swap_delete_recreate_collections ============== [2021-02-22 23:06:26,816] - [basetestcase] [140689256060672] - WARNING - Alerts were found: [{'msg': 'Hard out-of-memory error: Bucket "src_bucket" on node 172.23.123.65 is full. No memory currently allocated to this bucket can be easily released.', 'serverTime': '2021-02-22T22:51:45.000Z'}] [2021-02-22 23:06:26,876] - [bucket_helper] [140689256060672] - INFO - deleting existing buckets ['dst_bucket', 'metadata', 'src_bucket'] on 172.23.123.65 [2021-02-22 23:06:28,350] - [bucket_helper] [140689256060672] - INFO - waiting for bucket deletion to complete.... [2021-02-22 23:06:28,391] - [rest_client] [140689256060672] - INFO - node 172.23.123.65 existing buckets : ['metadata', 'src_bucket'] [2021-02-22 23:06:28,394] - [bucket_helper] [140689256060672] - INFO - deleted bucket : dst_bucket from 172.23.123.65 [2021-02-22 23:06:29,011] - [bucket_helper] [140689256060672] - INFO - waiting for bucket deletion to complete.... [2021-02-22 23:06:29,029] - [rest_client] [140689256060672] - INFO - node 172.23.123.65 existing buckets : ['src_bucket'] [2021-02-22 23:06:29,030] - [bucket_helper] [140689256060672] - INFO - deleted bucket : metadata from 172.23.123.65 [2021-02-22 23:06:29,762] - [bucket_helper] [140689256060672] - INFO - waiting for bucket deletion to complete.... [2021-02-22 23:06:29,766] - [rest_client] [140689256060672] - INFO - node 172.23.123.65 existing buckets : [] [2021-02-22 23:06:29,767] - [bucket_helper] [140689256060672] - INFO - deleted bucket : src_bucket from 172.23.123.65 [2021-02-22 23:06:29,781] - [bucket_helper] [140689256060672] - INFO - Could not find any buckets for node 172.23.122.182, nothing to delete [2021-02-22 23:06:29,791] - [bucket_helper] [140689256060672] - INFO - Could not find any buckets for node 172.23.122.179, nothing to delete [2021-02-22 23:06:29,799] - [bucket_helper] [140689256060672] - INFO - Could not find any buckets for node 172.23.122.153, nothing to delete [2021-02-22 23:06:29,809] - [bucket_helper] [140689256060672] - INFO - Could not find any buckets for node 172.23.122.139, nothing to delete [2021-02-22 23:06:29,819] - [bucket_helper] [140689256060672] - INFO - Could not find any buckets for node 172.23.122.140, nothing to delete [2021-02-22 23:06:29,827] - [bucket_helper] [140689256060672] - INFO - Could not find any buckets for node 172.23.122.138, nothing to delete [2021-02-22 23:06:29,835] - [bucket_helper] [140689256060672] - INFO - Could not find any buckets for node 172.23.122.141, nothing to delete [2021-02-22 23:06:29,842] - [rest_client] [140689256060672] - INFO - -->is_ns_server_running? [2021-02-22 23:06:29,856] - [cluster_helper] [140689256060672] - INFO - rebalancing all nodes in order to remove nodes [2021-02-22 23:06:29,859] - [rest_client] [140689256060672] - INFO - rebalance params : {'knownNodes': 'ns_1@172.23.122.139,ns_1@172.23.122.140,ns_1@172.23.122.153,ns_1@172.23.123.65', 'ejectedNodes': 'ns_1@172.23.122.139,ns_1@172.23.122.140,ns_1@172.23.122.153', 'user': 'Administrator', 'password': 'password'} [2021-02-22 23:06:29,890] - [rest_client] [140689256060672] - INFO - rebalance operation started [2021-02-22 23:06:29,894] - [rest_client] [140689256060672] - INFO - rebalance percentage : 0.00 % [2021-02-22 23:06:39,899] - [rest_client] [140689256060672] - INFO - rebalance percentage : 50.00 % [2021-02-22 23:06:49,903] - [rest_client] [140689256060672] - INFO - rebalance percentage : 75.00 % [2021-02-22 23:07:09,926] - [rest_client] [140689256060672] - INFO - rebalance progress took 40.04 seconds [2021-02-22 23:07:09,926] - [rest_client] [140689256060672] - INFO - sleep for 10 seconds after rebalance... [2021-02-22 23:07:19,956] - [cluster_helper] [140689256060672] - INFO - removed all the nodes from cluster associated with ip:172.23.123.65 port:8091 ssh_username:root ? [('ns_1@172.23.122.139', 8091), ('ns_1@172.23.122.140', 8091), ('ns_1@172.23.122.153', 8091)] [2021-02-22 23:07:19,962] - [cluster_helper] [140689256060672] - INFO - waiting for ns_server @ 172.23.123.65:8091 [2021-02-22 23:07:19,962] - [rest_client] [140689256060672] - INFO - -->is_ns_server_running? [2021-02-22 23:07:19,967] - [cluster_helper] [140689256060672] - INFO - ns_server @ 172.23.123.65:8091 is running [2021-02-22 23:07:19,972] - [cluster_helper] [140689256060672] - INFO - waiting for ns_server @ 172.23.122.182:8091 [2021-02-22 23:07:19,972] - [rest_client] [140689256060672] - INFO - -->is_ns_server_running? [2021-02-22 23:07:19,976] - [cluster_helper] [140689256060672] - INFO - ns_server @ 172.23.122.182:8091 is running [2021-02-22 23:07:19,980] - [cluster_helper] [140689256060672] - INFO - waiting for ns_server @ 172.23.122.179:8091 [2021-02-22 23:07:19,981] - [rest_client] [140689256060672] - INFO - -->is_ns_server_running? [2021-02-22 23:07:19,985] - [cluster_helper] [140689256060672] - INFO - ns_server @ 172.23.122.179:8091 is running [2021-02-22 23:07:19,989] - [cluster_helper] [140689256060672] - INFO - waiting for ns_server @ 172.23.122.153:8091 [2021-02-22 23:07:19,989] - [rest_client] [140689256060672] - INFO - -->is_ns_server_running? [2021-02-22 23:07:19,993] - [cluster_helper] [140689256060672] - INFO - ns_server @ 172.23.122.153:8091 is running [2021-02-22 23:07:19,997] - [cluster_helper] [140689256060672] - INFO - waiting for ns_server @ 172.23.122.139:8091 [2021-02-22 23:07:19,997] - [rest_client] [140689256060672] - INFO - -->is_ns_server_running? [2021-02-22 23:07:20,002] - [cluster_helper] [140689256060672] - INFO - ns_server @ 172.23.122.139:8091 is running [2021-02-22 23:07:20,006] - [cluster_helper] [140689256060672] - INFO - waiting for ns_server @ 172.23.122.140:8091 [2021-02-22 23:07:20,007] - [rest_client] [140689256060672] - INFO - -->is_ns_server_running? [2021-02-22 23:07:20,011] - [cluster_helper] [140689256060672] - INFO - ns_server @ 172.23.122.140:8091 is running [2021-02-22 23:07:20,016] - [cluster_helper] [140689256060672] - INFO - waiting for ns_server @ 172.23.122.138:8091 [2021-02-22 23:07:20,016] - [rest_client] [140689256060672] - INFO - -->is_ns_server_running? [2021-02-22 23:07:20,020] - [cluster_helper] [140689256060672] - INFO - ns_server @ 172.23.122.138:8091 is running [2021-02-22 23:07:20,025] - [cluster_helper] [140689256060672] - INFO - waiting for ns_server @ 172.23.122.141:8091 [2021-02-22 23:07:20,025] - [rest_client] [140689256060672] - INFO - -->is_ns_server_running? [2021-02-22 23:07:20,029] - [cluster_helper] [140689256060672] - INFO - ns_server @ 172.23.122.141:8091 is running [2021-02-22 23:07:20,030] - [basetestcase] [140689256060672] - INFO - ============== basetestcase cleanup was finished for test #14 test_eventing_rebalance_swap_delete_recreate_collections ============== [2021-02-22 23:07:20,030] - [basetestcase] [140689256060672] - INFO - closing all ssh connections [2021-02-22 23:07:20,051] - [basetestcase] [140689256060672] - INFO - closing all memcached connections [2021-02-22 23:07:20,055] - [remote_util] [140689256060672] - INFO - SSH Connecting to 172.23.123.65 with username:root, attempt#1 of 5 [2021-02-22 23:07:20,156] - [remote_util] [140689256060672] - INFO - SSH Connected to 172.23.123.65 as root [2021-02-22 23:07:20,442] - [remote_util] [140689256060672] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 23:07:20,745] - [remote_util] [140689256060672] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 23:07:20,759] - [remote_util] [140689256060672] - INFO - SSH Connecting to 172.23.123.65 with username:root, attempt#1 of 5 [2021-02-22 23:07:20,857] - [remote_util] [140689256060672] - INFO - SSH Connected to 172.23.123.65 as root [2021-02-22 23:07:21,131] - [remote_util] [140689256060672] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 23:07:21,433] - [remote_util] [140689256060672] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 23:07:21,496] - [remote_util] [140689256060672] - INFO - 172.23.123.65 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-22 23:07:21,601] - [remote_util] [140689256060672] - INFO - Running systemd command on this server [2021-02-22 23:07:21,602] - [remote_util] [140689256060672] - INFO - running command.raw on 172.23.123.65: systemctl stop couchbase-server.service [2021-02-22 23:07:23,340] - [remote_util] [140689256060672] - INFO - command executed successfully with root [2021-02-22 23:07:23,341] - [basetestcase] [140689256060672] - INFO - Couchbase stopped [2021-02-22 23:07:23,341] - [basetestcase] [140689256060672] - INFO - sleep for 5 secs. ... [2021-02-22 23:07:28,346] - [remote_util] [140689256060672] - INFO - running command.raw on 172.23.123.65: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-22 23:07:28,364] - [remote_util] [140689256060672] - INFO - command executed successfully with root [2021-02-22 23:07:28,364] - [remote_util] [140689256060672] - INFO - running command.raw on 172.23.123.65: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-22 23:07:28,420] - [remote_util] [140689256060672] - INFO - command executed successfully with root [2021-02-22 23:07:28,424] - [remote_util] [140689256060672] - INFO - SSH Connecting to 172.23.123.65 with username:root, attempt#1 of 5 [2021-02-22 23:07:28,525] - [remote_util] [140689256060672] - INFO - SSH Connected to 172.23.123.65 as root [2021-02-22 23:07:28,808] - [remote_util] [140689256060672] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 23:07:29,117] - [remote_util] [140689256060672] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 23:07:29,178] - [remote_util] [140689256060672] - INFO - 172.23.123.65 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-22 23:07:29,179] - [remote_util] [140689256060672] - INFO - Checking for process beam.smp on linux [2021-02-22 23:07:29,258] - [remote_util] [140689256060672] - INFO - Starting couchbase server [2021-02-22 23:07:29,359] - [remote_util] [140689256060672] - INFO - Running systemd command on this server [2021-02-22 23:07:29,359] - [remote_util] [140689256060672] - INFO - running command.raw on 172.23.123.65: systemctl start couchbase-server.service [2021-02-22 23:07:29,383] - [remote_util] [140689256060672] - INFO - command executed successfully with root [2021-02-22 23:07:29,383] - [remote_util] [140689256060672] - INFO - 172.23.123.65:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-22 23:07:34,389] - [remote_util] [140689256060672] - INFO - running command.raw on 172.23.123.65: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-22 23:07:34,418] - [remote_util] [140689256060672] - INFO - command executed successfully with root [2021-02-22 23:07:34,418] - [remote_util] [140689256060672] - INFO - Couchbase server status: [' Process: 25857 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-22 23:07:34,419] - [remote_util] [140689256060672] - INFO - Checking for process beam.smp on linux [2021-02-22 23:07:34,488] - [remote_util] [140689256060672] - INFO - process beam.smp is running on 172.23.123.65: with pid 26000 [2021-02-22 23:07:34,488] - [basetestcase] [140689256060672] - INFO - Couchbase started [2021-02-22 23:07:34,492] - [remote_util] [140689256060672] - INFO - SSH Connecting to 172.23.122.182 with username:root, attempt#1 of 5 [2021-02-22 23:07:34,596] - [remote_util] [140689256060672] - INFO - SSH Connected to 172.23.122.182 as root [2021-02-22 23:07:34,857] - [remote_util] [140689256060672] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 23:07:35,165] - [remote_util] [140689256060672] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 23:07:35,177] - [remote_util] [140689256060672] - INFO - SSH Connecting to 172.23.122.182 with username:root, attempt#1 of 5 [2021-02-22 23:07:35,278] - [remote_util] [140689256060672] - INFO - SSH Connected to 172.23.122.182 as root [2021-02-22 23:07:35,542] - [remote_util] [140689256060672] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 23:07:35,845] - [remote_util] [140689256060672] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 23:07:35,907] - [remote_util] [140689256060672] - INFO - 172.23.122.182 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-22 23:07:36,009] - [remote_util] [140689256060672] - INFO - Running systemd command on this server [2021-02-22 23:07:36,009] - [remote_util] [140689256060672] - INFO - running command.raw on 172.23.122.182: systemctl stop couchbase-server.service [2021-02-22 23:07:37,721] - [remote_util] [140689256060672] - INFO - command executed successfully with root [2021-02-22 23:07:37,722] - [basetestcase] [140689256060672] - INFO - Couchbase stopped [2021-02-22 23:07:37,722] - [basetestcase] [140689256060672] - INFO - sleep for 5 secs. ... [2021-02-22 23:07:42,724] - [remote_util] [140689256060672] - INFO - running command.raw on 172.23.122.182: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-22 23:07:42,745] - [remote_util] [140689256060672] - INFO - command executed successfully with root [2021-02-22 23:07:42,745] - [remote_util] [140689256060672] - INFO - running command.raw on 172.23.122.182: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-22 23:07:42,798] - [remote_util] [140689256060672] - INFO - command executed successfully with root [2021-02-22 23:07:42,801] - [remote_util] [140689256060672] - INFO - SSH Connecting to 172.23.122.182 with username:root, attempt#1 of 5 [2021-02-22 23:07:42,901] - [remote_util] [140689256060672] - INFO - SSH Connected to 172.23.122.182 as root [2021-02-22 23:07:43,157] - [remote_util] [140689256060672] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 23:07:43,464] - [remote_util] [140689256060672] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 23:07:43,526] - [remote_util] [140689256060672] - INFO - 172.23.122.182 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-22 23:07:43,526] - [remote_util] [140689256060672] - INFO - Checking for process beam.smp on linux [2021-02-22 23:07:43,603] - [remote_util] [140689256060672] - INFO - Starting couchbase server [2021-02-22 23:07:43,707] - [remote_util] [140689256060672] - INFO - Running systemd command on this server [2021-02-22 23:07:43,707] - [remote_util] [140689256060672] - INFO - running command.raw on 172.23.122.182: systemctl start couchbase-server.service [2021-02-22 23:07:43,731] - [remote_util] [140689256060672] - INFO - command executed successfully with root [2021-02-22 23:07:43,731] - [remote_util] [140689256060672] - INFO - 172.23.122.182:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-22 23:07:48,736] - [remote_util] [140689256060672] - INFO - running command.raw on 172.23.122.182: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-22 23:07:48,763] - [remote_util] [140689256060672] - INFO - command executed successfully with root [2021-02-22 23:07:48,763] - [remote_util] [140689256060672] - INFO - Couchbase server status: [' Process: 60726 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-22 23:07:48,763] - [remote_util] [140689256060672] - INFO - Checking for process beam.smp on linux [2021-02-22 23:07:48,836] - [remote_util] [140689256060672] - INFO - process beam.smp is running on 172.23.122.182: with pid 60870 [2021-02-22 23:07:48,837] - [basetestcase] [140689256060672] - INFO - Couchbase started [2021-02-22 23:07:48,840] - [remote_util] [140689256060672] - INFO - SSH Connecting to 172.23.122.179 with username:root, attempt#1 of 5 [2021-02-22 23:07:48,953] - [remote_util] [140689256060672] - INFO - SSH Connected to 172.23.122.179 as root [2021-02-22 23:07:49,239] - [remote_util] [140689256060672] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 23:07:49,551] - [remote_util] [140689256060672] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 23:07:49,565] - [remote_util] [140689256060672] - INFO - SSH Connecting to 172.23.122.179 with username:root, attempt#1 of 5 [2021-02-22 23:07:49,666] - [remote_util] [140689256060672] - INFO - SSH Connected to 172.23.122.179 as root [2021-02-22 23:07:49,948] - [remote_util] [140689256060672] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 23:07:50,266] - [remote_util] [140689256060672] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 23:07:50,329] - [remote_util] [140689256060672] - INFO - 172.23.122.179 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-22 23:07:50,394] - [remote_util] [140689256060672] - INFO - Running systemd command on this server [2021-02-22 23:07:50,395] - [remote_util] [140689256060672] - INFO - running command.raw on 172.23.122.179: systemctl stop couchbase-server.service [2021-02-22 23:07:52,142] - [remote_util] [140689256060672] - INFO - command executed successfully with root [2021-02-22 23:07:52,142] - [basetestcase] [140689256060672] - INFO - Couchbase stopped [2021-02-22 23:07:52,143] - [basetestcase] [140689256060672] - INFO - sleep for 5 secs. ... [2021-02-22 23:07:57,149] - [remote_util] [140689256060672] - INFO - running command.raw on 172.23.122.179: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-22 23:07:57,171] - [remote_util] [140689256060672] - INFO - command executed successfully with root [2021-02-22 23:07:57,171] - [remote_util] [140689256060672] - INFO - running command.raw on 172.23.122.179: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-22 23:07:57,228] - [remote_util] [140689256060672] - INFO - command executed successfully with root [2021-02-22 23:07:57,231] - [remote_util] [140689256060672] - INFO - SSH Connecting to 172.23.122.179 with username:root, attempt#1 of 5 [2021-02-22 23:07:57,333] - [remote_util] [140689256060672] - INFO - SSH Connected to 172.23.122.179 as root [2021-02-22 23:07:57,613] - [remote_util] [140689256060672] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 23:07:57,923] - [remote_util] [140689256060672] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 23:07:57,986] - [remote_util] [140689256060672] - INFO - 172.23.122.179 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-22 23:07:57,986] - [remote_util] [140689256060672] - INFO - Checking for process beam.smp on linux [2021-02-22 23:07:58,067] - [remote_util] [140689256060672] - INFO - Starting couchbase server [2021-02-22 23:07:58,173] - [remote_util] [140689256060672] - INFO - Running systemd command on this server [2021-02-22 23:07:58,173] - [remote_util] [140689256060672] - INFO - running command.raw on 172.23.122.179: systemctl start couchbase-server.service [2021-02-22 23:07:58,198] - [remote_util] [140689256060672] - INFO - command executed successfully with root [2021-02-22 23:07:58,198] - [remote_util] [140689256060672] - INFO - 172.23.122.179:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-22 23:08:03,201] - [remote_util] [140689256060672] - INFO - running command.raw on 172.23.122.179: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-22 23:08:03,229] - [remote_util] [140689256060672] - INFO - command executed successfully with root [2021-02-22 23:08:03,229] - [remote_util] [140689256060672] - INFO - Couchbase server status: [' Process: 91067 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-22 23:08:03,230] - [remote_util] [140689256060672] - INFO - Checking for process beam.smp on linux [2021-02-22 23:08:03,300] - [remote_util] [140689256060672] - INFO - process beam.smp is running on 172.23.122.179: with pid 91212 [2021-02-22 23:08:03,301] - [basetestcase] [140689256060672] - INFO - Couchbase started [2021-02-22 23:08:03,304] - [remote_util] [140689256060672] - INFO - SSH Connecting to 172.23.122.153 with username:root, attempt#1 of 5 [2021-02-22 23:08:03,408] - [remote_util] [140689256060672] - INFO - SSH Connected to 172.23.122.153 as root [2021-02-22 23:08:03,692] - [remote_util] [140689256060672] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 23:08:03,990] - [remote_util] [140689256060672] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 23:08:04,006] - [remote_util] [140689256060672] - INFO - SSH Connecting to 172.23.122.153 with username:root, attempt#1 of 5 [2021-02-22 23:08:04,106] - [remote_util] [140689256060672] - INFO - SSH Connected to 172.23.122.153 as root [2021-02-22 23:08:04,377] - [remote_util] [140689256060672] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 23:08:04,679] - [remote_util] [140689256060672] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 23:08:04,703] - [remote_util] [140689256060672] - INFO - 172.23.122.153 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-22 23:08:04,844] - [remote_util] [140689256060672] - INFO - Running systemd command on this server [2021-02-22 23:08:04,845] - [remote_util] [140689256060672] - INFO - running command.raw on 172.23.122.153: systemctl stop couchbase-server.service [2021-02-22 23:08:06,654] - [remote_util] [140689256060672] - INFO - command executed successfully with root [2021-02-22 23:08:06,655] - [basetestcase] [140689256060672] - INFO - Couchbase stopped [2021-02-22 23:08:06,656] - [basetestcase] [140689256060672] - INFO - sleep for 5 secs. ... [2021-02-22 23:08:11,661] - [remote_util] [140689256060672] - INFO - running command.raw on 172.23.122.153: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-22 23:08:11,680] - [remote_util] [140689256060672] - INFO - command executed successfully with root [2021-02-22 23:08:11,681] - [remote_util] [140689256060672] - INFO - running command.raw on 172.23.122.153: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-22 23:08:11,734] - [remote_util] [140689256060672] - INFO - command executed successfully with root [2021-02-22 23:08:11,737] - [remote_util] [140689256060672] - INFO - SSH Connecting to 172.23.122.153 with username:root, attempt#1 of 5 [2021-02-22 23:08:11,838] - [remote_util] [140689256060672] - INFO - SSH Connected to 172.23.122.153 as root [2021-02-22 23:08:12,106] - [remote_util] [140689256060672] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 23:08:12,447] - [remote_util] [140689256060672] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 23:08:12,508] - [remote_util] [140689256060672] - INFO - 172.23.122.153 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-22 23:08:12,508] - [remote_util] [140689256060672] - INFO - Checking for process beam.smp on linux [2021-02-22 23:08:12,586] - [remote_util] [140689256060672] - INFO - Starting couchbase server [2021-02-22 23:08:12,766] - [remote_util] [140689256060672] - INFO - Running systemd command on this server [2021-02-22 23:08:12,767] - [remote_util] [140689256060672] - INFO - running command.raw on 172.23.122.153: systemctl start couchbase-server.service [2021-02-22 23:08:12,790] - [remote_util] [140689256060672] - INFO - command executed successfully with root [2021-02-22 23:08:12,790] - [remote_util] [140689256060672] - INFO - 172.23.122.153:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-22 23:08:17,796] - [remote_util] [140689256060672] - INFO - running command.raw on 172.23.122.153: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-22 23:08:17,820] - [remote_util] [140689256060672] - INFO - command executed successfully with root [2021-02-22 23:08:17,820] - [remote_util] [140689256060672] - INFO - Couchbase server status: [' Process: 120681 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-22 23:08:17,820] - [remote_util] [140689256060672] - INFO - Checking for process beam.smp on linux [2021-02-22 23:08:17,887] - [remote_util] [140689256060672] - INFO - process beam.smp is running on 172.23.122.153: with pid 120825 [2021-02-22 23:08:17,888] - [basetestcase] [140689256060672] - INFO - Couchbase started [2021-02-22 23:08:17,891] - [remote_util] [140689256060672] - INFO - SSH Connecting to 172.23.122.139 with username:root, attempt#1 of 5 [2021-02-22 23:08:18,003] - [remote_util] [140689256060672] - INFO - SSH Connected to 172.23.122.139 as root [2021-02-22 23:08:18,299] - [remote_util] [140689256060672] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 23:08:18,620] - [remote_util] [140689256060672] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 23:08:18,632] - [remote_util] [140689256060672] - INFO - SSH Connecting to 172.23.122.139 with username:root, attempt#1 of 5 [2021-02-22 23:08:18,737] - [remote_util] [140689256060672] - INFO - SSH Connected to 172.23.122.139 as root [2021-02-22 23:08:19,031] - [remote_util] [140689256060672] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 23:08:19,349] - [remote_util] [140689256060672] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 23:08:19,414] - [remote_util] [140689256060672] - INFO - 172.23.122.139 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-22 23:08:19,527] - [remote_util] [140689256060672] - INFO - Running systemd command on this server [2021-02-22 23:08:19,528] - [remote_util] [140689256060672] - INFO - running command.raw on 172.23.122.139: systemctl stop couchbase-server.service [2021-02-22 23:08:21,263] - [remote_util] [140689256060672] - INFO - command executed successfully with root [2021-02-22 23:08:21,263] - [basetestcase] [140689256060672] - INFO - Couchbase stopped [2021-02-22 23:08:21,264] - [basetestcase] [140689256060672] - INFO - sleep for 5 secs. ... [2021-02-22 23:08:26,270] - [remote_util] [140689256060672] - INFO - running command.raw on 172.23.122.139: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-22 23:08:26,288] - [remote_util] [140689256060672] - INFO - command executed successfully with root [2021-02-22 23:08:26,289] - [remote_util] [140689256060672] - INFO - running command.raw on 172.23.122.139: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-22 23:08:26,343] - [remote_util] [140689256060672] - INFO - command executed successfully with root [2021-02-22 23:08:26,347] - [remote_util] [140689256060672] - INFO - SSH Connecting to 172.23.122.139 with username:root, attempt#1 of 5 [2021-02-22 23:08:26,453] - [remote_util] [140689256060672] - INFO - SSH Connected to 172.23.122.139 as root [2021-02-22 23:08:26,733] - [remote_util] [140689256060672] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 23:08:27,042] - [remote_util] [140689256060672] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 23:08:27,106] - [remote_util] [140689256060672] - INFO - 172.23.122.139 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-22 23:08:27,106] - [remote_util] [140689256060672] - INFO - Checking for process beam.smp on linux [2021-02-22 23:08:27,187] - [remote_util] [140689256060672] - INFO - Starting couchbase server [2021-02-22 23:08:27,295] - [remote_util] [140689256060672] - INFO - Running systemd command on this server [2021-02-22 23:08:27,295] - [remote_util] [140689256060672] - INFO - running command.raw on 172.23.122.139: systemctl start couchbase-server.service [2021-02-22 23:08:27,320] - [remote_util] [140689256060672] - INFO - command executed successfully with root [2021-02-22 23:08:27,320] - [remote_util] [140689256060672] - INFO - 172.23.122.139:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-22 23:08:32,326] - [remote_util] [140689256060672] - INFO - running command.raw on 172.23.122.139: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-22 23:08:32,353] - [remote_util] [140689256060672] - INFO - command executed successfully with root [2021-02-22 23:08:32,353] - [remote_util] [140689256060672] - INFO - Couchbase server status: [' Process: 74942 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-22 23:08:32,354] - [remote_util] [140689256060672] - INFO - Checking for process beam.smp on linux [2021-02-22 23:08:32,426] - [remote_util] [140689256060672] - INFO - process beam.smp is running on 172.23.122.139: with pid 75086 [2021-02-22 23:08:32,426] - [basetestcase] [140689256060672] - INFO - Couchbase started [2021-02-22 23:08:32,430] - [remote_util] [140689256060672] - INFO - SSH Connecting to 172.23.122.140 with username:root, attempt#1 of 5 [2021-02-22 23:08:32,537] - [remote_util] [140689256060672] - INFO - SSH Connected to 172.23.122.140 as root [2021-02-22 23:08:32,835] - [remote_util] [140689256060672] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 23:08:33,142] - [remote_util] [140689256060672] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 23:08:33,156] - [remote_util] [140689256060672] - INFO - SSH Connecting to 172.23.122.140 with username:root, attempt#1 of 5 [2021-02-22 23:08:33,258] - [remote_util] [140689256060672] - INFO - SSH Connected to 172.23.122.140 as root [2021-02-22 23:08:33,550] - [remote_util] [140689256060672] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 23:08:33,861] - [remote_util] [140689256060672] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 23:08:33,925] - [remote_util] [140689256060672] - INFO - 172.23.122.140 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-22 23:08:34,034] - [remote_util] [140689256060672] - INFO - Running systemd command on this server [2021-02-22 23:08:34,034] - [remote_util] [140689256060672] - INFO - running command.raw on 172.23.122.140: systemctl stop couchbase-server.service [2021-02-22 23:08:35,850] - [remote_util] [140689256060672] - INFO - command executed successfully with root [2021-02-22 23:08:35,851] - [basetestcase] [140689256060672] - INFO - Couchbase stopped [2021-02-22 23:08:35,852] - [basetestcase] [140689256060672] - INFO - sleep for 5 secs. ... [2021-02-22 23:08:40,857] - [remote_util] [140689256060672] - INFO - running command.raw on 172.23.122.140: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-22 23:08:40,876] - [remote_util] [140689256060672] - INFO - command executed successfully with root [2021-02-22 23:08:40,876] - [remote_util] [140689256060672] - INFO - running command.raw on 172.23.122.140: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-22 23:08:40,931] - [remote_util] [140689256060672] - INFO - command executed successfully with root [2021-02-22 23:08:40,935] - [remote_util] [140689256060672] - INFO - SSH Connecting to 172.23.122.140 with username:root, attempt#1 of 5 [2021-02-22 23:08:41,039] - [remote_util] [140689256060672] - INFO - SSH Connected to 172.23.122.140 as root [2021-02-22 23:08:41,340] - [remote_util] [140689256060672] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 23:08:41,647] - [remote_util] [140689256060672] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 23:08:41,709] - [remote_util] [140689256060672] - INFO - 172.23.122.140 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-22 23:08:41,709] - [remote_util] [140689256060672] - INFO - Checking for process beam.smp on linux [2021-02-22 23:08:41,790] - [remote_util] [140689256060672] - INFO - Starting couchbase server [2021-02-22 23:08:41,893] - [remote_util] [140689256060672] - INFO - Running systemd command on this server [2021-02-22 23:08:41,894] - [remote_util] [140689256060672] - INFO - running command.raw on 172.23.122.140: systemctl start couchbase-server.service [2021-02-22 23:08:41,917] - [remote_util] [140689256060672] - INFO - command executed successfully with root [2021-02-22 23:08:41,918] - [remote_util] [140689256060672] - INFO - 172.23.122.140:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-22 23:08:46,923] - [remote_util] [140689256060672] - INFO - running command.raw on 172.23.122.140: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-22 23:08:46,950] - [remote_util] [140689256060672] - INFO - command executed successfully with root [2021-02-22 23:08:46,950] - [remote_util] [140689256060672] - INFO - Couchbase server status: [' Process: 8445 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-22 23:08:46,951] - [remote_util] [140689256060672] - INFO - Checking for process beam.smp on linux [2021-02-22 23:08:47,021] - [remote_util] [140689256060672] - INFO - process beam.smp is running on 172.23.122.140: with pid 8588 [2021-02-22 23:08:47,022] - [basetestcase] [140689256060672] - INFO - Couchbase started [2021-02-22 23:08:47,025] - [remote_util] [140689256060672] - INFO - SSH Connecting to 172.23.122.138 with username:root, attempt#1 of 5 [2021-02-22 23:08:47,134] - [remote_util] [140689256060672] - INFO - SSH Connected to 172.23.122.138 as root [2021-02-22 23:08:47,439] - [remote_util] [140689256060672] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 23:08:47,754] - [remote_util] [140689256060672] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 23:08:47,767] - [remote_util] [140689256060672] - INFO - SSH Connecting to 172.23.122.138 with username:root, attempt#1 of 5 [2021-02-22 23:08:47,870] - [remote_util] [140689256060672] - INFO - SSH Connected to 172.23.122.138 as root [2021-02-22 23:08:48,160] - [remote_util] [140689256060672] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 23:08:48,476] - [remote_util] [140689256060672] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 23:08:48,540] - [remote_util] [140689256060672] - INFO - 172.23.122.138 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-22 23:08:48,648] - [remote_util] [140689256060672] - INFO - Running systemd command on this server [2021-02-22 23:08:48,648] - [remote_util] [140689256060672] - INFO - running command.raw on 172.23.122.138: systemctl stop couchbase-server.service [2021-02-22 23:08:50,557] - [remote_util] [140689256060672] - INFO - command executed successfully with root [2021-02-22 23:08:50,557] - [basetestcase] [140689256060672] - INFO - Couchbase stopped [2021-02-22 23:08:50,558] - [basetestcase] [140689256060672] - INFO - sleep for 5 secs. ... [2021-02-22 23:08:55,563] - [remote_util] [140689256060672] - INFO - running command.raw on 172.23.122.138: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-22 23:08:55,582] - [remote_util] [140689256060672] - INFO - command executed successfully with root [2021-02-22 23:08:55,582] - [remote_util] [140689256060672] - INFO - running command.raw on 172.23.122.138: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-22 23:08:55,638] - [remote_util] [140689256060672] - INFO - command executed successfully with root [2021-02-22 23:08:55,642] - [remote_util] [140689256060672] - INFO - SSH Connecting to 172.23.122.138 with username:root, attempt#1 of 5 [2021-02-22 23:08:55,749] - [remote_util] [140689256060672] - INFO - SSH Connected to 172.23.122.138 as root [2021-02-22 23:08:56,045] - [remote_util] [140689256060672] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 23:08:56,355] - [remote_util] [140689256060672] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 23:08:56,418] - [remote_util] [140689256060672] - INFO - 172.23.122.138 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-22 23:08:56,418] - [remote_util] [140689256060672] - INFO - Checking for process beam.smp on linux [2021-02-22 23:08:56,501] - [remote_util] [140689256060672] - INFO - Starting couchbase server [2021-02-22 23:08:56,610] - [remote_util] [140689256060672] - INFO - Running systemd command on this server [2021-02-22 23:08:56,610] - [remote_util] [140689256060672] - INFO - running command.raw on 172.23.122.138: systemctl start couchbase-server.service [2021-02-22 23:08:56,637] - [remote_util] [140689256060672] - INFO - command executed successfully with root [2021-02-22 23:08:56,637] - [remote_util] [140689256060672] - INFO - 172.23.122.138:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-22 23:09:01,643] - [remote_util] [140689256060672] - INFO - running command.raw on 172.23.122.138: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-22 23:09:01,673] - [remote_util] [140689256060672] - INFO - command executed successfully with root [2021-02-22 23:09:01,674] - [remote_util] [140689256060672] - INFO - Couchbase server status: [' Process: 94290 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-22 23:09:01,674] - [remote_util] [140689256060672] - INFO - Checking for process beam.smp on linux [2021-02-22 23:09:01,747] - [remote_util] [140689256060672] - INFO - process beam.smp is running on 172.23.122.138: with pid 94435 [2021-02-22 23:09:01,747] - [basetestcase] [140689256060672] - INFO - Couchbase started [2021-02-22 23:09:01,751] - [remote_util] [140689256060672] - INFO - SSH Connecting to 172.23.122.141 with username:root, attempt#1 of 5 [2021-02-22 23:09:01,856] - [remote_util] [140689256060672] - INFO - SSH Connected to 172.23.122.141 as root [2021-02-22 23:09:02,130] - [remote_util] [140689256060672] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 23:09:02,434] - [remote_util] [140689256060672] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 23:09:02,447] - [remote_util] [140689256060672] - INFO - SSH Connecting to 172.23.122.141 with username:root, attempt#1 of 5 [2021-02-22 23:09:02,552] - [remote_util] [140689256060672] - INFO - SSH Connected to 172.23.122.141 as root [2021-02-22 23:09:02,820] - [remote_util] [140689256060672] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 23:09:03,130] - [remote_util] [140689256060672] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 23:09:03,192] - [remote_util] [140689256060672] - INFO - 172.23.122.141 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-22 23:09:03,260] - [remote_util] [140689256060672] - INFO - Running systemd command on this server [2021-02-22 23:09:03,260] - [remote_util] [140689256060672] - INFO - running command.raw on 172.23.122.141: systemctl stop couchbase-server.service [2021-02-22 23:09:05,225] - [remote_util] [140689256060672] - INFO - command executed successfully with root [2021-02-22 23:09:05,225] - [basetestcase] [140689256060672] - INFO - Couchbase stopped [2021-02-22 23:09:05,226] - [basetestcase] [140689256060672] - INFO - sleep for 5 secs. ... [2021-02-22 23:09:10,231] - [remote_util] [140689256060672] - INFO - running command.raw on 172.23.122.141: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-22 23:09:10,248] - [remote_util] [140689256060672] - INFO - command executed successfully with root [2021-02-22 23:09:10,248] - [remote_util] [140689256060672] - INFO - running command.raw on 172.23.122.141: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-22 23:09:10,304] - [remote_util] [140689256060672] - INFO - command executed successfully with root [2021-02-22 23:09:10,307] - [remote_util] [140689256060672] - INFO - SSH Connecting to 172.23.122.141 with username:root, attempt#1 of 5 [2021-02-22 23:09:10,408] - [remote_util] [140689256060672] - INFO - SSH Connected to 172.23.122.141 as root [2021-02-22 23:09:10,684] - [remote_util] [140689256060672] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 23:09:10,986] - [remote_util] [140689256060672] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 23:09:11,048] - [remote_util] [140689256060672] - INFO - 172.23.122.141 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-22 23:09:11,049] - [remote_util] [140689256060672] - INFO - Checking for process beam.smp on linux [2021-02-22 23:09:11,126] - [remote_util] [140689256060672] - INFO - Starting couchbase server [2021-02-22 23:09:11,231] - [remote_util] [140689256060672] - INFO - Running systemd command on this server [2021-02-22 23:09:11,231] - [remote_util] [140689256060672] - INFO - running command.raw on 172.23.122.141: systemctl start couchbase-server.service [2021-02-22 23:09:11,254] - [remote_util] [140689256060672] - INFO - command executed successfully with root [2021-02-22 23:09:11,255] - [remote_util] [140689256060672] - INFO - 172.23.122.141:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-22 23:09:16,261] - [remote_util] [140689256060672] - INFO - running command.raw on 172.23.122.141: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-22 23:09:16,289] - [remote_util] [140689256060672] - INFO - command executed successfully with root [2021-02-22 23:09:16,289] - [remote_util] [140689256060672] - INFO - Couchbase server status: [' Process: 61436 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-22 23:09:16,289] - [remote_util] [140689256060672] - INFO - Checking for process beam.smp on linux [2021-02-22 23:09:16,356] - [remote_util] [140689256060672] - INFO - process beam.smp is running on 172.23.122.141: with pid 61581 [2021-02-22 23:09:16,357] - [basetestcase] [140689256060672] - INFO - Couchbase started [2021-02-22 23:09:16,357] - [basetestcase] [140689256060672] - INFO - sleep for 10 secs. ... [2021-02-22 23:09:26,408] - [remote_util] [140689256060672] - INFO - SSH Connecting to 172.23.123.65 with username:root, attempt#1 of 5 [2021-02-22 23:09:26,416] - [remote_util] [140689002182400] - INFO - SSH Connecting to 172.23.122.182 with username:root, attempt#1 of 5 [2021-02-22 23:09:26,417] - [remote_util] [140688993789696] - INFO - SSH Connecting to 172.23.122.179 with username:root, attempt#1 of 5 [2021-02-22 23:09:26,420] - [remote_util] [140688985396992] - INFO - SSH Connecting to 172.23.122.153 with username:root, attempt#1 of 5 [2021-02-22 23:09:26,423] - [remote_util] [140688423380736] - INFO - SSH Connecting to 172.23.122.139 with username:root, attempt#1 of 5 [2021-02-22 23:09:26,428] - [remote_util] [140688384304896] - INFO - SSH Connecting to 172.23.122.138 with username:root, attempt#1 of 5 [2021-02-22 23:09:26,435] - [remote_util] [140686814869248] - INFO - SSH Connecting to 172.23.122.141 with username:root, attempt#1 of 5 [2021-02-22 23:09:26,439] - [remote_util] [140688407123712] - INFO - SSH Connecting to 172.23.122.140 with username:root, attempt#1 of 5 [2021-02-22 23:09:26,536] - [remote_util] [140688993789696] - INFO - SSH Connected to 172.23.122.179 as root [2021-02-22 23:09:26,541] - [remote_util] [140689002182400] - INFO - SSH Connected to 172.23.122.182 as root [2021-02-22 23:09:26,547] - [remote_util] [140688985396992] - INFO - SSH Connected to 172.23.122.153 as root [2021-02-22 23:09:26,554] - [remote_util] [140688423380736] - INFO - SSH Connected to 172.23.122.139 as root [2021-02-22 23:09:26,557] - [remote_util] [140689256060672] - INFO - SSH Connected to 172.23.123.65 as root [2021-02-22 23:09:26,561] - [remote_util] [140688407123712] - INFO - SSH Connected to 172.23.122.140 as root [2021-02-22 23:09:26,587] - [remote_util] [140686814869248] - INFO - SSH Connected to 172.23.122.141 as root [2021-02-22 23:09:26,589] - [remote_util] [140688384304896] - INFO - SSH Connected to 172.23.122.138 as root [2021-02-22 23:09:26,899] - [remote_util] [140688985396992] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 23:09:26,909] - [remote_util] [140688993789696] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 23:09:26,937] - [remote_util] [140689256060672] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 23:09:26,954] - [remote_util] [140689002182400] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 23:09:26,960] - [remote_util] [140688423380736] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 23:09:26,969] - [remote_util] [140688407123712] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 23:09:26,981] - [remote_util] [140686814869248] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 23:09:27,006] - [remote_util] [140688384304896] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 23:09:27,293] - [remote_util] [140688993789696] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 23:09:27,295] - [remote_util] [140688993789696] - INFO - running command.raw on 172.23.122.179: /opt/couchbase/bin/cbcollect_info 172.23.122.179-20210222-2309-diag.zip [2021-02-22 23:09:27,301] - [remote_util] [140688985396992] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 23:09:27,304] - [remote_util] [140688985396992] - INFO - running command.raw on 172.23.122.153: /opt/couchbase/bin/cbcollect_info 172.23.122.153-20210222-2309-diag.zip [2021-02-22 23:09:27,307] - [remote_util] [140689256060672] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 23:09:27,308] - [remote_util] [140689256060672] - INFO - running command.raw on 172.23.123.65: /opt/couchbase/bin/cbcollect_info 172.23.123.65-20210222-2309-diag.zip [2021-02-22 23:09:27,325] - [remote_util] [140688384304896] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 23:09:27,326] - [remote_util] [140688384304896] - INFO - running command.raw on 172.23.122.138: /opt/couchbase/bin/cbcollect_info 172.23.122.138-20210222-2309-diag.zip [2021-02-22 23:09:27,337] - [remote_util] [140688423380736] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 23:09:27,340] - [remote_util] [140688423380736] - INFO - running command.raw on 172.23.122.139: /opt/couchbase/bin/cbcollect_info 172.23.122.139-20210222-2309-diag.zip [2021-02-22 23:09:27,346] - [remote_util] [140689002182400] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 23:09:27,348] - [remote_util] [140689002182400] - INFO - running command.raw on 172.23.122.182: /opt/couchbase/bin/cbcollect_info 172.23.122.182-20210222-2309-diag.zip [2021-02-22 23:09:27,355] - [remote_util] [140688407123712] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 23:09:27,359] - [remote_util] [140688407123712] - INFO - running command.raw on 172.23.122.140: /opt/couchbase/bin/cbcollect_info 172.23.122.140-20210222-2309-diag.zip [2021-02-22 23:09:27,361] - [remote_util] [140686814869248] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 23:09:27,363] - [remote_util] [140686814869248] - INFO - running command.raw on 172.23.122.141: /opt/couchbase/bin/cbcollect_info 172.23.122.141-20210222-2309-diag.zip [2021-02-22 23:10:54,533] - [remote_util] [140686814869248] - INFO - command executed successfully with root [2021-02-22 23:10:55,524] - [remote_util] [140686814869248] - INFO - running command.raw on 172.23.122.141: rm -f /root/172.23.122.141-20210222-2309-diag.zip [2021-02-22 23:10:55,584] - [remote_util] [140686814869248] - INFO - command executed successfully with root [2021-02-22 23:11:05,329] - [remote_util] [140688993789696] - INFO - command executed successfully with root [2021-02-22 23:11:06,825] - [remote_util] [140688985396992] - INFO - command executed successfully with root [2021-02-22 23:11:07,436] - [remote_util] [140689002182400] - INFO - command executed successfully with root [2021-02-22 23:11:08,563] - [remote_util] [140688993789696] - INFO - running command.raw on 172.23.122.179: rm -f /root/172.23.122.179-20210222-2309-diag.zip [2021-02-22 23:11:08,631] - [remote_util] [140688993789696] - INFO - command executed successfully with root [2021-02-22 23:11:10,292] - [remote_util] [140688384304896] - INFO - command executed successfully with root [2021-02-22 23:11:12,824] - [remote_util] [140688985396992] - INFO - running command.raw on 172.23.122.153: rm -f /root/172.23.122.153-20210222-2309-diag.zip [2021-02-22 23:11:12,894] - [remote_util] [140688985396992] - INFO - command executed successfully with root [2021-02-22 23:11:13,344] - [remote_util] [140688423380736] - INFO - command executed successfully with root [2021-02-22 23:11:13,546] - [remote_util] [140689002182400] - INFO - running command.raw on 172.23.122.182: rm -f /root/172.23.122.182-20210222-2309-diag.zip [2021-02-22 23:11:13,613] - [remote_util] [140689002182400] - INFO - command executed successfully with root [2021-02-22 23:11:13,793] - [remote_util] [140689256060672] - INFO - command executed successfully with root [2021-02-22 23:11:15,396] - [remote_util] [140688384304896] - INFO - running command.raw on 172.23.122.138: rm -f /root/172.23.122.138-20210222-2309-diag.zip [2021-02-22 23:11:15,457] - [remote_util] [140688384304896] - INFO - command executed successfully with root [2021-02-22 23:11:15,843] - [remote_util] [140688407123712] - INFO - command executed successfully with root [2021-02-22 23:11:20,088] - [remote_util] [140688423380736] - INFO - running command.raw on 172.23.122.139: rm -f /root/172.23.122.139-20210222-2309-diag.zip [2021-02-22 23:11:20,152] - [remote_util] [140688423380736] - INFO - command executed successfully with root [2021-02-22 23:11:21,348] - [remote_util] [140689256060672] - INFO - running command.raw on 172.23.123.65: rm -f /root/172.23.123.65-20210222-2309-diag.zip [2021-02-22 23:11:21,411] - [remote_util] [140689256060672] - INFO - command executed successfully with root [2021-02-22 23:11:21,710] - [remote_util] [140688407123712] - INFO - running command.raw on 172.23.122.140: rm -f /root/172.23.122.140-20210222-2309-diag.zip [2021-02-22 23:11:21,771] - [remote_util] [140688407123712] - INFO - command executed successfully with root [2021-02-22 23:11:22,066] - [eventing_base] [140689549858624] - INFO - Starting Test: suite_tearDown [2021-02-22 23:11:22,070] - [remote_util] [140689549858624] - INFO - SSH Connecting to 172.23.123.65 with username:root, attempt#1 of 5 [2021-02-22 23:11:22,173] - [remote_util] [140689549858624] - INFO - SSH Connected to 172.23.123.65 as root [2021-02-22 23:11:22,451] - [remote_util] [140689549858624] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 23:11:22,792] - [remote_util] [140689549858624] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 23:11:22,798] - [rest_client] [140689549858624] - INFO - -->is_ns_server_running? [2021-02-22 23:11:22,808] - [rest_client] [140689549858624] - INFO - Node version in cluster 7.0.0-4514-enterprise [2021-02-22 23:11:22,814] - [rest_client] [140689549858624] - ERROR - GET http://172.23.123.65:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-02-22 23:11:22,815] - [rest_client] [140689549858624] - INFO - Node versions in cluster [] [2021-02-22 23:11:22,815] - [basetestcase] [140689549858624] - INFO - ============== basetestcase setup was started for test #14 suite_tearDown============== [2021-02-22 23:11:22,821] - [rest_client] [140689549858624] - ERROR - GET http://172.23.123.65:8091/pools/default/ body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-02-22 23:11:22,838] - [bucket_helper] [140689549858624] - INFO - Could not find any buckets for node 172.23.123.65, nothing to delete [2021-02-22 23:11:22,850] - [bucket_helper] [140689549858624] - INFO - Could not find any buckets for node 172.23.122.182, nothing to delete [2021-02-22 23:11:22,862] - [bucket_helper] [140689549858624] - INFO - Could not find any buckets for node 172.23.122.179, nothing to delete [2021-02-22 23:11:22,875] - [bucket_helper] [140689549858624] - INFO - Could not find any buckets for node 172.23.122.153, nothing to delete [2021-02-22 23:11:22,887] - [bucket_helper] [140689549858624] - INFO - Could not find any buckets for node 172.23.122.139, nothing to delete [2021-02-22 23:11:22,899] - [bucket_helper] [140689549858624] - INFO - Could not find any buckets for node 172.23.122.140, nothing to delete [2021-02-22 23:11:22,911] - [bucket_helper] [140689549858624] - INFO - Could not find any buckets for node 172.23.122.138, nothing to delete [2021-02-22 23:11:22,922] - [bucket_helper] [140689549858624] - INFO - Could not find any buckets for node 172.23.122.141, nothing to delete [2021-02-22 23:11:22,926] - [rest_client] [140689549858624] - INFO - -->is_ns_server_running? [2021-02-22 23:11:22,938] - [cluster_helper] [140689549858624] - INFO - waiting for ns_server @ 172.23.123.65:8091 [2021-02-22 23:11:22,939] - [rest_client] [140689549858624] - INFO - -->is_ns_server_running? [2021-02-22 23:11:22,942] - [cluster_helper] [140689549858624] - INFO - ns_server @ 172.23.123.65:8091 is running [2021-02-22 23:11:22,947] - [cluster_helper] [140689549858624] - INFO - waiting for ns_server @ 172.23.122.182:8091 [2021-02-22 23:11:22,947] - [rest_client] [140689549858624] - INFO - -->is_ns_server_running? [2021-02-22 23:11:22,951] - [cluster_helper] [140689549858624] - INFO - ns_server @ 172.23.122.182:8091 is running [2021-02-22 23:11:22,955] - [cluster_helper] [140689549858624] - INFO - waiting for ns_server @ 172.23.122.179:8091 [2021-02-22 23:11:22,956] - [rest_client] [140689549858624] - INFO - -->is_ns_server_running? [2021-02-22 23:11:22,959] - [cluster_helper] [140689549858624] - INFO - ns_server @ 172.23.122.179:8091 is running [2021-02-22 23:11:22,963] - [cluster_helper] [140689549858624] - INFO - waiting for ns_server @ 172.23.122.153:8091 [2021-02-22 23:11:22,963] - [rest_client] [140689549858624] - INFO - -->is_ns_server_running? [2021-02-22 23:11:22,968] - [cluster_helper] [140689549858624] - INFO - ns_server @ 172.23.122.153:8091 is running [2021-02-22 23:11:22,972] - [cluster_helper] [140689549858624] - INFO - waiting for ns_server @ 172.23.122.139:8091 [2021-02-22 23:11:22,972] - [rest_client] [140689549858624] - INFO - -->is_ns_server_running? [2021-02-22 23:11:22,976] - [cluster_helper] [140689549858624] - INFO - ns_server @ 172.23.122.139:8091 is running [2021-02-22 23:11:22,979] - [cluster_helper] [140689549858624] - INFO - waiting for ns_server @ 172.23.122.140:8091 [2021-02-22 23:11:22,980] - [rest_client] [140689549858624] - INFO - -->is_ns_server_running? [2021-02-22 23:11:22,983] - [cluster_helper] [140689549858624] - INFO - ns_server @ 172.23.122.140:8091 is running [2021-02-22 23:11:22,986] - [cluster_helper] [140689549858624] - INFO - waiting for ns_server @ 172.23.122.138:8091 [2021-02-22 23:11:22,987] - [rest_client] [140689549858624] - INFO - -->is_ns_server_running? [2021-02-22 23:11:22,990] - [cluster_helper] [140689549858624] - INFO - ns_server @ 172.23.122.138:8091 is running [2021-02-22 23:11:22,993] - [cluster_helper] [140689549858624] - INFO - waiting for ns_server @ 172.23.122.141:8091 [2021-02-22 23:11:22,993] - [rest_client] [140689549858624] - INFO - -->is_ns_server_running? [2021-02-22 23:11:22,996] - [cluster_helper] [140689549858624] - INFO - ns_server @ 172.23.122.141:8091 is running [2021-02-22 23:11:22,997] - [basetestcase] [140689549858624] - INFO - Building docker image with java sdk client [2021-02-22 23:11:32,701] - [basetestcase] [140689549858624] - INFO - initializing cluster [2021-02-22 23:11:32,708] - [remote_util] [140689549858624] - INFO - SSH Connecting to 172.23.123.65 with username:root, attempt#1 of 5 [2021-02-22 23:11:32,818] - [remote_util] [140689549858624] - INFO - SSH Connected to 172.23.123.65 as root [2021-02-22 23:11:33,110] - [remote_util] [140689549858624] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 23:11:33,459] - [remote_util] [140689549858624] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 23:11:33,473] - [remote_util] [140689549858624] - INFO - SSH Connecting to 172.23.123.65 with username:root, attempt#1 of 5 [2021-02-22 23:11:33,573] - [remote_util] [140689549858624] - INFO - SSH Connected to 172.23.123.65 as root [2021-02-22 23:11:33,854] - [remote_util] [140689549858624] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 23:11:34,164] - [remote_util] [140689549858624] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 23:11:34,227] - [remote_util] [140689549858624] - INFO - 172.23.123.65 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-22 23:11:34,341] - [remote_util] [140689549858624] - INFO - Running systemd command on this server [2021-02-22 23:11:34,342] - [remote_util] [140689549858624] - INFO - running command.raw on 172.23.123.65: systemctl stop couchbase-server.service [2021-02-22 23:11:36,083] - [remote_util] [140689549858624] - INFO - command executed successfully with root [2021-02-22 23:11:36,083] - [basetestcase] [140689549858624] - INFO - Couchbase stopped [2021-02-22 23:11:36,085] - [basetestcase] [140689549858624] - INFO - sleep for 5 secs. ... [2021-02-22 23:11:41,091] - [remote_util] [140689549858624] - INFO - running command.raw on 172.23.123.65: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-22 23:11:41,110] - [remote_util] [140689549858624] - INFO - command executed successfully with root [2021-02-22 23:11:41,110] - [remote_util] [140689549858624] - INFO - running command.raw on 172.23.123.65: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-22 23:11:41,165] - [remote_util] [140689549858624] - INFO - command executed successfully with root [2021-02-22 23:11:41,169] - [remote_util] [140689549858624] - INFO - SSH Connecting to 172.23.123.65 with username:root, attempt#1 of 5 [2021-02-22 23:11:41,273] - [remote_util] [140689549858624] - INFO - SSH Connected to 172.23.123.65 as root [2021-02-22 23:11:41,555] - [remote_util] [140689549858624] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 23:11:41,902] - [remote_util] [140689549858624] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 23:11:41,966] - [remote_util] [140689549858624] - INFO - 172.23.123.65 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-22 23:11:41,968] - [remote_util] [140689549858624] - INFO - Checking for process beam.smp on linux [2021-02-22 23:11:42,048] - [remote_util] [140689549858624] - INFO - Starting couchbase server [2021-02-22 23:11:42,156] - [remote_util] [140689549858624] - INFO - Running systemd command on this server [2021-02-22 23:11:42,156] - [remote_util] [140689549858624] - INFO - running command.raw on 172.23.123.65: systemctl start couchbase-server.service [2021-02-22 23:11:42,180] - [remote_util] [140689549858624] - INFO - command executed successfully with root [2021-02-22 23:11:42,180] - [remote_util] [140689549858624] - INFO - 172.23.123.65:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-22 23:11:47,182] - [remote_util] [140689549858624] - INFO - running command.raw on 172.23.123.65: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-22 23:11:47,211] - [remote_util] [140689549858624] - INFO - command executed successfully with root [2021-02-22 23:11:47,212] - [remote_util] [140689549858624] - INFO - Couchbase server status: [' Process: 28821 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-22 23:11:47,212] - [remote_util] [140689549858624] - INFO - Checking for process beam.smp on linux [2021-02-22 23:11:47,283] - [remote_util] [140689549858624] - INFO - process beam.smp is running on 172.23.123.65: with pid 28963 [2021-02-22 23:11:47,283] - [basetestcase] [140689549858624] - INFO - Couchbase started [2021-02-22 23:11:47,287] - [remote_util] [140689549858624] - INFO - SSH Connecting to 172.23.122.182 with username:root, attempt#1 of 5 [2021-02-22 23:11:47,388] - [remote_util] [140689549858624] - INFO - SSH Connected to 172.23.122.182 as root [2021-02-22 23:11:47,662] - [remote_util] [140689549858624] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 23:11:47,970] - [remote_util] [140689549858624] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 23:11:48,000] - [remote_util] [140689549858624] - INFO - SSH Connecting to 172.23.122.182 with username:root, attempt#1 of 5 [2021-02-22 23:11:48,103] - [remote_util] [140689549858624] - INFO - SSH Connected to 172.23.122.182 as root [2021-02-22 23:11:48,376] - [remote_util] [140689549858624] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 23:11:48,726] - [remote_util] [140689549858624] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 23:11:48,789] - [remote_util] [140689549858624] - INFO - 172.23.122.182 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-22 23:11:48,979] - [remote_util] [140689549858624] - INFO - Running systemd command on this server [2021-02-22 23:11:48,979] - [remote_util] [140689549858624] - INFO - running command.raw on 172.23.122.182: systemctl stop couchbase-server.service [2021-02-22 23:11:50,801] - [remote_util] [140689549858624] - INFO - command executed successfully with root [2021-02-22 23:11:50,802] - [basetestcase] [140689549858624] - INFO - Couchbase stopped [2021-02-22 23:11:50,802] - [basetestcase] [140689549858624] - INFO - sleep for 5 secs. ... [2021-02-22 23:11:55,807] - [remote_util] [140689549858624] - INFO - running command.raw on 172.23.122.182: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-22 23:11:55,825] - [remote_util] [140689549858624] - INFO - command executed successfully with root [2021-02-22 23:11:55,825] - [remote_util] [140689549858624] - INFO - running command.raw on 172.23.122.182: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-22 23:11:55,891] - [remote_util] [140689549858624] - INFO - command executed successfully with root [2021-02-22 23:11:55,895] - [remote_util] [140689549858624] - INFO - SSH Connecting to 172.23.122.182 with username:root, attempt#1 of 5 [2021-02-22 23:11:55,993] - [remote_util] [140689549858624] - INFO - SSH Connected to 172.23.122.182 as root [2021-02-22 23:11:56,255] - [remote_util] [140689549858624] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 23:11:56,558] - [remote_util] [140689549858624] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 23:11:56,621] - [remote_util] [140689549858624] - INFO - 172.23.122.182 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-22 23:11:56,622] - [remote_util] [140689549858624] - INFO - Checking for process beam.smp on linux [2021-02-22 23:11:56,703] - [remote_util] [140689549858624] - INFO - Starting couchbase server [2021-02-22 23:11:56,814] - [remote_util] [140689549858624] - INFO - Running systemd command on this server [2021-02-22 23:11:56,814] - [remote_util] [140689549858624] - INFO - running command.raw on 172.23.122.182: systemctl start couchbase-server.service [2021-02-22 23:11:56,840] - [remote_util] [140689549858624] - INFO - command executed successfully with root [2021-02-22 23:11:56,840] - [remote_util] [140689549858624] - INFO - 172.23.122.182:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-22 23:12:01,846] - [remote_util] [140689549858624] - INFO - running command.raw on 172.23.122.182: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-22 23:12:01,875] - [remote_util] [140689549858624] - INFO - command executed successfully with root [2021-02-22 23:12:01,876] - [remote_util] [140689549858624] - INFO - Couchbase server status: [' Process: 63777 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-22 23:12:01,876] - [remote_util] [140689549858624] - INFO - Checking for process beam.smp on linux [2021-02-22 23:12:01,946] - [remote_util] [140689549858624] - INFO - process beam.smp is running on 172.23.122.182: with pid 63922 [2021-02-22 23:12:01,947] - [basetestcase] [140689549858624] - INFO - Couchbase started [2021-02-22 23:12:01,950] - [remote_util] [140689549858624] - INFO - SSH Connecting to 172.23.122.179 with username:root, attempt#1 of 5 [2021-02-22 23:12:02,057] - [remote_util] [140689549858624] - INFO - SSH Connected to 172.23.122.179 as root [2021-02-22 23:12:02,348] - [remote_util] [140689549858624] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 23:12:02,669] - [remote_util] [140689549858624] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 23:12:02,681] - [remote_util] [140689549858624] - INFO - SSH Connecting to 172.23.122.179 with username:root, attempt#1 of 5 [2021-02-22 23:12:02,784] - [remote_util] [140689549858624] - INFO - SSH Connected to 172.23.122.179 as root [2021-02-22 23:12:03,072] - [remote_util] [140689549858624] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 23:12:03,391] - [remote_util] [140689549858624] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 23:12:03,427] - [remote_util] [140689549858624] - INFO - 172.23.122.179 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-22 23:12:03,567] - [remote_util] [140689549858624] - INFO - Running systemd command on this server [2021-02-22 23:12:03,567] - [remote_util] [140689549858624] - INFO - running command.raw on 172.23.122.179: systemctl stop couchbase-server.service [2021-02-22 23:12:05,506] - [remote_util] [140689549858624] - INFO - command executed successfully with root [2021-02-22 23:12:05,506] - [basetestcase] [140689549858624] - INFO - Couchbase stopped [2021-02-22 23:12:05,506] - [basetestcase] [140689549858624] - INFO - sleep for 5 secs. ... [2021-02-22 23:12:10,508] - [remote_util] [140689549858624] - INFO - running command.raw on 172.23.122.179: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-22 23:12:10,526] - [remote_util] [140689549858624] - INFO - command executed successfully with root [2021-02-22 23:12:10,526] - [remote_util] [140689549858624] - INFO - running command.raw on 172.23.122.179: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-22 23:12:10,582] - [remote_util] [140689549858624] - INFO - command executed successfully with root [2021-02-22 23:12:10,586] - [remote_util] [140689549858624] - INFO - SSH Connecting to 172.23.122.179 with username:root, attempt#1 of 5 [2021-02-22 23:12:10,692] - [remote_util] [140689549858624] - INFO - SSH Connected to 172.23.122.179 as root [2021-02-22 23:12:10,995] - [remote_util] [140689549858624] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 23:12:11,369] - [remote_util] [140689549858624] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 23:12:11,454] - [remote_util] [140689549858624] - INFO - 172.23.122.179 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-22 23:12:11,454] - [remote_util] [140689549858624] - INFO - Checking for process beam.smp on linux [2021-02-22 23:12:11,505] - [remote_util] [140689549858624] - INFO - Starting couchbase server [2021-02-22 23:12:11,684] - [remote_util] [140689549858624] - INFO - Running systemd command on this server [2021-02-22 23:12:11,684] - [remote_util] [140689549858624] - INFO - running command.raw on 172.23.122.179: systemctl start couchbase-server.service [2021-02-22 23:12:11,715] - [remote_util] [140689549858624] - INFO - command executed successfully with root [2021-02-22 23:12:11,716] - [remote_util] [140689549858624] - INFO - 172.23.122.179:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-22 23:12:16,720] - [remote_util] [140689549858624] - INFO - running command.raw on 172.23.122.179: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-22 23:12:16,751] - [remote_util] [140689549858624] - INFO - command executed successfully with root [2021-02-22 23:12:16,751] - [remote_util] [140689549858624] - INFO - Couchbase server status: [' Process: 94117 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-22 23:12:16,752] - [remote_util] [140689549858624] - INFO - Checking for process beam.smp on linux [2021-02-22 23:12:16,820] - [remote_util] [140689549858624] - INFO - process beam.smp is running on 172.23.122.179: with pid 94261 [2021-02-22 23:12:16,820] - [basetestcase] [140689549858624] - INFO - Couchbase started [2021-02-22 23:12:16,823] - [remote_util] [140689549858624] - INFO - SSH Connecting to 172.23.122.153 with username:root, attempt#1 of 5 [2021-02-22 23:12:16,927] - [remote_util] [140689549858624] - INFO - SSH Connected to 172.23.122.153 as root [2021-02-22 23:12:17,199] - [remote_util] [140689549858624] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 23:12:17,499] - [remote_util] [140689549858624] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 23:12:17,516] - [remote_util] [140689549858624] - INFO - SSH Connecting to 172.23.122.153 with username:root, attempt#1 of 5 [2021-02-22 23:12:17,615] - [remote_util] [140689549858624] - INFO - SSH Connected to 172.23.122.153 as root [2021-02-22 23:12:17,891] - [remote_util] [140689549858624] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 23:12:18,195] - [remote_util] [140689549858624] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 23:12:18,256] - [remote_util] [140689549858624] - INFO - 172.23.122.153 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-22 23:12:18,396] - [remote_util] [140689549858624] - INFO - Running systemd command on this server [2021-02-22 23:12:18,397] - [remote_util] [140689549858624] - INFO - running command.raw on 172.23.122.153: systemctl stop couchbase-server.service [2021-02-22 23:12:20,031] - [remote_util] [140689549858624] - INFO - command executed successfully with root [2021-02-22 23:12:20,031] - [basetestcase] [140689549858624] - INFO - Couchbase stopped [2021-02-22 23:12:20,032] - [basetestcase] [140689549858624] - INFO - sleep for 5 secs. ... [2021-02-22 23:12:25,037] - [remote_util] [140689549858624] - INFO - running command.raw on 172.23.122.153: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-22 23:12:25,053] - [remote_util] [140689549858624] - INFO - command executed successfully with root [2021-02-22 23:12:25,054] - [remote_util] [140689549858624] - INFO - running command.raw on 172.23.122.153: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-22 23:12:25,108] - [remote_util] [140689549858624] - INFO - command executed successfully with root [2021-02-22 23:12:25,111] - [remote_util] [140689549858624] - INFO - SSH Connecting to 172.23.122.153 with username:root, attempt#1 of 5 [2021-02-22 23:12:25,213] - [remote_util] [140689549858624] - INFO - SSH Connected to 172.23.122.153 as root [2021-02-22 23:12:25,486] - [remote_util] [140689549858624] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 23:12:25,785] - [remote_util] [140689549858624] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 23:12:25,855] - [remote_util] [140689549858624] - INFO - 172.23.122.153 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-22 23:12:25,856] - [remote_util] [140689549858624] - INFO - Checking for process beam.smp on linux [2021-02-22 23:12:25,931] - [remote_util] [140689549858624] - INFO - Starting couchbase server [2021-02-22 23:12:26,072] - [remote_util] [140689549858624] - INFO - Running systemd command on this server [2021-02-22 23:12:26,072] - [remote_util] [140689549858624] - INFO - running command.raw on 172.23.122.153: systemctl start couchbase-server.service [2021-02-22 23:12:26,095] - [remote_util] [140689549858624] - INFO - command executed successfully with root [2021-02-22 23:12:26,095] - [remote_util] [140689549858624] - INFO - 172.23.122.153:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-22 23:12:31,101] - [remote_util] [140689549858624] - INFO - running command.raw on 172.23.122.153: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-22 23:12:31,125] - [remote_util] [140689549858624] - INFO - command executed successfully with root [2021-02-22 23:12:31,126] - [remote_util] [140689549858624] - INFO - Couchbase server status: [' Process: 123727 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-22 23:12:31,127] - [remote_util] [140689549858624] - INFO - Checking for process beam.smp on linux [2021-02-22 23:12:31,196] - [remote_util] [140689549858624] - INFO - process beam.smp is running on 172.23.122.153: with pid 123872 [2021-02-22 23:12:31,197] - [basetestcase] [140689549858624] - INFO - Couchbase started [2021-02-22 23:12:31,201] - [remote_util] [140689549858624] - INFO - SSH Connecting to 172.23.122.139 with username:root, attempt#1 of 5 [2021-02-22 23:12:31,311] - [remote_util] [140689549858624] - INFO - SSH Connected to 172.23.122.139 as root [2021-02-22 23:12:31,596] - [remote_util] [140689549858624] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 23:12:31,902] - [remote_util] [140689549858624] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 23:12:31,913] - [remote_util] [140689549858624] - INFO - SSH Connecting to 172.23.122.139 with username:root, attempt#1 of 5 [2021-02-22 23:12:32,011] - [remote_util] [140689549858624] - INFO - SSH Connected to 172.23.122.139 as root [2021-02-22 23:12:32,281] - [remote_util] [140689549858624] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 23:12:32,592] - [remote_util] [140689549858624] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 23:12:32,656] - [remote_util] [140689549858624] - INFO - 172.23.122.139 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-22 23:12:32,759] - [remote_util] [140689549858624] - INFO - Running systemd command on this server [2021-02-22 23:12:32,759] - [remote_util] [140689549858624] - INFO - running command.raw on 172.23.122.139: systemctl stop couchbase-server.service [2021-02-22 23:12:34,453] - [remote_util] [140689549858624] - INFO - command executed successfully with root [2021-02-22 23:12:34,454] - [basetestcase] [140689549858624] - INFO - Couchbase stopped [2021-02-22 23:12:34,455] - [basetestcase] [140689549858624] - INFO - sleep for 5 secs. ... [2021-02-22 23:12:39,460] - [remote_util] [140689549858624] - INFO - running command.raw on 172.23.122.139: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-22 23:12:39,477] - [remote_util] [140689549858624] - INFO - command executed successfully with root [2021-02-22 23:12:39,478] - [remote_util] [140689549858624] - INFO - running command.raw on 172.23.122.139: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-22 23:12:39,533] - [remote_util] [140689549858624] - INFO - command executed successfully with root [2021-02-22 23:12:39,536] - [remote_util] [140689549858624] - INFO - SSH Connecting to 172.23.122.139 with username:root, attempt#1 of 5 [2021-02-22 23:12:39,639] - [remote_util] [140689549858624] - INFO - SSH Connected to 172.23.122.139 as root [2021-02-22 23:12:39,930] - [remote_util] [140689549858624] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 23:12:40,238] - [remote_util] [140689549858624] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 23:12:40,300] - [remote_util] [140689549858624] - INFO - 172.23.122.139 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-22 23:12:40,300] - [remote_util] [140689549858624] - INFO - Checking for process beam.smp on linux [2021-02-22 23:12:40,382] - [remote_util] [140689549858624] - INFO - Starting couchbase server [2021-02-22 23:12:40,489] - [remote_util] [140689549858624] - INFO - Running systemd command on this server [2021-02-22 23:12:40,489] - [remote_util] [140689549858624] - INFO - running command.raw on 172.23.122.139: systemctl start couchbase-server.service [2021-02-22 23:12:40,516] - [remote_util] [140689549858624] - INFO - command executed successfully with root [2021-02-22 23:12:40,516] - [remote_util] [140689549858624] - INFO - 172.23.122.139:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-22 23:12:45,518] - [remote_util] [140689549858624] - INFO - running command.raw on 172.23.122.139: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-22 23:12:45,543] - [remote_util] [140689549858624] - INFO - command executed successfully with root [2021-02-22 23:12:45,543] - [remote_util] [140689549858624] - INFO - Couchbase server status: [' Process: 77990 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-22 23:12:45,544] - [remote_util] [140689549858624] - INFO - Checking for process beam.smp on linux [2021-02-22 23:12:45,614] - [remote_util] [140689549858624] - INFO - process beam.smp is running on 172.23.122.139: with pid 78167 [2021-02-22 23:12:45,615] - [basetestcase] [140689549858624] - INFO - Couchbase started [2021-02-22 23:12:45,618] - [remote_util] [140689549858624] - INFO - SSH Connecting to 172.23.122.140 with username:root, attempt#1 of 5 [2021-02-22 23:12:45,725] - [remote_util] [140689549858624] - INFO - SSH Connected to 172.23.122.140 as root [2021-02-22 23:12:46,025] - [remote_util] [140689549858624] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 23:12:46,335] - [remote_util] [140689549858624] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 23:12:46,347] - [remote_util] [140689549858624] - INFO - SSH Connecting to 172.23.122.140 with username:root, attempt#1 of 5 [2021-02-22 23:12:46,451] - [remote_util] [140689549858624] - INFO - SSH Connected to 172.23.122.140 as root [2021-02-22 23:12:46,740] - [remote_util] [140689549858624] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 23:12:47,045] - [remote_util] [140689549858624] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 23:12:47,109] - [remote_util] [140689549858624] - INFO - 172.23.122.140 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-22 23:12:47,251] - [remote_util] [140689549858624] - INFO - Running systemd command on this server [2021-02-22 23:12:47,251] - [remote_util] [140689549858624] - INFO - running command.raw on 172.23.122.140: systemctl stop couchbase-server.service [2021-02-22 23:12:48,977] - [remote_util] [140689549858624] - INFO - command executed successfully with root [2021-02-22 23:12:48,977] - [basetestcase] [140689549858624] - INFO - Couchbase stopped [2021-02-22 23:12:48,978] - [basetestcase] [140689549858624] - INFO - sleep for 5 secs. ... [2021-02-22 23:12:53,985] - [remote_util] [140689549858624] - INFO - running command.raw on 172.23.122.140: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-22 23:12:54,002] - [remote_util] [140689549858624] - INFO - command executed successfully with root [2021-02-22 23:12:54,002] - [remote_util] [140689549858624] - INFO - running command.raw on 172.23.122.140: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-22 23:12:54,057] - [remote_util] [140689549858624] - INFO - command executed successfully with root [2021-02-22 23:12:54,064] - [remote_util] [140689549858624] - INFO - SSH Connecting to 172.23.122.140 with username:root, attempt#1 of 5 [2021-02-22 23:12:54,164] - [remote_util] [140689549858624] - INFO - SSH Connected to 172.23.122.140 as root [2021-02-22 23:12:54,448] - [remote_util] [140689549858624] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 23:12:54,757] - [remote_util] [140689549858624] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 23:12:54,818] - [remote_util] [140689549858624] - INFO - 172.23.122.140 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-22 23:12:54,818] - [remote_util] [140689549858624] - INFO - Checking for process beam.smp on linux [2021-02-22 23:12:54,896] - [remote_util] [140689549858624] - INFO - Starting couchbase server [2021-02-22 23:12:54,999] - [remote_util] [140689549858624] - INFO - Running systemd command on this server [2021-02-22 23:12:54,999] - [remote_util] [140689549858624] - INFO - running command.raw on 172.23.122.140: systemctl start couchbase-server.service [2021-02-22 23:12:55,024] - [remote_util] [140689549858624] - INFO - command executed successfully with root [2021-02-22 23:12:55,025] - [remote_util] [140689549858624] - INFO - 172.23.122.140:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-22 23:13:00,031] - [remote_util] [140689549858624] - INFO - running command.raw on 172.23.122.140: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-22 23:13:00,059] - [remote_util] [140689549858624] - INFO - command executed successfully with root [2021-02-22 23:13:00,059] - [remote_util] [140689549858624] - INFO - Couchbase server status: [' Process: 11492 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-22 23:13:00,059] - [remote_util] [140689549858624] - INFO - Checking for process beam.smp on linux [2021-02-22 23:13:00,130] - [remote_util] [140689549858624] - INFO - process beam.smp is running on 172.23.122.140: with pid 11638 [2021-02-22 23:13:00,131] - [basetestcase] [140689549858624] - INFO - Couchbase started [2021-02-22 23:13:00,136] - [remote_util] [140689549858624] - INFO - SSH Connecting to 172.23.122.138 with username:root, attempt#1 of 5 [2021-02-22 23:13:00,241] - [remote_util] [140689549858624] - INFO - SSH Connected to 172.23.122.138 as root [2021-02-22 23:13:00,530] - [remote_util] [140689549858624] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 23:13:00,847] - [remote_util] [140689549858624] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 23:13:00,860] - [remote_util] [140689549858624] - INFO - SSH Connecting to 172.23.122.138 with username:root, attempt#1 of 5 [2021-02-22 23:13:00,962] - [remote_util] [140689549858624] - INFO - SSH Connected to 172.23.122.138 as root [2021-02-22 23:13:01,257] - [remote_util] [140689549858624] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 23:13:01,589] - [remote_util] [140689549858624] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 23:13:01,655] - [remote_util] [140689549858624] - INFO - 172.23.122.138 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-22 23:13:01,759] - [remote_util] [140689549858624] - INFO - Running systemd command on this server [2021-02-22 23:13:01,759] - [remote_util] [140689549858624] - INFO - running command.raw on 172.23.122.138: systemctl stop couchbase-server.service [2021-02-22 23:13:03,382] - [remote_util] [140689549858624] - INFO - command executed successfully with root [2021-02-22 23:13:03,382] - [basetestcase] [140689549858624] - INFO - Couchbase stopped [2021-02-22 23:13:03,383] - [basetestcase] [140689549858624] - INFO - sleep for 5 secs. ... [2021-02-22 23:13:08,388] - [remote_util] [140689549858624] - INFO - running command.raw on 172.23.122.138: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-22 23:13:08,406] - [remote_util] [140689549858624] - INFO - command executed successfully with root [2021-02-22 23:13:08,407] - [remote_util] [140689549858624] - INFO - running command.raw on 172.23.122.138: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-22 23:13:08,463] - [remote_util] [140689549858624] - INFO - command executed successfully with root [2021-02-22 23:13:08,466] - [remote_util] [140689549858624] - INFO - SSH Connecting to 172.23.122.138 with username:root, attempt#1 of 5 [2021-02-22 23:13:08,571] - [remote_util] [140689549858624] - INFO - SSH Connected to 172.23.122.138 as root [2021-02-22 23:13:08,874] - [remote_util] [140689549858624] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 23:13:09,197] - [remote_util] [140689549858624] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 23:13:09,261] - [remote_util] [140689549858624] - INFO - 172.23.122.138 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-22 23:13:09,261] - [remote_util] [140689549858624] - INFO - Checking for process beam.smp on linux [2021-02-22 23:13:09,347] - [remote_util] [140689549858624] - INFO - Starting couchbase server [2021-02-22 23:13:09,496] - [remote_util] [140689549858624] - INFO - Running systemd command on this server [2021-02-22 23:13:09,496] - [remote_util] [140689549858624] - INFO - running command.raw on 172.23.122.138: systemctl start couchbase-server.service [2021-02-22 23:13:09,522] - [remote_util] [140689549858624] - INFO - command executed successfully with root [2021-02-22 23:13:09,523] - [remote_util] [140689549858624] - INFO - 172.23.122.138:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-22 23:13:14,527] - [remote_util] [140689549858624] - INFO - running command.raw on 172.23.122.138: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-22 23:13:14,553] - [remote_util] [140689549858624] - INFO - command executed successfully with root [2021-02-22 23:13:14,554] - [remote_util] [140689549858624] - INFO - Couchbase server status: [' Process: 97337 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-22 23:13:14,554] - [remote_util] [140689549858624] - INFO - Checking for process beam.smp on linux [2021-02-22 23:13:14,626] - [remote_util] [140689549858624] - INFO - process beam.smp is running on 172.23.122.138: with pid 97526 [2021-02-22 23:13:14,627] - [basetestcase] [140689549858624] - INFO - Couchbase started [2021-02-22 23:13:14,630] - [remote_util] [140689549858624] - INFO - SSH Connecting to 172.23.122.141 with username:root, attempt#1 of 5 [2021-02-22 23:13:14,742] - [remote_util] [140689549858624] - INFO - SSH Connected to 172.23.122.141 as root [2021-02-22 23:13:15,020] - [remote_util] [140689549858624] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 23:13:15,329] - [remote_util] [140689549858624] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 23:13:15,342] - [remote_util] [140689549858624] - INFO - SSH Connecting to 172.23.122.141 with username:root, attempt#1 of 5 [2021-02-22 23:13:15,444] - [remote_util] [140689549858624] - INFO - SSH Connected to 172.23.122.141 as root [2021-02-22 23:13:15,716] - [remote_util] [140689549858624] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 23:13:16,024] - [remote_util] [140689549858624] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 23:13:16,088] - [remote_util] [140689549858624] - INFO - 172.23.122.141 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-22 23:13:16,193] - [remote_util] [140689549858624] - INFO - Running systemd command on this server [2021-02-22 23:13:16,194] - [remote_util] [140689549858624] - INFO - running command.raw on 172.23.122.141: systemctl stop couchbase-server.service [2021-02-22 23:13:17,862] - [remote_util] [140689549858624] - INFO - command executed successfully with root [2021-02-22 23:13:17,862] - [basetestcase] [140689549858624] - INFO - Couchbase stopped [2021-02-22 23:13:17,863] - [basetestcase] [140689549858624] - INFO - sleep for 5 secs. ... [2021-02-22 23:13:22,868] - [remote_util] [140689549858624] - INFO - running command.raw on 172.23.122.141: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-22 23:13:22,890] - [remote_util] [140689549858624] - INFO - command executed successfully with root [2021-02-22 23:13:22,891] - [remote_util] [140689549858624] - INFO - running command.raw on 172.23.122.141: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-22 23:13:22,951] - [remote_util] [140689549858624] - INFO - command executed successfully with root [2021-02-22 23:13:22,954] - [remote_util] [140689549858624] - INFO - SSH Connecting to 172.23.122.141 with username:root, attempt#1 of 5 [2021-02-22 23:13:23,050] - [remote_util] [140689549858624] - INFO - SSH Connected to 172.23.122.141 as root [2021-02-22 23:13:23,317] - [remote_util] [140689549858624] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 23:13:23,624] - [remote_util] [140689549858624] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 23:13:23,686] - [remote_util] [140689549858624] - INFO - 172.23.122.141 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-22 23:13:23,686] - [remote_util] [140689549858624] - INFO - Checking for process beam.smp on linux [2021-02-22 23:13:23,764] - [remote_util] [140689549858624] - INFO - Starting couchbase server [2021-02-22 23:13:23,864] - [remote_util] [140689549858624] - INFO - Running systemd command on this server [2021-02-22 23:13:23,865] - [remote_util] [140689549858624] - INFO - running command.raw on 172.23.122.141: systemctl start couchbase-server.service [2021-02-22 23:13:23,887] - [remote_util] [140689549858624] - INFO - command executed successfully with root [2021-02-22 23:13:23,887] - [remote_util] [140689549858624] - INFO - 172.23.122.141:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-22 23:13:28,892] - [remote_util] [140689549858624] - INFO - running command.raw on 172.23.122.141: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-22 23:13:28,925] - [remote_util] [140689549858624] - INFO - command executed successfully with root [2021-02-22 23:13:28,925] - [remote_util] [140689549858624] - INFO - Couchbase server status: [' Process: 64481 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-22 23:13:28,926] - [remote_util] [140689549858624] - INFO - Checking for process beam.smp on linux [2021-02-22 23:13:29,002] - [remote_util] [140689549858624] - INFO - process beam.smp is running on 172.23.122.141: with pid 64634 [2021-02-22 23:13:29,003] - [basetestcase] [140689549858624] - INFO - Couchbase started [2021-02-22 23:13:29,003] - [basetestcase] [140689549858624] - INFO - sleep for 10 secs. ... [2021-02-22 23:13:39,230] - [task] [140686669633280] - INFO - server: ip:172.23.123.65 port:8091 ssh_username:root, nodes/self [2021-02-22 23:13:39,234] - [task] [140686669633280] - INFO - {'uptime': '115', 'memoryTotal': 4201672704, 'memoryFree': 3535433728, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.123.65:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4514-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.123.65', 'ip': '172.23.123.65', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} [2021-02-22 23:13:39,237] - [rest_client] [140686669633280] - ERROR - GET http://172.23.123.65:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-02-22 23:13:39,238] - [rest_client] [140686669633280] - INFO - pools/default params : memoryQuota=2147 [2021-02-22 23:13:39,245] - [rest_client] [140686669633280] - INFO - --> init_node_services(Administrator,password,172.23.123.65,8091,['kv']) [2021-02-22 23:13:39,246] - [rest_client] [140686669633280] - INFO - /node/controller/setupServices params on 172.23.123.65: 8091:hostname=172.23.123.65&user=Administrator&password=password&services=kv [2021-02-22 23:13:39,275] - [rest_client] [140686669633280] - INFO - --> in init_cluster...Administrator,password,8091 [2021-02-22 23:13:39,276] - [rest_client] [140686669633280] - INFO - settings/web params on 172.23.123.65:8091:port=8091&username=Administrator&password=password [2021-02-22 23:13:39,331] - [rest_client] [140686669633280] - INFO - --> status:True [2021-02-22 23:13:39,333] - [remote_util] [140686669633280] - INFO - SSH Connecting to 172.23.123.65 with username:root, attempt#1 of 5 [2021-02-22 23:13:39,440] - [remote_util] [140686669633280] - INFO - SSH Connected to 172.23.123.65 as root [2021-02-22 23:13:39,725] - [remote_util] [140686669633280] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 23:13:40,036] - [remote_util] [140686669633280] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 23:13:40,046] - [remote_util] [140686669633280] - INFO - running command.raw on 172.23.123.65: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-02-22 23:13:40,103] - [remote_util] [140686669633280] - INFO - command executed successfully with root [2021-02-22 23:13:40,103] - [remote_util] [140686669633280] - INFO - ['ok'] [2021-02-22 23:13:40,107] - [rest_client] [140686669633280] - INFO - /diag/eval status on 172.23.123.65:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-22 23:13:40,109] - [rest_client] [140686669633280] - INFO - /diag/eval status on 172.23.123.65:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-22 23:13:40,117] - [rest_client] [140686669633280] - INFO - settings/indexes params : storageMode=plasma [2021-02-22 23:13:40,137] - [task] [140686669633280] - INFO - server: ip:172.23.122.182 port:8091 ssh_username:root, nodes/self [2021-02-22 23:13:40,141] - [task] [140686669633280] - INFO - {'uptime': '100', 'memoryTotal': 4201684992, 'memoryFree': 3569602560, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.122.182:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4514-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.122.182', 'ip': '172.23.122.182', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} [2021-02-22 23:13:40,143] - [rest_client] [140686669633280] - ERROR - GET http://172.23.122.182:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-02-22 23:13:40,144] - [rest_client] [140686669633280] - INFO - pools/default params : memoryQuota=2147 [2021-02-22 23:13:40,150] - [rest_client] [140686669633280] - INFO - --> in init_cluster...Administrator,password,8091 [2021-02-22 23:13:40,150] - [rest_client] [140686669633280] - INFO - settings/web params on 172.23.122.182:8091:port=8091&username=Administrator&password=password [2021-02-22 23:13:40,206] - [rest_client] [140686669633280] - INFO - --> status:True [2021-02-22 23:13:40,209] - [remote_util] [140686669633280] - INFO - SSH Connecting to 172.23.122.182 with username:root, attempt#1 of 5 [2021-02-22 23:13:40,313] - [remote_util] [140686669633280] - INFO - SSH Connected to 172.23.122.182 as root [2021-02-22 23:13:40,570] - [remote_util] [140686669633280] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 23:13:40,877] - [remote_util] [140686669633280] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 23:13:40,888] - [remote_util] [140686669633280] - INFO - running command.raw on 172.23.122.182: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-02-22 23:13:40,943] - [remote_util] [140686669633280] - INFO - command executed successfully with root [2021-02-22 23:13:40,944] - [remote_util] [140686669633280] - INFO - ['ok'] [2021-02-22 23:13:40,948] - [rest_client] [140686669633280] - INFO - /diag/eval status on 172.23.122.182:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-22 23:13:40,951] - [rest_client] [140686669633280] - INFO - /diag/eval status on 172.23.122.182:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-22 23:13:40,957] - [rest_client] [140686669633280] - INFO - settings/indexes params : storageMode=plasma [2021-02-22 23:13:40,977] - [task] [140686669633280] - INFO - server: ip:172.23.122.179 port:8091 ssh_username:root, nodes/self [2021-02-22 23:13:40,981] - [task] [140686669633280] - INFO - {'uptime': '85', 'memoryTotal': 4201684992, 'memoryFree': 3610542080, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.122.179:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4514-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.122.179', 'ip': '172.23.122.179', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} [2021-02-22 23:13:40,984] - [rest_client] [140686669633280] - ERROR - GET http://172.23.122.179:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-02-22 23:13:40,984] - [rest_client] [140686669633280] - INFO - pools/default params : memoryQuota=2147 [2021-02-22 23:13:40,991] - [rest_client] [140686669633280] - INFO - --> in init_cluster...Administrator,password,8091 [2021-02-22 23:13:40,991] - [rest_client] [140686669633280] - INFO - settings/web params on 172.23.122.179:8091:port=8091&username=Administrator&password=password [2021-02-22 23:13:41,047] - [rest_client] [140686669633280] - INFO - --> status:True [2021-02-22 23:13:41,050] - [remote_util] [140686669633280] - INFO - SSH Connecting to 172.23.122.179 with username:root, attempt#1 of 5 [2021-02-22 23:13:41,158] - [remote_util] [140686669633280] - INFO - SSH Connected to 172.23.122.179 as root [2021-02-22 23:13:41,439] - [remote_util] [140686669633280] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 23:13:41,753] - [remote_util] [140686669633280] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 23:13:41,767] - [remote_util] [140686669633280] - INFO - running command.raw on 172.23.122.179: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-02-22 23:13:41,821] - [remote_util] [140686669633280] - INFO - command executed successfully with root [2021-02-22 23:13:41,821] - [remote_util] [140686669633280] - INFO - ['ok'] [2021-02-22 23:13:41,825] - [rest_client] [140686669633280] - INFO - /diag/eval status on 172.23.122.179:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-22 23:13:41,829] - [rest_client] [140686669633280] - INFO - /diag/eval status on 172.23.122.179:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-22 23:13:41,835] - [rest_client] [140686669633280] - INFO - settings/indexes params : storageMode=plasma [2021-02-22 23:13:41,858] - [task] [140686669633280] - INFO - server: ip:172.23.122.153 port:8091 ssh_username:root, nodes/self [2021-02-22 23:13:41,862] - [task] [140686669633280] - INFO - {'uptime': '69', 'memoryTotal': 4201684992, 'memoryFree': 3627978752, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.122.153:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4514-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.122.153', 'ip': '172.23.122.153', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} [2021-02-22 23:13:41,866] - [rest_client] [140686669633280] - ERROR - GET http://172.23.122.153:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-02-22 23:13:41,867] - [rest_client] [140686669633280] - INFO - pools/default params : memoryQuota=2147 [2021-02-22 23:13:41,874] - [rest_client] [140686669633280] - INFO - --> in init_cluster...Administrator,password,8091 [2021-02-22 23:13:41,875] - [rest_client] [140686669633280] - INFO - settings/web params on 172.23.122.153:8091:port=8091&username=Administrator&password=password [2021-02-22 23:13:41,931] - [rest_client] [140686669633280] - INFO - --> status:True [2021-02-22 23:13:41,935] - [remote_util] [140686669633280] - INFO - SSH Connecting to 172.23.122.153 with username:root, attempt#1 of 5 [2021-02-22 23:13:42,034] - [remote_util] [140686669633280] - INFO - SSH Connected to 172.23.122.153 as root [2021-02-22 23:13:42,309] - [remote_util] [140686669633280] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 23:13:42,612] - [remote_util] [140686669633280] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 23:13:42,624] - [remote_util] [140686669633280] - INFO - running command.raw on 172.23.122.153: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-02-22 23:13:42,677] - [remote_util] [140686669633280] - INFO - command executed successfully with root [2021-02-22 23:13:42,678] - [remote_util] [140686669633280] - INFO - ['ok'] [2021-02-22 23:13:42,682] - [rest_client] [140686669633280] - INFO - /diag/eval status on 172.23.122.153:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-22 23:13:42,685] - [rest_client] [140686669633280] - INFO - /diag/eval status on 172.23.122.153:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-22 23:13:42,689] - [rest_client] [140686669633280] - INFO - settings/indexes params : storageMode=plasma [2021-02-22 23:13:42,713] - [task] [140686669633280] - INFO - server: ip:172.23.122.139 port:8091 ssh_username:root, nodes/self [2021-02-22 23:13:42,717] - [task] [140686669633280] - INFO - {'uptime': '60', 'memoryTotal': 4201684992, 'memoryFree': 3589521408, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.122.139:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4514-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.122.139', 'ip': '172.23.122.139', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} [2021-02-22 23:13:42,719] - [rest_client] [140686669633280] - ERROR - GET http://172.23.122.139:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-02-22 23:13:42,720] - [rest_client] [140686669633280] - INFO - pools/default params : memoryQuota=2147 [2021-02-22 23:13:42,727] - [rest_client] [140686669633280] - INFO - --> in init_cluster...Administrator,password,8091 [2021-02-22 23:13:42,727] - [rest_client] [140686669633280] - INFO - settings/web params on 172.23.122.139:8091:port=8091&username=Administrator&password=password [2021-02-22 23:13:42,783] - [rest_client] [140686669633280] - INFO - --> status:True [2021-02-22 23:13:42,786] - [remote_util] [140686669633280] - INFO - SSH Connecting to 172.23.122.139 with username:root, attempt#1 of 5 [2021-02-22 23:13:42,896] - [remote_util] [140686669633280] - INFO - SSH Connected to 172.23.122.139 as root [2021-02-22 23:13:43,180] - [remote_util] [140686669633280] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 23:13:43,496] - [remote_util] [140686669633280] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 23:13:43,507] - [remote_util] [140686669633280] - INFO - running command.raw on 172.23.122.139: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-02-22 23:13:43,563] - [remote_util] [140686669633280] - INFO - command executed successfully with root [2021-02-22 23:13:43,563] - [remote_util] [140686669633280] - INFO - ['ok'] [2021-02-22 23:13:43,568] - [rest_client] [140686669633280] - INFO - /diag/eval status on 172.23.122.139:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-22 23:13:43,572] - [rest_client] [140686669633280] - INFO - /diag/eval status on 172.23.122.139:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-22 23:13:43,577] - [rest_client] [140686669633280] - INFO - settings/indexes params : storageMode=plasma [2021-02-22 23:13:43,600] - [task] [140686669633280] - INFO - server: ip:172.23.122.140 port:8091 ssh_username:root, nodes/self [2021-02-22 23:13:43,605] - [task] [140686669633280] - INFO - {'uptime': '44', 'memoryTotal': 4201684992, 'memoryFree': 3587076096, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.122.140:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4514-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.122.140', 'ip': '172.23.122.140', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} [2021-02-22 23:13:43,608] - [rest_client] [140686669633280] - ERROR - GET http://172.23.122.140:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-02-22 23:13:43,608] - [rest_client] [140686669633280] - INFO - pools/default params : memoryQuota=2147 [2021-02-22 23:13:43,615] - [rest_client] [140686669633280] - INFO - --> in init_cluster...Administrator,password,8091 [2021-02-22 23:13:43,615] - [rest_client] [140686669633280] - INFO - settings/web params on 172.23.122.140:8091:port=8091&username=Administrator&password=password [2021-02-22 23:13:43,675] - [rest_client] [140686669633280] - INFO - --> status:True [2021-02-22 23:13:43,679] - [remote_util] [140686669633280] - INFO - SSH Connecting to 172.23.122.140 with username:root, attempt#1 of 5 [2021-02-22 23:13:43,781] - [remote_util] [140686669633280] - INFO - SSH Connected to 172.23.122.140 as root [2021-02-22 23:13:44,070] - [remote_util] [140686669633280] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 23:13:44,417] - [remote_util] [140686669633280] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 23:13:44,432] - [remote_util] [140686669633280] - INFO - running command.raw on 172.23.122.140: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-02-22 23:13:44,487] - [remote_util] [140686669633280] - INFO - command executed successfully with root [2021-02-22 23:13:44,487] - [remote_util] [140686669633280] - INFO - ['ok'] [2021-02-22 23:13:44,494] - [rest_client] [140686669633280] - INFO - /diag/eval status on 172.23.122.140:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-22 23:13:44,501] - [rest_client] [140686669633280] - INFO - /diag/eval status on 172.23.122.140:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-22 23:13:44,510] - [rest_client] [140686669633280] - INFO - settings/indexes params : storageMode=plasma [2021-02-22 23:13:44,543] - [task] [140686669633280] - INFO - server: ip:172.23.122.138 port:8091 ssh_username:root, nodes/self [2021-02-22 23:13:44,547] - [task] [140686669633280] - INFO - {'uptime': '29', 'memoryTotal': 4201684992, 'memoryFree': 3585945600, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.122.138:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4514-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.122.138', 'ip': '172.23.122.138', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} [2021-02-22 23:13:44,550] - [rest_client] [140686669633280] - ERROR - GET http://172.23.122.138:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-02-22 23:13:44,550] - [rest_client] [140686669633280] - INFO - pools/default params : memoryQuota=2147 [2021-02-22 23:13:44,558] - [rest_client] [140686669633280] - INFO - --> in init_cluster...Administrator,password,8091 [2021-02-22 23:13:44,558] - [rest_client] [140686669633280] - INFO - settings/web params on 172.23.122.138:8091:port=8091&username=Administrator&password=password [2021-02-22 23:13:44,616] - [rest_client] [140686669633280] - INFO - --> status:True [2021-02-22 23:13:44,619] - [remote_util] [140686669633280] - INFO - SSH Connecting to 172.23.122.138 with username:root, attempt#1 of 5 [2021-02-22 23:13:44,776] - [remote_util] [140686669633280] - INFO - SSH Connected to 172.23.122.138 as root [2021-02-22 23:13:45,133] - [remote_util] [140686669633280] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 23:13:45,495] - [remote_util] [140686669633280] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 23:13:45,507] - [remote_util] [140686669633280] - INFO - running command.raw on 172.23.122.138: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-02-22 23:13:45,561] - [remote_util] [140686669633280] - INFO - command executed successfully with root [2021-02-22 23:13:45,562] - [remote_util] [140686669633280] - INFO - ['ok'] [2021-02-22 23:13:45,567] - [rest_client] [140686669633280] - INFO - /diag/eval status on 172.23.122.138:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-22 23:13:45,570] - [rest_client] [140686669633280] - INFO - /diag/eval status on 172.23.122.138:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-22 23:13:45,577] - [rest_client] [140686669633280] - INFO - settings/indexes params : storageMode=plasma [2021-02-22 23:13:45,602] - [task] [140686669633280] - INFO - server: ip:172.23.122.141 port:8091 ssh_username:root, nodes/self [2021-02-22 23:13:45,608] - [task] [140686669633280] - INFO - {'uptime': '14', 'memoryTotal': 4201689088, 'memoryFree': 3609870336, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.122.141:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4514-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@cb.local', 'ip': '172.23.122.141', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} [2021-02-22 23:13:45,611] - [rest_client] [140686669633280] - ERROR - GET http://172.23.122.141:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-02-22 23:13:45,612] - [rest_client] [140686669633280] - INFO - pools/default params : memoryQuota=2147 [2021-02-22 23:13:45,620] - [rest_client] [140686669633280] - INFO - --> in init_cluster...Administrator,password,8091 [2021-02-22 23:13:45,621] - [rest_client] [140686669633280] - INFO - settings/web params on 172.23.122.141:8091:port=8091&username=Administrator&password=password [2021-02-22 23:13:45,678] - [rest_client] [140686669633280] - INFO - --> status:True [2021-02-22 23:13:45,683] - [remote_util] [140686669633280] - INFO - SSH Connecting to 172.23.122.141 with username:root, attempt#1 of 5 [2021-02-22 23:13:45,791] - [remote_util] [140686669633280] - INFO - SSH Connected to 172.23.122.141 as root [2021-02-22 23:13:46,065] - [remote_util] [140686669633280] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 23:13:46,369] - [remote_util] [140686669633280] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 23:13:46,383] - [remote_util] [140686669633280] - INFO - running command.raw on 172.23.122.141: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-02-22 23:13:46,436] - [remote_util] [140686669633280] - INFO - command executed successfully with root [2021-02-22 23:13:46,436] - [remote_util] [140686669633280] - INFO - ['ok'] [2021-02-22 23:13:46,440] - [rest_client] [140686669633280] - INFO - /diag/eval status on 172.23.122.141:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-22 23:13:46,444] - [rest_client] [140686669633280] - INFO - /diag/eval status on 172.23.122.141:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-22 23:13:46,452] - [rest_client] [140686669633280] - INFO - settings/indexes params : storageMode=plasma [2021-02-22 23:13:46,482] - [basetestcase] [140689549858624] - INFO - **** add built-in 'cbadminbucket' user to node 172.23.123.65 **** [2021-02-22 23:13:46,499] - [rest_client] [140689549858624] - ERROR - DELETE http://172.23.123.65:8091/settings/rbac/users/local/cbadminbucket body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:password [2021-02-22 23:13:46,501] - [internal_user] [140689549858624] - INFO - Exception while deleting user. Exception is -b'"User was not found."' [2021-02-22 23:13:46,568] - [basetestcase] [140689549858624] - INFO - sleep for 5 secs. ... [2021-02-22 23:13:51,573] - [basetestcase] [140689549858624] - INFO - **** add 'admin' role to 'cbadminbucket' user **** [2021-02-22 23:13:51,587] - [basetestcase] [140689549858624] - INFO - done initializing cluster [2021-02-22 23:13:52,482] - [task] [140686669633280] - INFO - adding node 172.23.122.182:8091 to cluster [2021-02-22 23:13:52,482] - [rest_client] [140686669633280] - INFO - adding remote node @172.23.122.182:8091 to this cluster @172.23.123.65:8091 [2021-02-22 23:14:02,498] - [rest_client] [140686669633280] - INFO - rebalance progress took 10.02 seconds [2021-02-22 23:14:02,499] - [rest_client] [140686669633280] - INFO - sleep for 10 seconds after rebalance... [2021-02-22 23:14:14,658] - [task] [140686669633280] - INFO - adding node 172.23.122.179:8091 to cluster [2021-02-22 23:14:14,659] - [rest_client] [140686669633280] - INFO - adding remote node @172.23.122.179:8091 to this cluster @172.23.123.65:8091 [2021-02-22 23:14:24,667] - [rest_client] [140686669633280] - INFO - rebalance progress took 10.01 seconds [2021-02-22 23:14:24,667] - [rest_client] [140686669633280] - INFO - sleep for 10 seconds after rebalance... [2021-02-22 23:14:37,067] - [task] [140686669633280] - INFO - adding node 172.23.122.153:8091 to cluster [2021-02-22 23:14:37,068] - [rest_client] [140686669633280] - INFO - adding remote node @172.23.122.153:8091 to this cluster @172.23.123.65:8091 [2021-02-22 23:14:47,085] - [rest_client] [140686669633280] - INFO - rebalance progress took 10.02 seconds [2021-02-22 23:14:47,086] - [rest_client] [140686669633280] - INFO - sleep for 10 seconds after rebalance... [2021-02-22 23:14:59,456] - [task] [140686669633280] - INFO - adding node 172.23.122.139:8091 to cluster [2021-02-22 23:14:59,456] - [rest_client] [140686669633280] - INFO - adding remote node @172.23.122.139:8091 to this cluster @172.23.123.65:8091 [2021-02-22 23:15:09,473] - [rest_client] [140686669633280] - INFO - rebalance progress took 10.02 seconds [2021-02-22 23:15:09,473] - [rest_client] [140686669633280] - INFO - sleep for 10 seconds after rebalance... [2021-02-22 23:15:21,755] - [rest_client] [140686669633280] - INFO - rebalance params : {'knownNodes': 'ns_1@172.23.122.139,ns_1@172.23.122.153,ns_1@172.23.122.179,ns_1@172.23.122.182,ns_1@172.23.123.65', 'ejectedNodes': '', 'user': 'Administrator', 'password': 'password'} [2021-02-22 23:15:21,785] - [rest_client] [140686669633280] - INFO - rebalance operation started [2021-02-22 23:15:21,788] - [rest_client] [140686669633280] - INFO - rebalance percentage : 0.00 % [2021-02-22 23:15:21,788] - [task] [140686669633280] - INFO - Rebalance - status: running, progress: 0.00% [2021-02-22 23:15:31,818] - [task] [140686669633280] - INFO - Rebalance - status: none, progress: 100.00% [2021-02-22 23:15:31,831] - [task] [140686669633280] - INFO - rebalancing was completed with progress: 100% in 10.046032905578613 sec [2021-02-22 23:15:31,837] - [remote_util] [140689549858624] - INFO - SSH Connecting to 172.23.123.65 with username:root, attempt#1 of 5 [2021-02-22 23:15:31,984] - [remote_util] [140689549858624] - INFO - SSH Connected to 172.23.123.65 as root [2021-02-22 23:15:32,278] - [remote_util] [140689549858624] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 23:15:32,620] - [remote_util] [140689549858624] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 23:15:32,634] - [remote_util] [140689549858624] - INFO - running command.raw on 172.23.123.65: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-02-22 23:15:32,690] - [remote_util] [140689549858624] - INFO - command executed successfully with root [2021-02-22 23:15:32,690] - [remote_util] [140689549858624] - INFO - ['ok'] [2021-02-22 23:15:32,690] - [basetestcase] [140689549858624] - INFO - Enabled diag/eval for non-local hosts from 172.23.123.65 [2021-02-22 23:15:32,690] - [basetestcase] [140689549858624] - INFO - ============== basetestcase setup was finished for test #14 suite_tearDown ============== [2021-02-22 23:15:32,701] - [remote_util] [140689549858624] - INFO - SSH Connecting to 172.23.123.65 with username:root, attempt#1 of 5 [2021-02-22 23:15:32,861] - [remote_util] [140689549858624] - INFO - SSH Connected to 172.23.123.65 as root [2021-02-22 23:15:33,179] - [remote_util] [140689549858624] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 23:15:33,511] - [remote_util] [140689549858624] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 23:15:33,515] - [remote_util] [140689549858624] - INFO - SSH Connecting to 172.23.122.182 with username:root, attempt#1 of 5 [2021-02-22 23:15:33,620] - [remote_util] [140689549858624] - INFO - SSH Connected to 172.23.122.182 as root [2021-02-22 23:15:33,895] - [remote_util] [140689549858624] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 23:15:34,263] - [remote_util] [140689549858624] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 23:15:34,267] - [remote_util] [140689549858624] - INFO - SSH Connecting to 172.23.122.179 with username:root, attempt#1 of 5 [2021-02-22 23:15:34,376] - [remote_util] [140689549858624] - INFO - SSH Connected to 172.23.122.179 as root [2021-02-22 23:15:34,657] - [remote_util] [140689549858624] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 23:15:34,962] - [remote_util] [140689549858624] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 23:15:34,965] - [remote_util] [140689549858624] - INFO - SSH Connecting to 172.23.122.153 with username:root, attempt#1 of 5 [2021-02-22 23:15:35,073] - [remote_util] [140689549858624] - INFO - SSH Connected to 172.23.122.153 as root [2021-02-22 23:15:35,346] - [remote_util] [140689549858624] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 23:15:35,687] - [remote_util] [140689549858624] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 23:15:35,693] - [remote_util] [140689549858624] - INFO - SSH Connecting to 172.23.122.139 with username:root, attempt#1 of 5 [2021-02-22 23:15:35,801] - [remote_util] [140689549858624] - INFO - SSH Connected to 172.23.122.139 as root [2021-02-22 23:15:36,100] - [remote_util] [140689549858624] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 23:15:36,425] - [remote_util] [140689549858624] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 23:15:36,428] - [remote_util] [140689549858624] - INFO - SSH Connecting to 172.23.122.140 with username:root, attempt#1 of 5 [2021-02-22 23:15:36,539] - [remote_util] [140689549858624] - INFO - SSH Connected to 172.23.122.140 as root [2021-02-22 23:15:36,850] - [remote_util] [140689549858624] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 23:15:37,160] - [remote_util] [140689549858624] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 23:15:37,168] - [remote_util] [140689549858624] - INFO - SSH Connecting to 172.23.122.138 with username:root, attempt#1 of 5 [2021-02-22 23:15:37,274] - [remote_util] [140689549858624] - INFO - SSH Connected to 172.23.122.138 as root [2021-02-22 23:15:37,576] - [remote_util] [140689549858624] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 23:15:37,938] - [remote_util] [140689549858624] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 23:15:37,944] - [remote_util] [140689549858624] - INFO - SSH Connecting to 172.23.122.141 with username:root, attempt#1 of 5 [2021-02-22 23:15:38,052] - [remote_util] [140689549858624] - INFO - SSH Connected to 172.23.122.141 as root [2021-02-22 23:15:38,324] - [remote_util] [140689549858624] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 23:15:38,641] - [remote_util] [140689549858624] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 23:17:31,332] - [basetestcase] [140689549858624] - INFO - ------- Cluster statistics ------- [2021-02-22 23:17:31,332] - [basetestcase] [140689549858624] - INFO - 172.23.122.139:8091 => {'services': ['n1ql'], 'cpu_utilization': 0.703694395576778, 'mem_free': 3540959232, 'mem_total': 4201684992, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-02-22 23:17:31,332] - [basetestcase] [140689549858624] - INFO - 172.23.122.153:8091 => {'services': ['index'], 'cpu_utilization': 0.6259389083625438, 'mem_free': 3574194176, 'mem_total': 4201684992, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-02-22 23:17:31,333] - [basetestcase] [140689549858624] - INFO - 172.23.122.179:8091 => {'services': ['eventing'], 'cpu_utilization': 0.6517924291802457, 'mem_free': 3575123968, 'mem_total': 4201684992, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-02-22 23:17:31,333] - [basetestcase] [140689549858624] - INFO - 172.23.122.182:8091 => {'services': ['eventing'], 'cpu_utilization': 0.8260325406758448, 'mem_free': 3530670080, 'mem_total': 4201684992, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-02-22 23:17:31,333] - [basetestcase] [140689549858624] - INFO - 172.23.123.65:8091 => {'services': ['kv'], 'cpu_utilization': 0.7537688442211056, 'mem_free': 3493773312, 'mem_total': 4201672704, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-02-22 23:17:31,333] - [basetestcase] [140689549858624] - INFO - --- End of cluster statistics --- [2021-02-22 23:17:31,337] - [remote_util] [140689549858624] - INFO - SSH Connecting to 172.23.123.65 with username:root, attempt#1 of 5 [2021-02-22 23:17:31,442] - [remote_util] [140689549858624] - INFO - SSH Connected to 172.23.123.65 as root [2021-02-22 23:17:31,728] - [remote_util] [140689549858624] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 23:17:32,039] - [remote_util] [140689549858624] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 23:17:38,741] - [basetestcase] [140689549858624] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.139 port:8091 ssh_username:root] [2021-02-22 23:17:38,750] - [basetestcase] [140689549858624] - INFO - list of eventing nodes in cluster: [ip:172.23.122.179 port:8091 ssh_username:root, ip:172.23.122.182 port:8091 ssh_username:root] [2021-02-22 23:17:38,756] - [rest_client] [140689549858624] - INFO - settings/indexes params : storageMode=plasma [2021-02-22 23:17:38,762] - [eventing_base] [140689549858624] - INFO - Setting the min possible memory quota so that adding mode nodes to the cluster wouldn't be a problem. [2021-02-22 23:17:38,762] - [rest_client] [140689549858624] - INFO - pools/default params : memoryQuota=330 [2021-02-22 23:17:38,772] - [rest_client] [140689549858624] - INFO - pools/default params : indexMemoryQuota=256 [2021-02-22 23:17:38,778] - [rest_client] [140689549858624] - INFO - pools/default params : eventingMemoryQuota=512 [2021-02-22 23:17:38,798] - [basetestcase] [140689549858624] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.139 port:8091 ssh_username:root] [2021-02-22 23:17:38,804] - [rest_client] [140689549858624] - INFO - pools/default params : memoryQuota=700 [2021-02-22 23:17:38,811] - [eventing_rebalance_collection] [140689549858624] - INFO - 200 [2021-02-22 23:17:39,691] - [rest_client] [140686669633280] - INFO - http://172.23.123.65:8091/pools/default/buckets with param: name=src_bucket&ramQuotaMB=200&authType=none&replicaNumber=0&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore [2021-02-22 23:17:39,721] - [rest_client] [140686669633280] - INFO - 0.03 seconds to create bucket src_bucket [2021-02-22 23:17:39,722] - [bucket_helper] [140686669633280] - INFO - waiting for memcached bucket : src_bucket in 172.23.123.65 to accept set ops [2021-02-22 23:17:40,396] - [data_helper] [140686669633280] - INFO - creating direct client 172.23.123.65:11210 src_bucket [2021-02-22 23:17:40,865] - [data_helper] [140686669633280] - INFO - creating direct client 172.23.123.65:11210 src_bucket [2021-02-22 23:17:40,962] - [task] [140686669633280] - WARNING - vbucket map not ready after try 0 [2021-02-22 23:17:40,963] - [bucket_helper] [140686669633280] - INFO - waiting for memcached bucket : src_bucket in 172.23.123.65 to accept set ops [2021-02-22 23:17:41,114] - [data_helper] [140686669633280] - INFO - creating direct client 172.23.123.65:11210 src_bucket [2021-02-22 23:17:41,271] - [data_helper] [140686669633280] - INFO - creating direct client 172.23.123.65:11210 src_bucket [2021-02-22 23:17:41,348] - [task] [140686669633280] - WARNING - vbucket map not ready after try 1 [2021-02-22 23:17:41,349] - [bucket_helper] [140686669633280] - INFO - waiting for memcached bucket : src_bucket in 172.23.123.65 to accept set ops [2021-02-22 23:17:41,446] - [data_helper] [140686669633280] - INFO - creating direct client 172.23.123.65:11210 src_bucket [2021-02-22 23:17:41,864] - [data_helper] [140686669633280] - INFO - creating direct client 172.23.123.65:11210 src_bucket [2021-02-22 23:17:41,918] - [task] [140686669633280] - WARNING - vbucket map not ready after try 2 [2021-02-22 23:17:41,919] - [bucket_helper] [140686669633280] - INFO - waiting for memcached bucket : src_bucket in 172.23.123.65 to accept set ops [2021-02-22 23:17:42,013] - [data_helper] [140686669633280] - INFO - creating direct client 172.23.123.65:11210 src_bucket [2021-02-22 23:17:42,106] - [data_helper] [140686669633280] - INFO - creating direct client 172.23.123.65:11210 src_bucket [2021-02-22 23:17:42,159] - [task] [140686669633280] - WARNING - vbucket map not ready after try 3 [2021-02-22 23:17:42,160] - [bucket_helper] [140686669633280] - INFO - waiting for memcached bucket : src_bucket in 172.23.123.65 to accept set ops [2021-02-22 23:17:42,576] - [data_helper] [140686669633280] - INFO - creating direct client 172.23.123.65:11210 src_bucket [2021-02-22 23:17:42,703] - [data_helper] [140686669633280] - INFO - creating direct client 172.23.123.65:11210 src_bucket [2021-02-22 23:17:42,773] - [task] [140686669633280] - WARNING - vbucket map not ready after try 4 [2021-02-22 23:17:42,774] - [bucket_helper] [140686669633280] - INFO - waiting for memcached bucket : src_bucket in 172.23.123.65 to accept set ops [2021-02-22 23:17:42,894] - [data_helper] [140686669633280] - INFO - creating direct client 172.23.123.65:11210 src_bucket [2021-02-22 23:17:43,310] - [data_helper] [140686669633280] - INFO - creating direct client 172.23.123.65:11210 src_bucket [2021-02-22 23:17:43,367] - [task] [140686669633280] - WARNING - vbucket map not ready after try 5 [2021-02-22 23:17:44,403] - [rest_client] [140686669633280] - INFO - http://172.23.123.65:8091/pools/default/buckets with param: name=dst_bucket&ramQuotaMB=200&authType=none&replicaNumber=0&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore [2021-02-22 23:17:44,434] - [rest_client] [140686669633280] - INFO - 0.03 seconds to create bucket dst_bucket [2021-02-22 23:17:44,435] - [bucket_helper] [140686669633280] - INFO - waiting for memcached bucket : dst_bucket in 172.23.123.65 to accept set ops [2021-02-22 23:17:45,123] - [data_helper] [140686669633280] - INFO - creating direct client 172.23.123.65:11210 dst_bucket [2021-02-22 23:17:45,281] - [data_helper] [140686669633280] - INFO - creating direct client 172.23.123.65:11210 dst_bucket [2021-02-22 23:17:45,371] - [task] [140686669633280] - WARNING - vbucket map not ready after try 0 [2021-02-22 23:17:45,371] - [bucket_helper] [140686669633280] - INFO - waiting for memcached bucket : dst_bucket in 172.23.123.65 to accept set ops [2021-02-22 23:17:46,025] - [data_helper] [140686669633280] - INFO - creating direct client 172.23.123.65:11210 dst_bucket [2021-02-22 23:17:46,202] - [data_helper] [140686669633280] - INFO - creating direct client 172.23.123.65:11210 dst_bucket [2021-02-22 23:17:46,303] - [task] [140686669633280] - WARNING - vbucket map not ready after try 1 [2021-02-22 23:17:46,304] - [bucket_helper] [140686669633280] - INFO - waiting for memcached bucket : dst_bucket in 172.23.123.65 to accept set ops [2021-02-22 23:17:46,461] - [data_helper] [140686669633280] - INFO - creating direct client 172.23.123.65:11210 dst_bucket [2021-02-22 23:17:46,610] - [data_helper] [140686669633280] - INFO - creating direct client 172.23.123.65:11210 dst_bucket [2021-02-22 23:17:47,034] - [task] [140686669633280] - WARNING - vbucket map not ready after try 2 [2021-02-22 23:17:47,034] - [bucket_helper] [140686669633280] - INFO - waiting for memcached bucket : dst_bucket in 172.23.123.65 to accept set ops [2021-02-22 23:17:47,175] - [data_helper] [140686669633280] - INFO - creating direct client 172.23.123.65:11210 dst_bucket [2021-02-22 23:17:47,326] - [data_helper] [140686669633280] - INFO - creating direct client 172.23.123.65:11210 dst_bucket [2021-02-22 23:17:47,414] - [task] [140686669633280] - WARNING - vbucket map not ready after try 3 [2021-02-22 23:17:47,415] - [bucket_helper] [140686669633280] - INFO - waiting for memcached bucket : dst_bucket in 172.23.123.65 to accept set ops [2021-02-22 23:17:47,880] - [data_helper] [140686669633280] - INFO - creating direct client 172.23.123.65:11210 dst_bucket [2021-02-22 23:17:48,031] - [data_helper] [140686669633280] - INFO - creating direct client 172.23.123.65:11210 dst_bucket [2021-02-22 23:17:48,126] - [task] [140686669633280] - WARNING - vbucket map not ready after try 4 [2021-02-22 23:17:48,126] - [bucket_helper] [140686669633280] - INFO - waiting for memcached bucket : dst_bucket in 172.23.123.65 to accept set ops [2021-02-22 23:17:48,265] - [data_helper] [140686669633280] - INFO - creating direct client 172.23.123.65:11210 dst_bucket [2021-02-22 23:17:48,443] - [data_helper] [140686669633280] - INFO - creating direct client 172.23.123.65:11210 dst_bucket [2021-02-22 23:17:48,540] - [task] [140686669633280] - WARNING - vbucket map not ready after try 5 [2021-02-22 23:17:49,575] - [rest_client] [140686669633280] - INFO - http://172.23.123.65:8091/pools/default/buckets with param: name=metadata&ramQuotaMB=200&authType=none&replicaNumber=0&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore [2021-02-22 23:17:49,605] - [rest_client] [140686669633280] - INFO - 0.03 seconds to create bucket metadata [2021-02-22 23:17:49,605] - [bucket_helper] [140686669633280] - INFO - waiting for memcached bucket : metadata in 172.23.123.65 to accept set ops [2021-02-22 23:17:50,629] - [data_helper] [140686669633280] - INFO - creating direct client 172.23.123.65:11210 metadata [2021-02-22 23:17:50,786] - [data_helper] [140686669633280] - INFO - creating direct client 172.23.123.65:11210 metadata [2021-02-22 23:17:50,891] - [task] [140686669633280] - WARNING - vbucket map not ready after try 0 [2021-02-22 23:17:50,892] - [bucket_helper] [140686669633280] - INFO - waiting for memcached bucket : metadata in 172.23.123.65 to accept set ops [2021-02-22 23:17:51,055] - [data_helper] [140686669633280] - INFO - creating direct client 172.23.123.65:11210 metadata [2021-02-22 23:17:51,545] - [data_helper] [140686669633280] - INFO - creating direct client 172.23.123.65:11210 metadata [2021-02-22 23:17:51,608] - [task] [140686669633280] - WARNING - vbucket map not ready after try 1 [2021-02-22 23:17:51,609] - [bucket_helper] [140686669633280] - INFO - waiting for memcached bucket : metadata in 172.23.123.65 to accept set ops [2021-02-22 23:17:51,702] - [data_helper] [140686669633280] - INFO - creating direct client 172.23.123.65:11210 metadata [2021-02-22 23:17:51,811] - [data_helper] [140686669633280] - INFO - creating direct client 172.23.123.65:11210 metadata [2021-02-22 23:17:51,871] - [task] [140686669633280] - WARNING - vbucket map not ready after try 2 [2021-02-22 23:17:51,871] - [bucket_helper] [140686669633280] - INFO - waiting for memcached bucket : metadata in 172.23.123.65 to accept set ops [2021-02-22 23:17:52,375] - [data_helper] [140686669633280] - INFO - creating direct client 172.23.123.65:11210 metadata [2021-02-22 23:17:52,514] - [data_helper] [140686669633280] - INFO - creating direct client 172.23.123.65:11210 metadata [2021-02-22 23:17:52,589] - [task] [140686669633280] - WARNING - vbucket map not ready after try 3 [2021-02-22 23:17:52,590] - [bucket_helper] [140686669633280] - INFO - waiting for memcached bucket : metadata in 172.23.123.65 to accept set ops [2021-02-22 23:17:52,695] - [data_helper] [140686669633280] - INFO - creating direct client 172.23.123.65:11210 metadata [2021-02-22 23:17:52,789] - [data_helper] [140686669633280] - INFO - creating direct client 172.23.123.65:11210 metadata [2021-02-22 23:17:53,162] - [task] [140686669633280] - WARNING - vbucket map not ready after try 4 [2021-02-22 23:17:53,162] - [bucket_helper] [140686669633280] - INFO - waiting for memcached bucket : metadata in 172.23.123.65 to accept set ops [2021-02-22 23:17:53,252] - [data_helper] [140686669633280] - INFO - creating direct client 172.23.123.65:11210 metadata [2021-02-22 23:17:53,350] - [data_helper] [140686669633280] - INFO - creating direct client 172.23.123.65:11210 metadata [2021-02-22 23:17:53,444] - [task] [140686669633280] - WARNING - vbucket map not ready after try 5 [2021-02-22 23:17:53,540] - [rest_client] [140689549858624] - INFO - http://172.23.122.179:8091/pools/default/buckets/dst_bucket/scopes with params: name=scope_0 [2021-02-22 23:17:53,540] - [rest_client] [140689549858624] - INFO - Scope created dst_bucket->scope_0 {'uid': '1'} [2021-02-22 23:17:53,576] - [rest_client] [140689549858624] - INFO - http://172.23.122.179:8091/pools/default/buckets/dst_bucket/scopes with params: name=scope_1 [2021-02-22 23:17:53,576] - [rest_client] [140689549858624] - INFO - Scope created dst_bucket->scope_1 {'uid': '2'} [2021-02-22 23:17:53,607] - [rest_client] [140689549858624] - INFO - http://172.23.122.179:8091/pools/default/buckets/dst_bucket/scopes with params: name=scope_2 [2021-02-22 23:17:53,607] - [rest_client] [140689549858624] - INFO - Scope created dst_bucket->scope_2 {'uid': '3'} [2021-02-22 23:17:53,636] - [rest_client] [140689549858624] - INFO - http://172.23.122.179:8091/pools/default/buckets/dst_bucket/scopes with params: name=scope_3 [2021-02-22 23:17:53,636] - [rest_client] [140689549858624] - INFO - Scope created dst_bucket->scope_3 {'uid': '4'} [2021-02-22 23:17:53,668] - [rest_client] [140689549858624] - INFO - http://172.23.122.179:8091/pools/default/buckets/dst_bucket/scopes with params: name=scope_4 [2021-02-22 23:17:53,668] - [rest_client] [140689549858624] - INFO - Scope created dst_bucket->scope_4 {'uid': '5'} [2021-02-22 23:17:53,697] - [rest_client] [140689549858624] - INFO - http://172.23.122.179:8091/pools/default/buckets/src_bucket/scopes with params: name=scope_0 [2021-02-22 23:17:53,697] - [rest_client] [140689549858624] - INFO - Scope created src_bucket->scope_0 {'uid': '1'} [2021-02-22 23:17:53,730] - [rest_client] [140689549858624] - INFO - http://172.23.122.179:8091/pools/default/buckets/src_bucket/scopes with params: name=scope_1 [2021-02-22 23:17:53,730] - [rest_client] [140689549858624] - INFO - Scope created src_bucket->scope_1 {'uid': '2'} [2021-02-22 23:17:53,761] - [rest_client] [140689549858624] - INFO - http://172.23.122.179:8091/pools/default/buckets/src_bucket/scopes with params: name=scope_2 [2021-02-22 23:17:53,761] - [rest_client] [140689549858624] - INFO - Scope created src_bucket->scope_2 {'uid': '3'} [2021-02-22 23:17:53,791] - [rest_client] [140689549858624] - INFO - http://172.23.122.179:8091/pools/default/buckets/src_bucket/scopes with params: name=scope_3 [2021-02-22 23:17:53,791] - [rest_client] [140689549858624] - INFO - Scope created src_bucket->scope_3 {'uid': '4'} [2021-02-22 23:17:53,823] - [rest_client] [140689549858624] - INFO - http://172.23.122.179:8091/pools/default/buckets/src_bucket/scopes with params: name=scope_4 [2021-02-22 23:17:53,823] - [rest_client] [140689549858624] - INFO - Scope created src_bucket->scope_4 {'uid': '5'} [2021-02-22 23:17:53,852] - [rest_client] [140689549858624] - INFO - http://172.23.122.179:8091/pools/default/buckets/dst_bucket/scopes/scope_1/collections with params: name=coll_0 [2021-02-22 23:17:53,853] - [rest_client] [140689549858624] - INFO - Collection created dst_bucket->scope_1->coll_0 manifest:{'uid': '6'} [2021-02-22 23:17:53,884] - [rest_client] [140689549858624] - INFO - http://172.23.122.179:8091/pools/default/buckets/dst_bucket/scopes/scope_1/collections with params: name=coll_1 [2021-02-22 23:17:53,885] - [rest_client] [140689549858624] - INFO - Collection created dst_bucket->scope_1->coll_1 manifest:{'uid': '7'} [2021-02-22 23:17:53,916] - [rest_client] [140689549858624] - INFO - http://172.23.122.179:8091/pools/default/buckets/dst_bucket/scopes/scope_1/collections with params: name=coll_2 [2021-02-22 23:17:53,917] - [rest_client] [140689549858624] - INFO - Collection created dst_bucket->scope_1->coll_2 manifest:{'uid': '8'} [2021-02-22 23:17:53,947] - [rest_client] [140689549858624] - INFO - http://172.23.122.179:8091/pools/default/buckets/dst_bucket/scopes/scope_1/collections with params: name=coll_3 [2021-02-22 23:17:53,948] - [rest_client] [140689549858624] - INFO - Collection created dst_bucket->scope_1->coll_3 manifest:{'uid': '9'} [2021-02-22 23:17:53,978] - [rest_client] [140689549858624] - INFO - http://172.23.122.179:8091/pools/default/buckets/dst_bucket/scopes/scope_1/collections with params: name=coll_4 [2021-02-22 23:17:53,978] - [rest_client] [140689549858624] - INFO - Collection created dst_bucket->scope_1->coll_4 manifest:{'uid': 'a'} [2021-02-22 23:17:54,009] - [rest_client] [140689549858624] - INFO - http://172.23.122.179:8091/pools/default/buckets/src_bucket/scopes/scope_1/collections with params: name=coll_0 [2021-02-22 23:17:54,010] - [rest_client] [140689549858624] - INFO - Collection created src_bucket->scope_1->coll_0 manifest:{'uid': '6'} [2021-02-22 23:17:54,047] - [rest_client] [140689549858624] - INFO - http://172.23.122.179:8091/pools/default/buckets/src_bucket/scopes/scope_1/collections with params: name=coll_1 [2021-02-22 23:17:54,047] - [rest_client] [140689549858624] - INFO - Collection created src_bucket->scope_1->coll_1 manifest:{'uid': '7'} [2021-02-22 23:17:54,077] - [rest_client] [140689549858624] - INFO - http://172.23.122.179:8091/pools/default/buckets/src_bucket/scopes/scope_1/collections with params: name=coll_2 [2021-02-22 23:17:54,078] - [rest_client] [140689549858624] - INFO - Collection created src_bucket->scope_1->coll_2 manifest:{'uid': '8'} [2021-02-22 23:17:54,113] - [rest_client] [140689549858624] - INFO - http://172.23.122.179:8091/pools/default/buckets/src_bucket/scopes/scope_1/collections with params: name=coll_3 [2021-02-22 23:17:54,113] - [rest_client] [140689549858624] - INFO - Collection created src_bucket->scope_1->coll_3 manifest:{'uid': '9'} [2021-02-22 23:17:54,148] - [rest_client] [140689549858624] - INFO - http://172.23.122.179:8091/pools/default/buckets/src_bucket/scopes/scope_1/collections with params: name=coll_4 [2021-02-22 23:17:54,149] - [rest_client] [140689549858624] - INFO - Collection created src_bucket->scope_1->coll_4 manifest:{'uid': 'a'} [2021-02-22 23:17:54,167] - [basetestcase] [140689549858624] - INFO - list of eventing nodes in cluster: [ip:172.23.122.179 port:8091 ssh_username:root, ip:172.23.122.182 port:8091 ssh_username:root] [2021-02-22 23:17:54,178] - [eventing_base] [140689549858624] - INFO - Stats for Node 172.23.122.179 is [] [2021-02-22 23:17:54,189] - [eventing_base] [140689549858624] - INFO - Stats for Node 172.23.122.182 is [] [2021-02-22 23:17:54,217] - [basetestcase] [140689549858624] - INFO - list of eventing nodes in cluster: [ip:172.23.122.179 port:8091 ssh_username:root, ip:172.23.122.182 port:8091 ssh_username:root] [2021-02-22 23:17:54,220] - [remote_util] [140689549858624] - INFO - SSH Connecting to 172.23.122.179 with username:root, attempt#1 of 5 [2021-02-22 23:17:54,327] - [remote_util] [140689549858624] - INFO - SSH Connected to 172.23.122.179 as root [2021-02-22 23:17:54,618] - [remote_util] [140689549858624] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 23:17:54,928] - [remote_util] [140689549858624] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 23:17:54,938] - [rest_client] [140689549858624] - INFO - /diag/eval status on 172.23.122.179:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). [2021-02-22 23:17:54,939] - [remote_util] [140689549858624] - INFO - running command.raw on 172.23.122.179: zgrep "panic" "/opt/couchbase/var/lib/couchbase/logs"/eventing.log* | wc -l [2021-02-22 23:17:55,602] - [remote_util] [140689549858624] - INFO - command executed successfully with root [2021-02-22 23:17:55,602] - [remote_util] [140689549858624] - INFO - running command.raw on 172.23.122.179: ls "/opt/couchbase/var/lib/couchbase/logs"/../crash/| wc -l [2021-02-22 23:17:55,618] - [remote_util] [140689549858624] - INFO - command executed successfully with root [2021-02-22 23:17:55,622] - [remote_util] [140689549858624] - INFO - SSH Connecting to 172.23.122.182 with username:root, attempt#1 of 5 [2021-02-22 23:17:55,726] - [remote_util] [140689549858624] - INFO - SSH Connected to 172.23.122.182 as root [2021-02-22 23:17:55,986] - [remote_util] [140689549858624] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 23:17:56,327] - [remote_util] [140689549858624] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 23:17:56,339] - [rest_client] [140689549858624] - INFO - /diag/eval status on 172.23.122.182:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). [2021-02-22 23:17:56,339] - [remote_util] [140689549858624] - INFO - running command.raw on 172.23.122.182: zgrep "panic" "/opt/couchbase/var/lib/couchbase/logs"/eventing.log* | wc -l [2021-02-22 23:17:57,936] - [remote_util] [140689549858624] - INFO - command executed successfully with root [2021-02-22 23:17:57,937] - [remote_util] [140689549858624] - INFO - running command.raw on 172.23.122.182: ls "/opt/couchbase/var/lib/couchbase/logs"/../crash/| wc -l [2021-02-22 23:17:57,991] - [remote_util] [140689549858624] - INFO - command executed successfully with root [2021-02-22 23:17:58,390] - [basetestcase] [140689549858624] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.139 port:8091 ssh_username:root] [2021-02-22 23:17:58,398] - [rest_client] [140689549858624] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+metadata [2021-02-22 23:17:58,448] - [eventing_base] [140689549858624] - INFO - number of documents in metadata bucket 0 [2021-02-22 23:17:58,459] - [basetestcase] [140689549858624] - INFO - ------- Cluster statistics ------- [2021-02-22 23:17:58,460] - [basetestcase] [140689549858624] - INFO - 172.23.122.139:8091 => {'services': ['n1ql'], 'cpu_utilization': 1.432160804020101, 'mem_free': 3508436992, 'mem_total': 4201684992, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-02-22 23:17:58,460] - [basetestcase] [140689549858624] - INFO - 172.23.122.153:8091 => {'services': ['index'], 'cpu_utilization': 4.546596332579754, 'mem_free': 3452321792, 'mem_total': 4201684992, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-02-22 23:17:58,460] - [basetestcase] [140689549858624] - INFO - 172.23.122.179:8091 => {'services': ['eventing'], 'cpu_utilization': 1.705970898143502, 'mem_free': 3543068672, 'mem_total': 4201684992, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-02-22 23:17:58,461] - [basetestcase] [140689549858624] - INFO - 172.23.122.182:8091 => {'services': ['eventing'], 'cpu_utilization': 2.506893958385561, 'mem_free': 3486265344, 'mem_total': 4201684992, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-02-22 23:17:58,461] - [basetestcase] [140689549858624] - INFO - 172.23.123.65:8091 => {'services': ['kv'], 'cpu_utilization': 14.73362222788682, 'mem_free': 3301687296, 'mem_total': 4201672704, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-02-22 23:17:58,461] - [basetestcase] [140689549858624] - INFO - --- End of cluster statistics --- [2021-02-22 23:17:58,465] - [remote_util] [140689549858624] - INFO - SSH Connecting to 172.23.123.65 with username:root, attempt#1 of 5 [2021-02-22 23:17:58,567] - [remote_util] [140689549858624] - INFO - SSH Connected to 172.23.123.65 as root [2021-02-22 23:17:58,854] - [remote_util] [140689549858624] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 23:17:59,208] - [remote_util] [140689549858624] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 23:17:59,212] - [remote_util] [140689549858624] - INFO - SSH Connecting to 172.23.122.182 with username:root, attempt#1 of 5 [2021-02-22 23:17:59,312] - [remote_util] [140689549858624] - INFO - SSH Connected to 172.23.122.182 as root [2021-02-22 23:17:59,577] - [remote_util] [140689549858624] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 23:17:59,879] - [remote_util] [140689549858624] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 23:17:59,885] - [remote_util] [140689549858624] - INFO - SSH Connecting to 172.23.122.179 with username:root, attempt#1 of 5 [2021-02-22 23:17:59,984] - [remote_util] [140689549858624] - INFO - SSH Connected to 172.23.122.179 as root [2021-02-22 23:18:00,254] - [remote_util] [140689549858624] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 23:18:00,556] - [remote_util] [140689549858624] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 23:18:00,561] - [remote_util] [140689549858624] - INFO - SSH Connecting to 172.23.122.153 with username:root, attempt#1 of 5 [2021-02-22 23:18:00,664] - [remote_util] [140689549858624] - INFO - SSH Connected to 172.23.122.153 as root [2021-02-22 23:18:00,944] - [remote_util] [140689549858624] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 23:18:01,247] - [remote_util] [140689549858624] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 23:18:01,255] - [remote_util] [140689549858624] - INFO - SSH Connecting to 172.23.122.139 with username:root, attempt#1 of 5 [2021-02-22 23:18:01,359] - [remote_util] [140689549858624] - INFO - SSH Connected to 172.23.122.139 as root [2021-02-22 23:18:01,642] - [remote_util] [140689549858624] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 23:18:01,950] - [remote_util] [140689549858624] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 23:18:01,954] - [remote_util] [140689549858624] - INFO - SSH Connecting to 172.23.122.140 with username:root, attempt#1 of 5 [2021-02-22 23:18:02,055] - [remote_util] [140689549858624] - INFO - SSH Connected to 172.23.122.140 as root [2021-02-22 23:18:02,345] - [remote_util] [140689549858624] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 23:18:02,648] - [remote_util] [140689549858624] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 23:18:02,652] - [remote_util] [140689549858624] - INFO - SSH Connecting to 172.23.122.138 with username:root, attempt#1 of 5 [2021-02-22 23:18:02,760] - [remote_util] [140689549858624] - INFO - SSH Connected to 172.23.122.138 as root [2021-02-22 23:18:03,051] - [remote_util] [140689549858624] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 23:18:03,355] - [remote_util] [140689549858624] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 23:18:03,359] - [remote_util] [140689549858624] - INFO - SSH Connecting to 172.23.122.141 with username:root, attempt#1 of 5 [2021-02-22 23:18:03,463] - [remote_util] [140689549858624] - INFO - SSH Connected to 172.23.122.141 as root [2021-02-22 23:18:03,741] - [remote_util] [140689549858624] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 23:18:04,042] - [remote_util] [140689549858624] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 23:19:56,555] - [basetestcase] [140689549858624] - INFO - ============== basetestcase cleanup was started for test #14 suite_tearDown ============== [2021-02-22 23:19:56,626] - [bucket_helper] [140689549858624] - INFO - deleting existing buckets ['dst_bucket', 'metadata', 'src_bucket'] on 172.23.123.65 [2021-02-22 23:19:57,017] - [bucket_helper] [140689549858624] - INFO - waiting for bucket deletion to complete.... [2021-02-22 23:19:57,051] - [rest_client] [140689549858624] - INFO - node 172.23.123.65 existing buckets : ['metadata', 'src_bucket'] [2021-02-22 23:19:57,053] - [bucket_helper] [140689549858624] - INFO - deleted bucket : dst_bucket from 172.23.123.65 [2021-02-22 23:19:57,439] - [bucket_helper] [140689549858624] - INFO - waiting for bucket deletion to complete.... [2021-02-22 23:19:57,457] - [rest_client] [140689549858624] - INFO - node 172.23.123.65 existing buckets : ['src_bucket'] [2021-02-22 23:19:57,459] - [bucket_helper] [140689549858624] - INFO - deleted bucket : metadata from 172.23.123.65 [2021-02-22 23:19:57,851] - [bucket_helper] [140689549858624] - INFO - waiting for bucket deletion to complete.... [2021-02-22 23:19:57,855] - [rest_client] [140689549858624] - INFO - node 172.23.123.65 existing buckets : [] [2021-02-22 23:19:57,855] - [bucket_helper] [140689549858624] - INFO - deleted bucket : src_bucket from 172.23.123.65 [2021-02-22 23:19:57,866] - [bucket_helper] [140689549858624] - INFO - Could not find any buckets for node 172.23.122.182, nothing to delete [2021-02-22 23:19:57,875] - [bucket_helper] [140689549858624] - INFO - Could not find any buckets for node 172.23.122.179, nothing to delete [2021-02-22 23:19:57,883] - [bucket_helper] [140689549858624] - INFO - Could not find any buckets for node 172.23.122.153, nothing to delete [2021-02-22 23:19:57,892] - [bucket_helper] [140689549858624] - INFO - Could not find any buckets for node 172.23.122.139, nothing to delete [2021-02-22 23:19:57,899] - [bucket_helper] [140689549858624] - INFO - Could not find any buckets for node 172.23.122.140, nothing to delete [2021-02-22 23:19:57,907] - [bucket_helper] [140689549858624] - INFO - Could not find any buckets for node 172.23.122.138, nothing to delete [2021-02-22 23:19:57,913] - [bucket_helper] [140689549858624] - INFO - Could not find any buckets for node 172.23.122.141, nothing to delete [2021-02-22 23:19:57,920] - [rest_client] [140689549858624] - INFO - -->is_ns_server_running? [2021-02-22 23:19:57,938] - [cluster_helper] [140689549858624] - INFO - rebalancing all nodes in order to remove nodes [2021-02-22 23:19:57,941] - [rest_client] [140689549858624] - INFO - rebalance params : {'knownNodes': 'ns_1@172.23.122.139,ns_1@172.23.122.153,ns_1@172.23.122.179,ns_1@172.23.122.182,ns_1@172.23.123.65', 'ejectedNodes': 'ns_1@172.23.122.139,ns_1@172.23.122.153,ns_1@172.23.122.179,ns_1@172.23.122.182', 'user': 'Administrator', 'password': 'password'} [2021-02-22 23:19:57,972] - [rest_client] [140689549858624] - INFO - rebalance operation started [2021-02-22 23:19:57,976] - [rest_client] [140689549858624] - INFO - rebalance percentage : 0.00 % [2021-02-22 23:20:07,994] - [rest_client] [140689549858624] - INFO - rebalance percentage : 80.00 % [2021-02-22 23:20:18,008] - [rest_client] [140689549858624] - INFO - rebalance percentage : 80.00 % [2021-02-22 23:20:38,029] - [rest_client] [140689549858624] - INFO - rebalance progress took 40.06 seconds [2021-02-22 23:20:38,030] - [rest_client] [140689549858624] - INFO - sleep for 10 seconds after rebalance... [2021-02-22 23:20:48,082] - [cluster_helper] [140689549858624] - INFO - removed all the nodes from cluster associated with ip:172.23.123.65 port:8091 ssh_username:root ? [('ns_1@172.23.122.139', 8091), ('ns_1@172.23.122.153', 8091), ('ns_1@172.23.122.179', 8091), ('ns_1@172.23.122.182', 8091)] [2021-02-22 23:20:48,088] - [cluster_helper] [140689549858624] - INFO - waiting for ns_server @ 172.23.123.65:8091 [2021-02-22 23:20:48,089] - [rest_client] [140689549858624] - INFO - -->is_ns_server_running? [2021-02-22 23:20:48,094] - [cluster_helper] [140689549858624] - INFO - ns_server @ 172.23.123.65:8091 is running [2021-02-22 23:20:48,098] - [cluster_helper] [140689549858624] - INFO - waiting for ns_server @ 172.23.122.182:8091 [2021-02-22 23:20:48,098] - [rest_client] [140689549858624] - INFO - -->is_ns_server_running? [2021-02-22 23:20:48,102] - [cluster_helper] [140689549858624] - INFO - ns_server @ 172.23.122.182:8091 is running [2021-02-22 23:20:48,107] - [cluster_helper] [140689549858624] - INFO - waiting for ns_server @ 172.23.122.179:8091 [2021-02-22 23:20:48,107] - [rest_client] [140689549858624] - INFO - -->is_ns_server_running? [2021-02-22 23:20:48,111] - [cluster_helper] [140689549858624] - INFO - ns_server @ 172.23.122.179:8091 is running [2021-02-22 23:20:48,115] - [cluster_helper] [140689549858624] - INFO - waiting for ns_server @ 172.23.122.153:8091 [2021-02-22 23:20:48,116] - [rest_client] [140689549858624] - INFO - -->is_ns_server_running? [2021-02-22 23:20:48,120] - [cluster_helper] [140689549858624] - INFO - ns_server @ 172.23.122.153:8091 is running [2021-02-22 23:20:48,124] - [cluster_helper] [140689549858624] - INFO - waiting for ns_server @ 172.23.122.139:8091 [2021-02-22 23:20:48,124] - [rest_client] [140689549858624] - INFO - -->is_ns_server_running? [2021-02-22 23:20:48,128] - [cluster_helper] [140689549858624] - INFO - ns_server @ 172.23.122.139:8091 is running [2021-02-22 23:20:48,132] - [cluster_helper] [140689549858624] - INFO - waiting for ns_server @ 172.23.122.140:8091 [2021-02-22 23:20:48,132] - [rest_client] [140689549858624] - INFO - -->is_ns_server_running? [2021-02-22 23:20:48,136] - [cluster_helper] [140689549858624] - INFO - ns_server @ 172.23.122.140:8091 is running [2021-02-22 23:20:48,141] - [cluster_helper] [140689549858624] - INFO - waiting for ns_server @ 172.23.122.138:8091 [2021-02-22 23:20:48,141] - [rest_client] [140689549858624] - INFO - -->is_ns_server_running? [2021-02-22 23:20:48,145] - [cluster_helper] [140689549858624] - INFO - ns_server @ 172.23.122.138:8091 is running [2021-02-22 23:20:48,149] - [cluster_helper] [140689549858624] - INFO - waiting for ns_server @ 172.23.122.141:8091 [2021-02-22 23:20:48,150] - [rest_client] [140689549858624] - INFO - -->is_ns_server_running? [2021-02-22 23:20:48,153] - [cluster_helper] [140689549858624] - INFO - ns_server @ 172.23.122.141:8091 is running [2021-02-22 23:20:48,154] - [basetestcase] [140689549858624] - INFO - ============== basetestcase cleanup was finished for test #14 suite_tearDown ============== [2021-02-22 23:20:48,154] - [basetestcase] [140689549858624] - INFO - closing all ssh connections [2021-02-22 23:20:48,159] - [basetestcase] [140689549858624] - INFO - closing all memcached connections [2021-02-22 23:20:48,162] - [remote_util] [140689549858624] - INFO - SSH Connecting to 172.23.123.65 with username:root, attempt#1 of 5 [2021-02-22 23:20:48,265] - [remote_util] [140689549858624] - INFO - SSH Connected to 172.23.123.65 as root [2021-02-22 23:20:48,561] - [remote_util] [140689549858624] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 23:20:48,912] - [remote_util] [140689549858624] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 23:20:48,929] - [remote_util] [140689549858624] - INFO - SSH Connecting to 172.23.123.65 with username:root, attempt#1 of 5 [2021-02-22 23:20:49,032] - [remote_util] [140689549858624] - INFO - SSH Connected to 172.23.123.65 as root [2021-02-22 23:20:49,313] - [remote_util] [140689549858624] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 23:20:49,621] - [remote_util] [140689549858624] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 23:20:49,683] - [remote_util] [140689549858624] - INFO - 172.23.123.65 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-22 23:20:49,784] - [remote_util] [140689549858624] - INFO - Running systemd command on this server [2021-02-22 23:20:49,784] - [remote_util] [140689549858624] - INFO - running command.raw on 172.23.123.65: systemctl stop couchbase-server.service [2021-02-22 23:20:51,492] - [remote_util] [140689549858624] - INFO - command executed successfully with root [2021-02-22 23:20:51,492] - [basetestcase] [140689549858624] - INFO - Couchbase stopped [2021-02-22 23:20:51,493] - [basetestcase] [140689549858624] - INFO - sleep for 5 secs. ... [2021-02-22 23:20:56,498] - [remote_util] [140689549858624] - INFO - running command.raw on 172.23.123.65: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-22 23:20:56,515] - [remote_util] [140689549858624] - INFO - command executed successfully with root [2021-02-22 23:20:56,515] - [remote_util] [140689549858624] - INFO - running command.raw on 172.23.123.65: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-22 23:20:56,570] - [remote_util] [140689549858624] - INFO - command executed successfully with root [2021-02-22 23:20:56,573] - [remote_util] [140689549858624] - INFO - SSH Connecting to 172.23.123.65 with username:root, attempt#1 of 5 [2021-02-22 23:20:56,675] - [remote_util] [140689549858624] - INFO - SSH Connected to 172.23.123.65 as root [2021-02-22 23:20:56,961] - [remote_util] [140689549858624] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 23:20:57,271] - [remote_util] [140689549858624] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 23:20:57,334] - [remote_util] [140689549858624] - INFO - 172.23.123.65 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-22 23:20:57,334] - [remote_util] [140689549858624] - INFO - Checking for process beam.smp on linux [2021-02-22 23:20:57,415] - [remote_util] [140689549858624] - INFO - Starting couchbase server [2021-02-22 23:20:57,522] - [remote_util] [140689549858624] - INFO - Running systemd command on this server [2021-02-22 23:20:57,523] - [remote_util] [140689549858624] - INFO - running command.raw on 172.23.123.65: systemctl start couchbase-server.service [2021-02-22 23:20:57,546] - [remote_util] [140689549858624] - INFO - command executed successfully with root [2021-02-22 23:20:57,546] - [remote_util] [140689549858624] - INFO - 172.23.123.65:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-22 23:21:02,548] - [remote_util] [140689549858624] - INFO - running command.raw on 172.23.123.65: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-22 23:21:02,577] - [remote_util] [140689549858624] - INFO - command executed successfully with root [2021-02-22 23:21:02,578] - [remote_util] [140689549858624] - INFO - Couchbase server status: [' Process: 39697 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-22 23:21:02,578] - [remote_util] [140689549858624] - INFO - Checking for process beam.smp on linux [2021-02-22 23:21:02,651] - [remote_util] [140689549858624] - INFO - process beam.smp is running on 172.23.123.65: with pid 39839 [2021-02-22 23:21:02,652] - [basetestcase] [140689549858624] - INFO - Couchbase started [2021-02-22 23:21:02,655] - [remote_util] [140689549858624] - INFO - SSH Connecting to 172.23.122.182 with username:root, attempt#1 of 5 [2021-02-22 23:21:02,761] - [remote_util] [140689549858624] - INFO - SSH Connected to 172.23.122.182 as root [2021-02-22 23:21:03,039] - [remote_util] [140689549858624] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 23:21:03,358] - [remote_util] [140689549858624] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 23:21:03,374] - [remote_util] [140689549858624] - INFO - SSH Connecting to 172.23.122.182 with username:root, attempt#1 of 5 [2021-02-22 23:21:03,480] - [remote_util] [140689549858624] - INFO - SSH Connected to 172.23.122.182 as root [2021-02-22 23:21:03,757] - [remote_util] [140689549858624] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 23:21:04,075] - [remote_util] [140689549858624] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 23:21:04,140] - [remote_util] [140689549858624] - INFO - 172.23.122.182 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-22 23:21:04,208] - [remote_util] [140689549858624] - INFO - Running systemd command on this server [2021-02-22 23:21:04,208] - [remote_util] [140689549858624] - INFO - running command.raw on 172.23.122.182: systemctl stop couchbase-server.service [2021-02-22 23:21:05,940] - [remote_util] [140689549858624] - INFO - command executed successfully with root [2021-02-22 23:21:05,941] - [basetestcase] [140689549858624] - INFO - Couchbase stopped [2021-02-22 23:21:05,942] - [basetestcase] [140689549858624] - INFO - sleep for 5 secs. ... [2021-02-22 23:21:10,948] - [remote_util] [140689549858624] - INFO - running command.raw on 172.23.122.182: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-22 23:21:10,968] - [remote_util] [140689549858624] - INFO - command executed successfully with root [2021-02-22 23:21:10,968] - [remote_util] [140689549858624] - INFO - running command.raw on 172.23.122.182: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-22 23:21:11,030] - [remote_util] [140689549858624] - INFO - command executed successfully with root [2021-02-22 23:21:11,033] - [remote_util] [140689549858624] - INFO - SSH Connecting to 172.23.122.182 with username:root, attempt#1 of 5 [2021-02-22 23:21:11,147] - [remote_util] [140689549858624] - INFO - SSH Connected to 172.23.122.182 as root [2021-02-22 23:21:11,431] - [remote_util] [140689549858624] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 23:21:11,751] - [remote_util] [140689549858624] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 23:21:11,816] - [remote_util] [140689549858624] - INFO - 172.23.122.182 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-22 23:21:11,817] - [remote_util] [140689549858624] - INFO - Checking for process beam.smp on linux [2021-02-22 23:21:11,900] - [remote_util] [140689549858624] - INFO - Starting couchbase server [2021-02-22 23:21:12,051] - [remote_util] [140689549858624] - INFO - Running systemd command on this server [2021-02-22 23:21:12,052] - [remote_util] [140689549858624] - INFO - running command.raw on 172.23.122.182: systemctl start couchbase-server.service [2021-02-22 23:21:12,077] - [remote_util] [140689549858624] - INFO - command executed successfully with root [2021-02-22 23:21:12,077] - [remote_util] [140689549858624] - INFO - 172.23.122.182:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-22 23:21:17,082] - [remote_util] [140689549858624] - INFO - running command.raw on 172.23.122.182: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-22 23:21:17,108] - [remote_util] [140689549858624] - INFO - command executed successfully with root [2021-02-22 23:21:17,109] - [remote_util] [140689549858624] - INFO - Couchbase server status: [' Process: 72555 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-22 23:21:17,109] - [remote_util] [140689549858624] - INFO - Checking for process beam.smp on linux [2021-02-22 23:21:17,179] - [remote_util] [140689549858624] - INFO - process beam.smp is running on 172.23.122.182: with pid 72699 [2021-02-22 23:21:17,180] - [basetestcase] [140689549858624] - INFO - Couchbase started [2021-02-22 23:21:17,183] - [remote_util] [140689549858624] - INFO - SSH Connecting to 172.23.122.179 with username:root, attempt#1 of 5 [2021-02-22 23:21:17,288] - [remote_util] [140689549858624] - INFO - SSH Connected to 172.23.122.179 as root [2021-02-22 23:21:17,582] - [remote_util] [140689549858624] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 23:21:17,935] - [remote_util] [140689549858624] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 23:21:17,951] - [remote_util] [140689549858624] - INFO - SSH Connecting to 172.23.122.179 with username:root, attempt#1 of 5 [2021-02-22 23:21:18,055] - [remote_util] [140689549858624] - INFO - SSH Connected to 172.23.122.179 as root [2021-02-22 23:21:18,339] - [remote_util] [140689549858624] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 23:21:18,653] - [remote_util] [140689549858624] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 23:21:18,716] - [remote_util] [140689549858624] - INFO - 172.23.122.179 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-22 23:21:18,820] - [remote_util] [140689549858624] - INFO - Running systemd command on this server [2021-02-22 23:21:18,820] - [remote_util] [140689549858624] - INFO - running command.raw on 172.23.122.179: systemctl stop couchbase-server.service [2021-02-22 23:21:20,538] - [remote_util] [140689549858624] - INFO - command executed successfully with root [2021-02-22 23:21:20,540] - [basetestcase] [140689549858624] - INFO - Couchbase stopped [2021-02-22 23:21:20,540] - [basetestcase] [140689549858624] - INFO - sleep for 5 secs. ... [2021-02-22 23:21:25,545] - [remote_util] [140689549858624] - INFO - running command.raw on 172.23.122.179: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-22 23:21:25,564] - [remote_util] [140689549858624] - INFO - command executed successfully with root [2021-02-22 23:21:25,565] - [remote_util] [140689549858624] - INFO - running command.raw on 172.23.122.179: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-22 23:21:25,620] - [remote_util] [140689549858624] - INFO - command executed successfully with root [2021-02-22 23:21:25,623] - [remote_util] [140689549858624] - INFO - SSH Connecting to 172.23.122.179 with username:root, attempt#1 of 5 [2021-02-22 23:21:25,728] - [remote_util] [140689549858624] - INFO - SSH Connected to 172.23.122.179 as root [2021-02-22 23:21:26,015] - [remote_util] [140689549858624] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 23:21:26,333] - [remote_util] [140689549858624] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 23:21:26,401] - [remote_util] [140689549858624] - INFO - 172.23.122.179 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-22 23:21:26,402] - [remote_util] [140689549858624] - INFO - Checking for process beam.smp on linux [2021-02-22 23:21:26,482] - [remote_util] [140689549858624] - INFO - Starting couchbase server [2021-02-22 23:21:26,599] - [remote_util] [140689549858624] - INFO - Running systemd command on this server [2021-02-22 23:21:26,599] - [remote_util] [140689549858624] - INFO - running command.raw on 172.23.122.179: systemctl start couchbase-server.service [2021-02-22 23:21:26,624] - [remote_util] [140689549858624] - INFO - command executed successfully with root [2021-02-22 23:21:26,625] - [remote_util] [140689549858624] - INFO - 172.23.122.179:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-22 23:21:31,630] - [remote_util] [140689549858624] - INFO - running command.raw on 172.23.122.179: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-22 23:21:31,655] - [remote_util] [140689549858624] - INFO - command executed successfully with root [2021-02-22 23:21:31,655] - [remote_util] [140689549858624] - INFO - Couchbase server status: [' Process: 101674 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-22 23:21:31,655] - [remote_util] [140689549858624] - INFO - Checking for process beam.smp on linux [2021-02-22 23:21:31,724] - [remote_util] [140689549858624] - INFO - process beam.smp is running on 172.23.122.179: with pid 101819 [2021-02-22 23:21:31,724] - [basetestcase] [140689549858624] - INFO - Couchbase started [2021-02-22 23:21:31,727] - [remote_util] [140689549858624] - INFO - SSH Connecting to 172.23.122.153 with username:root, attempt#1 of 5 [2021-02-22 23:21:31,833] - [remote_util] [140689549858624] - INFO - SSH Connected to 172.23.122.153 as root [2021-02-22 23:21:32,115] - [remote_util] [140689549858624] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 23:21:32,414] - [remote_util] [140689549858624] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 23:21:32,426] - [remote_util] [140689549858624] - INFO - SSH Connecting to 172.23.122.153 with username:root, attempt#1 of 5 [2021-02-22 23:21:32,530] - [remote_util] [140689549858624] - INFO - SSH Connected to 172.23.122.153 as root [2021-02-22 23:21:32,799] - [remote_util] [140689549858624] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 23:21:33,100] - [remote_util] [140689549858624] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 23:21:33,162] - [remote_util] [140689549858624] - INFO - 172.23.122.153 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-22 23:21:33,258] - [remote_util] [140689549858624] - INFO - Running systemd command on this server [2021-02-22 23:21:33,259] - [remote_util] [140689549858624] - INFO - running command.raw on 172.23.122.153: systemctl stop couchbase-server.service [2021-02-22 23:21:35,051] - [remote_util] [140689549858624] - INFO - command executed successfully with root [2021-02-22 23:21:35,052] - [basetestcase] [140689549858624] - INFO - Couchbase stopped [2021-02-22 23:21:35,052] - [basetestcase] [140689549858624] - INFO - sleep for 5 secs. ... [2021-02-22 23:21:40,057] - [remote_util] [140689549858624] - INFO - running command.raw on 172.23.122.153: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-22 23:21:40,074] - [remote_util] [140689549858624] - INFO - command executed successfully with root [2021-02-22 23:21:40,074] - [remote_util] [140689549858624] - INFO - running command.raw on 172.23.122.153: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-22 23:21:40,129] - [remote_util] [140689549858624] - INFO - command executed successfully with root [2021-02-22 23:21:40,132] - [remote_util] [140689549858624] - INFO - SSH Connecting to 172.23.122.153 with username:root, attempt#1 of 5 [2021-02-22 23:21:40,233] - [remote_util] [140689549858624] - INFO - SSH Connected to 172.23.122.153 as root [2021-02-22 23:21:40,514] - [remote_util] [140689549858624] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 23:21:40,815] - [remote_util] [140689549858624] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 23:21:40,876] - [remote_util] [140689549858624] - INFO - 172.23.122.153 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-22 23:21:40,876] - [remote_util] [140689549858624] - INFO - Checking for process beam.smp on linux [2021-02-22 23:21:40,951] - [remote_util] [140689549858624] - INFO - Starting couchbase server [2021-02-22 23:21:41,053] - [remote_util] [140689549858624] - INFO - Running systemd command on this server [2021-02-22 23:21:41,053] - [remote_util] [140689549858624] - INFO - running command.raw on 172.23.122.153: systemctl start couchbase-server.service [2021-02-22 23:21:41,078] - [remote_util] [140689549858624] - INFO - command executed successfully with root [2021-02-22 23:21:41,078] - [remote_util] [140689549858624] - INFO - 172.23.122.153:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-22 23:21:46,084] - [remote_util] [140689549858624] - INFO - running command.raw on 172.23.122.153: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-22 23:21:46,107] - [remote_util] [140689549858624] - INFO - command executed successfully with root [2021-02-22 23:21:46,107] - [remote_util] [140689549858624] - INFO - Couchbase server status: [' Process: 608 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-22 23:21:46,108] - [remote_util] [140689549858624] - INFO - Checking for process beam.smp on linux [2021-02-22 23:21:46,176] - [remote_util] [140689549858624] - INFO - process beam.smp is running on 172.23.122.153: with pid 755 [2021-02-22 23:21:46,177] - [basetestcase] [140689549858624] - INFO - Couchbase started [2021-02-22 23:21:46,180] - [remote_util] [140689549858624] - INFO - SSH Connecting to 172.23.122.139 with username:root, attempt#1 of 5 [2021-02-22 23:21:46,281] - [remote_util] [140689549858624] - INFO - SSH Connected to 172.23.122.139 as root [2021-02-22 23:21:46,558] - [remote_util] [140689549858624] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 23:21:46,857] - [remote_util] [140689549858624] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 23:21:46,870] - [remote_util] [140689549858624] - INFO - SSH Connecting to 172.23.122.139 with username:root, attempt#1 of 5 [2021-02-22 23:21:46,967] - [remote_util] [140689549858624] - INFO - SSH Connected to 172.23.122.139 as root [2021-02-22 23:21:47,232] - [remote_util] [140689549858624] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 23:21:47,533] - [remote_util] [140689549858624] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 23:21:47,595] - [remote_util] [140689549858624] - INFO - 172.23.122.139 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-22 23:21:47,697] - [remote_util] [140689549858624] - INFO - Running systemd command on this server [2021-02-22 23:21:47,697] - [remote_util] [140689549858624] - INFO - running command.raw on 172.23.122.139: systemctl stop couchbase-server.service [2021-02-22 23:21:49,447] - [remote_util] [140689549858624] - INFO - command executed successfully with root [2021-02-22 23:21:49,448] - [basetestcase] [140689549858624] - INFO - Couchbase stopped [2021-02-22 23:21:49,448] - [basetestcase] [140689549858624] - INFO - sleep for 5 secs. ... [2021-02-22 23:21:54,454] - [remote_util] [140689549858624] - INFO - running command.raw on 172.23.122.139: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-22 23:21:54,471] - [remote_util] [140689549858624] - INFO - command executed successfully with root [2021-02-22 23:21:54,471] - [remote_util] [140689549858624] - INFO - running command.raw on 172.23.122.139: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-22 23:21:54,525] - [remote_util] [140689549858624] - INFO - command executed successfully with root [2021-02-22 23:21:54,528] - [remote_util] [140689549858624] - INFO - SSH Connecting to 172.23.122.139 with username:root, attempt#1 of 5 [2021-02-22 23:21:54,628] - [remote_util] [140689549858624] - INFO - SSH Connected to 172.23.122.139 as root [2021-02-22 23:21:54,895] - [remote_util] [140689549858624] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 23:21:55,191] - [remote_util] [140689549858624] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 23:21:55,251] - [remote_util] [140689549858624] - INFO - 172.23.122.139 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-22 23:21:55,251] - [remote_util] [140689549858624] - INFO - Checking for process beam.smp on linux [2021-02-22 23:21:55,330] - [remote_util] [140689549858624] - INFO - Starting couchbase server [2021-02-22 23:21:55,424] - [remote_util] [140689549858624] - INFO - Running systemd command on this server [2021-02-22 23:21:55,425] - [remote_util] [140689549858624] - INFO - running command.raw on 172.23.122.139: systemctl start couchbase-server.service [2021-02-22 23:21:55,447] - [remote_util] [140689549858624] - INFO - command executed successfully with root [2021-02-22 23:21:55,447] - [remote_util] [140689549858624] - INFO - 172.23.122.139:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-22 23:22:00,452] - [remote_util] [140689549858624] - INFO - running command.raw on 172.23.122.139: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-22 23:22:00,479] - [remote_util] [140689549858624] - INFO - command executed successfully with root [2021-02-22 23:22:00,480] - [remote_util] [140689549858624] - INFO - Couchbase server status: [' Process: 85560 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-22 23:22:00,480] - [remote_util] [140689549858624] - INFO - Checking for process beam.smp on linux [2021-02-22 23:22:00,548] - [remote_util] [140689549858624] - INFO - process beam.smp is running on 172.23.122.139: with pid 85704 [2021-02-22 23:22:00,549] - [basetestcase] [140689549858624] - INFO - Couchbase started [2021-02-22 23:22:00,551] - [remote_util] [140689549858624] - INFO - SSH Connecting to 172.23.122.140 with username:root, attempt#1 of 5 [2021-02-22 23:22:00,659] - [remote_util] [140689549858624] - INFO - SSH Connected to 172.23.122.140 as root [2021-02-22 23:22:00,940] - [remote_util] [140689549858624] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 23:22:01,237] - [remote_util] [140689549858624] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 23:22:01,251] - [remote_util] [140689549858624] - INFO - SSH Connecting to 172.23.122.140 with username:root, attempt#1 of 5 [2021-02-22 23:22:01,351] - [remote_util] [140689549858624] - INFO - SSH Connected to 172.23.122.140 as root [2021-02-22 23:22:01,643] - [remote_util] [140689549858624] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 23:22:01,951] - [remote_util] [140689549858624] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 23:22:02,013] - [remote_util] [140689549858624] - INFO - 172.23.122.140 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-22 23:22:02,114] - [remote_util] [140689549858624] - INFO - Running systemd command on this server [2021-02-22 23:22:02,114] - [remote_util] [140689549858624] - INFO - running command.raw on 172.23.122.140: systemctl stop couchbase-server.service [2021-02-22 23:22:03,813] - [remote_util] [140689549858624] - INFO - command executed successfully with root [2021-02-22 23:22:03,813] - [basetestcase] [140689549858624] - INFO - Couchbase stopped [2021-02-22 23:22:03,814] - [basetestcase] [140689549858624] - INFO - sleep for 5 secs. ... [2021-02-22 23:22:08,821] - [remote_util] [140689549858624] - INFO - running command.raw on 172.23.122.140: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-22 23:22:08,838] - [remote_util] [140689549858624] - INFO - command executed successfully with root [2021-02-22 23:22:08,839] - [remote_util] [140689549858624] - INFO - running command.raw on 172.23.122.140: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-22 23:22:08,894] - [remote_util] [140689549858624] - INFO - command executed successfully with root [2021-02-22 23:22:08,899] - [remote_util] [140689549858624] - INFO - SSH Connecting to 172.23.122.140 with username:root, attempt#1 of 5 [2021-02-22 23:22:09,007] - [remote_util] [140689549858624] - INFO - SSH Connected to 172.23.122.140 as root [2021-02-22 23:22:09,317] - [remote_util] [140689549858624] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 23:22:09,638] - [remote_util] [140689549858624] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 23:22:09,714] - [remote_util] [140689549858624] - INFO - 172.23.122.140 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-22 23:22:09,715] - [remote_util] [140689549858624] - INFO - Checking for process beam.smp on linux [2021-02-22 23:22:09,799] - [remote_util] [140689549858624] - INFO - Starting couchbase server [2021-02-22 23:22:09,931] - [remote_util] [140689549858624] - INFO - Running systemd command on this server [2021-02-22 23:22:09,931] - [remote_util] [140689549858624] - INFO - running command.raw on 172.23.122.140: systemctl start couchbase-server.service [2021-02-22 23:22:09,959] - [remote_util] [140689549858624] - INFO - command executed successfully with root [2021-02-22 23:22:09,959] - [remote_util] [140689549858624] - INFO - 172.23.122.140:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-22 23:22:14,963] - [remote_util] [140689549858624] - INFO - running command.raw on 172.23.122.140: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-22 23:22:15,009] - [remote_util] [140689549858624] - INFO - command executed successfully with root [2021-02-22 23:22:15,009] - [remote_util] [140689549858624] - INFO - Couchbase server status: [' Process: 18886 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-22 23:22:15,010] - [remote_util] [140689549858624] - INFO - Checking for process beam.smp on linux [2021-02-22 23:22:15,086] - [remote_util] [140689549858624] - INFO - process beam.smp is running on 172.23.122.140: with pid 19031 [2021-02-22 23:22:15,087] - [basetestcase] [140689549858624] - INFO - Couchbase started [2021-02-22 23:22:15,090] - [remote_util] [140689549858624] - INFO - SSH Connecting to 172.23.122.138 with username:root, attempt#1 of 5 [2021-02-22 23:22:15,203] - [remote_util] [140689549858624] - INFO - SSH Connected to 172.23.122.138 as root [2021-02-22 23:22:15,535] - [remote_util] [140689549858624] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 23:22:15,934] - [remote_util] [140689549858624] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 23:22:15,949] - [remote_util] [140689549858624] - INFO - SSH Connecting to 172.23.122.138 with username:root, attempt#1 of 5 [2021-02-22 23:22:16,061] - [remote_util] [140689549858624] - INFO - SSH Connected to 172.23.122.138 as root [2021-02-22 23:22:16,381] - [remote_util] [140689549858624] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 23:22:16,707] - [remote_util] [140689549858624] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 23:22:16,776] - [remote_util] [140689549858624] - INFO - 172.23.122.138 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-22 23:22:16,898] - [remote_util] [140689549858624] - INFO - Running systemd command on this server [2021-02-22 23:22:16,898] - [remote_util] [140689549858624] - INFO - running command.raw on 172.23.122.138: systemctl stop couchbase-server.service [2021-02-22 23:22:18,750] - [remote_util] [140689549858624] - INFO - command executed successfully with root [2021-02-22 23:22:18,751] - [basetestcase] [140689549858624] - INFO - Couchbase stopped [2021-02-22 23:22:18,752] - [basetestcase] [140689549858624] - INFO - sleep for 5 secs. ... [2021-02-22 23:22:23,757] - [remote_util] [140689549858624] - INFO - running command.raw on 172.23.122.138: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-22 23:22:23,774] - [remote_util] [140689549858624] - INFO - command executed successfully with root [2021-02-22 23:22:23,774] - [remote_util] [140689549858624] - INFO - running command.raw on 172.23.122.138: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-22 23:22:23,828] - [remote_util] [140689549858624] - INFO - command executed successfully with root [2021-02-22 23:22:23,831] - [remote_util] [140689549858624] - INFO - SSH Connecting to 172.23.122.138 with username:root, attempt#1 of 5 [2021-02-22 23:22:23,935] - [remote_util] [140689549858624] - INFO - SSH Connected to 172.23.122.138 as root [2021-02-22 23:22:24,222] - [remote_util] [140689549858624] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 23:22:24,523] - [remote_util] [140689549858624] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 23:22:24,586] - [remote_util] [140689549858624] - INFO - 172.23.122.138 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-22 23:22:24,586] - [remote_util] [140689549858624] - INFO - Checking for process beam.smp on linux [2021-02-22 23:22:24,665] - [remote_util] [140689549858624] - INFO - Starting couchbase server [2021-02-22 23:22:24,766] - [remote_util] [140689549858624] - INFO - Running systemd command on this server [2021-02-22 23:22:24,767] - [remote_util] [140689549858624] - INFO - running command.raw on 172.23.122.138: systemctl start couchbase-server.service [2021-02-22 23:22:24,790] - [remote_util] [140689549858624] - INFO - command executed successfully with root [2021-02-22 23:22:24,791] - [remote_util] [140689549858624] - INFO - 172.23.122.138:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-22 23:22:29,796] - [remote_util] [140689549858624] - INFO - running command.raw on 172.23.122.138: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-22 23:22:29,823] - [remote_util] [140689549858624] - INFO - command executed successfully with root [2021-02-22 23:22:29,823] - [remote_util] [140689549858624] - INFO - Couchbase server status: [' Process: 104426 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-22 23:22:29,823] - [remote_util] [140689549858624] - INFO - Checking for process beam.smp on linux [2021-02-22 23:22:29,894] - [remote_util] [140689549858624] - INFO - process beam.smp is running on 172.23.122.138: with pid 104571 [2021-02-22 23:22:29,894] - [basetestcase] [140689549858624] - INFO - Couchbase started [2021-02-22 23:22:29,898] - [remote_util] [140689549858624] - INFO - SSH Connecting to 172.23.122.141 with username:root, attempt#1 of 5 [2021-02-22 23:22:30,002] - [remote_util] [140689549858624] - INFO - SSH Connected to 172.23.122.141 as root [2021-02-22 23:22:30,272] - [remote_util] [140689549858624] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 23:22:30,576] - [remote_util] [140689549858624] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 23:22:30,590] - [remote_util] [140689549858624] - INFO - SSH Connecting to 172.23.122.141 with username:root, attempt#1 of 5 [2021-02-22 23:22:30,690] - [remote_util] [140689549858624] - INFO - SSH Connected to 172.23.122.141 as root [2021-02-22 23:22:30,961] - [remote_util] [140689549858624] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 23:22:31,270] - [remote_util] [140689549858624] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 23:22:31,333] - [remote_util] [140689549858624] - INFO - 172.23.122.141 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-22 23:22:31,437] - [remote_util] [140689549858624] - INFO - Running systemd command on this server [2021-02-22 23:22:31,437] - [remote_util] [140689549858624] - INFO - running command.raw on 172.23.122.141: systemctl stop couchbase-server.service [2021-02-22 23:22:33,064] - [remote_util] [140689549858624] - INFO - command executed successfully with root [2021-02-22 23:22:33,065] - [basetestcase] [140689549858624] - INFO - Couchbase stopped [2021-02-22 23:22:33,065] - [basetestcase] [140689549858624] - INFO - sleep for 5 secs. ... [2021-02-22 23:22:38,071] - [remote_util] [140689549858624] - INFO - running command.raw on 172.23.122.141: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-22 23:22:38,089] - [remote_util] [140689549858624] - INFO - command executed successfully with root [2021-02-22 23:22:38,090] - [remote_util] [140689549858624] - INFO - running command.raw on 172.23.122.141: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-22 23:22:38,143] - [remote_util] [140689549858624] - INFO - command executed successfully with root [2021-02-22 23:22:38,147] - [remote_util] [140689549858624] - INFO - SSH Connecting to 172.23.122.141 with username:root, attempt#1 of 5 [2021-02-22 23:22:38,248] - [remote_util] [140689549858624] - INFO - SSH Connected to 172.23.122.141 as root [2021-02-22 23:22:38,536] - [remote_util] [140689549858624] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-22 23:22:38,890] - [remote_util] [140689549858624] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-22 23:22:38,953] - [remote_util] [140689549858624] - INFO - 172.23.122.141 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-22 23:22:38,953] - [remote_util] [140689549858624] - INFO - Checking for process beam.smp on linux [2021-02-22 23:22:39,029] - [remote_util] [140689549858624] - INFO - Starting couchbase server [2021-02-22 23:22:39,133] - [remote_util] [140689549858624] - INFO - Running systemd command on this server [2021-02-22 23:22:39,133] - [remote_util] [140689549858624] - INFO - running command.raw on 172.23.122.141: systemctl start couchbase-server.service [2021-02-22 23:22:39,156] - [remote_util] [140689549858624] - INFO - command executed successfully with root [2021-02-22 23:22:39,156] - [remote_util] [140689549858624] - INFO - 172.23.122.141:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-22 23:22:44,161] - [remote_util] [140689549858624] - INFO - running command.raw on 172.23.122.141: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-22 23:22:44,186] - [remote_util] [140689549858624] - INFO - command executed successfully with root [2021-02-22 23:22:44,187] - [remote_util] [140689549858624] - INFO - Couchbase server status: [' Process: 70987 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-22 23:22:44,187] - [remote_util] [140689549858624] - INFO - Checking for process beam.smp on linux [2021-02-22 23:22:44,256] - [remote_util] [140689549858624] - INFO - process beam.smp is running on 172.23.122.141: with pid 71131 [2021-02-22 23:22:44,257] - [basetestcase] [140689549858624] - INFO - Couchbase started [2021-02-22 23:22:44,257] - [basetestcase] [140689549858624] - INFO - sleep for 10 secs. ...