[2021-07-27 22:10:09,776] - [eventing_base] [140687446664960] - INFO - ============== EventingBaseTest setup has started ============== [2021-07-27 22:10:09,777] - [eventing_base] [140687446664960] - INFO - Starting Test: test_source_scope_recreate [2021-07-27 22:10:09,780] - [remote_util] [140687446664960] - INFO - SSH Connecting to 172.23.123.103 with username:root, attempt#1 of 5 [2021-07-27 22:10:09,939] - [remote_util] [140687446664960] - INFO - SSH Connected to 172.23.123.103 as root [2021-07-27 22:10:10,322] - [remote_util] [140687446664960] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-07-27 22:10:10,744] - [remote_util] [140687446664960] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-07-27 22:10:10,763] - [rest_client] [140687446664960] - INFO - -->is_ns_server_running? [2021-07-27 22:10:10,825] - [rest_client] [140687446664960] - INFO - Node version in cluster 7.1.0-1085-enterprise [2021-07-27 22:10:10,865] - [rest_client] [140687446664960] - ERROR - GET http://172.23.123.103:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-07-27 22:10:10,866] - [rest_client] [140687446664960] - INFO - Node versions in cluster [] [2021-07-27 22:10:10,866] - [basetestcase] [140687446664960] - INFO - ============== basetestcase setup was started for test #2 test_source_scope_recreate============== [2021-07-27 22:10:10,905] - [rest_client] [140687446664960] - ERROR - GET http://172.23.123.103:8091/pools/default/ body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-07-27 22:10:10,950] - [bucket_helper] [140687446664960] - INFO - Could not find any buckets for node 172.23.123.103, nothing to delete [2021-07-27 22:10:11,011] - [bucket_helper] [140687446664960] - INFO - Could not find any buckets for node 172.23.123.151, nothing to delete [2021-07-27 22:10:11,042] - [bucket_helper] [140687446664960] - INFO - Could not find any buckets for node 172.23.123.147, nothing to delete [2021-07-27 22:10:11,071] - [bucket_helper] [140687446664960] - INFO - Could not find any buckets for node 172.23.123.146, nothing to delete [2021-07-27 22:10:11,091] - [rest_client] [140687446664960] - INFO - -->is_ns_server_running? [2021-07-27 22:10:11,175] - [cluster_helper] [140687446664960] - INFO - waiting for ns_server @ 172.23.123.103:8091 [2021-07-27 22:10:11,176] - [rest_client] [140687446664960] - INFO - -->is_ns_server_running? [2021-07-27 22:10:11,192] - [cluster_helper] [140687446664960] - INFO - ns_server @ 172.23.123.103:8091 is running [2021-07-27 22:10:11,208] - [cluster_helper] [140687446664960] - INFO - waiting for ns_server @ 172.23.123.151:8091 [2021-07-27 22:10:11,208] - [rest_client] [140687446664960] - INFO - -->is_ns_server_running? [2021-07-27 22:10:11,231] - [cluster_helper] [140687446664960] - INFO - ns_server @ 172.23.123.151:8091 is running [2021-07-27 22:10:11,256] - [cluster_helper] [140687446664960] - INFO - waiting for ns_server @ 172.23.123.147:8091 [2021-07-27 22:10:11,256] - [rest_client] [140687446664960] - INFO - -->is_ns_server_running? [2021-07-27 22:10:11,275] - [cluster_helper] [140687446664960] - INFO - ns_server @ 172.23.123.147:8091 is running [2021-07-27 22:10:11,314] - [cluster_helper] [140687446664960] - INFO - waiting for ns_server @ 172.23.123.146:8091 [2021-07-27 22:10:11,314] - [rest_client] [140687446664960] - INFO - -->is_ns_server_running? [2021-07-27 22:10:11,336] - [cluster_helper] [140687446664960] - INFO - ns_server @ 172.23.123.146:8091 is running [2021-07-27 22:10:11,337] - [basetestcase] [140687446664960] - INFO - Building docker image with java sdk client [2021-07-27 22:10:35,751] - [basetestcase] [140687446664960] - INFO - initializing cluster [2021-07-27 22:10:35,756] - [remote_util] [140687446664960] - INFO - SSH Connecting to 172.23.123.103 with username:root, attempt#1 of 5 [2021-07-27 22:10:35,971] - [remote_util] [140687446664960] - INFO - SSH Connected to 172.23.123.103 as root [2021-07-27 22:10:36,448] - [remote_util] [140687446664960] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-07-27 22:10:36,932] - [remote_util] [140687446664960] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-07-27 22:10:36,977] - [remote_util] [140687446664960] - INFO - SSH Connecting to 172.23.123.103 with username:root, attempt#1 of 5 [2021-07-27 22:10:37,156] - [remote_util] [140687446664960] - INFO - SSH Connected to 172.23.123.103 as root [2021-07-27 22:10:37,529] - [remote_util] [140687446664960] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-07-27 22:10:37,995] - [remote_util] [140687446664960] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-07-27 22:10:38,089] - [remote_util] [140687446664960] - INFO - 172.23.123.103 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-07-27 22:10:38,327] - [remote_util] [140687446664960] - INFO - Running systemd command on this server [2021-07-27 22:10:38,329] - [remote_util] [140687446664960] - INFO - running command.raw on 172.23.123.103: systemctl stop couchbase-server.service [2021-07-27 22:10:40,452] - [remote_util] [140687446664960] - INFO - command executed successfully with root [2021-07-27 22:10:40,453] - [basetestcase] [140687446664960] - INFO - Couchbase stopped [2021-07-27 22:10:40,453] - [basetestcase] [140687446664960] - INFO - sleep for 5 secs. ... [2021-07-27 22:10:45,463] - [remote_util] [140687446664960] - INFO - running command.raw on 172.23.123.103: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-07-27 22:10:45,499] - [remote_util] [140687446664960] - INFO - command executed successfully with root [2021-07-27 22:10:45,500] - [remote_util] [140687446664960] - INFO - running command.raw on 172.23.123.103: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-07-27 22:10:45,558] - [remote_util] [140687446664960] - INFO - command executed successfully with root [2021-07-27 22:10:45,560] - [remote_util] [140687446664960] - INFO - SSH Connecting to 172.23.123.103 with username:root, attempt#1 of 5 [2021-07-27 22:10:45,690] - [remote_util] [140687446664960] - INFO - SSH Connected to 172.23.123.103 as root [2021-07-27 22:10:46,084] - [remote_util] [140687446664960] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-07-27 22:10:46,560] - [remote_util] [140687446664960] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-07-27 22:10:46,676] - [remote_util] [140687446664960] - INFO - 172.23.123.103 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-07-27 22:10:46,677] - [remote_util] [140687446664960] - INFO - Checking for process beam.smp on linux [2021-07-27 22:10:46,741] - [remote_util] [140687446664960] - INFO - Starting couchbase server [2021-07-27 22:10:46,967] - [remote_util] [140687446664960] - INFO - Running systemd command on this server [2021-07-27 22:10:46,967] - [remote_util] [140687446664960] - INFO - running command.raw on 172.23.123.103: systemctl start couchbase-server.service [2021-07-27 22:10:47,006] - [remote_util] [140687446664960] - INFO - command executed successfully with root [2021-07-27 22:10:47,007] - [remote_util] [140687446664960] - INFO - 172.23.123.103:sleep for 5 secs. waiting for couchbase server to come up ... [2021-07-27 22:10:52,013] - [remote_util] [140687446664960] - INFO - running command.raw on 172.23.123.103: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-07-27 22:10:52,081] - [remote_util] [140687446664960] - INFO - command executed successfully with root [2021-07-27 22:10:52,081] - [remote_util] [140687446664960] - INFO - Couchbase server status: [' Process: 100651 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-07-27 22:10:52,082] - [remote_util] [140687446664960] - INFO - Checking for process beam.smp on linux [2021-07-27 22:10:52,185] - [remote_util] [140687446664960] - INFO - process beam.smp is running on 172.23.123.103: with pid 100794 [2021-07-27 22:10:52,186] - [basetestcase] [140687446664960] - INFO - Couchbase started [2021-07-27 22:10:52,189] - [remote_util] [140687446664960] - INFO - SSH Connecting to 172.23.123.151 with username:root, attempt#1 of 5 [2021-07-27 22:10:52,348] - [remote_util] [140687446664960] - INFO - SSH Connected to 172.23.123.151 as root [2021-07-27 22:10:52,687] - [remote_util] [140687446664960] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-07-27 22:10:53,017] - [remote_util] [140687446664960] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-07-27 22:10:53,050] - [remote_util] [140687446664960] - INFO - SSH Connecting to 172.23.123.151 with username:root, attempt#1 of 5 [2021-07-27 22:10:53,219] - [remote_util] [140687446664960] - INFO - SSH Connected to 172.23.123.151 as root [2021-07-27 22:10:53,628] - [remote_util] [140687446664960] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-07-27 22:10:54,031] - [remote_util] [140687446664960] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-07-27 22:10:54,105] - [remote_util] [140687446664960] - INFO - 172.23.123.151 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-07-27 22:10:54,390] - [remote_util] [140687446664960] - INFO - Running systemd command on this server [2021-07-27 22:10:54,391] - [remote_util] [140687446664960] - INFO - running command.raw on 172.23.123.151: systemctl stop couchbase-server.service [2021-07-27 22:10:56,467] - [remote_util] [140687446664960] - INFO - command executed successfully with root [2021-07-27 22:10:56,468] - [basetestcase] [140687446664960] - INFO - Couchbase stopped [2021-07-27 22:10:56,469] - [basetestcase] [140687446664960] - INFO - sleep for 5 secs. ... [2021-07-27 22:11:01,475] - [remote_util] [140687446664960] - INFO - running command.raw on 172.23.123.151: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-07-27 22:11:01,502] - [remote_util] [140687446664960] - INFO - command executed successfully with root [2021-07-27 22:11:01,503] - [remote_util] [140687446664960] - INFO - running command.raw on 172.23.123.151: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-07-27 22:11:01,562] - [remote_util] [140687446664960] - INFO - command executed successfully with root [2021-07-27 22:11:01,565] - [remote_util] [140687446664960] - INFO - SSH Connecting to 172.23.123.151 with username:root, attempt#1 of 5 [2021-07-27 22:11:01,762] - [remote_util] [140687446664960] - INFO - SSH Connected to 172.23.123.151 as root [2021-07-27 22:11:02,151] - [remote_util] [140687446664960] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-07-27 22:11:02,538] - [remote_util] [140687446664960] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-07-27 22:11:02,629] - [remote_util] [140687446664960] - INFO - 172.23.123.151 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-07-27 22:11:02,629] - [remote_util] [140687446664960] - INFO - Checking for process beam.smp on linux [2021-07-27 22:11:02,709] - [remote_util] [140687446664960] - INFO - Starting couchbase server [2021-07-27 22:11:02,887] - [remote_util] [140687446664960] - INFO - Running systemd command on this server [2021-07-27 22:11:02,887] - [remote_util] [140687446664960] - INFO - running command.raw on 172.23.123.151: systemctl start couchbase-server.service [2021-07-27 22:11:02,915] - [remote_util] [140687446664960] - INFO - command executed successfully with root [2021-07-27 22:11:02,915] - [remote_util] [140687446664960] - INFO - 172.23.123.151:sleep for 5 secs. waiting for couchbase server to come up ... [2021-07-27 22:11:07,918] - [remote_util] [140687446664960] - INFO - running command.raw on 172.23.123.151: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-07-27 22:11:07,951] - [remote_util] [140687446664960] - INFO - command executed successfully with root [2021-07-27 22:11:07,951] - [remote_util] [140687446664960] - INFO - Couchbase server status: [' Process: 4016 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-07-27 22:11:07,951] - [remote_util] [140687446664960] - INFO - Checking for process beam.smp on linux [2021-07-27 22:11:08,024] - [remote_util] [140687446664960] - INFO - process beam.smp is running on 172.23.123.151: with pid 4159 [2021-07-27 22:11:08,024] - [basetestcase] [140687446664960] - INFO - Couchbase started [2021-07-27 22:11:08,026] - [remote_util] [140687446664960] - INFO - SSH Connecting to 172.23.123.147 with username:root, attempt#1 of 5 [2021-07-27 22:11:08,137] - [remote_util] [140687446664960] - INFO - SSH Connected to 172.23.123.147 as root [2021-07-27 22:11:08,546] - [remote_util] [140687446664960] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-07-27 22:11:08,981] - [remote_util] [140687446664960] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-07-27 22:11:09,034] - [remote_util] [140687446664960] - INFO - SSH Connecting to 172.23.123.147 with username:root, attempt#1 of 5 [2021-07-27 22:11:09,163] - [remote_util] [140687446664960] - INFO - SSH Connected to 172.23.123.147 as root [2021-07-27 22:11:09,588] - [remote_util] [140687446664960] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-07-27 22:11:10,044] - [remote_util] [140687446664960] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-07-27 22:11:10,147] - [remote_util] [140687446664960] - INFO - 172.23.123.147 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-07-27 22:11:10,462] - [remote_util] [140687446664960] - INFO - Running systemd command on this server [2021-07-27 22:11:10,463] - [remote_util] [140687446664960] - INFO - running command.raw on 172.23.123.147: systemctl stop couchbase-server.service [2021-07-27 22:11:12,697] - [remote_util] [140687446664960] - INFO - command executed successfully with root [2021-07-27 22:11:12,698] - [basetestcase] [140687446664960] - INFO - Couchbase stopped [2021-07-27 22:11:12,699] - [basetestcase] [140687446664960] - INFO - sleep for 5 secs. ... [2021-07-27 22:11:17,702] - [remote_util] [140687446664960] - INFO - running command.raw on 172.23.123.147: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-07-27 22:11:17,731] - [remote_util] [140687446664960] - INFO - command executed successfully with root [2021-07-27 22:11:17,732] - [remote_util] [140687446664960] - INFO - running command.raw on 172.23.123.147: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-07-27 22:11:17,802] - [remote_util] [140687446664960] - INFO - command executed successfully with root [2021-07-27 22:11:17,805] - [remote_util] [140687446664960] - INFO - SSH Connecting to 172.23.123.147 with username:root, attempt#1 of 5 [2021-07-27 22:11:17,980] - [remote_util] [140687446664960] - INFO - SSH Connected to 172.23.123.147 as root [2021-07-27 22:11:18,355] - [remote_util] [140687446664960] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-07-27 22:11:18,726] - [remote_util] [140687446664960] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-07-27 22:11:18,789] - [remote_util] [140687446664960] - INFO - 172.23.123.147 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-07-27 22:11:18,789] - [remote_util] [140687446664960] - INFO - Checking for process beam.smp on linux [2021-07-27 22:11:18,877] - [remote_util] [140687446664960] - INFO - Starting couchbase server [2021-07-27 22:11:19,072] - [remote_util] [140687446664960] - INFO - Running systemd command on this server [2021-07-27 22:11:19,073] - [remote_util] [140687446664960] - INFO - running command.raw on 172.23.123.147: systemctl start couchbase-server.service [2021-07-27 22:11:19,115] - [remote_util] [140687446664960] - INFO - command executed successfully with root [2021-07-27 22:11:19,116] - [remote_util] [140687446664960] - INFO - 172.23.123.147:sleep for 5 secs. waiting for couchbase server to come up ... [2021-07-27 22:11:24,121] - [remote_util] [140687446664960] - INFO - running command.raw on 172.23.123.147: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-07-27 22:11:24,177] - [remote_util] [140687446664960] - INFO - command executed successfully with root [2021-07-27 22:11:24,179] - [remote_util] [140687446664960] - INFO - Couchbase server status: [' Process: 113699 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-07-27 22:11:24,180] - [remote_util] [140687446664960] - INFO - Checking for process beam.smp on linux [2021-07-27 22:11:24,265] - [remote_util] [140687446664960] - INFO - process beam.smp is running on 172.23.123.147: with pid 113841 [2021-07-27 22:11:24,265] - [basetestcase] [140687446664960] - INFO - Couchbase started [2021-07-27 22:11:24,268] - [remote_util] [140687446664960] - INFO - SSH Connecting to 172.23.123.146 with username:root, attempt#1 of 5 [2021-07-27 22:11:24,452] - [remote_util] [140687446664960] - INFO - SSH Connected to 172.23.123.146 as root [2021-07-27 22:11:24,855] - [remote_util] [140687446664960] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-07-27 22:11:25,274] - [remote_util] [140687446664960] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-07-27 22:11:25,312] - [remote_util] [140687446664960] - INFO - SSH Connecting to 172.23.123.146 with username:root, attempt#1 of 5 [2021-07-27 22:11:25,486] - [remote_util] [140687446664960] - INFO - SSH Connected to 172.23.123.146 as root [2021-07-27 22:11:25,795] - [remote_util] [140687446664960] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-07-27 22:11:26,172] - [remote_util] [140687446664960] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-07-27 22:11:26,244] - [remote_util] [140687446664960] - INFO - 172.23.123.146 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-07-27 22:11:26,462] - [remote_util] [140687446664960] - INFO - Running systemd command on this server [2021-07-27 22:11:26,462] - [remote_util] [140687446664960] - INFO - running command.raw on 172.23.123.146: systemctl stop couchbase-server.service [2021-07-27 22:11:28,487] - [remote_util] [140687446664960] - INFO - command executed successfully with root [2021-07-27 22:11:28,487] - [basetestcase] [140687446664960] - INFO - Couchbase stopped [2021-07-27 22:11:28,488] - [basetestcase] [140687446664960] - INFO - sleep for 5 secs. ... [2021-07-27 22:11:33,493] - [remote_util] [140687446664960] - INFO - running command.raw on 172.23.123.146: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-07-27 22:11:33,525] - [remote_util] [140687446664960] - INFO - command executed successfully with root [2021-07-27 22:11:33,525] - [remote_util] [140687446664960] - INFO - running command.raw on 172.23.123.146: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-07-27 22:11:33,600] - [remote_util] [140687446664960] - INFO - command executed successfully with root [2021-07-27 22:11:33,606] - [remote_util] [140687446664960] - INFO - SSH Connecting to 172.23.123.146 with username:root, attempt#1 of 5 [2021-07-27 22:11:33,795] - [remote_util] [140687446664960] - INFO - SSH Connected to 172.23.123.146 as root [2021-07-27 22:11:34,187] - [remote_util] [140687446664960] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-07-27 22:11:34,619] - [remote_util] [140687446664960] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-07-27 22:11:34,735] - [remote_util] [140687446664960] - INFO - 172.23.123.146 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-07-27 22:11:34,735] - [remote_util] [140687446664960] - INFO - Checking for process beam.smp on linux [2021-07-27 22:11:34,801] - [remote_util] [140687446664960] - INFO - Starting couchbase server [2021-07-27 22:11:35,029] - [remote_util] [140687446664960] - INFO - Running systemd command on this server [2021-07-27 22:11:35,030] - [remote_util] [140687446664960] - INFO - running command.raw on 172.23.123.146: systemctl start couchbase-server.service [2021-07-27 22:11:35,069] - [remote_util] [140687446664960] - INFO - command executed successfully with root [2021-07-27 22:11:35,070] - [remote_util] [140687446664960] - INFO - 172.23.123.146:sleep for 5 secs. waiting for couchbase server to come up ... [2021-07-27 22:11:40,072] - [remote_util] [140687446664960] - INFO - running command.raw on 172.23.123.146: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-07-27 22:11:40,109] - [remote_util] [140687446664960] - INFO - command executed successfully with root [2021-07-27 22:11:40,109] - [remote_util] [140687446664960] - INFO - Couchbase server status: [' Process: 11387 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-07-27 22:11:40,109] - [remote_util] [140687446664960] - INFO - Checking for process beam.smp on linux [2021-07-27 22:11:40,195] - [remote_util] [140687446664960] - INFO - process beam.smp is running on 172.23.123.146: with pid 11531 [2021-07-27 22:11:40,196] - [basetestcase] [140687446664960] - INFO - Couchbase started [2021-07-27 22:11:40,196] - [basetestcase] [140687446664960] - INFO - sleep for 10 secs. ... [2021-07-27 22:11:51,087] - [task] [140687072810752] - INFO - server: ip:172.23.123.103 port:8091 ssh_username:root, nodes/self [2021-07-27 22:11:51,100] - [task] [140687072810752] - INFO - {'uptime': '60', 'memoryTotal': 4201672704, 'memoryFree': 3614126080, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.123.103:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-1085-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.123.103', 'ip': '172.23.123.103', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 4343} [2021-07-27 22:11:51,111] - [rest_client] [140687072810752] - ERROR - GET http://172.23.123.103:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-07-27 22:11:51,112] - [rest_client] [140687072810752] - INFO - pools/default params : memoryQuota=2147 [2021-07-27 22:11:51,127] - [rest_client] [140687072810752] - INFO - --> init_node_services(Administrator,password,172.23.123.103,8091,['kv']) [2021-07-27 22:11:51,128] - [rest_client] [140687072810752] - INFO - /node/controller/setupServices params on 172.23.123.103: 8091:hostname=172.23.123.103&user=Administrator&password=password&services=kv [2021-07-27 22:11:51,158] - [rest_client] [140687072810752] - INFO - --> in init_cluster...Administrator,password,8091 [2021-07-27 22:11:51,159] - [rest_client] [140687072810752] - INFO - settings/web params on 172.23.123.103:8091:port=8091&username=Administrator&password=password [2021-07-27 22:11:51,230] - [rest_client] [140687072810752] - INFO - --> status:True [2021-07-27 22:11:51,232] - [remote_util] [140687072810752] - INFO - SSH Connecting to 172.23.123.103 with username:root, attempt#1 of 5 [2021-07-27 22:11:51,340] - [remote_util] [140687072810752] - INFO - SSH Connected to 172.23.123.103 as root [2021-07-27 22:11:51,622] - [remote_util] [140687072810752] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-07-27 22:11:51,977] - [remote_util] [140687072810752] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-07-27 22:11:51,987] - [remote_util] [140687072810752] - INFO - running command.raw on 172.23.123.103: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-07-27 22:11:52,043] - [remote_util] [140687072810752] - INFO - command executed successfully with root [2021-07-27 22:11:52,043] - [remote_util] [140687072810752] - INFO - ['ok'] [2021-07-27 22:11:52,048] - [rest_client] [140687072810752] - INFO - /diag/eval status on 172.23.123.103:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2021-07-27 22:11:52,052] - [rest_client] [140687072810752] - INFO - /diag/eval status on 172.23.123.103:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2021-07-27 22:11:52,059] - [rest_client] [140687072810752] - INFO - settings/indexes params : storageMode=plasma [2021-07-27 22:11:52,097] - [task] [140687072810752] - INFO - server: ip:172.23.123.151 port:8091 ssh_username:root, nodes/self [2021-07-27 22:11:52,102] - [task] [140687072810752] - INFO - {'uptime': '46', 'memoryTotal': 4201672704, 'memoryFree': 3625521152, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.123.151:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-1085-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.123.151', 'ip': '172.23.123.151', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} [2021-07-27 22:11:52,105] - [rest_client] [140687072810752] - ERROR - GET http://172.23.123.151:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-07-27 22:11:52,106] - [rest_client] [140687072810752] - INFO - pools/default params : memoryQuota=2147 [2021-07-27 22:11:52,114] - [rest_client] [140687072810752] - INFO - --> in init_cluster...Administrator,password,8091 [2021-07-27 22:11:52,115] - [rest_client] [140687072810752] - INFO - settings/web params on 172.23.123.151:8091:port=8091&username=Administrator&password=password [2021-07-27 22:11:52,176] - [rest_client] [140687072810752] - INFO - --> status:True [2021-07-27 22:11:52,178] - [remote_util] [140687072810752] - INFO - SSH Connecting to 172.23.123.151 with username:root, attempt#1 of 5 [2021-07-27 22:11:52,288] - [remote_util] [140687072810752] - INFO - SSH Connected to 172.23.123.151 as root [2021-07-27 22:11:52,572] - [remote_util] [140687072810752] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-07-27 22:11:52,935] - [remote_util] [140687072810752] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-07-27 22:11:52,964] - [remote_util] [140687072810752] - INFO - running command.raw on 172.23.123.151: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-07-27 22:11:53,002] - [remote_util] [140687072810752] - INFO - command executed successfully with root [2021-07-27 22:11:53,003] - [remote_util] [140687072810752] - INFO - ['ok'] [2021-07-27 22:11:53,014] - [rest_client] [140687072810752] - INFO - /diag/eval status on 172.23.123.151:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2021-07-27 22:11:53,025] - [rest_client] [140687072810752] - INFO - /diag/eval status on 172.23.123.151:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2021-07-27 22:11:53,040] - [rest_client] [140687072810752] - INFO - settings/indexes params : storageMode=plasma [2021-07-27 22:11:53,080] - [task] [140687072810752] - INFO - server: ip:172.23.123.147 port:8091 ssh_username:root, nodes/self [2021-07-27 22:11:53,085] - [task] [140687072810752] - INFO - {'uptime': '31', 'memoryTotal': 4201672704, 'memoryFree': 3562663936, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.123.147:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-1085-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.123.147', 'ip': '172.23.123.147', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} [2021-07-27 22:11:53,088] - [rest_client] [140687072810752] - ERROR - GET http://172.23.123.147:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-07-27 22:11:53,089] - [rest_client] [140687072810752] - INFO - pools/default params : memoryQuota=2147 [2021-07-27 22:11:53,096] - [rest_client] [140687072810752] - INFO - --> in init_cluster...Administrator,password,8091 [2021-07-27 22:11:53,097] - [rest_client] [140687072810752] - INFO - settings/web params on 172.23.123.147:8091:port=8091&username=Administrator&password=password [2021-07-27 22:11:53,158] - [rest_client] [140687072810752] - INFO - --> status:True [2021-07-27 22:11:53,162] - [remote_util] [140687072810752] - INFO - SSH Connecting to 172.23.123.147 with username:root, attempt#1 of 5 [2021-07-27 22:11:53,275] - [remote_util] [140687072810752] - INFO - SSH Connected to 172.23.123.147 as root [2021-07-27 22:11:53,605] - [remote_util] [140687072810752] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-07-27 22:11:53,935] - [remote_util] [140687072810752] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-07-27 22:11:53,949] - [remote_util] [140687072810752] - INFO - running command.raw on 172.23.123.147: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-07-27 22:11:54,004] - [remote_util] [140687072810752] - INFO - command executed successfully with root [2021-07-27 22:11:54,005] - [remote_util] [140687072810752] - INFO - ['ok'] [2021-07-27 22:11:54,010] - [rest_client] [140687072810752] - INFO - /diag/eval status on 172.23.123.147:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2021-07-27 22:11:54,015] - [rest_client] [140687072810752] - INFO - /diag/eval status on 172.23.123.147:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2021-07-27 22:11:54,021] - [rest_client] [140687072810752] - INFO - settings/indexes params : storageMode=plasma [2021-07-27 22:11:54,063] - [task] [140687072810752] - INFO - server: ip:172.23.123.146 port:8091 ssh_username:root, nodes/self [2021-07-27 22:11:54,068] - [task] [140687072810752] - INFO - {'uptime': '15', 'memoryTotal': 4201672704, 'memoryFree': 3579387904, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.123.146:8091', 'clusterCompatibility': 458753, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.1.0-1085-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.123.146', 'ip': '172.23.123.146', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} [2021-07-27 22:11:54,071] - [rest_client] [140687072810752] - ERROR - GET http://172.23.123.146:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-07-27 22:11:54,072] - [rest_client] [140687072810752] - INFO - pools/default params : memoryQuota=2147 [2021-07-27 22:11:54,080] - [rest_client] [140687072810752] - INFO - --> in init_cluster...Administrator,password,8091 [2021-07-27 22:11:54,080] - [rest_client] [140687072810752] - INFO - settings/web params on 172.23.123.146:8091:port=8091&username=Administrator&password=password [2021-07-27 22:11:54,147] - [rest_client] [140687072810752] - INFO - --> status:True [2021-07-27 22:11:54,149] - [remote_util] [140687072810752] - INFO - SSH Connecting to 172.23.123.146 with username:root, attempt#1 of 5 [2021-07-27 22:11:54,318] - [remote_util] [140687072810752] - INFO - SSH Connected to 172.23.123.146 as root [2021-07-27 22:11:54,698] - [remote_util] [140687072810752] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-07-27 22:11:55,110] - [remote_util] [140687072810752] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-07-27 22:11:55,122] - [remote_util] [140687072810752] - INFO - running command.raw on 172.23.123.146: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-07-27 22:11:55,193] - [remote_util] [140687072810752] - INFO - command executed successfully with root [2021-07-27 22:11:55,193] - [remote_util] [140687072810752] - INFO - ['ok'] [2021-07-27 22:11:55,199] - [rest_client] [140687072810752] - INFO - /diag/eval status on 172.23.123.146:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2021-07-27 22:11:55,203] - [rest_client] [140687072810752] - INFO - /diag/eval status on 172.23.123.146:8091: True content: [7,1] command: cluster_compat_mode:get_compat_version(). [2021-07-27 22:11:55,210] - [rest_client] [140687072810752] - INFO - settings/indexes params : storageMode=plasma [2021-07-27 22:11:55,237] - [basetestcase] [140687446664960] - INFO - **** add built-in 'cbadminbucket' user to node 172.23.123.103 **** [2021-07-27 22:11:55,251] - [rest_client] [140687446664960] - ERROR - DELETE http://172.23.123.103:8091/settings/rbac/users/local/cbadminbucket body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:password [2021-07-27 22:11:55,252] - [internal_user] [140687446664960] - INFO - Exception while deleting user. Exception is -b'"User was not found."' [2021-07-27 22:11:55,322] - [basetestcase] [140687446664960] - INFO - sleep for 5 secs. ... [2021-07-27 22:12:00,327] - [basetestcase] [140687446664960] - INFO - **** add 'admin' role to 'cbadminbucket' user **** [2021-07-27 22:12:00,364] - [basetestcase] [140687446664960] - INFO - done initializing cluster [2021-07-27 22:12:00,367] - [remote_util] [140687446664960] - INFO - SSH Connecting to 172.23.123.103 with username:root, attempt#1 of 5 [2021-07-27 22:12:00,472] - [remote_util] [140687446664960] - INFO - SSH Connected to 172.23.123.103 as root [2021-07-27 22:12:00,740] - [remote_util] [140687446664960] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-07-27 22:12:01,052] - [remote_util] [140687446664960] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-07-27 22:12:01,166] - [remote_util] [140687446664960] - INFO - command to run: echo y | /opt/couchbase/bin/couchbase-cli enable-developer-preview -c http://172.23.123.103:8091 -u Administrator -p password --enable [2021-07-27 22:12:01,385] - [remote_util] [140687446664960] - INFO - SSH Connecting to 172.23.123.151 with username:root, attempt#1 of 5 [2021-07-27 22:12:01,490] - [remote_util] [140687446664960] - INFO - SSH Connected to 172.23.123.151 as root [2021-07-27 22:12:01,775] - [remote_util] [140687446664960] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-07-27 22:12:02,142] - [remote_util] [140687446664960] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-07-27 22:12:02,339] - [remote_util] [140687446664960] - INFO - command to run: echo y | /opt/couchbase/bin/couchbase-cli enable-developer-preview -c http://172.23.123.151:8091 -u Administrator -p password --enable [2021-07-27 22:12:02,576] - [remote_util] [140687446664960] - INFO - SSH Connecting to 172.23.123.147 with username:root, attempt#1 of 5 [2021-07-27 22:12:02,687] - [remote_util] [140687446664960] - INFO - SSH Connected to 172.23.123.147 as root [2021-07-27 22:12:02,994] - [remote_util] [140687446664960] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-07-27 22:12:03,316] - [remote_util] [140687446664960] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-07-27 22:12:03,468] - [remote_util] [140687446664960] - INFO - command to run: echo y | /opt/couchbase/bin/couchbase-cli enable-developer-preview -c http://172.23.123.147:8091 -u Administrator -p password --enable [2021-07-27 22:12:03,698] - [remote_util] [140687446664960] - INFO - SSH Connecting to 172.23.123.146 with username:root, attempt#1 of 5 [2021-07-27 22:12:03,806] - [remote_util] [140687446664960] - INFO - SSH Connected to 172.23.123.146 as root [2021-07-27 22:12:04,099] - [remote_util] [140687446664960] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-07-27 22:12:04,455] - [remote_util] [140687446664960] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-07-27 22:12:04,628] - [remote_util] [140687446664960] - INFO - command to run: echo y | /opt/couchbase/bin/couchbase-cli enable-developer-preview -c http://172.23.123.146:8091 -u Administrator -p password --enable [2021-07-27 22:12:05,236] - [task] [140687072810752] - INFO - adding node 172.23.123.151:8091 to cluster [2021-07-27 22:12:05,236] - [rest_client] [140687072810752] - INFO - adding remote node @172.23.123.151:8091 to this cluster @172.23.123.103:8091 [2021-07-27 22:12:15,281] - [rest_client] [140687072810752] - INFO - rebalance progress took 10.04 seconds [2021-07-27 22:12:15,281] - [rest_client] [140687072810752] - INFO - sleep for 10 seconds after rebalance... [2021-07-27 22:12:28,141] - [task] [140687072810752] - INFO - adding node 172.23.123.147:8091 to cluster [2021-07-27 22:12:28,141] - [rest_client] [140687072810752] - INFO - adding remote node @172.23.123.147:8091 to this cluster @172.23.123.103:8091 [2021-07-27 22:12:38,168] - [rest_client] [140687072810752] - INFO - rebalance progress took 10.03 seconds [2021-07-27 22:12:38,169] - [rest_client] [140687072810752] - INFO - sleep for 10 seconds after rebalance... [2021-07-27 22:12:51,437] - [task] [140687072810752] - INFO - adding node 172.23.123.146:8091 to cluster [2021-07-27 22:12:51,438] - [rest_client] [140687072810752] - INFO - adding remote node @172.23.123.146:8091 to this cluster @172.23.123.103:8091 [2021-07-27 22:13:01,464] - [rest_client] [140687072810752] - INFO - rebalance progress took 10.03 seconds [2021-07-27 22:13:01,464] - [rest_client] [140687072810752] - INFO - sleep for 10 seconds after rebalance... [2021-07-27 22:13:14,639] - [rest_client] [140687072810752] - INFO - rebalance params : {'knownNodes': 'ns_1@172.23.123.103,ns_1@172.23.123.146,ns_1@172.23.123.147,ns_1@172.23.123.151', 'ejectedNodes': '', 'user': 'Administrator', 'password': 'password'} [2021-07-27 22:13:14,690] - [rest_client] [140687072810752] - INFO - rebalance operation started [2021-07-27 22:13:24,706] - [task] [140687072810752] - INFO - Rebalance - status: none, progress: 100.00% [2021-07-27 22:13:24,715] - [task] [140687072810752] - INFO - rebalancing was completed with progress: 100% in 10.024826049804688 sec [2021-07-27 22:13:24,719] - [remote_util] [140687446664960] - INFO - SSH Connecting to 172.23.123.103 with username:root, attempt#1 of 5 [2021-07-27 22:13:24,888] - [remote_util] [140687446664960] - INFO - SSH Connected to 172.23.123.103 as root [2021-07-27 22:13:25,255] - [remote_util] [140687446664960] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-07-27 22:13:25,670] - [remote_util] [140687446664960] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-07-27 22:13:25,685] - [remote_util] [140687446664960] - INFO - running command.raw on 172.23.123.103: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-07-27 22:13:25,747] - [remote_util] [140687446664960] - INFO - command executed successfully with root [2021-07-27 22:13:25,747] - [remote_util] [140687446664960] - INFO - ['ok'] [2021-07-27 22:13:25,748] - [basetestcase] [140687446664960] - INFO - Enabled diag/eval for non-local hosts from 172.23.123.103 [2021-07-27 22:13:25,748] - [basetestcase] [140687446664960] - INFO - ============== basetestcase setup was finished for test #2 test_source_scope_recreate ============== [2021-07-27 22:13:25,770] - [remote_util] [140687446664960] - INFO - SSH Connecting to 172.23.123.103 with username:root, attempt#1 of 5 [2021-07-27 22:13:25,880] - [remote_util] [140687446664960] - INFO - SSH Connected to 172.23.123.103 as root [2021-07-27 22:13:26,162] - [remote_util] [140687446664960] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-07-27 22:13:26,483] - [remote_util] [140687446664960] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-07-27 22:13:26,488] - [remote_util] [140687446664960] - INFO - SSH Connecting to 172.23.123.151 with username:root, attempt#1 of 5 [2021-07-27 22:13:26,659] - [remote_util] [140687446664960] - INFO - SSH Connected to 172.23.123.151 as root [2021-07-27 22:13:27,016] - [remote_util] [140687446664960] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-07-27 22:13:27,425] - [remote_util] [140687446664960] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-07-27 22:13:27,429] - [remote_util] [140687446664960] - INFO - SSH Connecting to 172.23.123.147 with username:root, attempt#1 of 5 [2021-07-27 22:13:27,542] - [remote_util] [140687446664960] - INFO - SSH Connected to 172.23.123.147 as root [2021-07-27 22:13:27,855] - [remote_util] [140687446664960] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-07-27 22:13:28,171] - [remote_util] [140687446664960] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-07-27 22:13:28,177] - [remote_util] [140687446664960] - INFO - SSH Connecting to 172.23.123.146 with username:root, attempt#1 of 5 [2021-07-27 22:13:28,289] - [remote_util] [140687446664960] - INFO - SSH Connected to 172.23.123.146 as root [2021-07-27 22:13:28,582] - [remote_util] [140687446664960] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-07-27 22:13:28,943] - [remote_util] [140687446664960] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-07-27 22:13:32,070] - [basetestcase] [140687446664960] - INFO - ------- Cluster statistics ------- [2021-07-27 22:13:32,070] - [basetestcase] [140687446664960] - INFO - 172.23.123.103:8091 => {'services': ['kv'], 'cpu_utilization': 2.427427427427427, 'mem_free': 3582103552, 'mem_total': 4201672704, 'swap_mem_used': 83886080, 'swap_mem_total': 3758092288} [2021-07-27 22:13:32,070] - [basetestcase] [140687446664960] - INFO - 172.23.123.146:8091 => {'services': ['n1ql'], 'cpu_utilization': 0.2260738507912585, 'mem_free': 3579908096, 'mem_total': 4201672704, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-07-27 22:13:32,071] - [basetestcase] [140687446664960] - INFO - 172.23.123.147:8091 => {'services': ['index'], 'cpu_utilization': 3.573225968797181, 'mem_free': 3529818112, 'mem_total': 4201672704, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-07-27 22:13:32,071] - [basetestcase] [140687446664960] - INFO - 172.23.123.151:8091 => {'services': ['eventing'], 'cpu_utilization': 2.164066431806744, 'mem_free': 3601408000, 'mem_total': 4201672704, 'swap_mem_used': 63438848, 'swap_mem_total': 3758092288} [2021-07-27 22:13:32,071] - [basetestcase] [140687446664960] - INFO - --- End of cluster statistics --- [2021-07-27 22:13:32,074] - [remote_util] [140687446664960] - INFO - SSH Connecting to 172.23.123.103 with username:root, attempt#1 of 5 [2021-07-27 22:13:32,194] - [remote_util] [140687446664960] - INFO - SSH Connected to 172.23.123.103 as root [2021-07-27 22:13:32,510] - [remote_util] [140687446664960] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-07-27 22:13:32,861] - [remote_util] [140687446664960] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-07-27 22:13:55,795] - [basetestcase] [140687446664960] - INFO - list of n1ql nodes in cluster: [ip:172.23.123.146 port:8091 ssh_username:root] [2021-07-27 22:13:55,804] - [basetestcase] [140687446664960] - INFO - list of eventing nodes in cluster: [ip:172.23.123.151 port:8091 ssh_username:root] [2021-07-27 22:13:55,810] - [rest_client] [140687446664960] - INFO - settings/indexes params : storageMode=plasma [2021-07-27 22:13:55,816] - [eventing_base] [140687446664960] - INFO - Setting the min possible memory quota so that adding mode nodes to the cluster wouldn't be a problem. [2021-07-27 22:13:55,816] - [rest_client] [140687446664960] - INFO - pools/default params : memoryQuota=330 [2021-07-27 22:13:55,824] - [rest_client] [140687446664960] - INFO - pools/default params : indexMemoryQuota=256 [2021-07-27 22:13:55,830] - [rest_client] [140687446664960] - INFO - pools/default params : eventingMemoryQuota=512 [2021-07-27 22:13:55,848] - [basetestcase] [140687446664960] - INFO - list of n1ql nodes in cluster: [ip:172.23.123.146 port:8091 ssh_username:root] [2021-07-27 22:13:55,853] - [eventing_base] [140687446664960] - INFO - ============== EventingBaseTest setup has completed ============== [2021-07-27 22:13:55,854] - [rest_client] [140687446664960] - INFO - pools/default params : memoryQuota=1400 [2021-07-27 22:13:55,860] - [eventing_collections] [140687446664960] - INFO - 250 [2021-07-27 22:13:56,408] - [rest_client] [140687072810752] - INFO - http://172.23.123.103:8091/pools/default/buckets with param: name=src_bucket&ramQuotaMB=250&replicaNumber=0&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=magma [2021-07-27 22:13:56,438] - [rest_client] [140687072810752] - INFO - 0.03 seconds to create bucket src_bucket [2021-07-27 22:13:56,438] - [bucket_helper] [140687072810752] - INFO - waiting for memcached bucket : src_bucket in 172.23.123.103 to accept set ops [2021-07-27 22:13:57,060] - [data_helper] [140687072810752] - INFO - creating direct client 172.23.123.103:11210 src_bucket [2021-07-27 22:13:57,168] - [data_helper] [140687072810752] - INFO - creating direct client 172.23.123.103:11210 src_bucket [2021-07-27 22:13:57,221] - [task] [140687072810752] - WARNING - vbucket map not ready after try 0 [2021-07-27 22:13:57,222] - [bucket_helper] [140687072810752] - INFO - waiting for memcached bucket : src_bucket in 172.23.123.103 to accept set ops [2021-07-27 22:13:57,316] - [data_helper] [140687072810752] - INFO - creating direct client 172.23.123.103:11210 src_bucket [2021-07-27 22:13:57,412] - [data_helper] [140687072810752] - INFO - creating direct client 172.23.123.103:11210 src_bucket [2021-07-27 22:13:57,957] - [task] [140687072810752] - WARNING - vbucket map not ready after try 1 [2021-07-27 22:13:57,957] - [bucket_helper] [140687072810752] - INFO - waiting for memcached bucket : src_bucket in 172.23.123.103 to accept set ops [2021-07-27 22:13:58,049] - [data_helper] [140687072810752] - INFO - creating direct client 172.23.123.103:11210 src_bucket [2021-07-27 22:13:58,150] - [data_helper] [140687072810752] - INFO - creating direct client 172.23.123.103:11210 src_bucket [2021-07-27 22:13:58,207] - [task] [140687072810752] - WARNING - vbucket map not ready after try 2 [2021-07-27 22:13:58,207] - [bucket_helper] [140687072810752] - INFO - waiting for memcached bucket : src_bucket in 172.23.123.103 to accept set ops [2021-07-27 22:13:58,294] - [data_helper] [140687072810752] - INFO - creating direct client 172.23.123.103:11210 src_bucket [2021-07-27 22:13:58,391] - [data_helper] [140687072810752] - INFO - creating direct client 172.23.123.103:11210 src_bucket [2021-07-27 22:13:58,450] - [task] [140687072810752] - WARNING - vbucket map not ready after try 3 [2021-07-27 22:13:58,450] - [bucket_helper] [140687072810752] - INFO - waiting for memcached bucket : src_bucket in 172.23.123.103 to accept set ops [2021-07-27 22:13:58,546] - [data_helper] [140687072810752] - INFO - creating direct client 172.23.123.103:11210 src_bucket [2021-07-27 22:13:59,181] - [data_helper] [140687072810752] - INFO - creating direct client 172.23.123.103:11210 src_bucket [2021-07-27 22:13:59,237] - [task] [140687072810752] - WARNING - vbucket map not ready after try 4 [2021-07-27 22:13:59,238] - [bucket_helper] [140687072810752] - INFO - waiting for memcached bucket : src_bucket in 172.23.123.103 to accept set ops [2021-07-27 22:13:59,332] - [data_helper] [140687072810752] - INFO - creating direct client 172.23.123.103:11210 src_bucket [2021-07-27 22:13:59,428] - [data_helper] [140687072810752] - INFO - creating direct client 172.23.123.103:11210 src_bucket [2021-07-27 22:13:59,483] - [task] [140687072810752] - WARNING - vbucket map not ready after try 5 [2021-07-27 22:14:00,557] - [rest_client] [140687072810752] - INFO - http://172.23.123.103:8091/pools/default/buckets with param: name=dst_bucket&ramQuotaMB=250&replicaNumber=0&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=magma [2021-07-27 22:14:00,600] - [rest_client] [140687072810752] - INFO - 0.04 seconds to create bucket dst_bucket [2021-07-27 22:14:00,600] - [bucket_helper] [140687072810752] - INFO - waiting for memcached bucket : dst_bucket in 172.23.123.103 to accept set ops [2021-07-27 22:14:01,252] - [data_helper] [140687072810752] - INFO - creating direct client 172.23.123.103:11210 dst_bucket [2021-07-27 22:14:01,390] - [data_helper] [140687072810752] - INFO - creating direct client 172.23.123.103:11210 dst_bucket [2021-07-27 22:14:01,459] - [task] [140687072810752] - WARNING - vbucket map not ready after try 0 [2021-07-27 22:14:01,460] - [bucket_helper] [140687072810752] - INFO - waiting for memcached bucket : dst_bucket in 172.23.123.103 to accept set ops [2021-07-27 22:14:02,087] - [data_helper] [140687072810752] - INFO - creating direct client 172.23.123.103:11210 dst_bucket [2021-07-27 22:14:02,226] - [data_helper] [140687072810752] - INFO - creating direct client 172.23.123.103:11210 dst_bucket [2021-07-27 22:14:02,334] - [task] [140687072810752] - WARNING - vbucket map not ready after try 1 [2021-07-27 22:14:02,334] - [bucket_helper] [140687072810752] - INFO - waiting for memcached bucket : dst_bucket in 172.23.123.103 to accept set ops [2021-07-27 22:14:02,441] - [data_helper] [140687072810752] - INFO - creating direct client 172.23.123.103:11210 dst_bucket [2021-07-27 22:14:02,610] - [data_helper] [140687072810752] - INFO - creating direct client 172.23.123.103:11210 dst_bucket [2021-07-27 22:14:02,700] - [task] [140687072810752] - WARNING - vbucket map not ready after try 2 [2021-07-27 22:14:02,701] - [bucket_helper] [140687072810752] - INFO - waiting for memcached bucket : dst_bucket in 172.23.123.103 to accept set ops [2021-07-27 22:14:02,792] - [data_helper] [140687072810752] - INFO - creating direct client 172.23.123.103:11210 dst_bucket [2021-07-27 22:14:03,445] - [data_helper] [140687072810752] - INFO - creating direct client 172.23.123.103:11210 dst_bucket [2021-07-27 22:14:03,543] - [task] [140687072810752] - WARNING - vbucket map not ready after try 3 [2021-07-27 22:14:03,544] - [bucket_helper] [140687072810752] - INFO - waiting for memcached bucket : dst_bucket in 172.23.123.103 to accept set ops [2021-07-27 22:14:03,650] - [data_helper] [140687072810752] - INFO - creating direct client 172.23.123.103:11210 dst_bucket [2021-07-27 22:14:03,813] - [data_helper] [140687072810752] - INFO - creating direct client 172.23.123.103:11210 dst_bucket [2021-07-27 22:14:03,871] - [task] [140687072810752] - WARNING - vbucket map not ready after try 4 [2021-07-27 22:14:03,871] - [bucket_helper] [140687072810752] - INFO - waiting for memcached bucket : dst_bucket in 172.23.123.103 to accept set ops [2021-07-27 22:14:03,988] - [data_helper] [140687072810752] - INFO - creating direct client 172.23.123.103:11210 dst_bucket [2021-07-27 22:14:04,193] - [data_helper] [140687072810752] - INFO - creating direct client 172.23.123.103:11210 dst_bucket [2021-07-27 22:14:04,261] - [task] [140687072810752] - WARNING - vbucket map not ready after try 5 [2021-07-27 22:14:05,288] - [rest_client] [140687072810752] - INFO - http://172.23.123.103:8091/pools/default/buckets with param: name=metadata&ramQuotaMB=250&replicaNumber=0&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=magma [2021-07-27 22:14:05,323] - [rest_client] [140687072810752] - INFO - 0.03 seconds to create bucket metadata [2021-07-27 22:14:05,324] - [bucket_helper] [140687072810752] - INFO - waiting for memcached bucket : metadata in 172.23.123.103 to accept set ops [2021-07-27 22:14:06,503] - [data_helper] [140687072810752] - INFO - creating direct client 172.23.123.103:11210 metadata [2021-07-27 22:14:06,629] - [data_helper] [140687072810752] - INFO - creating direct client 172.23.123.103:11210 metadata [2021-07-27 22:14:06,748] - [task] [140687072810752] - WARNING - vbucket map not ready after try 0 [2021-07-27 22:14:06,748] - [bucket_helper] [140687072810752] - INFO - waiting for memcached bucket : metadata in 172.23.123.103 to accept set ops [2021-07-27 22:14:06,903] - [data_helper] [140687072810752] - INFO - creating direct client 172.23.123.103:11210 metadata [2021-07-27 22:14:07,023] - [data_helper] [140687072810752] - INFO - creating direct client 172.23.123.103:11210 metadata [2021-07-27 22:14:07,113] - [task] [140687072810752] - WARNING - vbucket map not ready after try 1 [2021-07-27 22:14:07,114] - [bucket_helper] [140687072810752] - INFO - waiting for memcached bucket : metadata in 172.23.123.103 to accept set ops [2021-07-27 22:14:07,208] - [data_helper] [140687072810752] - INFO - creating direct client 172.23.123.103:11210 metadata [2021-07-27 22:14:07,365] - [data_helper] [140687072810752] - INFO - creating direct client 172.23.123.103:11210 metadata [2021-07-27 22:14:07,438] - [task] [140687072810752] - WARNING - vbucket map not ready after try 2 [2021-07-27 22:14:07,438] - [bucket_helper] [140687072810752] - INFO - waiting for memcached bucket : metadata in 172.23.123.103 to accept set ops [2021-07-27 22:14:08,113] - [data_helper] [140687072810752] - INFO - creating direct client 172.23.123.103:11210 metadata [2021-07-27 22:14:08,304] - [data_helper] [140687072810752] - INFO - creating direct client 172.23.123.103:11210 metadata [2021-07-27 22:14:08,376] - [task] [140687072810752] - WARNING - vbucket map not ready after try 3 [2021-07-27 22:14:08,377] - [bucket_helper] [140687072810752] - INFO - waiting for memcached bucket : metadata in 172.23.123.103 to accept set ops [2021-07-27 22:14:08,485] - [data_helper] [140687072810752] - INFO - creating direct client 172.23.123.103:11210 metadata [2021-07-27 22:14:08,655] - [data_helper] [140687072810752] - INFO - creating direct client 172.23.123.103:11210 metadata [2021-07-27 22:14:08,749] - [task] [140687072810752] - WARNING - vbucket map not ready after try 4 [2021-07-27 22:14:08,749] - [bucket_helper] [140687072810752] - INFO - waiting for memcached bucket : metadata in 172.23.123.103 to accept set ops [2021-07-27 22:14:08,877] - [data_helper] [140687072810752] - INFO - creating direct client 172.23.123.103:11210 metadata [2021-07-27 22:14:09,495] - [data_helper] [140687072810752] - INFO - creating direct client 172.23.123.103:11210 metadata [2021-07-27 22:14:09,550] - [task] [140687072810752] - WARNING - vbucket map not ready after try 5 [2021-07-27 22:14:09,608] - [tuq_helper] [140687446664960] - INFO - RUN QUERY create primary index on src_bucket [2021-07-27 22:14:09,621] - [rest_client] [140687446664960] - INFO - query params : statement=create+primary+index+on+src_bucket [2021-07-27 22:14:11,769] - [tuq_helper] [140687446664960] - INFO - TOTAL ELAPSED TIME: 2.136884968s [2021-07-27 22:14:11,784] - [tuq_helper] [140687446664960] - INFO - RUN QUERY create primary index on dst_bucket [2021-07-27 22:14:11,798] - [rest_client] [140687446664960] - INFO - query params : statement=create+primary+index+on+dst_bucket [2021-07-27 22:14:14,142] - [tuq_helper] [140687446664960] - INFO - TOTAL ELAPSED TIME: 2.333563729s [2021-07-27 22:14:14,148] - [tuq_helper] [140687446664960] - INFO - RUN QUERY create primary index on metadata [2021-07-27 22:14:14,153] - [rest_client] [140687446664960] - INFO - query params : statement=create+primary+index+on+metadata [2021-07-27 22:14:16,467] - [tuq_helper] [140687446664960] - INFO - TOTAL ELAPSED TIME: 2.310362412s [2021-07-27 22:14:16,530] - [rest_client] [140687446664960] - INFO - http://172.23.123.103:8091/pools/default/buckets/src_bucket/scopes with params: name=src_bucket [2021-07-27 22:14:16,531] - [rest_client] [140687446664960] - INFO - Scope created src_bucket->src_bucket {'uid': '1'} [2021-07-27 22:14:16,594] - [rest_client] [140687446664960] - INFO - http://172.23.123.103:8091/pools/default/buckets/src_bucket/scopes/src_bucket/collections with params: name=src_bucket [2021-07-27 22:14:16,595] - [rest_client] [140687446664960] - INFO - Collection created src_bucket->src_bucket->src_bucket manifest:{'uid': '2'} [2021-07-27 22:14:16,677] - [rest_client] [140687446664960] - INFO - http://172.23.123.103:8091/pools/default/buckets/metadata/scopes with params: name=metadata [2021-07-27 22:14:16,678] - [rest_client] [140687446664960] - INFO - Scope created metadata->metadata {'uid': '1'} [2021-07-27 22:14:16,754] - [rest_client] [140687446664960] - INFO - http://172.23.123.103:8091/pools/default/buckets/metadata/scopes/metadata/collections with params: name=metadata [2021-07-27 22:14:16,755] - [rest_client] [140687446664960] - INFO - Collection created metadata->metadata->metadata manifest:{'uid': '2'} [2021-07-27 22:14:16,831] - [rest_client] [140687446664960] - INFO - http://172.23.123.103:8091/pools/default/buckets/dst_bucket/scopes with params: name=dst_bucket [2021-07-27 22:14:16,831] - [rest_client] [140687446664960] - INFO - Scope created dst_bucket->dst_bucket {'uid': '1'} [2021-07-27 22:14:16,914] - [rest_client] [140687446664960] - INFO - http://172.23.123.103:8091/pools/default/buckets/dst_bucket/scopes/dst_bucket/collections with params: name=dst_bucket [2021-07-27 22:14:16,915] - [rest_client] [140687446664960] - INFO - Collection created dst_bucket->dst_bucket->dst_bucket manifest:{'uid': '2'} [2021-07-27 22:14:17,559] - [task] [140687072810752] - INFO - java -jar java_sdk_client/collections/target/javaclient/javaclient.jar -i 172.23.123.103 -u Administrator -p password -b src_bucket -s src_bucket -c src_bucket -n 2016 -pc 100 -pu 0 -pd 0 -l uniform -dsn 1 -dpx doc_ -dt Person -de 0 -ds 500 -ac False -st 0 -en 2015 -o False -sd False [2021-07-27 22:14:27,341] - [eventing_base] [140687446664960] - INFO - saving function b'{\n "code": 0,\n "info": {\n "status": "Stored function: \'Function_539913202_test_source_scope_recreate\' in metakv",\n "warnings": null\n }\n}' [2021-07-27 22:14:28,519] - [eventing_base] [140687446664960] - INFO - Deploying the following handler code : Function_539913202_test_source_scope_recreate with bindings: {'buckets': [{'alias': 'dst_bucket', 'bucket_name': 'dst_bucket', 'scope_name': 'dst_bucket', 'collection_name': 'dst_bucket', 'access': 'rw'}, {'alias': 'src_bucket', 'bucket_name': 'src_bucket', 'scope_name': 'src_bucket', 'collection_name': 'src_bucket'}], 'metadata_bucket': 'metadata', 'metadata_scope': 'metadata', 'metadata_collection': 'metadata', 'source_bucket': 'src_bucket', 'source_scope': 'src_bucket', 'source_collection': 'src_bucket', 'curl': []} and settings: {'checkpoint_interval': 20000, 'cleanup_timers': False, 'dcp_stream_boundary': 'everything', 'deployment_status': False, 'description': 'Sample Description', 'log_level': 'INFO', 'skip_timer_threshold': 86400, 'sock_batch_size': 1, 'tick_duration': 5000, 'timer_processing_tick_interval': 500, 'timer_worker_pool_size': 3, 'worker_count': 3, 'processing_status': False, 'cpp_worker_thread_count': 1, 'execution_timeout': 20, 'data_chan_size': 10000, 'worker_queue_cap': 100000, 'use_memory_manager': True, 'deadline_timeout': 21, 'timer_storage_chan_size': 10000, 'dcp_gen_chan_size': 10000, 'language_compatibility': '6.6.2'} [2021-07-27 22:14:28,519] - [eventing_base] [140687446664960] - INFO - function OnUpdate(doc, meta) { log("Doc created/updated", meta.id); try{ var result1= couchbase.insert(dst_bucket,meta,doc); log(result1); }catch(e){ log("error:",e); } } function OnDelete(meta, options) { log("Doc deleted/expired", meta.id); var doc={"id":meta.id} var result = couchbase.delete(dst_bucket,doc); log(result); } [2021-07-27 22:14:29,714] - [eventing_base] [140687446664960] - INFO - deploy Application : b'' [2021-07-27 22:14:29,715] - [basetestcase] [140687446664960] - INFO - sleep for 5 secs. Waiting for Function_539913202_test_source_scope_recreate to deployed... ... [2021-07-27 22:14:34,743] - [basetestcase] [140687446664960] - INFO - sleep for 5 secs. Waiting for Function_539913202_test_source_scope_recreate to deployed... ... [2021-07-27 22:14:39,764] - [basetestcase] [140687446664960] - INFO - sleep for 5 secs. Waiting for Function_539913202_test_source_scope_recreate to deployed... ... [2021-07-27 22:14:44,779] - [basetestcase] [140687446664960] - INFO - sleep for 5 secs. Waiting for Function_539913202_test_source_scope_recreate to deployed... ... [2021-07-27 22:14:49,839] - [basetestcase] [140687446664960] - INFO - list of eventing nodes in cluster: [ip:172.23.123.151 port:8091 ssh_username:root] [2021-07-27 22:14:49,863] - [basetestcase] [140687446664960] - INFO - list of n1ql nodes in cluster: [ip:172.23.123.146 port:8091 ssh_username:root] [2021-07-27 22:14:49,869] - [tuq_helper] [140687446664960] - INFO - RUN QUERY create primary index on dst_bucket.dst_bucket.dst_bucket [2021-07-27 22:14:49,875] - [rest_client] [140687446664960] - INFO - query params : statement=create+primary+index+on+dst_bucket.dst_bucket.dst_bucket [2021-07-27 22:14:53,433] - [tuq_helper] [140687446664960] - INFO - TOTAL ELAPSED TIME: 3.545821274s [2021-07-27 22:14:53,463] - [basetestcase] [140687446664960] - INFO - list of n1ql nodes in cluster: [ip:172.23.123.146 port:8091 ssh_username:root] [2021-07-27 22:14:53,477] - [rest_client] [140687446664960] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.dst_bucket.dst_bucket [2021-07-27 22:14:53,499] - [eventing_base] [140687446664960] - INFO - Final docs count... Current : 2016 Expected : 2016 [2021-07-27 22:14:56,224] - [task] [140687072810752] - INFO - java -jar java_sdk_client/collections/target/javaclient/javaclient.jar -i 172.23.123.103 -u Administrator -p password -b src_bucket -s src_bucket -c src_bucket -n 2016 -pc 0 -pu 0 -pd 100 -l uniform -dsn 1 -dpx doc_ -dt Person -de 0 -ds 500 -ac False -st 0 -en 2015 -o False -sd False [2021-07-27 22:15:01,540] - [basetestcase] [140687446664960] - INFO - list of eventing nodes in cluster: [ip:172.23.123.151 port:8091 ssh_username:root] [2021-07-27 22:15:01,569] - [basetestcase] [140687446664960] - INFO - list of n1ql nodes in cluster: [ip:172.23.123.146 port:8091 ssh_username:root] [2021-07-27 22:15:01,579] - [tuq_helper] [140687446664960] - INFO - RUN QUERY create primary index on dst_bucket.dst_bucket.dst_bucket [2021-07-27 22:15:01,589] - [rest_client] [140687446664960] - INFO - query params : statement=create+primary+index+on+dst_bucket.dst_bucket.dst_bucket [2021-07-27 22:15:01,601] - [rest_client] [140687446664960] - ERROR - POST http://172.23.123.146:8093/query?statement=create+primary+index+on+dst_bucket.dst_bucket.dst_bucket body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 500 reason: unknown b'{\n"requestID": "68a82019-9964-4241-a9be-1b71f2f8675a",\n"signature": null,\n"results": [\n],\n"errors": [{"code":5000,"msg":"GSI CreatePrimaryIndex() - cause: Index #primary already exists."}],\n"status": "errors",\n"metrics": {"elapsedTime": "1.019366ms","executionTime": "912.825\xc2\xb5s","resultCount": 0,"resultSize": 0,"serviceLoad": 6,"errorCount": 1}\n}\n' auth: Administrator:password [2021-07-27 22:15:01,638] - [basetestcase] [140687446664960] - INFO - list of n1ql nodes in cluster: [ip:172.23.123.146 port:8091 ssh_username:root] [2021-07-27 22:15:01,649] - [rest_client] [140687446664960] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.dst_bucket.dst_bucket [2021-07-27 22:15:01,668] - [basetestcase] [140687446664960] - INFO - sleep for 30 secs. Waiting for handler code dst_bucket.dst_bucket.dst_bucket to complete bucket operations... Current : 399 Expected : 0 ... [2021-07-27 22:15:31,719] - [basetestcase] [140687446664960] - INFO - list of n1ql nodes in cluster: [ip:172.23.123.146 port:8091 ssh_username:root] [2021-07-27 22:15:31,731] - [rest_client] [140687446664960] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.dst_bucket.dst_bucket [2021-07-27 22:15:31,766] - [basetestcase] [140687446664960] - INFO - list of n1ql nodes in cluster: [ip:172.23.123.146 port:8091 ssh_username:root] [2021-07-27 22:15:31,771] - [rest_client] [140687446664960] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.dst_bucket.dst_bucket [2021-07-27 22:15:31,782] - [eventing_base] [140687446664960] - INFO - Final docs count... Current : 0 Expected : 0 [2021-07-27 22:15:31,805] - [basetestcase] [140687446664960] - INFO - list of eventing nodes in cluster: [ip:172.23.123.151 port:8091 ssh_username:root] [2021-07-27 22:15:31,859] - [eventing_base] [140687446664960] - INFO - Pause Application : Function_539913202_test_source_scope_recreate [2021-07-27 22:15:31,859] - [basetestcase] [140687446664960] - INFO - sleep for 5 secs. Waiting for Function_539913202_test_source_scope_recreate to paused... ... [2021-07-27 22:15:36,883] - [basetestcase] [140687446664960] - INFO - sleep for 5 secs. Waiting for Function_539913202_test_source_scope_recreate to paused... ... [2021-07-27 22:15:41,945] - [rest_client] [140687446664960] - INFO - http://172.23.123.103:8091/pools/default/buckets/src_bucket/scopes/src_bucket [2021-07-27 22:15:41,945] - [basetestcase] [140687446664960] - INFO - sleep for 5 secs. Waiting for Function_539913202_test_source_scope_recreate to undeployed... ... [2021-07-27 22:15:46,970] - [basetestcase] [140687446664960] - INFO - sleep for 5 secs. Waiting for Function_539913202_test_source_scope_recreate to undeployed... ... [2021-07-27 22:15:51,984] - [basetestcase] [140687446664960] - INFO - sleep for 5 secs. Waiting for Function_539913202_test_source_scope_recreate to undeployed... ... [2021-07-27 22:15:57,006] - [basetestcase] [140687446664960] - INFO - sleep for 5 secs. Waiting for Function_539913202_test_source_scope_recreate to undeployed... ... [2021-07-27 22:16:02,051] - [rest_client] [140687446664960] - INFO - http://172.23.123.103:8091/pools/default/buckets/src_bucket/scopes with params: name=src_bucket [2021-07-27 22:16:02,051] - [rest_client] [140687446664960] - INFO - Scope created src_bucket->src_bucket {'uid': '4'} [2021-07-27 22:16:02,137] - [rest_client] [140687446664960] - INFO - http://172.23.123.103:8091/pools/default/buckets/src_bucket/scopes/src_bucket/collections with params: name=src_bucket [2021-07-27 22:16:02,137] - [rest_client] [140687446664960] - INFO - Collection created src_bucket->src_bucket->src_bucket manifest:{'uid': '5'} [2021-07-27 22:16:02,138] - [eventing_base] [140687446664960] - INFO - Deploying the following handler code : Function_539913202_test_source_scope_recreate with bindings: {'buckets': [{'alias': 'dst_bucket', 'bucket_name': 'dst_bucket', 'scope_name': 'dst_bucket', 'collection_name': 'dst_bucket', 'access': 'rw'}, {'alias': 'src_bucket', 'bucket_name': 'src_bucket', 'scope_name': 'src_bucket', 'collection_name': 'src_bucket'}], 'metadata_bucket': 'metadata', 'metadata_scope': 'metadata', 'metadata_collection': 'metadata', 'source_bucket': 'src_bucket', 'source_scope': 'src_bucket', 'source_collection': 'src_bucket', 'curl': []} and settings: {'checkpoint_interval': 20000, 'cleanup_timers': False, 'dcp_stream_boundary': 'everything', 'deployment_status': False, 'description': 'Sample Description', 'log_level': 'INFO', 'skip_timer_threshold': 86400, 'sock_batch_size': 1, 'tick_duration': 5000, 'timer_processing_tick_interval': 500, 'timer_worker_pool_size': 3, 'worker_count': 3, 'processing_status': False, 'cpp_worker_thread_count': 1, 'execution_timeout': 20, 'data_chan_size': 10000, 'worker_queue_cap': 100000, 'use_memory_manager': True, 'deadline_timeout': 21, 'timer_storage_chan_size': 10000, 'dcp_gen_chan_size': 10000, 'language_compatibility': '6.6.2'} [2021-07-27 22:16:02,138] - [eventing_base] [140687446664960] - INFO - function OnUpdate(doc, meta) { log("Doc created/updated", meta.id); try{ var result1= couchbase.insert(dst_bucket,meta,doc); log(result1); }catch(e){ log("error:",e); } } function OnDelete(meta, options) { log("Doc deleted/expired", meta.id); var doc={"id":meta.id} var result = couchbase.delete(dst_bucket,doc); log(result); } [2021-07-27 22:16:03,324] - [eventing_base] [140687446664960] - INFO - deploy Application : b'' [2021-07-27 22:16:03,324] - [basetestcase] [140687446664960] - INFO - sleep for 5 secs. Waiting for Function_539913202_test_source_scope_recreate to deployed... ... [2021-07-27 22:16:08,357] - [basetestcase] [140687446664960] - INFO - sleep for 5 secs. Waiting for Function_539913202_test_source_scope_recreate to deployed... ... [2021-07-27 22:16:13,371] - [basetestcase] [140687446664960] - INFO - sleep for 5 secs. Waiting for Function_539913202_test_source_scope_recreate to deployed... ... [2021-07-27 22:16:18,387] - [basetestcase] [140687446664960] - INFO - sleep for 5 secs. Waiting for Function_539913202_test_source_scope_recreate to deployed... ... [2021-07-27 22:16:23,573] - [task] [140687072810752] - INFO - java -jar java_sdk_client/collections/target/javaclient/javaclient.jar -i 172.23.123.103 -u Administrator -p password -b src_bucket -s src_bucket -c src_bucket -n 2016 -pc 100 -pu 0 -pd 0 -l uniform -dsn 1 -dpx doc_ -dt Person -de 0 -ds 500 -ac False -st 0 -en 2015 -o False -sd False [2021-07-27 22:16:32,147] - [basetestcase] [140687446664960] - INFO - list of eventing nodes in cluster: [ip:172.23.123.151 port:8091 ssh_username:root] [2021-07-27 22:16:32,172] - [basetestcase] [140687446664960] - INFO - list of n1ql nodes in cluster: [ip:172.23.123.146 port:8091 ssh_username:root] [2021-07-27 22:16:32,186] - [tuq_helper] [140687446664960] - INFO - RUN QUERY create primary index on dst_bucket.dst_bucket.dst_bucket [2021-07-27 22:16:32,198] - [rest_client] [140687446664960] - INFO - query params : statement=create+primary+index+on+dst_bucket.dst_bucket.dst_bucket [2021-07-27 22:16:32,204] - [rest_client] [140687446664960] - ERROR - POST http://172.23.123.146:8093/query?statement=create+primary+index+on+dst_bucket.dst_bucket.dst_bucket body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 500 reason: unknown b'{\n"requestID": "e32047ec-7b03-4698-b1c5-25c8cbea6450",\n"signature": null,\n"results": [\n],\n"errors": [{"code":5000,"msg":"GSI CreatePrimaryIndex() - cause: Index #primary already exists."}],\n"status": "errors",\n"metrics": {"elapsedTime": "868.459\xc2\xb5s","executionTime": "781.103\xc2\xb5s","resultCount": 0,"resultSize": 0,"serviceLoad": 6,"errorCount": 1}\n}\n' auth: Administrator:password [2021-07-27 22:16:32,232] - [basetestcase] [140687446664960] - INFO - list of n1ql nodes in cluster: [ip:172.23.123.146 port:8091 ssh_username:root] [2021-07-27 22:16:32,248] - [rest_client] [140687446664960] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.dst_bucket.dst_bucket [2021-07-27 22:16:32,259] - [basetestcase] [140687446664960] - INFO - sleep for 30 secs. Waiting for handler code dst_bucket.dst_bucket.dst_bucket to complete bucket operations... Current : 1914 Expected : 2016 ... [2021-07-27 22:17:02,283] - [basetestcase] [140687446664960] - INFO - list of n1ql nodes in cluster: [ip:172.23.123.146 port:8091 ssh_username:root] [2021-07-27 22:17:02,290] - [rest_client] [140687446664960] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.dst_bucket.dst_bucket [2021-07-27 22:17:02,311] - [basetestcase] [140687446664960] - INFO - list of n1ql nodes in cluster: [ip:172.23.123.146 port:8091 ssh_username:root] [2021-07-27 22:17:02,317] - [rest_client] [140687446664960] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.dst_bucket.dst_bucket [2021-07-27 22:17:02,322] - [eventing_base] [140687446664960] - INFO - Final docs count... Current : 2016 Expected : 2016 [2021-07-27 22:17:02,334] - [basetestcase] [140687446664960] - INFO - list of eventing nodes in cluster: [ip:172.23.123.151 port:8091 ssh_username:root] [2021-07-27 22:17:02,381] - [eventing_base] [140687446664960] - INFO - Undeploy Application : Function_539913202_test_source_scope_recreate [2021-07-27 22:17:02,382] - [basetestcase] [140687446664960] - INFO - sleep for 5 secs. Waiting for Function_539913202_test_source_scope_recreate to undeployed... ... [2021-07-27 22:17:07,401] - [basetestcase] [140687446664960] - INFO - sleep for 5 secs. Waiting for Function_539913202_test_source_scope_recreate to undeployed... ... [2021-07-27 22:17:12,418] - [basetestcase] [140687446664960] - INFO - sleep for 5 secs. Waiting for Function_539913202_test_source_scope_recreate to undeployed... ... [2021-07-27 22:17:17,434] - [basetestcase] [140687446664960] - INFO - sleep for 5 secs. Waiting for Function_539913202_test_source_scope_recreate to undeployed... ... [2021-07-27 22:17:22,449] - [basetestcase] [140687446664960] - INFO - sleep for 5 secs. Waiting for Function_539913202_test_source_scope_recreate to undeployed... ... [2021-07-27 22:17:27,473] - [basetestcase] [140687446664960] - INFO - sleep for 5 secs. Waiting for Function_539913202_test_source_scope_recreate to undeployed... ... [2021-07-27 22:17:32,491] - [basetestcase] [140687446664960] - INFO - sleep for 5 secs. Waiting for Function_539913202_test_source_scope_recreate to undeployed... ... [2021-07-27 22:17:37,508] - [basetestcase] [140687446664960] - INFO - sleep for 5 secs. Waiting for Function_539913202_test_source_scope_recreate to undeployed... ... [2021-07-27 22:17:42,533] - [basetestcase] [140687446664960] - INFO - sleep for 5 secs. Waiting for Function_539913202_test_source_scope_recreate to undeployed... ... [2021-07-27 22:17:47,556] - [basetestcase] [140687446664960] - INFO - sleep for 5 secs. Waiting for Function_539913202_test_source_scope_recreate to undeployed... ... [2021-07-27 22:17:52,573] - [basetestcase] [140687446664960] - INFO - sleep for 5 secs. Waiting for Function_539913202_test_source_scope_recreate to undeployed... ... [2021-07-27 22:17:57,596] - [basetestcase] [140687446664960] - INFO - sleep for 5 secs. Waiting for Function_539913202_test_source_scope_recreate to undeployed... ... [2021-07-27 22:18:02,622] - [basetestcase] [140687446664960] - INFO - sleep for 5 secs. Waiting for Function_539913202_test_source_scope_recreate to undeployed... ... [2021-07-27 22:18:07,643] - [basetestcase] [140687446664960] - INFO - sleep for 5 secs. Waiting for Function_539913202_test_source_scope_recreate to undeployed... ... [2021-07-27 22:18:12,666] - [basetestcase] [140687446664960] - INFO - sleep for 5 secs. Waiting for Function_539913202_test_source_scope_recreate to undeployed... ... [2021-07-27 22:18:17,681] - [basetestcase] [140687446664960] - INFO - sleep for 5 secs. Waiting for Function_539913202_test_source_scope_recreate to undeployed... ... [2021-07-27 22:18:22,696] - [basetestcase] [140687446664960] - INFO - sleep for 5 secs. Waiting for Function_539913202_test_source_scope_recreate to undeployed... ... [2021-07-27 22:18:27,715] - [basetestcase] [140687446664960] - INFO - sleep for 5 secs. Waiting for Function_539913202_test_source_scope_recreate to undeployed... ... [2021-07-27 22:18:32,730] - [basetestcase] [140687446664960] - INFO - sleep for 5 secs. Waiting for Function_539913202_test_source_scope_recreate to undeployed... ... [2021-07-27 22:18:37,752] - [basetestcase] [140687446664960] - INFO - sleep for 5 secs. Waiting for Function_539913202_test_source_scope_recreate to undeployed... ... [2021-07-27 22:18:42,769] - [basetestcase] [140687446664960] - INFO - sleep for 5 secs. Waiting for Function_539913202_test_source_scope_recreate to undeployed... ... [2021-07-27 22:18:47,785] - [basetestcase] [140687446664960] - INFO - sleep for 5 secs. Waiting for Function_539913202_test_source_scope_recreate to undeployed... ... [2021-07-27 22:18:52,807] - [basetestcase] [140687446664960] - INFO - sleep for 5 secs. Waiting for Function_539913202_test_source_scope_recreate to undeployed... ... [2021-07-27 22:18:57,827] - [basetestcase] [140687446664960] - INFO - sleep for 5 secs. Waiting for Function_539913202_test_source_scope_recreate to undeployed... ... [2021-07-27 22:19:02,843] - [basetestcase] [140687446664960] - INFO - sleep for 5 secs. Waiting for Function_539913202_test_source_scope_recreate to undeployed... ... [2021-07-27 22:19:07,884] - [basetestcase] [140687446664960] - INFO - sleep for 5 secs. Waiting for Function_539913202_test_source_scope_recreate to undeployed... ... [2021-07-27 22:19:12,905] - [basetestcase] [140687446664960] - INFO - sleep for 5 secs. Waiting for Function_539913202_test_source_scope_recreate to undeployed... ... [2021-07-27 22:19:17,924] - [basetestcase] [140687446664960] - INFO - sleep for 5 secs. Waiting for Function_539913202_test_source_scope_recreate to undeployed... ... [2021-07-27 22:19:22,949] - [basetestcase] [140687446664960] - INFO - sleep for 5 secs. Waiting for Function_539913202_test_source_scope_recreate to undeployed... ... [2021-07-27 22:19:27,964] - [basetestcase] [140687446664960] - INFO - sleep for 5 secs. Waiting for Function_539913202_test_source_scope_recreate to undeployed... ... [2021-07-27 22:19:32,978] - [basetestcase] [140687446664960] - INFO - sleep for 5 secs. Waiting for Function_539913202_test_source_scope_recreate to undeployed... ... [2021-07-27 22:19:37,996] - [basetestcase] [140687446664960] - INFO - sleep for 5 secs. Waiting for Function_539913202_test_source_scope_recreate to undeployed... ... [2021-07-27 22:19:43,032] - [basetestcase] [140687446664960] - INFO - sleep for 5 secs. Waiting for Function_539913202_test_source_scope_recreate to undeployed... ... [2021-07-27 22:19:48,059] - [basetestcase] [140687446664960] - INFO - sleep for 5 secs. Waiting for Function_539913202_test_source_scope_recreate to undeployed... ... [2021-07-27 22:19:53,075] - [basetestcase] [140687446664960] - INFO - sleep for 5 secs. Waiting for Function_539913202_test_source_scope_recreate to undeployed... ... [2021-07-27 22:19:58,089] - [basetestcase] [140687446664960] - INFO - sleep for 5 secs. Waiting for Function_539913202_test_source_scope_recreate to undeployed... ... [2021-07-27 22:20:03,109] - [basetestcase] [140687446664960] - INFO - sleep for 5 secs. Waiting for Function_539913202_test_source_scope_recreate to undeployed... ... [2021-07-27 22:20:08,125] - [basetestcase] [140687446664960] - INFO - sleep for 5 secs. Waiting for Function_539913202_test_source_scope_recreate to undeployed... ... [2021-07-27 22:20:13,143] - [basetestcase] [140687446664960] - INFO - sleep for 5 secs. Waiting for Function_539913202_test_source_scope_recreate to undeployed... ... [2021-07-27 22:20:18,165] - [basetestcase] [140687446664960] - INFO - sleep for 5 secs. Waiting for Function_539913202_test_source_scope_recreate to undeployed... ... [2021-07-27 22:20:23,177] - [basetestcase] [140687446664960] - INFO - sleep for 5 secs. Waiting for Function_539913202_test_source_scope_recreate to undeployed... ... [2021-07-27 22:20:28,203] - [basetestcase] [140687446664960] - INFO - sleep for 5 secs. Waiting for Function_539913202_test_source_scope_recreate to undeployed... ... [2021-07-27 22:20:33,217] - [basetestcase] [140687446664960] - INFO - sleep for 5 secs. Waiting for Function_539913202_test_source_scope_recreate to undeployed... ... [2021-07-27 22:20:38,235] - [basetestcase] [140687446664960] - INFO - sleep for 5 secs. Waiting for Function_539913202_test_source_scope_recreate to undeployed... ... [2021-07-27 22:20:43,246] - [basetestcase] [140687446664960] - INFO - sleep for 5 secs. Waiting for Function_539913202_test_source_scope_recreate to undeployed... ... [2021-07-27 22:20:48,261] - [basetestcase] [140687446664960] - INFO - sleep for 5 secs. Waiting for Function_539913202_test_source_scope_recreate to undeployed... ... [2021-07-27 22:20:53,277] - [basetestcase] [140687446664960] - INFO - sleep for 5 secs. Waiting for Function_539913202_test_source_scope_recreate to undeployed... ... [2021-07-27 22:20:58,295] - [basetestcase] [140687446664960] - INFO - sleep for 5 secs. Waiting for Function_539913202_test_source_scope_recreate to undeployed... ... [2021-07-27 22:21:03,314] - [basetestcase] [140687446664960] - INFO - sleep for 5 secs. Waiting for Function_539913202_test_source_scope_recreate to undeployed... ... [2021-07-27 22:21:08,338] - [basetestcase] [140687446664960] - INFO - sleep for 5 secs. Waiting for Function_539913202_test_source_scope_recreate to undeployed... ... [2021-07-27 22:21:13,368] - [basetestcase] [140687446664960] - INFO - sleep for 5 secs. Waiting for Function_539913202_test_source_scope_recreate to undeployed... ... [2021-07-27 22:21:18,390] - [basetestcase] [140687446664960] - INFO - sleep for 5 secs. Waiting for Function_539913202_test_source_scope_recreate to undeployed... ... [2021-07-27 22:21:23,415] - [basetestcase] [140687446664960] - INFO - sleep for 5 secs. Waiting for Function_539913202_test_source_scope_recreate to undeployed... ... [2021-07-27 22:21:28,441] - [basetestcase] [140687446664960] - INFO - sleep for 5 secs. Waiting for Function_539913202_test_source_scope_recreate to undeployed... ... [2021-07-27 22:21:33,454] - [basetestcase] [140687446664960] - INFO - sleep for 5 secs. Waiting for Function_539913202_test_source_scope_recreate to undeployed... ... [2021-07-27 22:21:38,472] - [basetestcase] [140687446664960] - INFO - sleep for 5 secs. Waiting for Function_539913202_test_source_scope_recreate to undeployed... ... [2021-07-27 22:21:43,493] - [basetestcase] [140687446664960] - INFO - sleep for 5 secs. Waiting for Function_539913202_test_source_scope_recreate to undeployed... ... [2021-07-27 22:21:48,518] - [basetestcase] [140687446664960] - INFO - sleep for 5 secs. Waiting for Function_539913202_test_source_scope_recreate to undeployed... ... [2021-07-27 22:21:53,536] - [basetestcase] [140687446664960] - INFO - sleep for 5 secs. Waiting for Function_539913202_test_source_scope_recreate to undeployed... ... [2021-07-27 22:21:58,550] - [basetestcase] [140687446664960] - INFO - sleep for 5 secs. Waiting for Function_539913202_test_source_scope_recreate to undeployed... ... [2021-07-27 22:22:03,565] - [basetestcase] [140687446664960] - INFO - sleep for 5 secs. Waiting for Function_539913202_test_source_scope_recreate to undeployed... ... [2021-07-27 22:22:08,588] - [basetestcase] [140687446664960] - INFO - sleep for 5 secs. Waiting for Function_539913202_test_source_scope_recreate to undeployed... ... [2021-07-27 22:22:13,613] - [basetestcase] [140687446664960] - INFO - sleep for 5 secs. Waiting for Function_539913202_test_source_scope_recreate to undeployed... ... [2021-07-27 22:22:19,635] - [basetestcase] [140687446664960] - INFO - sleep for 5 secs. Waiting for Function_539913202_test_source_scope_recreate to undeployed... ... [2021-07-27 22:22:24,655] - [basetestcase] [140687446664960] - INFO - sleep for 5 secs. Waiting for Function_539913202_test_source_scope_recreate to undeployed... ... [2021-07-27 22:22:29,671] - [basetestcase] [140687446664960] - INFO - sleep for 5 secs. Waiting for Function_539913202_test_source_scope_recreate to undeployed... ... [2021-07-27 22:22:34,687] - [basetestcase] [140687446664960] - INFO - sleep for 5 secs. Waiting for Function_539913202_test_source_scope_recreate to undeployed... ... [2021-07-27 22:22:39,701] - [basetestcase] [140687446664960] - INFO - sleep for 5 secs. Waiting for Function_539913202_test_source_scope_recreate to undeployed... ... [2021-07-27 22:22:44,718] - [basetestcase] [140687446664960] - INFO - sleep for 5 secs. Waiting for Function_539913202_test_source_scope_recreate to undeployed... ... [2021-07-27 22:22:49,735] - [basetestcase] [140687446664960] - INFO - sleep for 5 secs. Waiting for Function_539913202_test_source_scope_recreate to undeployed... ... [2021-07-27 22:22:54,753] - [basetestcase] [140687446664960] - INFO - sleep for 5 secs. Waiting for Function_539913202_test_source_scope_recreate to undeployed... ... [2021-07-27 22:22:59,770] - [basetestcase] [140687446664960] - INFO - sleep for 5 secs. Waiting for Function_539913202_test_source_scope_recreate to undeployed... ... [2021-07-27 22:23:04,788] - [basetestcase] [140687446664960] - INFO - sleep for 5 secs. Waiting for Function_539913202_test_source_scope_recreate to undeployed... ... [2021-07-27 22:23:09,805] - [basetestcase] [140687446664960] - INFO - sleep for 5 secs. Waiting for Function_539913202_test_source_scope_recreate to undeployed... ... [2021-07-27 22:23:14,823] - [basetestcase] [140687446664960] - INFO - sleep for 5 secs. Waiting for Function_539913202_test_source_scope_recreate to undeployed... ... [2021-07-27 22:23:19,848] - [basetestcase] [140687446664960] - INFO - sleep for 5 secs. Waiting for Function_539913202_test_source_scope_recreate to undeployed... ... [2021-07-27 22:23:24,864] - [basetestcase] [140687446664960] - INFO - sleep for 5 secs. Waiting for Function_539913202_test_source_scope_recreate to undeployed... ... [2021-07-27 22:23:29,881] - [basetestcase] [140687446664960] - INFO - sleep for 5 secs. Waiting for Function_539913202_test_source_scope_recreate to undeployed... ... [2021-07-27 22:23:34,899] - [basetestcase] [140687446664960] - INFO - sleep for 5 secs. Waiting for Function_539913202_test_source_scope_recreate to undeployed... ... [2021-07-27 22:23:39,915] - [basetestcase] [140687446664960] - INFO - sleep for 5 secs. Waiting for Function_539913202_test_source_scope_recreate to undeployed... ... [2021-07-27 22:23:44,932] - [basetestcase] [140687446664960] - INFO - sleep for 5 secs. Waiting for Function_539913202_test_source_scope_recreate to undeployed... ... [2021-07-27 22:23:49,946] - [basetestcase] [140687446664960] - INFO - sleep for 5 secs. Waiting for Function_539913202_test_source_scope_recreate to undeployed... ... [2021-07-27 22:23:54,959] - [basetestcase] [140687446664960] - INFO - sleep for 5 secs. Waiting for Function_539913202_test_source_scope_recreate to undeployed... ... [2021-07-27 22:23:59,974] - [basetestcase] [140687446664960] - INFO - sleep for 5 secs. Waiting for Function_539913202_test_source_scope_recreate to undeployed... ... [2021-07-27 22:24:04,993] - [basetestcase] [140687446664960] - INFO - sleep for 5 secs. Waiting for Function_539913202_test_source_scope_recreate to undeployed... ... [2021-07-27 22:24:10,020] - [basetestcase] [140687446664960] - INFO - sleep for 5 secs. Waiting for Function_539913202_test_source_scope_recreate to undeployed... ... [2021-07-27 22:24:15,036] - [basetestcase] [140687446664960] - INFO - sleep for 5 secs. Waiting for Function_539913202_test_source_scope_recreate to undeployed... ... [2021-07-27 22:24:20,052] - [basetestcase] [140687446664960] - INFO - sleep for 5 secs. Waiting for Function_539913202_test_source_scope_recreate to undeployed... ... [2021-07-27 22:24:25,069] - [basetestcase] [140687446664960] - INFO - sleep for 5 secs. Waiting for Function_539913202_test_source_scope_recreate to undeployed... ... [2021-07-27 22:24:30,084] - [basetestcase] [140687446664960] - INFO - sleep for 5 secs. Waiting for Function_539913202_test_source_scope_recreate to undeployed... ... [2021-07-27 22:24:35,100] - [basetestcase] [140687446664960] - INFO - sleep for 5 secs. Waiting for Function_539913202_test_source_scope_recreate to undeployed... ... [2021-07-27 22:24:40,117] - [basetestcase] [140687446664960] - INFO - sleep for 5 secs. Waiting for Function_539913202_test_source_scope_recreate to undeployed... ... [2021-07-27 22:24:45,128] - [basetestcase] [140687446664960] - INFO - sleep for 5 secs. Waiting for Function_539913202_test_source_scope_recreate to undeployed... ... [2021-07-27 22:24:50,147] - [basetestcase] [140687446664960] - INFO - sleep for 5 secs. Waiting for Function_539913202_test_source_scope_recreate to undeployed... ... [2021-07-27 22:24:55,164] - [basetestcase] [140687446664960] - INFO - sleep for 5 secs. Waiting for Function_539913202_test_source_scope_recreate to undeployed... ... [2021-07-27 22:25:00,180] - [basetestcase] [140687446664960] - INFO - sleep for 5 secs. Waiting for Function_539913202_test_source_scope_recreate to undeployed... ... [2021-07-27 22:25:05,196] - [basetestcase] [140687446664960] - INFO - sleep for 5 secs. Waiting for Function_539913202_test_source_scope_recreate to undeployed... ... [2021-07-27 22:25:10,212] - [basetestcase] [140687446664960] - INFO - sleep for 5 secs. Waiting for Function_539913202_test_source_scope_recreate to undeployed... ... [2021-07-27 22:25:15,228] - [basetestcase] [140687446664960] - INFO - sleep for 5 secs. Waiting for Function_539913202_test_source_scope_recreate to undeployed... ... [2021-07-27 22:25:20,252] - [basetestcase] [140687446664960] - INFO - sleep for 5 secs. Waiting for Function_539913202_test_source_scope_recreate to undeployed... ... [2021-07-27 22:25:25,276] - [basetestcase] [140687446664960] - INFO - sleep for 5 secs. Waiting for Function_539913202_test_source_scope_recreate to undeployed... ... [2021-07-27 22:25:30,291] - [basetestcase] [140687446664960] - INFO - sleep for 5 secs. Waiting for Function_539913202_test_source_scope_recreate to undeployed... ... [2021-07-27 22:25:35,307] - [basetestcase] [140687446664960] - INFO - sleep for 5 secs. Waiting for Function_539913202_test_source_scope_recreate to undeployed... ... [2021-07-27 22:25:40,323] - [basetestcase] [140687446664960] - INFO - sleep for 5 secs. Waiting for Function_539913202_test_source_scope_recreate to undeployed... ... [2021-07-27 22:25:45,339] - [basetestcase] [140687446664960] - INFO - sleep for 5 secs. Waiting for Function_539913202_test_source_scope_recreate to undeployed... ... [2021-07-27 22:25:50,355] - [basetestcase] [140687446664960] - INFO - sleep for 5 secs. Waiting for Function_539913202_test_source_scope_recreate to undeployed... ... [2021-07-27 22:25:55,367] - [basetestcase] [140687446664960] - INFO - sleep for 5 secs. Waiting for Function_539913202_test_source_scope_recreate to undeployed... ... [2021-07-27 22:26:00,385] - [basetestcase] [140687446664960] - INFO - sleep for 5 secs. Waiting for Function_539913202_test_source_scope_recreate to undeployed... ... [2021-07-27 22:26:05,397] - [basetestcase] [140687446664960] - INFO - sleep for 5 secs. Waiting for Function_539913202_test_source_scope_recreate to undeployed... ... [2021-07-27 22:26:10,412] - [basetestcase] [140687446664960] - INFO - sleep for 5 secs. Waiting for Function_539913202_test_source_scope_recreate to undeployed... ... [2021-07-27 22:26:15,428] - [basetestcase] [140687446664960] - INFO - sleep for 5 secs. Waiting for Function_539913202_test_source_scope_recreate to undeployed... ... [2021-07-27 22:26:20,441] - [basetestcase] [140687446664960] - INFO - sleep for 5 secs. Waiting for Function_539913202_test_source_scope_recreate to undeployed... ... [2021-07-27 22:26:25,454] - [basetestcase] [140687446664960] - INFO - sleep for 5 secs. Waiting for Function_539913202_test_source_scope_recreate to undeployed... ... [2021-07-27 22:26:30,471] - [basetestcase] [140687446664960] - INFO - sleep for 5 secs. Waiting for Function_539913202_test_source_scope_recreate to undeployed... ... [2021-07-27 22:26:35,489] - [basetestcase] [140687446664960] - INFO - sleep for 5 secs. Waiting for Function_539913202_test_source_scope_recreate to undeployed... ... [2021-07-27 22:26:40,507] - [basetestcase] [140687446664960] - INFO - sleep for 5 secs. Waiting for Function_539913202_test_source_scope_recreate to undeployed... ... [2021-07-27 22:26:45,521] - [basetestcase] [140687446664960] - INFO - sleep for 5 secs. Waiting for Function_539913202_test_source_scope_recreate to undeployed... ... [2021-07-27 22:26:50,539] - [basetestcase] [140687446664960] - INFO - sleep for 5 secs. Waiting for Function_539913202_test_source_scope_recreate to undeployed... ... [2021-07-27 22:26:55,554] - [basetestcase] [140687446664960] - INFO - sleep for 5 secs. Waiting for Function_539913202_test_source_scope_recreate to undeployed... ... [2021-07-27 22:27:00,568] - [basetestcase] [140687446664960] - INFO - sleep for 5 secs. Waiting for Function_539913202_test_source_scope_recreate to undeployed... ... [2021-07-27 22:27:05,585] - [basetestcase] [140687446664960] - INFO - sleep for 5 secs. Waiting for Function_539913202_test_source_scope_recreate to undeployed... ... [2021-07-27 22:27:10,600] - [eventing_base] [140687446664960] - INFO - ============== EventingBaseTest tearDown has started ============== [2021-07-27 22:27:10,639] - [basetestcase] [140687446664960] - INFO - list of eventing nodes in cluster: [ip:172.23.123.151 port:8091 ssh_username:root] [2021-07-27 22:27:10,644] - [remote_util] [140687446664960] - INFO - SSH Connecting to 172.23.123.151 with username:root, attempt#1 of 5 [2021-07-27 22:27:10,759] - [remote_util] [140687446664960] - INFO - SSH Connected to 172.23.123.151 as root [2021-07-27 22:27:11,064] - [remote_util] [140687446664960] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-07-27 22:27:11,388] - [remote_util] [140687446664960] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-07-27 22:27:11,400] - [rest_client] [140687446664960] - INFO - /diag/eval status on 172.23.123.151:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). [2021-07-27 22:27:11,401] - [remote_util] [140687446664960] - INFO - running command.raw on 172.23.123.151: zgrep "panic" "/opt/couchbase/var/lib/couchbase/logs"/eventing.log* | wc -l [2021-07-27 22:27:11,490] - [remote_util] [140687446664960] - INFO - command executed successfully with root [2021-07-27 22:27:11,491] - [remote_util] [140687446664960] - INFO - running command.raw on 172.23.123.151: ls "/opt/couchbase/var/lib/couchbase/logs"/../crash/| wc -l [2021-07-27 22:27:11,507] - [remote_util] [140687446664960] - INFO - command executed successfully with root [2021-07-27 22:27:11,574] - [basetestcase] [140687446664960] - INFO - list of n1ql nodes in cluster: [ip:172.23.123.146 port:8091 ssh_username:root] [2021-07-27 22:27:11,581] - [rest_client] [140687446664960] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+metadata [2021-07-27 22:27:11,594] - [eventing_base] [140687446664960] - INFO - number of documents in metadata bucket 0 [2021-07-27 22:27:11,605] - [basetestcase] [140687446664960] - INFO - ------- Cluster statistics ------- [2021-07-27 22:27:11,605] - [basetestcase] [140687446664960] - INFO - 172.23.123.103:8091 => {'services': ['kv'], 'cpu_utilization': 41.08507386653082, 'mem_free': 2676555776, 'mem_total': 4201672704, 'swap_mem_used': 83886080, 'swap_mem_total': 3758092288} [2021-07-27 22:27:11,606] - [basetestcase] [140687446664960] - INFO - 172.23.123.146:8091 => {'services': ['n1ql'], 'cpu_utilization': 0.4029211785444473, 'mem_free': 3487985664, 'mem_total': 4201672704, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-07-27 22:27:11,606] - [basetestcase] [140687446664960] - INFO - 172.23.123.147:8091 => {'services': ['index'], 'cpu_utilization': 1.296060991105464, 'mem_free': 3305865216, 'mem_total': 4201672704, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-07-27 22:27:11,607] - [basetestcase] [140687446664960] - INFO - 172.23.123.151:8091 => {'services': ['eventing'], 'cpu_utilization': 0.5531807895398542, 'mem_free': 3497721856, 'mem_total': 4201672704, 'swap_mem_used': 63438848, 'swap_mem_total': 3758092288} [2021-07-27 22:27:11,607] - [basetestcase] [140687446664960] - INFO - --- End of cluster statistics --- [2021-07-27 22:27:11,610] - [remote_util] [140687446664960] - INFO - SSH Connecting to 172.23.123.103 with username:root, attempt#1 of 5 [2021-07-27 22:27:11,715] - [remote_util] [140687446664960] - INFO - SSH Connected to 172.23.123.103 as root [2021-07-27 22:27:11,990] - [remote_util] [140687446664960] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-07-27 22:27:12,347] - [remote_util] [140687446664960] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-07-27 22:27:12,351] - [remote_util] [140687446664960] - INFO - SSH Connecting to 172.23.123.151 with username:root, attempt#1 of 5 [2021-07-27 22:27:12,467] - [remote_util] [140687446664960] - INFO - SSH Connected to 172.23.123.151 as root [2021-07-27 22:27:12,762] - [remote_util] [140687446664960] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-07-27 22:27:13,095] - [remote_util] [140687446664960] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-07-27 22:27:13,099] - [remote_util] [140687446664960] - INFO - SSH Connecting to 172.23.123.147 with username:root, attempt#1 of 5 [2021-07-27 22:27:13,211] - [remote_util] [140687446664960] - INFO - SSH Connected to 172.23.123.147 as root [2021-07-27 22:27:13,542] - [remote_util] [140687446664960] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-07-27 22:27:13,872] - [remote_util] [140687446664960] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-07-27 22:27:13,875] - [remote_util] [140687446664960] - INFO - SSH Connecting to 172.23.123.146 with username:root, attempt#1 of 5 [2021-07-27 22:27:13,991] - [remote_util] [140687446664960] - INFO - SSH Connected to 172.23.123.146 as root [2021-07-27 22:27:14,303] - [remote_util] [140687446664960] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-07-27 22:27:14,631] - [remote_util] [140687446664960] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-07-27 22:27:19,205] - [basetestcase] [140687446664960] - INFO - ============== basetestcase cleanup was started for test #2 test_source_scope_recreate ============== [2021-07-27 22:27:19,273] - [bucket_helper] [140687446664960] - INFO - deleting existing buckets ['dst_bucket', 'metadata', 'src_bucket'] on 172.23.123.103 [2021-07-27 22:27:24,750] - [bucket_helper] [140687446664960] - INFO - waiting for bucket deletion to complete.... [2021-07-27 22:27:24,785] - [rest_client] [140687446664960] - INFO - node 172.23.123.103 existing buckets : ['metadata', 'src_bucket'] [2021-07-27 22:27:24,788] - [bucket_helper] [140687446664960] - INFO - deleted bucket : dst_bucket from 172.23.123.103 [2021-07-27 22:27:32,718] - [bucket_helper] [140687446664960] - INFO - waiting for bucket deletion to complete.... [2021-07-27 22:27:32,732] - [rest_client] [140687446664960] - INFO - node 172.23.123.103 existing buckets : ['src_bucket'] [2021-07-27 22:27:32,733] - [bucket_helper] [140687446664960] - INFO - deleted bucket : metadata from 172.23.123.103 [2021-07-27 22:27:37,158] - [bucket_helper] [140687446664960] - INFO - waiting for bucket deletion to complete.... [2021-07-27 22:27:37,163] - [rest_client] [140687446664960] - INFO - node 172.23.123.103 existing buckets : [] [2021-07-27 22:27:37,163] - [bucket_helper] [140687446664960] - INFO - deleted bucket : src_bucket from 172.23.123.103 [2021-07-27 22:27:37,175] - [bucket_helper] [140687446664960] - INFO - Could not find any buckets for node 172.23.123.151, nothing to delete [2021-07-27 22:27:37,185] - [bucket_helper] [140687446664960] - INFO - Could not find any buckets for node 172.23.123.147, nothing to delete [2021-07-27 22:27:37,194] - [bucket_helper] [140687446664960] - INFO - Could not find any buckets for node 172.23.123.146, nothing to delete [2021-07-27 22:27:37,200] - [rest_client] [140687446664960] - INFO - -->is_ns_server_running? [2021-07-27 22:27:37,214] - [cluster_helper] [140687446664960] - INFO - rebalancing all nodes in order to remove nodes [2021-07-27 22:27:37,217] - [rest_client] [140687446664960] - INFO - rebalance params : {'knownNodes': 'ns_1@172.23.123.103,ns_1@172.23.123.146,ns_1@172.23.123.147,ns_1@172.23.123.151', 'ejectedNodes': 'ns_1@172.23.123.146,ns_1@172.23.123.147,ns_1@172.23.123.151', 'user': 'Administrator', 'password': 'password'} [2021-07-27 22:27:37,272] - [rest_client] [140687446664960] - INFO - rebalance operation started [2021-07-27 22:27:37,275] - [rest_client] [140687446664960] - INFO - rebalance percentage : 0.00 % [2021-07-27 22:27:47,290] - [rest_client] [140687446664960] - INFO - rebalance percentage : 50.00 % [2021-07-27 22:27:57,300] - [rest_client] [140687446664960] - INFO - rebalance percentage : 75.00 % [2021-07-27 22:28:17,323] - [rest_client] [140687446664960] - INFO - rebalance progress took 40.05 seconds [2021-07-27 22:28:17,323] - [rest_client] [140687446664960] - INFO - sleep for 10 seconds after rebalance... [2021-07-27 22:28:27,358] - [cluster_helper] [140687446664960] - INFO - removed all the nodes from cluster associated with ip:172.23.123.103 port:8091 ssh_username:root ? [('ns_1@172.23.123.146', 8091), ('ns_1@172.23.123.147', 8091), ('ns_1@172.23.123.151', 8091)] [2021-07-27 22:28:27,364] - [cluster_helper] [140687446664960] - INFO - waiting for ns_server @ 172.23.123.103:8091 [2021-07-27 22:28:27,364] - [rest_client] [140687446664960] - INFO - -->is_ns_server_running? [2021-07-27 22:28:27,369] - [cluster_helper] [140687446664960] - INFO - ns_server @ 172.23.123.103:8091 is running [2021-07-27 22:28:27,374] - [cluster_helper] [140687446664960] - INFO - waiting for ns_server @ 172.23.123.151:8091 [2021-07-27 22:28:27,374] - [rest_client] [140687446664960] - INFO - -->is_ns_server_running? [2021-07-27 22:28:27,378] - [cluster_helper] [140687446664960] - INFO - ns_server @ 172.23.123.151:8091 is running [2021-07-27 22:28:27,383] - [cluster_helper] [140687446664960] - INFO - waiting for ns_server @ 172.23.123.147:8091 [2021-07-27 22:28:27,383] - [rest_client] [140687446664960] - INFO - -->is_ns_server_running? [2021-07-27 22:28:27,387] - [cluster_helper] [140687446664960] - INFO - ns_server @ 172.23.123.147:8091 is running [2021-07-27 22:28:27,392] - [cluster_helper] [140687446664960] - INFO - waiting for ns_server @ 172.23.123.146:8091 [2021-07-27 22:28:27,392] - [rest_client] [140687446664960] - INFO - -->is_ns_server_running? [2021-07-27 22:28:27,397] - [cluster_helper] [140687446664960] - INFO - ns_server @ 172.23.123.146:8091 is running [2021-07-27 22:28:27,397] - [basetestcase] [140687446664960] - INFO - ============== basetestcase cleanup was finished for test #2 test_source_scope_recreate ============== [2021-07-27 22:28:27,397] - [basetestcase] [140687446664960] - INFO - closing all ssh connections [2021-07-27 22:28:27,401] - [basetestcase] [140687446664960] - INFO - closing all memcached connections [2021-07-27 22:28:27,403] - [remote_util] [140687446664960] - INFO - SSH Connecting to 172.23.123.103 with username:root, attempt#1 of 5 [2021-07-27 22:28:27,510] - [remote_util] [140687446664960] - INFO - SSH Connected to 172.23.123.103 as root [2021-07-27 22:28:27,794] - [remote_util] [140687446664960] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-07-27 22:28:28,103] - [remote_util] [140687446664960] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-07-27 22:28:28,116] - [remote_util] [140687446664960] - INFO - SSH Connecting to 172.23.123.103 with username:root, attempt#1 of 5 [2021-07-27 22:28:28,218] - [remote_util] [140687446664960] - INFO - SSH Connected to 172.23.123.103 as root [2021-07-27 22:28:28,477] - [remote_util] [140687446664960] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-07-27 22:28:28,788] - [remote_util] [140687446664960] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-07-27 22:28:28,851] - [remote_util] [140687446664960] - INFO - 172.23.123.103 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-07-27 22:28:28,963] - [remote_util] [140687446664960] - INFO - Running systemd command on this server [2021-07-27 22:28:28,963] - [remote_util] [140687446664960] - INFO - running command.raw on 172.23.123.103: systemctl stop couchbase-server.service [2021-07-27 22:28:31,309] - [remote_util] [140687446664960] - INFO - command executed successfully with root [2021-07-27 22:28:31,310] - [basetestcase] [140687446664960] - INFO - Couchbase stopped [2021-07-27 22:28:31,310] - [basetestcase] [140687446664960] - INFO - sleep for 5 secs. ... [2021-07-27 22:28:36,311] - [remote_util] [140687446664960] - INFO - running command.raw on 172.23.123.103: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-07-27 22:28:36,327] - [remote_util] [140687446664960] - INFO - command executed successfully with root [2021-07-27 22:28:36,328] - [remote_util] [140687446664960] - INFO - running command.raw on 172.23.123.103: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-07-27 22:28:36,387] - [remote_util] [140687446664960] - INFO - command executed successfully with root [2021-07-27 22:28:36,390] - [remote_util] [140687446664960] - INFO - SSH Connecting to 172.23.123.103 with username:root, attempt#1 of 5 [2021-07-27 22:28:36,495] - [remote_util] [140687446664960] - INFO - SSH Connected to 172.23.123.103 as root [2021-07-27 22:28:36,766] - [remote_util] [140687446664960] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-07-27 22:28:37,076] - [remote_util] [140687446664960] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-07-27 22:28:37,139] - [remote_util] [140687446664960] - INFO - 172.23.123.103 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-07-27 22:28:37,140] - [remote_util] [140687446664960] - INFO - Checking for process beam.smp on linux [2021-07-27 22:28:37,222] - [remote_util] [140687446664960] - INFO - Starting couchbase server [2021-07-27 22:28:37,324] - [remote_util] [140687446664960] - INFO - Running systemd command on this server [2021-07-27 22:28:37,325] - [remote_util] [140687446664960] - INFO - running command.raw on 172.23.123.103: systemctl start couchbase-server.service [2021-07-27 22:28:37,349] - [remote_util] [140687446664960] - INFO - command executed successfully with root [2021-07-27 22:28:37,351] - [remote_util] [140687446664960] - INFO - 172.23.123.103:sleep for 5 secs. waiting for couchbase server to come up ... [2021-07-27 22:28:42,355] - [remote_util] [140687446664960] - INFO - running command.raw on 172.23.123.103: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-07-27 22:28:42,385] - [remote_util] [140687446664960] - INFO - command executed successfully with root [2021-07-27 22:28:42,387] - [remote_util] [140687446664960] - INFO - Couchbase server status: [' Process: 105974 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-07-27 22:28:42,387] - [remote_util] [140687446664960] - INFO - Checking for process beam.smp on linux [2021-07-27 22:28:42,455] - [remote_util] [140687446664960] - INFO - process beam.smp is running on 172.23.123.103: with pid 106117 [2021-07-27 22:28:42,455] - [basetestcase] [140687446664960] - INFO - Couchbase started [2021-07-27 22:28:42,458] - [remote_util] [140687446664960] - INFO - SSH Connecting to 172.23.123.151 with username:root, attempt#1 of 5 [2021-07-27 22:28:42,572] - [remote_util] [140687446664960] - INFO - SSH Connected to 172.23.123.151 as root [2021-07-27 22:28:42,869] - [remote_util] [140687446664960] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-07-27 22:28:43,190] - [remote_util] [140687446664960] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-07-27 22:28:43,203] - [remote_util] [140687446664960] - INFO - SSH Connecting to 172.23.123.151 with username:root, attempt#1 of 5 [2021-07-27 22:28:43,315] - [remote_util] [140687446664960] - INFO - SSH Connected to 172.23.123.151 as root [2021-07-27 22:28:43,611] - [remote_util] [140687446664960] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-07-27 22:28:43,975] - [remote_util] [140687446664960] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-07-27 22:28:44,043] - [remote_util] [140687446664960] - INFO - 172.23.123.151 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-07-27 22:28:44,201] - [remote_util] [140687446664960] - INFO - Running systemd command on this server [2021-07-27 22:28:44,201] - [remote_util] [140687446664960] - INFO - running command.raw on 172.23.123.151: systemctl stop couchbase-server.service [2021-07-27 22:28:46,326] - [remote_util] [140687446664960] - INFO - command executed successfully with root [2021-07-27 22:28:46,326] - [basetestcase] [140687446664960] - INFO - Couchbase stopped [2021-07-27 22:28:46,327] - [basetestcase] [140687446664960] - INFO - sleep for 5 secs. ... [2021-07-27 22:28:51,330] - [remote_util] [140687446664960] - INFO - running command.raw on 172.23.123.151: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-07-27 22:28:51,352] - [remote_util] [140687446664960] - INFO - command executed successfully with root [2021-07-27 22:28:51,353] - [remote_util] [140687446664960] - INFO - running command.raw on 172.23.123.151: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-07-27 22:28:51,409] - [remote_util] [140687446664960] - INFO - command executed successfully with root [2021-07-27 22:28:51,412] - [remote_util] [140687446664960] - INFO - SSH Connecting to 172.23.123.151 with username:root, attempt#1 of 5 [2021-07-27 22:28:51,521] - [remote_util] [140687446664960] - INFO - SSH Connected to 172.23.123.151 as root [2021-07-27 22:28:51,819] - [remote_util] [140687446664960] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-07-27 22:28:52,141] - [remote_util] [140687446664960] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-07-27 22:28:52,206] - [remote_util] [140687446664960] - INFO - 172.23.123.151 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-07-27 22:28:52,206] - [remote_util] [140687446664960] - INFO - Checking for process beam.smp on linux [2021-07-27 22:28:52,289] - [remote_util] [140687446664960] - INFO - Starting couchbase server [2021-07-27 22:28:52,401] - [remote_util] [140687446664960] - INFO - Running systemd command on this server [2021-07-27 22:28:52,401] - [remote_util] [140687446664960] - INFO - running command.raw on 172.23.123.151: systemctl start couchbase-server.service [2021-07-27 22:28:52,428] - [remote_util] [140687446664960] - INFO - command executed successfully with root [2021-07-27 22:28:52,429] - [remote_util] [140687446664960] - INFO - 172.23.123.151:sleep for 5 secs. waiting for couchbase server to come up ... [2021-07-27 22:28:57,434] - [remote_util] [140687446664960] - INFO - running command.raw on 172.23.123.151: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-07-27 22:28:57,467] - [remote_util] [140687446664960] - INFO - command executed successfully with root [2021-07-27 22:28:57,468] - [remote_util] [140687446664960] - INFO - Couchbase server status: [' Process: 9240 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-07-27 22:28:57,468] - [remote_util] [140687446664960] - INFO - Checking for process beam.smp on linux [2021-07-27 22:28:57,539] - [remote_util] [140687446664960] - INFO - process beam.smp is running on 172.23.123.151: with pid 9382 [2021-07-27 22:28:57,539] - [basetestcase] [140687446664960] - INFO - Couchbase started [2021-07-27 22:28:57,542] - [remote_util] [140687446664960] - INFO - SSH Connecting to 172.23.123.147 with username:root, attempt#1 of 5 [2021-07-27 22:28:57,653] - [remote_util] [140687446664960] - INFO - SSH Connected to 172.23.123.147 as root [2021-07-27 22:28:57,975] - [remote_util] [140687446664960] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-07-27 22:28:58,296] - [remote_util] [140687446664960] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-07-27 22:28:58,307] - [remote_util] [140687446664960] - INFO - SSH Connecting to 172.23.123.147 with username:root, attempt#1 of 5 [2021-07-27 22:28:58,416] - [remote_util] [140687446664960] - INFO - SSH Connected to 172.23.123.147 as root [2021-07-27 22:28:58,726] - [remote_util] [140687446664960] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-07-27 22:28:59,055] - [remote_util] [140687446664960] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-07-27 22:28:59,121] - [remote_util] [140687446664960] - INFO - 172.23.123.147 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-07-27 22:28:59,189] - [remote_util] [140687446664960] - INFO - Running systemd command on this server [2021-07-27 22:28:59,189] - [remote_util] [140687446664960] - INFO - running command.raw on 172.23.123.147: systemctl stop couchbase-server.service [2021-07-27 22:29:01,307] - [remote_util] [140687446664960] - INFO - command executed successfully with root [2021-07-27 22:29:01,308] - [basetestcase] [140687446664960] - INFO - Couchbase stopped [2021-07-27 22:29:01,309] - [basetestcase] [140687446664960] - INFO - sleep for 5 secs. ... [2021-07-27 22:29:06,311] - [remote_util] [140687446664960] - INFO - running command.raw on 172.23.123.147: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-07-27 22:29:06,334] - [remote_util] [140687446664960] - INFO - command executed successfully with root [2021-07-27 22:29:06,335] - [remote_util] [140687446664960] - INFO - running command.raw on 172.23.123.147: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-07-27 22:29:06,394] - [remote_util] [140687446664960] - INFO - command executed successfully with root [2021-07-27 22:29:06,396] - [remote_util] [140687446664960] - INFO - SSH Connecting to 172.23.123.147 with username:root, attempt#1 of 5 [2021-07-27 22:29:06,506] - [remote_util] [140687446664960] - INFO - SSH Connected to 172.23.123.147 as root [2021-07-27 22:29:06,825] - [remote_util] [140687446664960] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-07-27 22:29:07,190] - [remote_util] [140687446664960] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-07-27 22:29:07,257] - [remote_util] [140687446664960] - INFO - 172.23.123.147 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-07-27 22:29:07,257] - [remote_util] [140687446664960] - INFO - Checking for process beam.smp on linux [2021-07-27 22:29:07,342] - [remote_util] [140687446664960] - INFO - Starting couchbase server [2021-07-27 22:29:07,457] - [remote_util] [140687446664960] - INFO - Running systemd command on this server [2021-07-27 22:29:07,458] - [remote_util] [140687446664960] - INFO - running command.raw on 172.23.123.147: systemctl start couchbase-server.service [2021-07-27 22:29:07,485] - [remote_util] [140687446664960] - INFO - command executed successfully with root [2021-07-27 22:29:07,485] - [remote_util] [140687446664960] - INFO - 172.23.123.147:sleep for 5 secs. waiting for couchbase server to come up ... [2021-07-27 22:29:12,491] - [remote_util] [140687446664960] - INFO - running command.raw on 172.23.123.147: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-07-27 22:29:12,525] - [remote_util] [140687446664960] - INFO - command executed successfully with root [2021-07-27 22:29:12,526] - [remote_util] [140687446664960] - INFO - Couchbase server status: [' Process: 118799 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-07-27 22:29:12,526] - [remote_util] [140687446664960] - INFO - Checking for process beam.smp on linux [2021-07-27 22:29:12,602] - [remote_util] [140687446664960] - INFO - process beam.smp is running on 172.23.123.147: with pid 118986 [2021-07-27 22:29:12,602] - [basetestcase] [140687446664960] - INFO - Couchbase started [2021-07-27 22:29:12,605] - [remote_util] [140687446664960] - INFO - SSH Connecting to 172.23.123.146 with username:root, attempt#1 of 5 [2021-07-27 22:29:12,719] - [remote_util] [140687446664960] - INFO - SSH Connected to 172.23.123.146 as root [2021-07-27 22:29:13,037] - [remote_util] [140687446664960] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-07-27 22:29:13,369] - [remote_util] [140687446664960] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-07-27 22:29:13,381] - [remote_util] [140687446664960] - INFO - SSH Connecting to 172.23.123.146 with username:root, attempt#1 of 5 [2021-07-27 22:29:13,489] - [remote_util] [140687446664960] - INFO - SSH Connected to 172.23.123.146 as root [2021-07-27 22:29:13,788] - [remote_util] [140687446664960] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-07-27 22:29:14,119] - [remote_util] [140687446664960] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-07-27 22:29:14,183] - [remote_util] [140687446664960] - INFO - 172.23.123.146 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-07-27 22:29:14,294] - [remote_util] [140687446664960] - INFO - Running systemd command on this server [2021-07-27 22:29:14,294] - [remote_util] [140687446664960] - INFO - running command.raw on 172.23.123.146: systemctl stop couchbase-server.service [2021-07-27 22:29:16,319] - [remote_util] [140687446664960] - INFO - command executed successfully with root [2021-07-27 22:29:16,320] - [basetestcase] [140687446664960] - INFO - Couchbase stopped [2021-07-27 22:29:16,320] - [basetestcase] [140687446664960] - INFO - sleep for 5 secs. ... [2021-07-27 22:29:21,323] - [remote_util] [140687446664960] - INFO - running command.raw on 172.23.123.146: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-07-27 22:29:21,343] - [remote_util] [140687446664960] - INFO - command executed successfully with root [2021-07-27 22:29:21,343] - [remote_util] [140687446664960] - INFO - running command.raw on 172.23.123.146: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-07-27 22:29:21,399] - [remote_util] [140687446664960] - INFO - command executed successfully with root [2021-07-27 22:29:21,402] - [remote_util] [140687446664960] - INFO - SSH Connecting to 172.23.123.146 with username:root, attempt#1 of 5 [2021-07-27 22:29:21,516] - [remote_util] [140687446664960] - INFO - SSH Connected to 172.23.123.146 as root [2021-07-27 22:29:21,823] - [remote_util] [140687446664960] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-07-27 22:29:22,140] - [remote_util] [140687446664960] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-07-27 22:29:22,208] - [remote_util] [140687446664960] - INFO - 172.23.123.146 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-07-27 22:29:22,208] - [remote_util] [140687446664960] - INFO - Checking for process beam.smp on linux [2021-07-27 22:29:22,294] - [remote_util] [140687446664960] - INFO - Starting couchbase server [2021-07-27 22:29:22,406] - [remote_util] [140687446664960] - INFO - Running systemd command on this server [2021-07-27 22:29:22,406] - [remote_util] [140687446664960] - INFO - running command.raw on 172.23.123.146: systemctl start couchbase-server.service [2021-07-27 22:29:22,433] - [remote_util] [140687446664960] - INFO - command executed successfully with root [2021-07-27 22:29:22,434] - [remote_util] [140687446664960] - INFO - 172.23.123.146:sleep for 5 secs. waiting for couchbase server to come up ... [2021-07-27 22:29:27,436] - [remote_util] [140687446664960] - INFO - running command.raw on 172.23.123.146: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-07-27 22:29:27,465] - [remote_util] [140687446664960] - INFO - command executed successfully with root [2021-07-27 22:29:27,466] - [remote_util] [140687446664960] - INFO - Couchbase server status: [' Process: 16463 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-07-27 22:29:27,467] - [remote_util] [140687446664960] - INFO - Checking for process beam.smp on linux [2021-07-27 22:29:27,536] - [remote_util] [140687446664960] - INFO - process beam.smp is running on 172.23.123.146: with pid 16605 [2021-07-27 22:29:27,537] - [basetestcase] [140687446664960] - INFO - Couchbase started [2021-07-27 22:29:27,537] - [basetestcase] [140687446664960] - INFO - sleep for 10 secs. ... [2021-07-27 22:29:37,565] - [eventing_base] [140687446664960] - INFO - ============== EventingBaseTest tearDown has completed ============== [2021-07-27 22:29:37,574] - [remote_util] [140687446664960] - INFO - SSH Connecting to 172.23.123.103 with username:root, attempt#1 of 5 [2021-07-27 22:29:37,580] - [remote_util] [140687471843072] - INFO - SSH Connecting to 172.23.123.151 with username:root, attempt#1 of 5 [2021-07-27 22:29:37,585] - [remote_util] [140687064418048] - INFO - SSH Connecting to 172.23.123.146 with username:root, attempt#1 of 5 [2021-07-27 22:29:37,589] - [remote_util] [140687455057664] - INFO - SSH Connecting to 172.23.123.147 with username:root, attempt#1 of 5 [2021-07-27 22:29:37,686] - [remote_util] [140687446664960] - INFO - SSH Connected to 172.23.123.103 as root [2021-07-27 22:29:37,694] - [remote_util] [140687471843072] - INFO - SSH Connected to 172.23.123.151 as root [2021-07-27 22:29:37,709] - [remote_util] [140687064418048] - INFO - SSH Connected to 172.23.123.146 as root [2021-07-27 22:29:37,713] - [remote_util] [140687455057664] - INFO - SSH Connected to 172.23.123.147 as root [2021-07-27 22:29:37,960] - [remote_util] [140687446664960] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-07-27 22:29:38,004] - [remote_util] [140687471843072] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-07-27 22:29:38,036] - [remote_util] [140687064418048] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-07-27 22:29:38,054] - [remote_util] [140687455057664] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-07-27 22:29:38,324] - [remote_util] [140687446664960] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-07-27 22:29:38,330] - [remote_util] [140687446664960] - INFO - running command.raw on 172.23.123.103: /opt/couchbase/bin/cbcollect_info 172.23.123.103-20210727-2229-diag.zip [2021-07-27 22:29:38,374] - [remote_util] [140687064418048] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-07-27 22:29:38,376] - [remote_util] [140687064418048] - INFO - running command.raw on 172.23.123.146: /opt/couchbase/bin/cbcollect_info 172.23.123.146-20210727-2229-diag.zip [2021-07-27 22:29:38,395] - [remote_util] [140687471843072] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-07-27 22:29:38,397] - [remote_util] [140687471843072] - INFO - running command.raw on 172.23.123.151: /opt/couchbase/bin/cbcollect_info 172.23.123.151-20210727-2229-diag.zip [2021-07-27 22:29:38,442] - [remote_util] [140687455057664] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-07-27 22:29:38,443] - [remote_util] [140687455057664] - INFO - running command.raw on 172.23.123.147: /opt/couchbase/bin/cbcollect_info 172.23.123.147-20210727-2229-diag.zip [2021-07-27 22:31:31,420] - [remote_util] [140687064418048] - INFO - command executed successfully with root [2021-07-27 22:31:31,778] - [remote_util] [140687064418048] - INFO - running command.raw on 172.23.123.146: rm -f /root/172.23.123.146-20210727-2229-diag.zip [2021-07-27 22:31:31,834] - [remote_util] [140687064418048] - INFO - command executed successfully with root [2021-07-27 22:31:32,124] - [remote_util] [140687471843072] - INFO - command executed successfully with root [2021-07-27 22:31:32,486] - [remote_util] [140687471843072] - INFO - running command.raw on 172.23.123.151: rm -f /root/172.23.123.151-20210727-2229-diag.zip [2021-07-27 22:31:32,541] - [remote_util] [140687471843072] - INFO - command executed successfully with root [2021-07-27 22:31:32,819] - [remote_util] [140687446664960] - INFO - command executed successfully with root [2021-07-27 22:31:33,395] - [remote_util] [140687446664960] - INFO - running command.raw on 172.23.123.103: rm -f /root/172.23.123.103-20210727-2229-diag.zip [2021-07-27 22:31:33,453] - [remote_util] [140687446664960] - INFO - command executed successfully with root [2021-07-27 22:31:33,477] - [remote_util] [140687455057664] - INFO - command executed successfully with root [2021-07-27 22:31:33,844] - [remote_util] [140687455057664] - INFO - running command.raw on 172.23.123.147: rm -f /root/172.23.123.147-20210727-2229-diag.zip [2021-07-27 22:31:33,900] - [remote_util] [140687455057664] - INFO - command executed successfully with root