[2021-02-12 23:30:35,706] - [eventing_base] [140096917202688] - INFO - Starting Test: test_eventing_rebalance_out_delete_recreate_collections [2021-02-12 23:30:35,707] - [remote_util] [140096917202688] - INFO - SSH Connecting to 172.23.123.61 with username:root, attempt#1 of 5 [2021-02-12 23:30:35,809] - [remote_util] [140096917202688] - INFO - SSH Connected to 172.23.123.61 as root [2021-02-12 23:30:36,074] - [remote_util] [140096917202688] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:30:36,369] - [remote_util] [140096917202688] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:30:36,374] - [rest_client] [140096917202688] - INFO - -->is_ns_server_running? [2021-02-12 23:30:36,382] - [rest_client] [140096917202688] - INFO - Node version in cluster 7.0.0-4454-enterprise [2021-02-12 23:30:36,388] - [rest_client] [140096917202688] - ERROR - GET http://172.23.123.61:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-02-12 23:30:36,388] - [rest_client] [140096917202688] - INFO - Node versions in cluster [] [2021-02-12 23:30:36,388] - [basetestcase] [140096917202688] - INFO - ============== basetestcase setup was started for test #10 test_eventing_rebalance_out_delete_recreate_collections============== [2021-02-12 23:30:36,392] - [rest_client] [140096917202688] - ERROR - GET http://172.23.123.61:8091/pools/default/ body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-02-12 23:30:36,405] - [bucket_helper] [140096917202688] - INFO - Could not find any buckets for node 172.23.123.61, nothing to delete [2021-02-12 23:30:36,415] - [bucket_helper] [140096917202688] - INFO - Could not find any buckets for node 172.23.123.62, nothing to delete [2021-02-12 23:30:36,426] - [bucket_helper] [140096917202688] - INFO - Could not find any buckets for node 172.23.122.90, nothing to delete [2021-02-12 23:30:36,436] - [bucket_helper] [140096917202688] - INFO - Could not find any buckets for node 172.23.122.117, nothing to delete [2021-02-12 23:30:36,445] - [bucket_helper] [140096917202688] - INFO - Could not find any buckets for node 172.23.122.97, nothing to delete [2021-02-12 23:30:36,455] - [bucket_helper] [140096917202688] - INFO - Could not find any buckets for node 172.23.122.82, nothing to delete [2021-02-12 23:30:36,465] - [bucket_helper] [140096917202688] - INFO - Could not find any buckets for node 172.23.122.118, nothing to delete [2021-02-12 23:30:36,475] - [bucket_helper] [140096917202688] - INFO - Could not find any buckets for node 172.23.122.116, nothing to delete [2021-02-12 23:30:36,478] - [rest_client] [140096917202688] - INFO - -->is_ns_server_running? [2021-02-12 23:30:36,488] - [cluster_helper] [140096917202688] - INFO - waiting for ns_server @ 172.23.123.61:8091 [2021-02-12 23:30:36,488] - [rest_client] [140096917202688] - INFO - -->is_ns_server_running? [2021-02-12 23:30:36,490] - [cluster_helper] [140096917202688] - INFO - ns_server @ 172.23.123.61:8091 is running [2021-02-12 23:30:36,493] - [cluster_helper] [140096917202688] - INFO - waiting for ns_server @ 172.23.123.62:8091 [2021-02-12 23:30:36,493] - [rest_client] [140096917202688] - INFO - -->is_ns_server_running? [2021-02-12 23:30:36,496] - [cluster_helper] [140096917202688] - INFO - ns_server @ 172.23.123.62:8091 is running [2021-02-12 23:30:36,499] - [cluster_helper] [140096917202688] - INFO - waiting for ns_server @ 172.23.122.90:8091 [2021-02-12 23:30:36,499] - [rest_client] [140096917202688] - INFO - -->is_ns_server_running? [2021-02-12 23:30:36,502] - [cluster_helper] [140096917202688] - INFO - ns_server @ 172.23.122.90:8091 is running [2021-02-12 23:30:36,507] - [cluster_helper] [140096917202688] - INFO - waiting for ns_server @ 172.23.122.117:8091 [2021-02-12 23:30:36,507] - [rest_client] [140096917202688] - INFO - -->is_ns_server_running? [2021-02-12 23:30:36,510] - [cluster_helper] [140096917202688] - INFO - ns_server @ 172.23.122.117:8091 is running [2021-02-12 23:30:36,513] - [cluster_helper] [140096917202688] - INFO - waiting for ns_server @ 172.23.122.97:8091 [2021-02-12 23:30:36,513] - [rest_client] [140096917202688] - INFO - -->is_ns_server_running? [2021-02-12 23:30:36,515] - [cluster_helper] [140096917202688] - INFO - ns_server @ 172.23.122.97:8091 is running [2021-02-12 23:30:36,519] - [cluster_helper] [140096917202688] - INFO - waiting for ns_server @ 172.23.122.82:8091 [2021-02-12 23:30:36,519] - [rest_client] [140096917202688] - INFO - -->is_ns_server_running? [2021-02-12 23:30:36,522] - [cluster_helper] [140096917202688] - INFO - ns_server @ 172.23.122.82:8091 is running [2021-02-12 23:30:36,525] - [cluster_helper] [140096917202688] - INFO - waiting for ns_server @ 172.23.122.118:8091 [2021-02-12 23:30:36,525] - [rest_client] [140096917202688] - INFO - -->is_ns_server_running? [2021-02-12 23:30:36,527] - [cluster_helper] [140096917202688] - INFO - ns_server @ 172.23.122.118:8091 is running [2021-02-12 23:30:36,530] - [cluster_helper] [140096917202688] - INFO - waiting for ns_server @ 172.23.122.116:8091 [2021-02-12 23:30:36,530] - [rest_client] [140096917202688] - INFO - -->is_ns_server_running? [2021-02-12 23:30:36,533] - [cluster_helper] [140096917202688] - INFO - ns_server @ 172.23.122.116:8091 is running [2021-02-12 23:30:36,533] - [basetestcase] [140096917202688] - INFO - Building docker image with java sdk client [2021-02-12 23:30:40,154] - [basetestcase] [140096917202688] - INFO - initializing cluster [2021-02-12 23:30:40,157] - [remote_util] [140096917202688] - INFO - SSH Connecting to 172.23.123.61 with username:root, attempt#1 of 5 [2021-02-12 23:30:40,257] - [remote_util] [140096917202688] - INFO - SSH Connected to 172.23.123.61 as root [2021-02-12 23:30:40,514] - [remote_util] [140096917202688] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:30:40,809] - [remote_util] [140096917202688] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:30:40,817] - [remote_util] [140096917202688] - INFO - SSH Connecting to 172.23.123.61 with username:root, attempt#1 of 5 [2021-02-12 23:30:40,917] - [remote_util] [140096917202688] - INFO - SSH Connected to 172.23.123.61 as root [2021-02-12 23:30:41,170] - [remote_util] [140096917202688] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:30:41,465] - [remote_util] [140096917202688] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:30:41,484] - [remote_util] [140096917202688] - INFO - 172.23.123.61 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 23:30:41,537] - [remote_util] [140096917202688] - INFO - Running systemd command on this server [2021-02-12 23:30:41,537] - [remote_util] [140096917202688] - INFO - running command.raw on 172.23.123.61: systemctl stop couchbase-server.service [2021-02-12 23:30:43,189] - [remote_util] [140096917202688] - INFO - command executed successfully with root [2021-02-12 23:30:43,189] - [basetestcase] [140096917202688] - INFO - Couchbase stopped [2021-02-12 23:30:43,189] - [basetestcase] [140096917202688] - INFO - sleep for 5 secs. ... [2021-02-12 23:30:48,195] - [remote_util] [140096917202688] - INFO - running command.raw on 172.23.123.61: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-12 23:30:48,209] - [remote_util] [140096917202688] - INFO - command executed successfully with root [2021-02-12 23:30:48,209] - [remote_util] [140096917202688] - INFO - running command.raw on 172.23.123.61: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-12 23:30:48,262] - [remote_util] [140096917202688] - INFO - command executed successfully with root [2021-02-12 23:30:48,264] - [remote_util] [140096917202688] - INFO - SSH Connecting to 172.23.123.61 with username:root, attempt#1 of 5 [2021-02-12 23:30:48,363] - [remote_util] [140096917202688] - INFO - SSH Connected to 172.23.123.61 as root [2021-02-12 23:30:48,625] - [remote_util] [140096917202688] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:30:48,931] - [remote_util] [140096917202688] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:30:48,988] - [remote_util] [140096917202688] - INFO - 172.23.123.61 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 23:30:48,989] - [remote_util] [140096917202688] - INFO - Checking for process beam.smp on linux [2021-02-12 23:30:49,023] - [remote_util] [140096917202688] - INFO - Starting couchbase server [2021-02-12 23:30:49,117] - [remote_util] [140096917202688] - INFO - Running systemd command on this server [2021-02-12 23:30:49,117] - [remote_util] [140096917202688] - INFO - running command.raw on 172.23.123.61: systemctl start couchbase-server.service [2021-02-12 23:30:49,139] - [remote_util] [140096917202688] - INFO - command executed successfully with root [2021-02-12 23:30:49,139] - [remote_util] [140096917202688] - INFO - 172.23.123.61:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-12 23:30:54,143] - [remote_util] [140096917202688] - INFO - running command.raw on 172.23.123.61: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-12 23:30:54,167] - [remote_util] [140096917202688] - INFO - command executed successfully with root [2021-02-12 23:30:54,167] - [remote_util] [140096917202688] - INFO - Couchbase server status: [' Process: 14359 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-12 23:30:54,167] - [remote_util] [140096917202688] - INFO - Checking for process beam.smp on linux [2021-02-12 23:30:54,234] - [remote_util] [140096917202688] - INFO - process beam.smp is running on 172.23.123.61: with pid 14504 [2021-02-12 23:30:54,235] - [basetestcase] [140096917202688] - INFO - Couchbase started [2021-02-12 23:30:54,236] - [remote_util] [140096917202688] - INFO - SSH Connecting to 172.23.123.62 with username:root, attempt#1 of 5 [2021-02-12 23:30:54,342] - [remote_util] [140096917202688] - INFO - SSH Connected to 172.23.123.62 as root [2021-02-12 23:30:54,623] - [remote_util] [140096917202688] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:30:54,924] - [remote_util] [140096917202688] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:30:54,932] - [remote_util] [140096917202688] - INFO - SSH Connecting to 172.23.123.62 with username:root, attempt#1 of 5 [2021-02-12 23:30:55,033] - [remote_util] [140096917202688] - INFO - SSH Connected to 172.23.123.62 as root [2021-02-12 23:30:55,314] - [remote_util] [140096917202688] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:30:55,620] - [remote_util] [140096917202688] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:30:55,640] - [remote_util] [140096917202688] - INFO - 172.23.123.62 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 23:30:55,697] - [remote_util] [140096917202688] - INFO - Running systemd command on this server [2021-02-12 23:30:55,697] - [remote_util] [140096917202688] - INFO - running command.raw on 172.23.123.62: systemctl stop couchbase-server.service [2021-02-12 23:30:57,281] - [remote_util] [140096917202688] - INFO - command executed successfully with root [2021-02-12 23:30:57,281] - [basetestcase] [140096917202688] - INFO - Couchbase stopped [2021-02-12 23:30:57,282] - [basetestcase] [140096917202688] - INFO - sleep for 5 secs. ... [2021-02-12 23:31:02,287] - [remote_util] [140096917202688] - INFO - running command.raw on 172.23.123.62: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-12 23:31:02,303] - [remote_util] [140096917202688] - INFO - command executed successfully with root [2021-02-12 23:31:02,303] - [remote_util] [140096917202688] - INFO - running command.raw on 172.23.123.62: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-12 23:31:02,356] - [remote_util] [140096917202688] - INFO - command executed successfully with root [2021-02-12 23:31:02,358] - [remote_util] [140096917202688] - INFO - SSH Connecting to 172.23.123.62 with username:root, attempt#1 of 5 [2021-02-12 23:31:02,460] - [remote_util] [140096917202688] - INFO - SSH Connected to 172.23.123.62 as root [2021-02-12 23:31:02,737] - [remote_util] [140096917202688] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:31:03,036] - [remote_util] [140096917202688] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:31:03,094] - [remote_util] [140096917202688] - INFO - 172.23.123.62 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 23:31:03,094] - [remote_util] [140096917202688] - INFO - Checking for process beam.smp on linux [2021-02-12 23:31:03,132] - [remote_util] [140096917202688] - INFO - Starting couchbase server [2021-02-12 23:31:03,227] - [remote_util] [140096917202688] - INFO - Running systemd command on this server [2021-02-12 23:31:03,227] - [remote_util] [140096917202688] - INFO - running command.raw on 172.23.123.62: systemctl start couchbase-server.service [2021-02-12 23:31:03,249] - [remote_util] [140096917202688] - INFO - command executed successfully with root [2021-02-12 23:31:03,249] - [remote_util] [140096917202688] - INFO - 172.23.123.62:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-12 23:31:08,255] - [remote_util] [140096917202688] - INFO - running command.raw on 172.23.123.62: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-12 23:31:08,279] - [remote_util] [140096917202688] - INFO - command executed successfully with root [2021-02-12 23:31:08,279] - [remote_util] [140096917202688] - INFO - Couchbase server status: [' Process: 61219 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-12 23:31:08,280] - [remote_util] [140096917202688] - INFO - Checking for process beam.smp on linux [2021-02-12 23:31:08,348] - [remote_util] [140096917202688] - INFO - process beam.smp is running on 172.23.123.62: with pid 61361 [2021-02-12 23:31:08,348] - [basetestcase] [140096917202688] - INFO - Couchbase started [2021-02-12 23:31:08,350] - [remote_util] [140096917202688] - INFO - SSH Connecting to 172.23.122.90 with username:root, attempt#1 of 5 [2021-02-12 23:31:08,450] - [remote_util] [140096917202688] - INFO - SSH Connected to 172.23.122.90 as root [2021-02-12 23:31:08,740] - [remote_util] [140096917202688] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:31:09,037] - [remote_util] [140096917202688] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:31:09,045] - [remote_util] [140096917202688] - INFO - SSH Connecting to 172.23.122.90 with username:root, attempt#1 of 5 [2021-02-12 23:31:09,142] - [remote_util] [140096917202688] - INFO - SSH Connected to 172.23.122.90 as root [2021-02-12 23:31:09,567] - [remote_util] [140096917202688] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:31:09,863] - [remote_util] [140096917202688] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:31:09,922] - [remote_util] [140096917202688] - INFO - 172.23.122.90 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 23:31:09,975] - [remote_util] [140096917202688] - INFO - Running systemd command on this server [2021-02-12 23:31:09,975] - [remote_util] [140096917202688] - INFO - running command.raw on 172.23.122.90: systemctl stop couchbase-server.service [2021-02-12 23:31:11,801] - [remote_util] [140096917202688] - INFO - command executed successfully with root [2021-02-12 23:31:11,801] - [basetestcase] [140096917202688] - INFO - Couchbase stopped [2021-02-12 23:31:11,801] - [basetestcase] [140096917202688] - INFO - sleep for 5 secs. ... [2021-02-12 23:31:16,807] - [remote_util] [140096917202688] - INFO - running command.raw on 172.23.122.90: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-12 23:31:16,821] - [remote_util] [140096917202688] - INFO - command executed successfully with root [2021-02-12 23:31:16,821] - [remote_util] [140096917202688] - INFO - running command.raw on 172.23.122.90: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-12 23:31:16,874] - [remote_util] [140096917202688] - INFO - command executed successfully with root [2021-02-12 23:31:16,875] - [remote_util] [140096917202688] - INFO - SSH Connecting to 172.23.122.90 with username:root, attempt#1 of 5 [2021-02-12 23:31:16,972] - [remote_util] [140096917202688] - INFO - SSH Connected to 172.23.122.90 as root [2021-02-12 23:31:17,242] - [remote_util] [140096917202688] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:31:17,530] - [remote_util] [140096917202688] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:31:17,549] - [remote_util] [140096917202688] - INFO - 172.23.122.90 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 23:31:17,549] - [remote_util] [140096917202688] - INFO - Checking for process beam.smp on linux [2021-02-12 23:31:17,586] - [remote_util] [140096917202688] - INFO - Starting couchbase server [2021-02-12 23:31:17,676] - [remote_util] [140096917202688] - INFO - Running systemd command on this server [2021-02-12 23:31:17,676] - [remote_util] [140096917202688] - INFO - running command.raw on 172.23.122.90: systemctl start couchbase-server.service [2021-02-12 23:31:17,697] - [remote_util] [140096917202688] - INFO - command executed successfully with root [2021-02-12 23:31:17,697] - [remote_util] [140096917202688] - INFO - 172.23.122.90:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-12 23:31:22,702] - [remote_util] [140096917202688] - INFO - running command.raw on 172.23.122.90: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-12 23:31:22,748] - [remote_util] [140096917202688] - INFO - command executed successfully with root [2021-02-12 23:31:22,749] - [remote_util] [140096917202688] - INFO - Couchbase server status: [' Process: 12200 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-12 23:31:22,749] - [remote_util] [140096917202688] - INFO - Checking for process beam.smp on linux [2021-02-12 23:31:22,816] - [remote_util] [140096917202688] - INFO - process beam.smp is running on 172.23.122.90: with pid 12345 [2021-02-12 23:31:22,816] - [basetestcase] [140096917202688] - INFO - Couchbase started [2021-02-12 23:31:22,818] - [remote_util] [140096917202688] - INFO - SSH Connecting to 172.23.122.117 with username:root, attempt#1 of 5 [2021-02-12 23:31:22,921] - [remote_util] [140096917202688] - INFO - SSH Connected to 172.23.122.117 as root [2021-02-12 23:31:23,181] - [remote_util] [140096917202688] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:31:23,472] - [remote_util] [140096917202688] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:31:23,480] - [remote_util] [140096917202688] - INFO - SSH Connecting to 172.23.122.117 with username:root, attempt#1 of 5 [2021-02-12 23:31:23,577] - [remote_util] [140096917202688] - INFO - SSH Connected to 172.23.122.117 as root [2021-02-12 23:31:23,827] - [remote_util] [140096917202688] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:31:24,128] - [remote_util] [140096917202688] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:31:24,147] - [remote_util] [140096917202688] - INFO - 172.23.122.117 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 23:31:24,201] - [remote_util] [140096917202688] - INFO - Running systemd command on this server [2021-02-12 23:31:24,201] - [remote_util] [140096917202688] - INFO - running command.raw on 172.23.122.117: systemctl stop couchbase-server.service [2021-02-12 23:31:25,853] - [remote_util] [140096917202688] - INFO - command executed successfully with root [2021-02-12 23:31:25,853] - [basetestcase] [140096917202688] - INFO - Couchbase stopped [2021-02-12 23:31:25,854] - [basetestcase] [140096917202688] - INFO - sleep for 5 secs. ... [2021-02-12 23:31:30,857] - [remote_util] [140096917202688] - INFO - running command.raw on 172.23.122.117: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-12 23:31:30,873] - [remote_util] [140096917202688] - INFO - command executed successfully with root [2021-02-12 23:31:30,873] - [remote_util] [140096917202688] - INFO - running command.raw on 172.23.122.117: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-12 23:31:30,926] - [remote_util] [140096917202688] - INFO - command executed successfully with root [2021-02-12 23:31:30,927] - [remote_util] [140096917202688] - INFO - SSH Connecting to 172.23.122.117 with username:root, attempt#1 of 5 [2021-02-12 23:31:31,027] - [remote_util] [140096917202688] - INFO - SSH Connected to 172.23.122.117 as root [2021-02-12 23:31:31,291] - [remote_util] [140096917202688] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:31:31,588] - [remote_util] [140096917202688] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:31:31,609] - [remote_util] [140096917202688] - INFO - 172.23.122.117 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 23:31:31,609] - [remote_util] [140096917202688] - INFO - Checking for process beam.smp on linux [2021-02-12 23:31:31,646] - [remote_util] [140096917202688] - INFO - Starting couchbase server [2021-02-12 23:31:31,741] - [remote_util] [140096917202688] - INFO - Running systemd command on this server [2021-02-12 23:31:31,741] - [remote_util] [140096917202688] - INFO - running command.raw on 172.23.122.117: systemctl start couchbase-server.service [2021-02-12 23:31:31,762] - [remote_util] [140096917202688] - INFO - command executed successfully with root [2021-02-12 23:31:31,762] - [remote_util] [140096917202688] - INFO - 172.23.122.117:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-12 23:31:36,763] - [remote_util] [140096917202688] - INFO - running command.raw on 172.23.122.117: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-12 23:31:36,787] - [remote_util] [140096917202688] - INFO - command executed successfully with root [2021-02-12 23:31:36,788] - [remote_util] [140096917202688] - INFO - Couchbase server status: [' Process: 12805 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-12 23:31:36,788] - [remote_util] [140096917202688] - INFO - Checking for process beam.smp on linux [2021-02-12 23:31:36,855] - [remote_util] [140096917202688] - INFO - process beam.smp is running on 172.23.122.117: with pid 12949 [2021-02-12 23:31:36,856] - [basetestcase] [140096917202688] - INFO - Couchbase started [2021-02-12 23:31:36,857] - [remote_util] [140096917202688] - INFO - SSH Connecting to 172.23.122.97 with username:root, attempt#1 of 5 [2021-02-12 23:31:36,962] - [remote_util] [140096917202688] - INFO - SSH Connected to 172.23.122.97 as root [2021-02-12 23:31:37,228] - [remote_util] [140096917202688] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:31:37,532] - [remote_util] [140096917202688] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:31:37,541] - [remote_util] [140096917202688] - INFO - SSH Connecting to 172.23.122.97 with username:root, attempt#1 of 5 [2021-02-12 23:31:37,640] - [remote_util] [140096917202688] - INFO - SSH Connected to 172.23.122.97 as root [2021-02-12 23:31:37,899] - [remote_util] [140096917202688] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:31:38,200] - [remote_util] [140096917202688] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:31:38,220] - [remote_util] [140096917202688] - INFO - 172.23.122.97 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 23:31:38,276] - [remote_util] [140096917202688] - INFO - Running systemd command on this server [2021-02-12 23:31:38,276] - [remote_util] [140096917202688] - INFO - running command.raw on 172.23.122.97: systemctl stop couchbase-server.service [2021-02-12 23:31:40,039] - [remote_util] [140096917202688] - INFO - command executed successfully with root [2021-02-12 23:31:40,040] - [basetestcase] [140096917202688] - INFO - Couchbase stopped [2021-02-12 23:31:40,040] - [basetestcase] [140096917202688] - INFO - sleep for 5 secs. ... [2021-02-12 23:31:45,045] - [remote_util] [140096917202688] - INFO - running command.raw on 172.23.122.97: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-12 23:31:45,061] - [remote_util] [140096917202688] - INFO - command executed successfully with root [2021-02-12 23:31:45,061] - [remote_util] [140096917202688] - INFO - running command.raw on 172.23.122.97: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-12 23:31:45,116] - [remote_util] [140096917202688] - INFO - command executed successfully with root [2021-02-12 23:31:45,117] - [remote_util] [140096917202688] - INFO - SSH Connecting to 172.23.122.97 with username:root, attempt#1 of 5 [2021-02-12 23:31:45,219] - [remote_util] [140096917202688] - INFO - SSH Connected to 172.23.122.97 as root [2021-02-12 23:31:45,479] - [remote_util] [140096917202688] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:31:45,776] - [remote_util] [140096917202688] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:31:45,794] - [remote_util] [140096917202688] - INFO - 172.23.122.97 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 23:31:45,794] - [remote_util] [140096917202688] - INFO - Checking for process beam.smp on linux [2021-02-12 23:31:45,831] - [remote_util] [140096917202688] - INFO - Starting couchbase server [2021-02-12 23:31:45,926] - [remote_util] [140096917202688] - INFO - Running systemd command on this server [2021-02-12 23:31:45,926] - [remote_util] [140096917202688] - INFO - running command.raw on 172.23.122.97: systemctl start couchbase-server.service [2021-02-12 23:31:45,950] - [remote_util] [140096917202688] - INFO - command executed successfully with root [2021-02-12 23:31:45,950] - [remote_util] [140096917202688] - INFO - 172.23.122.97:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-12 23:31:50,955] - [remote_util] [140096917202688] - INFO - running command.raw on 172.23.122.97: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-12 23:31:50,984] - [remote_util] [140096917202688] - INFO - command executed successfully with root [2021-02-12 23:31:50,984] - [remote_util] [140096917202688] - INFO - Couchbase server status: [' Process: 7816 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-12 23:31:50,984] - [remote_util] [140096917202688] - INFO - Checking for process beam.smp on linux [2021-02-12 23:31:51,049] - [remote_util] [140096917202688] - INFO - process beam.smp is running on 172.23.122.97: with pid 7960 [2021-02-12 23:31:51,049] - [basetestcase] [140096917202688] - INFO - Couchbase started [2021-02-12 23:31:51,051] - [remote_util] [140096917202688] - INFO - SSH Connecting to 172.23.122.82 with username:root, attempt#1 of 5 [2021-02-12 23:31:51,156] - [remote_util] [140096917202688] - INFO - SSH Connected to 172.23.122.82 as root [2021-02-12 23:31:51,422] - [remote_util] [140096917202688] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:31:51,721] - [remote_util] [140096917202688] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:31:51,729] - [remote_util] [140096917202688] - INFO - SSH Connecting to 172.23.122.82 with username:root, attempt#1 of 5 [2021-02-12 23:31:51,836] - [remote_util] [140096917202688] - INFO - SSH Connected to 172.23.122.82 as root [2021-02-12 23:31:52,107] - [remote_util] [140096917202688] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:31:52,413] - [remote_util] [140096917202688] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:31:52,476] - [remote_util] [140096917202688] - INFO - 172.23.122.82 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 23:31:52,539] - [remote_util] [140096917202688] - INFO - Running systemd command on this server [2021-02-12 23:31:52,539] - [remote_util] [140096917202688] - INFO - running command.raw on 172.23.122.82: systemctl stop couchbase-server.service [2021-02-12 23:31:54,289] - [remote_util] [140096917202688] - INFO - command executed successfully with root [2021-02-12 23:31:54,290] - [basetestcase] [140096917202688] - INFO - Couchbase stopped [2021-02-12 23:31:54,290] - [basetestcase] [140096917202688] - INFO - sleep for 5 secs. ... [2021-02-12 23:31:59,295] - [remote_util] [140096917202688] - INFO - running command.raw on 172.23.122.82: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-12 23:31:59,316] - [remote_util] [140096917202688] - INFO - command executed successfully with root [2021-02-12 23:31:59,316] - [remote_util] [140096917202688] - INFO - running command.raw on 172.23.122.82: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-12 23:31:59,373] - [remote_util] [140096917202688] - INFO - command executed successfully with root [2021-02-12 23:31:59,374] - [remote_util] [140096917202688] - INFO - SSH Connecting to 172.23.122.82 with username:root, attempt#1 of 5 [2021-02-12 23:31:59,478] - [remote_util] [140096917202688] - INFO - SSH Connected to 172.23.122.82 as root [2021-02-12 23:31:59,762] - [remote_util] [140096917202688] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:32:00,068] - [remote_util] [140096917202688] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:32:00,088] - [remote_util] [140096917202688] - INFO - 172.23.122.82 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 23:32:00,088] - [remote_util] [140096917202688] - INFO - Checking for process beam.smp on linux [2021-02-12 23:32:00,131] - [remote_util] [140096917202688] - INFO - Starting couchbase server [2021-02-12 23:32:00,233] - [remote_util] [140096917202688] - INFO - Running systemd command on this server [2021-02-12 23:32:00,233] - [remote_util] [140096917202688] - INFO - running command.raw on 172.23.122.82: systemctl start couchbase-server.service [2021-02-12 23:32:00,259] - [remote_util] [140096917202688] - INFO - command executed successfully with root [2021-02-12 23:32:00,260] - [remote_util] [140096917202688] - INFO - 172.23.122.82:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-12 23:32:05,261] - [remote_util] [140096917202688] - INFO - running command.raw on 172.23.122.82: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-12 23:32:05,290] - [remote_util] [140096917202688] - INFO - command executed successfully with root [2021-02-12 23:32:05,290] - [remote_util] [140096917202688] - INFO - Couchbase server status: [' Process: 101630 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-12 23:32:05,290] - [remote_util] [140096917202688] - INFO - Checking for process beam.smp on linux [2021-02-12 23:32:05,364] - [remote_util] [140096917202688] - INFO - process beam.smp is running on 172.23.122.82: with pid 101775 [2021-02-12 23:32:05,365] - [basetestcase] [140096917202688] - INFO - Couchbase started [2021-02-12 23:32:05,366] - [remote_util] [140096917202688] - INFO - SSH Connecting to 172.23.122.118 with username:root, attempt#1 of 5 [2021-02-12 23:32:05,471] - [remote_util] [140096917202688] - INFO - SSH Connected to 172.23.122.118 as root [2021-02-12 23:32:05,738] - [remote_util] [140096917202688] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:32:06,076] - [remote_util] [140096917202688] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:32:06,084] - [remote_util] [140096917202688] - INFO - SSH Connecting to 172.23.122.118 with username:root, attempt#1 of 5 [2021-02-12 23:32:06,183] - [remote_util] [140096917202688] - INFO - SSH Connected to 172.23.122.118 as root [2021-02-12 23:32:06,440] - [remote_util] [140096917202688] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:32:06,743] - [remote_util] [140096917202688] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:32:06,762] - [remote_util] [140096917202688] - INFO - 172.23.122.118 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 23:32:06,818] - [remote_util] [140096917202688] - INFO - Running systemd command on this server [2021-02-12 23:32:06,818] - [remote_util] [140096917202688] - INFO - running command.raw on 172.23.122.118: systemctl stop couchbase-server.service [2021-02-12 23:32:08,843] - [remote_util] [140096917202688] - INFO - command executed successfully with root [2021-02-12 23:32:08,843] - [basetestcase] [140096917202688] - INFO - Couchbase stopped [2021-02-12 23:32:08,843] - [basetestcase] [140096917202688] - INFO - sleep for 5 secs. ... [2021-02-12 23:32:13,848] - [remote_util] [140096917202688] - INFO - running command.raw on 172.23.122.118: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-12 23:32:13,863] - [remote_util] [140096917202688] - INFO - command executed successfully with root [2021-02-12 23:32:13,863] - [remote_util] [140096917202688] - INFO - running command.raw on 172.23.122.118: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-12 23:32:13,917] - [remote_util] [140096917202688] - INFO - command executed successfully with root [2021-02-12 23:32:13,919] - [remote_util] [140096917202688] - INFO - SSH Connecting to 172.23.122.118 with username:root, attempt#1 of 5 [2021-02-12 23:32:14,018] - [remote_util] [140096917202688] - INFO - SSH Connected to 172.23.122.118 as root [2021-02-12 23:32:14,271] - [remote_util] [140096917202688] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:32:14,567] - [remote_util] [140096917202688] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:32:14,586] - [remote_util] [140096917202688] - INFO - 172.23.122.118 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 23:32:14,586] - [remote_util] [140096917202688] - INFO - Checking for process beam.smp on linux [2021-02-12 23:32:14,622] - [remote_util] [140096917202688] - INFO - Starting couchbase server [2021-02-12 23:32:14,718] - [remote_util] [140096917202688] - INFO - Running systemd command on this server [2021-02-12 23:32:14,718] - [remote_util] [140096917202688] - INFO - running command.raw on 172.23.122.118: systemctl start couchbase-server.service [2021-02-12 23:32:14,740] - [remote_util] [140096917202688] - INFO - command executed successfully with root [2021-02-12 23:32:14,740] - [remote_util] [140096917202688] - INFO - 172.23.122.118:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-12 23:32:19,745] - [remote_util] [140096917202688] - INFO - running command.raw on 172.23.122.118: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-12 23:32:19,766] - [remote_util] [140096917202688] - INFO - command executed successfully with root [2021-02-12 23:32:19,766] - [remote_util] [140096917202688] - INFO - Couchbase server status: [' Process: 127218 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-12 23:32:19,766] - [remote_util] [140096917202688] - INFO - Checking for process beam.smp on linux [2021-02-12 23:32:19,832] - [remote_util] [140096917202688] - INFO - process beam.smp is running on 172.23.122.118: with pid 127361 [2021-02-12 23:32:19,832] - [basetestcase] [140096917202688] - INFO - Couchbase started [2021-02-12 23:32:19,833] - [remote_util] [140096917202688] - INFO - SSH Connecting to 172.23.122.116 with username:root, attempt#1 of 5 [2021-02-12 23:32:19,936] - [remote_util] [140096917202688] - INFO - SSH Connected to 172.23.122.116 as root [2021-02-12 23:32:20,212] - [remote_util] [140096917202688] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:32:20,507] - [remote_util] [140096917202688] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:32:20,515] - [remote_util] [140096917202688] - INFO - SSH Connecting to 172.23.122.116 with username:root, attempt#1 of 5 [2021-02-12 23:32:20,614] - [remote_util] [140096917202688] - INFO - SSH Connected to 172.23.122.116 as root [2021-02-12 23:32:20,879] - [remote_util] [140096917202688] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:32:21,174] - [remote_util] [140096917202688] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:32:21,193] - [remote_util] [140096917202688] - INFO - 172.23.122.116 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 23:32:21,246] - [remote_util] [140096917202688] - INFO - Running systemd command on this server [2021-02-12 23:32:21,246] - [remote_util] [140096917202688] - INFO - running command.raw on 172.23.122.116: systemctl stop couchbase-server.service [2021-02-12 23:32:23,057] - [remote_util] [140096917202688] - INFO - command executed successfully with root [2021-02-12 23:32:23,058] - [basetestcase] [140096917202688] - INFO - Couchbase stopped [2021-02-12 23:32:23,058] - [basetestcase] [140096917202688] - INFO - sleep for 5 secs. ... [2021-02-12 23:32:28,063] - [remote_util] [140096917202688] - INFO - running command.raw on 172.23.122.116: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-12 23:32:28,078] - [remote_util] [140096917202688] - INFO - command executed successfully with root [2021-02-12 23:32:28,078] - [remote_util] [140096917202688] - INFO - running command.raw on 172.23.122.116: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-12 23:32:28,132] - [remote_util] [140096917202688] - INFO - command executed successfully with root [2021-02-12 23:32:28,134] - [remote_util] [140096917202688] - INFO - SSH Connecting to 172.23.122.116 with username:root, attempt#1 of 5 [2021-02-12 23:32:28,236] - [remote_util] [140096917202688] - INFO - SSH Connected to 172.23.122.116 as root [2021-02-12 23:32:28,513] - [remote_util] [140096917202688] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:32:28,812] - [remote_util] [140096917202688] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:32:28,871] - [remote_util] [140096917202688] - INFO - 172.23.122.116 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 23:32:28,872] - [remote_util] [140096917202688] - INFO - Checking for process beam.smp on linux [2021-02-12 23:32:28,908] - [remote_util] [140096917202688] - INFO - Starting couchbase server [2021-02-12 23:32:29,002] - [remote_util] [140096917202688] - INFO - Running systemd command on this server [2021-02-12 23:32:29,002] - [remote_util] [140096917202688] - INFO - running command.raw on 172.23.122.116: systemctl start couchbase-server.service [2021-02-12 23:32:29,024] - [remote_util] [140096917202688] - INFO - command executed successfully with root [2021-02-12 23:32:29,024] - [remote_util] [140096917202688] - INFO - 172.23.122.116:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-12 23:32:34,030] - [remote_util] [140096917202688] - INFO - running command.raw on 172.23.122.116: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-12 23:32:34,053] - [remote_util] [140096917202688] - INFO - command executed successfully with root [2021-02-12 23:32:34,053] - [remote_util] [140096917202688] - INFO - Couchbase server status: [' Process: 86882 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-12 23:32:34,053] - [remote_util] [140096917202688] - INFO - Checking for process beam.smp on linux [2021-02-12 23:32:34,120] - [remote_util] [140096917202688] - INFO - process beam.smp is running on 172.23.122.116: with pid 87026 [2021-02-12 23:32:34,120] - [basetestcase] [140096917202688] - INFO - Couchbase started [2021-02-12 23:32:34,121] - [basetestcase] [140096917202688] - INFO - sleep for 10 secs. ... [2021-02-12 23:32:44,844] - [task] [140098712618752] - INFO - server: ip:172.23.123.61 port:8091 ssh_username:root, nodes/self [2021-02-12 23:32:44,847] - [task] [140098712618752] - INFO - {'uptime': '110', 'memoryTotal': 4201672704, 'memoryFree': 3531169792, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.123.61:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4454-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.123.61', 'ip': '172.23.123.61', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} [2021-02-12 23:32:44,849] - [rest_client] [140098712618752] - ERROR - GET http://172.23.123.61:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-02-12 23:32:44,849] - [rest_client] [140098712618752] - INFO - pools/default params : memoryQuota=2147 [2021-02-12 23:32:44,854] - [rest_client] [140098712618752] - INFO - --> init_node_services(Administrator,password,172.23.123.61,8091,['kv']) [2021-02-12 23:32:44,855] - [rest_client] [140098712618752] - INFO - /node/controller/setupServices params on 172.23.123.61: 8091:hostname=172.23.123.61&user=Administrator&password=password&services=kv [2021-02-12 23:32:44,884] - [rest_client] [140098712618752] - INFO - --> in init_cluster...Administrator,password,8091 [2021-02-12 23:32:44,884] - [rest_client] [140098712618752] - INFO - settings/web params on 172.23.123.61:8091:port=8091&username=Administrator&password=password [2021-02-12 23:32:44,939] - [rest_client] [140098712618752] - INFO - --> status:True [2021-02-12 23:32:44,940] - [remote_util] [140098712618752] - INFO - SSH Connecting to 172.23.123.61 with username:root, attempt#1 of 5 [2021-02-12 23:32:45,043] - [remote_util] [140098712618752] - INFO - SSH Connected to 172.23.123.61 as root [2021-02-12 23:32:45,307] - [remote_util] [140098712618752] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:32:45,607] - [remote_util] [140098712618752] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:32:45,614] - [remote_util] [140098712618752] - INFO - running command.raw on 172.23.123.61: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-02-12 23:32:45,671] - [remote_util] [140098712618752] - INFO - command executed successfully with root [2021-02-12 23:32:45,671] - [remote_util] [140098712618752] - INFO - ['ok'] [2021-02-12 23:32:45,673] - [rest_client] [140098712618752] - INFO - /diag/eval status on 172.23.123.61:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-12 23:32:45,675] - [rest_client] [140098712618752] - INFO - /diag/eval status on 172.23.123.61:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-12 23:32:45,679] - [rest_client] [140098712618752] - INFO - settings/indexes params : storageMode=plasma [2021-02-12 23:32:45,699] - [task] [140098712618752] - INFO - server: ip:172.23.123.62 port:8091 ssh_username:root, nodes/self [2021-02-12 23:32:45,702] - [task] [140098712618752] - INFO - {'uptime': '100', 'memoryTotal': 4201676800, 'memoryFree': 3486826496, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.123.62:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4454-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.123.62', 'ip': '172.23.123.62', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} [2021-02-12 23:32:45,704] - [rest_client] [140098712618752] - ERROR - GET http://172.23.123.62:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-02-12 23:32:45,704] - [rest_client] [140098712618752] - INFO - pools/default params : memoryQuota=2147 [2021-02-12 23:32:45,710] - [rest_client] [140098712618752] - INFO - --> in init_cluster...Administrator,password,8091 [2021-02-12 23:32:45,710] - [rest_client] [140098712618752] - INFO - settings/web params on 172.23.123.62:8091:port=8091&username=Administrator&password=password [2021-02-12 23:32:45,765] - [rest_client] [140098712618752] - INFO - --> status:True [2021-02-12 23:32:45,767] - [remote_util] [140098712618752] - INFO - SSH Connecting to 172.23.123.62 with username:root, attempt#1 of 5 [2021-02-12 23:32:45,870] - [remote_util] [140098712618752] - INFO - SSH Connected to 172.23.123.62 as root [2021-02-12 23:32:46,151] - [remote_util] [140098712618752] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:32:46,448] - [remote_util] [140098712618752] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:32:46,457] - [remote_util] [140098712618752] - INFO - running command.raw on 172.23.123.62: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-02-12 23:32:46,512] - [remote_util] [140098712618752] - INFO - command executed successfully with root [2021-02-12 23:32:46,512] - [remote_util] [140098712618752] - INFO - ['ok'] [2021-02-12 23:32:46,514] - [rest_client] [140098712618752] - INFO - /diag/eval status on 172.23.123.62:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-12 23:32:46,516] - [rest_client] [140098712618752] - INFO - /diag/eval status on 172.23.123.62:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-12 23:32:46,520] - [rest_client] [140098712618752] - INFO - settings/indexes params : storageMode=plasma [2021-02-12 23:32:46,543] - [task] [140098712618752] - INFO - server: ip:172.23.122.90 port:8091 ssh_username:root, nodes/self [2021-02-12 23:32:46,546] - [task] [140098712618752] - INFO - {'uptime': '85', 'memoryTotal': 4201684992, 'memoryFree': 3573051392, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.122.90:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4454-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.122.90', 'ip': '172.23.122.90', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} [2021-02-12 23:32:46,548] - [rest_client] [140098712618752] - ERROR - GET http://172.23.122.90:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-02-12 23:32:46,548] - [rest_client] [140098712618752] - INFO - pools/default params : memoryQuota=2147 [2021-02-12 23:32:46,554] - [rest_client] [140098712618752] - INFO - --> in init_cluster...Administrator,password,8091 [2021-02-12 23:32:46,554] - [rest_client] [140098712618752] - INFO - settings/web params on 172.23.122.90:8091:port=8091&username=Administrator&password=password [2021-02-12 23:32:46,609] - [rest_client] [140098712618752] - INFO - --> status:True [2021-02-12 23:32:46,610] - [remote_util] [140098712618752] - INFO - SSH Connecting to 172.23.122.90 with username:root, attempt#1 of 5 [2021-02-12 23:32:46,714] - [remote_util] [140098712618752] - INFO - SSH Connected to 172.23.122.90 as root [2021-02-12 23:32:46,991] - [remote_util] [140098712618752] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:32:47,287] - [remote_util] [140098712618752] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:32:47,294] - [remote_util] [140098712618752] - INFO - running command.raw on 172.23.122.90: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-02-12 23:32:47,351] - [remote_util] [140098712618752] - INFO - command executed successfully with root [2021-02-12 23:32:47,351] - [remote_util] [140098712618752] - INFO - ['ok'] [2021-02-12 23:32:47,353] - [rest_client] [140098712618752] - INFO - /diag/eval status on 172.23.122.90:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-12 23:32:47,355] - [rest_client] [140098712618752] - INFO - /diag/eval status on 172.23.122.90:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-12 23:32:47,359] - [rest_client] [140098712618752] - INFO - settings/indexes params : storageMode=plasma [2021-02-12 23:32:47,378] - [task] [140098712618752] - INFO - server: ip:172.23.122.117 port:8091 ssh_username:root, nodes/self [2021-02-12 23:32:47,380] - [task] [140098712618752] - INFO - {'uptime': '69', 'memoryTotal': 4201684992, 'memoryFree': 3532197888, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.122.117:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4454-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.122.117', 'ip': '172.23.122.117', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} [2021-02-12 23:32:47,382] - [rest_client] [140098712618752] - ERROR - GET http://172.23.122.117:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-02-12 23:32:47,382] - [rest_client] [140098712618752] - INFO - pools/default params : memoryQuota=2147 [2021-02-12 23:32:47,388] - [rest_client] [140098712618752] - INFO - --> in init_cluster...Administrator,password,8091 [2021-02-12 23:32:47,388] - [rest_client] [140098712618752] - INFO - settings/web params on 172.23.122.117:8091:port=8091&username=Administrator&password=password [2021-02-12 23:32:47,443] - [rest_client] [140098712618752] - INFO - --> status:True [2021-02-12 23:32:47,444] - [remote_util] [140098712618752] - INFO - SSH Connecting to 172.23.122.117 with username:root, attempt#1 of 5 [2021-02-12 23:32:47,544] - [remote_util] [140098712618752] - INFO - SSH Connected to 172.23.122.117 as root [2021-02-12 23:32:47,793] - [remote_util] [140098712618752] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:32:48,081] - [remote_util] [140098712618752] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:32:48,088] - [remote_util] [140098712618752] - INFO - running command.raw on 172.23.122.117: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-02-12 23:32:48,145] - [remote_util] [140098712618752] - INFO - command executed successfully with root [2021-02-12 23:32:48,145] - [remote_util] [140098712618752] - INFO - ['ok'] [2021-02-12 23:32:48,147] - [rest_client] [140098712618752] - INFO - /diag/eval status on 172.23.122.117:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-12 23:32:48,149] - [rest_client] [140098712618752] - INFO - /diag/eval status on 172.23.122.117:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-12 23:32:48,153] - [rest_client] [140098712618752] - INFO - settings/indexes params : storageMode=plasma [2021-02-12 23:32:48,172] - [task] [140098712618752] - INFO - server: ip:172.23.122.97 port:8091 ssh_username:root, nodes/self [2021-02-12 23:32:48,175] - [task] [140098712618752] - INFO - {'uptime': '59', 'memoryTotal': 4201684992, 'memoryFree': 3551608832, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.122.97:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4454-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.122.97', 'ip': '172.23.122.97', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} [2021-02-12 23:32:48,176] - [rest_client] [140098712618752] - ERROR - GET http://172.23.122.97:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-02-12 23:32:48,177] - [rest_client] [140098712618752] - INFO - pools/default params : memoryQuota=2147 [2021-02-12 23:32:48,182] - [rest_client] [140098712618752] - INFO - --> in init_cluster...Administrator,password,8091 [2021-02-12 23:32:48,182] - [rest_client] [140098712618752] - INFO - settings/web params on 172.23.122.97:8091:port=8091&username=Administrator&password=password [2021-02-12 23:32:48,236] - [rest_client] [140098712618752] - INFO - --> status:True [2021-02-12 23:32:48,238] - [remote_util] [140098712618752] - INFO - SSH Connecting to 172.23.122.97 with username:root, attempt#1 of 5 [2021-02-12 23:32:48,341] - [remote_util] [140098712618752] - INFO - SSH Connected to 172.23.122.97 as root [2021-02-12 23:32:48,606] - [remote_util] [140098712618752] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:32:48,903] - [remote_util] [140098712618752] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:32:48,911] - [remote_util] [140098712618752] - INFO - running command.raw on 172.23.122.97: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-02-12 23:32:48,968] - [remote_util] [140098712618752] - INFO - command executed successfully with root [2021-02-12 23:32:48,968] - [remote_util] [140098712618752] - INFO - ['ok'] [2021-02-12 23:32:48,970] - [rest_client] [140098712618752] - INFO - /diag/eval status on 172.23.122.97:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-12 23:32:48,972] - [rest_client] [140098712618752] - INFO - /diag/eval status on 172.23.122.97:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-12 23:32:48,978] - [rest_client] [140098712618752] - INFO - settings/indexes params : storageMode=plasma [2021-02-12 23:32:48,998] - [task] [140098712618752] - INFO - server: ip:172.23.122.82 port:8091 ssh_username:root, nodes/self [2021-02-12 23:32:49,000] - [task] [140098712618752] - INFO - {'uptime': '45', 'memoryTotal': 4201684992, 'memoryFree': 3581685760, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.122.82:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4454-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.122.82', 'ip': '172.23.122.82', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} [2021-02-12 23:32:49,002] - [rest_client] [140098712618752] - ERROR - GET http://172.23.122.82:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-02-12 23:32:49,002] - [rest_client] [140098712618752] - INFO - pools/default params : memoryQuota=2147 [2021-02-12 23:32:49,007] - [rest_client] [140098712618752] - INFO - --> in init_cluster...Administrator,password,8091 [2021-02-12 23:32:49,007] - [rest_client] [140098712618752] - INFO - settings/web params on 172.23.122.82:8091:port=8091&username=Administrator&password=password [2021-02-12 23:32:49,060] - [rest_client] [140098712618752] - INFO - --> status:True [2021-02-12 23:32:49,062] - [remote_util] [140098712618752] - INFO - SSH Connecting to 172.23.122.82 with username:root, attempt#1 of 5 [2021-02-12 23:32:49,164] - [remote_util] [140098712618752] - INFO - SSH Connected to 172.23.122.82 as root [2021-02-12 23:32:49,430] - [remote_util] [140098712618752] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:32:49,724] - [remote_util] [140098712618752] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:32:49,732] - [remote_util] [140098712618752] - INFO - running command.raw on 172.23.122.82: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-02-12 23:32:49,788] - [remote_util] [140098712618752] - INFO - command executed successfully with root [2021-02-12 23:32:49,788] - [remote_util] [140098712618752] - INFO - ['ok'] [2021-02-12 23:32:49,790] - [rest_client] [140098712618752] - INFO - /diag/eval status on 172.23.122.82:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-12 23:32:49,792] - [rest_client] [140098712618752] - INFO - /diag/eval status on 172.23.122.82:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-12 23:32:49,796] - [rest_client] [140098712618752] - INFO - settings/indexes params : storageMode=plasma [2021-02-12 23:32:49,814] - [task] [140098712618752] - INFO - server: ip:172.23.122.118 port:8091 ssh_username:root, nodes/self [2021-02-12 23:32:49,817] - [task] [140098712618752] - INFO - {'uptime': '29', 'memoryTotal': 4201689088, 'memoryFree': 3535654912, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.122.118:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4454-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.122.118', 'ip': '172.23.122.118', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} [2021-02-12 23:32:49,819] - [rest_client] [140098712618752] - ERROR - GET http://172.23.122.118:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-02-12 23:32:49,819] - [rest_client] [140098712618752] - INFO - pools/default params : memoryQuota=2147 [2021-02-12 23:32:49,825] - [rest_client] [140098712618752] - INFO - --> in init_cluster...Administrator,password,8091 [2021-02-12 23:32:49,825] - [rest_client] [140098712618752] - INFO - settings/web params on 172.23.122.118:8091:port=8091&username=Administrator&password=password [2021-02-12 23:32:49,877] - [rest_client] [140098712618752] - INFO - --> status:True [2021-02-12 23:32:49,879] - [remote_util] [140098712618752] - INFO - SSH Connecting to 172.23.122.118 with username:root, attempt#1 of 5 [2021-02-12 23:32:49,978] - [remote_util] [140098712618752] - INFO - SSH Connected to 172.23.122.118 as root [2021-02-12 23:32:50,224] - [remote_util] [140098712618752] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:32:50,515] - [remote_util] [140098712618752] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:32:50,522] - [remote_util] [140098712618752] - INFO - running command.raw on 172.23.122.118: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-02-12 23:32:50,578] - [remote_util] [140098712618752] - INFO - command executed successfully with root [2021-02-12 23:32:50,578] - [remote_util] [140098712618752] - INFO - ['ok'] [2021-02-12 23:32:50,580] - [rest_client] [140098712618752] - INFO - /diag/eval status on 172.23.122.118:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-12 23:32:50,582] - [rest_client] [140098712618752] - INFO - /diag/eval status on 172.23.122.118:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-12 23:32:50,586] - [rest_client] [140098712618752] - INFO - settings/indexes params : storageMode=plasma [2021-02-12 23:32:50,604] - [task] [140098712618752] - INFO - server: ip:172.23.122.116 port:8091 ssh_username:root, nodes/self [2021-02-12 23:32:50,607] - [task] [140098712618752] - INFO - {'uptime': '14', 'memoryTotal': 4201689088, 'memoryFree': 3564830720, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.122.116:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4454-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@cb.local', 'ip': '172.23.122.116', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} [2021-02-12 23:32:50,609] - [rest_client] [140098712618752] - ERROR - GET http://172.23.122.116:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-02-12 23:32:50,609] - [rest_client] [140098712618752] - INFO - pools/default params : memoryQuota=2147 [2021-02-12 23:32:50,615] - [rest_client] [140098712618752] - INFO - --> in init_cluster...Administrator,password,8091 [2021-02-12 23:32:50,615] - [rest_client] [140098712618752] - INFO - settings/web params on 172.23.122.116:8091:port=8091&username=Administrator&password=password [2021-02-12 23:32:50,670] - [rest_client] [140098712618752] - INFO - --> status:True [2021-02-12 23:32:50,671] - [remote_util] [140098712618752] - INFO - SSH Connecting to 172.23.122.116 with username:root, attempt#1 of 5 [2021-02-12 23:32:50,776] - [remote_util] [140098712618752] - INFO - SSH Connected to 172.23.122.116 as root [2021-02-12 23:32:51,050] - [remote_util] [140098712618752] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:32:51,341] - [remote_util] [140098712618752] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:32:51,348] - [remote_util] [140098712618752] - INFO - running command.raw on 172.23.122.116: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-02-12 23:32:51,404] - [remote_util] [140098712618752] - INFO - command executed successfully with root [2021-02-12 23:32:51,404] - [remote_util] [140098712618752] - INFO - ['ok'] [2021-02-12 23:32:51,406] - [rest_client] [140098712618752] - INFO - /diag/eval status on 172.23.122.116:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-12 23:32:51,409] - [rest_client] [140098712618752] - INFO - /diag/eval status on 172.23.122.116:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-02-12 23:32:51,413] - [rest_client] [140098712618752] - INFO - settings/indexes params : storageMode=plasma [2021-02-12 23:32:51,435] - [basetestcase] [140096917202688] - INFO - **** add built-in 'cbadminbucket' user to node 172.23.123.61 **** [2021-02-12 23:32:51,444] - [rest_client] [140096917202688] - ERROR - DELETE http://172.23.123.61:8091/settings/rbac/users/local/cbadminbucket body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:password [2021-02-12 23:32:51,444] - [internal_user] [140096917202688] - INFO - Exception while deleting user. Exception is -b'"User was not found."' [2021-02-12 23:32:51,506] - [basetestcase] [140096917202688] - INFO - sleep for 5 secs. ... [2021-02-12 23:32:56,511] - [basetestcase] [140096917202688] - INFO - **** add 'admin' role to 'cbadminbucket' user **** [2021-02-12 23:32:56,522] - [basetestcase] [140096917202688] - INFO - done initializing cluster [2021-02-12 23:32:57,433] - [task] [140098712618752] - INFO - adding node 172.23.123.62:8091 to cluster [2021-02-12 23:32:57,434] - [rest_client] [140098712618752] - INFO - adding remote node @172.23.123.62:8091 to this cluster @172.23.123.61:8091 [2021-02-12 23:32:59,630] - [task] [140098712618752] - INFO - adding node 172.23.122.90:8091 to cluster [2021-02-12 23:32:59,630] - [rest_client] [140098712618752] - INFO - adding remote node @172.23.122.90:8091 to this cluster @172.23.123.61:8091 [2021-02-12 23:33:01,813] - [task] [140098712618752] - INFO - adding node 172.23.122.117:8091 to cluster [2021-02-12 23:33:01,813] - [rest_client] [140098712618752] - INFO - adding remote node @172.23.122.117:8091 to this cluster @172.23.123.61:8091 [2021-02-12 23:33:04,346] - [task] [140098712618752] - INFO - adding node 172.23.122.97:8091 to cluster [2021-02-12 23:33:04,346] - [rest_client] [140098712618752] - INFO - adding remote node @172.23.122.97:8091 to this cluster @172.23.123.61:8091 [2021-02-12 23:33:06,859] - [task] [140098712618752] - INFO - adding node 172.23.122.82:8091 to cluster [2021-02-12 23:33:06,859] - [rest_client] [140098712618752] - INFO - adding remote node @172.23.122.82:8091 to this cluster @172.23.123.61:8091 [2021-02-12 23:33:09,240] - [rest_client] [140098712618752] - INFO - rebalance params : {'knownNodes': 'ns_1@172.23.122.117,ns_1@172.23.122.82,ns_1@172.23.122.90,ns_1@172.23.122.97,ns_1@172.23.123.61,ns_1@172.23.123.62', 'ejectedNodes': '', 'user': 'Administrator', 'password': 'password'} [2021-02-12 23:33:09,268] - [rest_client] [140098712618752] - INFO - rebalance operation started [2021-02-12 23:33:09,270] - [rest_client] [140098712618752] - INFO - rebalance percentage : 0.00 % [2021-02-12 23:33:09,270] - [task] [140098712618752] - INFO - Rebalance - status: running, progress: 0.00% [2021-02-12 23:33:19,290] - [task] [140098712618752] - INFO - Rebalance - status: none, progress: 100.00% [2021-02-12 23:33:19,298] - [task] [140098712618752] - INFO - rebalancing was completed with progress: 100% in 10.030362129211426 sec [2021-02-12 23:33:19,300] - [remote_util] [140096917202688] - INFO - SSH Connecting to 172.23.123.61 with username:root, attempt#1 of 5 [2021-02-12 23:33:19,404] - [remote_util] [140096917202688] - INFO - SSH Connected to 172.23.123.61 as root [2021-02-12 23:33:19,679] - [remote_util] [140096917202688] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:33:19,978] - [remote_util] [140096917202688] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:33:19,987] - [remote_util] [140096917202688] - INFO - running command.raw on 172.23.123.61: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-02-12 23:33:20,043] - [remote_util] [140096917202688] - INFO - command executed successfully with root [2021-02-12 23:33:20,044] - [remote_util] [140096917202688] - INFO - ['ok'] [2021-02-12 23:33:20,044] - [basetestcase] [140096917202688] - INFO - Enabled diag/eval for non-local hosts from 172.23.123.61 [2021-02-12 23:33:20,044] - [basetestcase] [140096917202688] - INFO - ============== basetestcase setup was finished for test #10 test_eventing_rebalance_out_delete_recreate_collections ============== [2021-02-12 23:33:20,052] - [remote_util] [140096917202688] - INFO - SSH Connecting to 172.23.123.61 with username:root, attempt#1 of 5 [2021-02-12 23:33:20,153] - [remote_util] [140096917202688] - INFO - SSH Connected to 172.23.123.61 as root [2021-02-12 23:33:20,414] - [remote_util] [140096917202688] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:33:20,711] - [remote_util] [140096917202688] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:33:20,712] - [remote_util] [140096917202688] - INFO - SSH Connecting to 172.23.123.62 with username:root, attempt#1 of 5 [2021-02-12 23:33:20,814] - [remote_util] [140096917202688] - INFO - SSH Connected to 172.23.123.62 as root [2021-02-12 23:33:21,084] - [remote_util] [140096917202688] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:33:21,379] - [remote_util] [140096917202688] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:33:21,382] - [remote_util] [140096917202688] - INFO - SSH Connecting to 172.23.122.90 with username:root, attempt#1 of 5 [2021-02-12 23:33:21,484] - [remote_util] [140096917202688] - INFO - SSH Connected to 172.23.122.90 as root [2021-02-12 23:33:21,764] - [remote_util] [140096917202688] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:33:22,057] - [remote_util] [140096917202688] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:33:22,058] - [remote_util] [140096917202688] - INFO - SSH Connecting to 172.23.122.117 with username:root, attempt#1 of 5 [2021-02-12 23:33:22,156] - [remote_util] [140096917202688] - INFO - SSH Connected to 172.23.122.117 as root [2021-02-12 23:33:22,404] - [remote_util] [140096917202688] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:33:22,693] - [remote_util] [140096917202688] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:33:22,695] - [remote_util] [140096917202688] - INFO - SSH Connecting to 172.23.122.97 with username:root, attempt#1 of 5 [2021-02-12 23:33:22,798] - [remote_util] [140096917202688] - INFO - SSH Connected to 172.23.122.97 as root [2021-02-12 23:33:23,060] - [remote_util] [140096917202688] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:33:23,356] - [remote_util] [140096917202688] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:33:23,358] - [remote_util] [140096917202688] - INFO - SSH Connecting to 172.23.122.82 with username:root, attempt#1 of 5 [2021-02-12 23:33:23,457] - [remote_util] [140096917202688] - INFO - SSH Connected to 172.23.122.82 as root [2021-02-12 23:33:23,712] - [remote_util] [140096917202688] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:33:23,998] - [remote_util] [140096917202688] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:33:24,000] - [remote_util] [140096917202688] - INFO - SSH Connecting to 172.23.122.118 with username:root, attempt#1 of 5 [2021-02-12 23:33:24,098] - [remote_util] [140096917202688] - INFO - SSH Connected to 172.23.122.118 as root [2021-02-12 23:33:24,336] - [remote_util] [140096917202688] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:33:24,622] - [remote_util] [140096917202688] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:33:24,624] - [remote_util] [140096917202688] - INFO - SSH Connecting to 172.23.122.116 with username:root, attempt#1 of 5 [2021-02-12 23:33:24,724] - [remote_util] [140096917202688] - INFO - SSH Connected to 172.23.122.116 as root [2021-02-12 23:33:24,993] - [remote_util] [140096917202688] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:33:25,284] - [remote_util] [140096917202688] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:34:50,498] - [basetestcase] [140096917202688] - INFO - ------- Cluster statistics ------- [2021-02-12 23:34:50,498] - [basetestcase] [140096917202688] - INFO - 172.23.122.117:8091 => {'services': ['eventing'], 'cpu_utilization': 0.6778809942254582, 'mem_free': 3493658624, 'mem_total': 4201684992, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-02-12 23:34:50,498] - [basetestcase] [140096917202688] - INFO - 172.23.122.82:8091 => {'services': ['n1ql'], 'cpu_utilization': 12.00101703534198, 'mem_free': 3519975424, 'mem_total': 4201684992, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-02-12 23:34:50,499] - [basetestcase] [140096917202688] - INFO - 172.23.122.90:8091 => {'services': ['eventing'], 'cpu_utilization': 0.8514901076884548, 'mem_free': 3533144064, 'mem_total': 4201684992, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-02-12 23:34:50,499] - [basetestcase] [140096917202688] - INFO - 172.23.122.97:8091 => {'services': ['index'], 'cpu_utilization': 13.14241881871644, 'mem_free': 3493900288, 'mem_total': 4201684992, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-02-12 23:34:50,499] - [basetestcase] [140096917202688] - INFO - 172.23.123.61:8091 => {'services': ['kv'], 'cpu_utilization': 0.6542526421741318, 'mem_free': 3491401728, 'mem_total': 4201672704, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-02-12 23:34:50,499] - [basetestcase] [140096917202688] - INFO - 172.23.123.62:8091 => {'services': ['eventing'], 'cpu_utilization': 0.5775991963837268, 'mem_free': 3447701504, 'mem_total': 4201676800, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-02-12 23:34:50,499] - [basetestcase] [140096917202688] - INFO - --- End of cluster statistics --- [2021-02-12 23:34:50,501] - [remote_util] [140096917202688] - INFO - SSH Connecting to 172.23.123.61 with username:root, attempt#1 of 5 [2021-02-12 23:34:50,603] - [remote_util] [140096917202688] - INFO - SSH Connected to 172.23.123.61 as root [2021-02-12 23:34:50,866] - [remote_util] [140096917202688] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:34:51,161] - [remote_util] [140096917202688] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:34:53,946] - [basetestcase] [140096917202688] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root] [2021-02-12 23:34:53,953] - [basetestcase] [140096917202688] - INFO - list of eventing nodes in cluster: [ip:172.23.122.117 port:8091 ssh_username:root, ip:172.23.122.90 port:8091 ssh_username:root, ip:172.23.123.62 port:8091 ssh_username:root] [2021-02-12 23:34:53,958] - [rest_client] [140096917202688] - INFO - settings/indexes params : storageMode=plasma [2021-02-12 23:34:53,963] - [eventing_base] [140096917202688] - INFO - Setting the min possible memory quota so that adding mode nodes to the cluster wouldn't be a problem. [2021-02-12 23:34:53,963] - [rest_client] [140096917202688] - INFO - pools/default params : memoryQuota=330 [2021-02-12 23:34:53,969] - [rest_client] [140096917202688] - INFO - pools/default params : indexMemoryQuota=256 [2021-02-12 23:34:53,974] - [rest_client] [140096917202688] - INFO - pools/default params : eventingMemoryQuota=512 [2021-02-12 23:34:53,996] - [basetestcase] [140096917202688] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root] [2021-02-12 23:34:54,001] - [rest_client] [140096917202688] - INFO - pools/default params : memoryQuota=700 [2021-02-12 23:34:54,006] - [eventing_rebalance_collection] [140096917202688] - INFO - 200 [2021-02-12 23:34:54,542] - [rest_client] [140098712618752] - INFO - http://172.23.123.61:8091/pools/default/buckets with param: name=src_bucket&ramQuotaMB=200&authType=none&replicaNumber=0&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore [2021-02-12 23:34:54,571] - [rest_client] [140098712618752] - INFO - 0.03 seconds to create bucket src_bucket [2021-02-12 23:34:54,571] - [bucket_helper] [140098712618752] - INFO - waiting for memcached bucket : src_bucket in 172.23.123.61 to accept set ops [2021-02-12 23:34:55,209] - [data_helper] [140098712618752] - INFO - creating direct client 172.23.123.61:11210 src_bucket [2021-02-12 23:34:55,317] - [data_helper] [140098712618752] - INFO - creating direct client 172.23.123.61:11210 src_bucket [2021-02-12 23:34:55,376] - [task] [140098712618752] - WARNING - vbucket map not ready after try 0 [2021-02-12 23:34:55,376] - [bucket_helper] [140098712618752] - INFO - waiting for memcached bucket : src_bucket in 172.23.123.61 to accept set ops [2021-02-12 23:34:55,491] - [data_helper] [140098712618752] - INFO - creating direct client 172.23.123.61:11210 src_bucket [2021-02-12 23:34:55,791] - [data_helper] [140098712618752] - INFO - creating direct client 172.23.123.61:11210 src_bucket [2021-02-12 23:34:55,849] - [task] [140098712618752] - WARNING - vbucket map not ready after try 1 [2021-02-12 23:34:55,849] - [bucket_helper] [140098712618752] - INFO - waiting for memcached bucket : src_bucket in 172.23.123.61 to accept set ops [2021-02-12 23:34:55,948] - [data_helper] [140098712618752] - INFO - creating direct client 172.23.123.61:11210 src_bucket [2021-02-12 23:34:56,055] - [data_helper] [140098712618752] - INFO - creating direct client 172.23.123.61:11210 src_bucket [2021-02-12 23:34:56,300] - [task] [140098712618752] - WARNING - vbucket map not ready after try 2 [2021-02-12 23:34:56,300] - [bucket_helper] [140098712618752] - INFO - waiting for memcached bucket : src_bucket in 172.23.123.61 to accept set ops [2021-02-12 23:34:56,406] - [data_helper] [140098712618752] - INFO - creating direct client 172.23.123.61:11210 src_bucket [2021-02-12 23:34:56,505] - [data_helper] [140098712618752] - INFO - creating direct client 172.23.123.61:11210 src_bucket [2021-02-12 23:34:56,563] - [task] [140098712618752] - WARNING - vbucket map not ready after try 3 [2021-02-12 23:34:56,563] - [bucket_helper] [140098712618752] - INFO - waiting for memcached bucket : src_bucket in 172.23.123.61 to accept set ops [2021-02-12 23:34:56,659] - [data_helper] [140098712618752] - INFO - creating direct client 172.23.123.61:11210 src_bucket [2021-02-12 23:34:56,948] - [data_helper] [140098712618752] - INFO - creating direct client 172.23.123.61:11210 src_bucket [2021-02-12 23:34:57,011] - [task] [140098712618752] - WARNING - vbucket map not ready after try 4 [2021-02-12 23:34:57,011] - [bucket_helper] [140098712618752] - INFO - waiting for memcached bucket : src_bucket in 172.23.123.61 to accept set ops [2021-02-12 23:34:57,110] - [data_helper] [140098712618752] - INFO - creating direct client 172.23.123.61:11210 src_bucket [2021-02-12 23:34:57,208] - [data_helper] [140098712618752] - INFO - creating direct client 172.23.123.61:11210 src_bucket [2021-02-12 23:34:57,266] - [task] [140098712618752] - WARNING - vbucket map not ready after try 5 [2021-02-12 23:34:58,290] - [rest_client] [140098712618752] - INFO - http://172.23.123.61:8091/pools/default/buckets with param: name=dst_bucket&ramQuotaMB=200&authType=none&replicaNumber=0&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore [2021-02-12 23:34:58,319] - [rest_client] [140098712618752] - INFO - 0.03 seconds to create bucket dst_bucket [2021-02-12 23:34:58,319] - [bucket_helper] [140098712618752] - INFO - waiting for memcached bucket : dst_bucket in 172.23.123.61 to accept set ops [2021-02-12 23:34:59,130] - [data_helper] [140098712618752] - INFO - creating direct client 172.23.123.61:11210 dst_bucket [2021-02-12 23:34:59,262] - [data_helper] [140098712618752] - INFO - creating direct client 172.23.123.61:11210 dst_bucket [2021-02-12 23:34:59,323] - [task] [140098712618752] - WARNING - vbucket map not ready after try 0 [2021-02-12 23:34:59,323] - [bucket_helper] [140098712618752] - INFO - waiting for memcached bucket : dst_bucket in 172.23.123.61 to accept set ops [2021-02-12 23:34:59,426] - [data_helper] [140098712618752] - INFO - creating direct client 172.23.123.61:11210 dst_bucket [2021-02-12 23:34:59,716] - [data_helper] [140098712618752] - INFO - creating direct client 172.23.123.61:11210 dst_bucket [2021-02-12 23:34:59,776] - [task] [140098712618752] - WARNING - vbucket map not ready after try 1 [2021-02-12 23:34:59,776] - [bucket_helper] [140098712618752] - INFO - waiting for memcached bucket : dst_bucket in 172.23.123.61 to accept set ops [2021-02-12 23:34:59,877] - [data_helper] [140098712618752] - INFO - creating direct client 172.23.123.61:11210 dst_bucket [2021-02-12 23:34:59,978] - [data_helper] [140098712618752] - INFO - creating direct client 172.23.123.61:11210 dst_bucket [2021-02-12 23:35:00,038] - [task] [140098712618752] - WARNING - vbucket map not ready after try 2 [2021-02-12 23:35:00,038] - [bucket_helper] [140098712618752] - INFO - waiting for memcached bucket : dst_bucket in 172.23.123.61 to accept set ops [2021-02-12 23:35:00,319] - [data_helper] [140098712618752] - INFO - creating direct client 172.23.123.61:11210 dst_bucket [2021-02-12 23:35:00,420] - [data_helper] [140098712618752] - INFO - creating direct client 172.23.123.61:11210 dst_bucket [2021-02-12 23:35:00,478] - [task] [140098712618752] - WARNING - vbucket map not ready after try 3 [2021-02-12 23:35:00,479] - [bucket_helper] [140098712618752] - INFO - waiting for memcached bucket : dst_bucket in 172.23.123.61 to accept set ops [2021-02-12 23:35:00,578] - [data_helper] [140098712618752] - INFO - creating direct client 172.23.123.61:11210 dst_bucket [2021-02-12 23:35:00,874] - [data_helper] [140098712618752] - INFO - creating direct client 172.23.123.61:11210 dst_bucket [2021-02-12 23:35:00,941] - [task] [140098712618752] - WARNING - vbucket map not ready after try 4 [2021-02-12 23:35:00,941] - [bucket_helper] [140098712618752] - INFO - waiting for memcached bucket : dst_bucket in 172.23.123.61 to accept set ops [2021-02-12 23:35:01,044] - [data_helper] [140098712618752] - INFO - creating direct client 172.23.123.61:11210 dst_bucket [2021-02-12 23:35:01,147] - [data_helper] [140098712618752] - INFO - creating direct client 172.23.123.61:11210 dst_bucket [2021-02-12 23:35:01,207] - [task] [140098712618752] - WARNING - vbucket map not ready after try 5 [2021-02-12 23:35:02,232] - [rest_client] [140098712618752] - INFO - http://172.23.123.61:8091/pools/default/buckets with param: name=metadata&ramQuotaMB=200&authType=none&replicaNumber=0&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore [2021-02-12 23:35:02,260] - [rest_client] [140098712618752] - INFO - 0.03 seconds to create bucket metadata [2021-02-12 23:35:02,261] - [bucket_helper] [140098712618752] - INFO - waiting for memcached bucket : metadata in 172.23.123.61 to accept set ops [2021-02-12 23:35:03,079] - [data_helper] [140098712618752] - INFO - creating direct client 172.23.123.61:11210 metadata [2021-02-12 23:35:03,214] - [data_helper] [140098712618752] - INFO - creating direct client 172.23.123.61:11210 metadata [2021-02-12 23:35:03,277] - [task] [140098712618752] - WARNING - vbucket map not ready after try 0 [2021-02-12 23:35:03,278] - [bucket_helper] [140098712618752] - INFO - waiting for memcached bucket : metadata in 172.23.123.61 to accept set ops [2021-02-12 23:35:03,389] - [data_helper] [140098712618752] - INFO - creating direct client 172.23.123.61:11210 metadata [2021-02-12 23:35:03,495] - [data_helper] [140098712618752] - INFO - creating direct client 172.23.123.61:11210 metadata [2021-02-12 23:35:03,741] - [task] [140098712618752] - WARNING - vbucket map not ready after try 1 [2021-02-12 23:35:03,742] - [bucket_helper] [140098712618752] - INFO - waiting for memcached bucket : metadata in 172.23.123.61 to accept set ops [2021-02-12 23:35:03,848] - [data_helper] [140098712618752] - INFO - creating direct client 172.23.123.61:11210 metadata [2021-02-12 23:35:03,955] - [data_helper] [140098712618752] - INFO - creating direct client 172.23.123.61:11210 metadata [2021-02-12 23:35:04,020] - [task] [140098712618752] - WARNING - vbucket map not ready after try 2 [2021-02-12 23:35:04,020] - [bucket_helper] [140098712618752] - INFO - waiting for memcached bucket : metadata in 172.23.123.61 to accept set ops [2021-02-12 23:35:04,134] - [data_helper] [140098712618752] - INFO - creating direct client 172.23.123.61:11210 metadata [2021-02-12 23:35:04,435] - [data_helper] [140098712618752] - INFO - creating direct client 172.23.123.61:11210 metadata [2021-02-12 23:35:04,502] - [task] [140098712618752] - WARNING - vbucket map not ready after try 3 [2021-02-12 23:35:04,502] - [bucket_helper] [140098712618752] - INFO - waiting for memcached bucket : metadata in 172.23.123.61 to accept set ops [2021-02-12 23:35:04,611] - [data_helper] [140098712618752] - INFO - creating direct client 172.23.123.61:11210 metadata [2021-02-12 23:35:04,729] - [data_helper] [140098712618752] - INFO - creating direct client 172.23.123.61:11210 metadata [2021-02-12 23:35:04,978] - [task] [140098712618752] - WARNING - vbucket map not ready after try 4 [2021-02-12 23:35:04,978] - [bucket_helper] [140098712618752] - INFO - waiting for memcached bucket : metadata in 172.23.123.61 to accept set ops [2021-02-12 23:35:05,084] - [data_helper] [140098712618752] - INFO - creating direct client 172.23.123.61:11210 metadata [2021-02-12 23:35:05,198] - [data_helper] [140098712618752] - INFO - creating direct client 172.23.123.61:11210 metadata [2021-02-12 23:35:05,266] - [task] [140098712618752] - WARNING - vbucket map not ready after try 5 [2021-02-12 23:35:05,336] - [rest_client] [140096917202688] - INFO - http://172.23.122.117:8091/pools/default/buckets/dst_bucket/scopes with params: name=scope_0 [2021-02-12 23:35:05,336] - [rest_client] [140096917202688] - INFO - Scope created dst_bucket->scope_0 {'uid': '1'} [2021-02-12 23:35:05,363] - [rest_client] [140096917202688] - INFO - http://172.23.122.117:8091/pools/default/buckets/dst_bucket/scopes with params: name=scope_1 [2021-02-12 23:35:05,363] - [rest_client] [140096917202688] - INFO - Scope created dst_bucket->scope_1 {'uid': '2'} [2021-02-12 23:35:05,393] - [rest_client] [140096917202688] - INFO - http://172.23.122.117:8091/pools/default/buckets/dst_bucket/scopes with params: name=scope_2 [2021-02-12 23:35:05,393] - [rest_client] [140096917202688] - INFO - Scope created dst_bucket->scope_2 {'uid': '3'} [2021-02-12 23:35:05,421] - [rest_client] [140096917202688] - INFO - http://172.23.122.117:8091/pools/default/buckets/dst_bucket/scopes with params: name=scope_3 [2021-02-12 23:35:05,421] - [rest_client] [140096917202688] - INFO - Scope created dst_bucket->scope_3 {'uid': '4'} [2021-02-12 23:35:05,453] - [rest_client] [140096917202688] - INFO - http://172.23.122.117:8091/pools/default/buckets/dst_bucket/scopes with params: name=scope_4 [2021-02-12 23:35:05,453] - [rest_client] [140096917202688] - INFO - Scope created dst_bucket->scope_4 {'uid': '5'} [2021-02-12 23:35:05,479] - [rest_client] [140096917202688] - INFO - http://172.23.122.117:8091/pools/default/buckets/src_bucket/scopes with params: name=scope_0 [2021-02-12 23:35:05,479] - [rest_client] [140096917202688] - INFO - Scope created src_bucket->scope_0 {'uid': '1'} [2021-02-12 23:35:05,511] - [rest_client] [140096917202688] - INFO - http://172.23.122.117:8091/pools/default/buckets/src_bucket/scopes with params: name=scope_1 [2021-02-12 23:35:05,511] - [rest_client] [140096917202688] - INFO - Scope created src_bucket->scope_1 {'uid': '2'} [2021-02-12 23:35:05,540] - [rest_client] [140096917202688] - INFO - http://172.23.122.117:8091/pools/default/buckets/src_bucket/scopes with params: name=scope_2 [2021-02-12 23:35:05,540] - [rest_client] [140096917202688] - INFO - Scope created src_bucket->scope_2 {'uid': '3'} [2021-02-12 23:35:05,568] - [rest_client] [140096917202688] - INFO - http://172.23.122.117:8091/pools/default/buckets/src_bucket/scopes with params: name=scope_3 [2021-02-12 23:35:05,568] - [rest_client] [140096917202688] - INFO - Scope created src_bucket->scope_3 {'uid': '4'} [2021-02-12 23:35:05,599] - [rest_client] [140096917202688] - INFO - http://172.23.122.117:8091/pools/default/buckets/src_bucket/scopes with params: name=scope_4 [2021-02-12 23:35:05,600] - [rest_client] [140096917202688] - INFO - Scope created src_bucket->scope_4 {'uid': '5'} [2021-02-12 23:35:05,627] - [rest_client] [140096917202688] - INFO - http://172.23.122.117:8091/pools/default/buckets/dst_bucket/scopes/scope_1/collections with params: name=coll_0 [2021-02-12 23:35:05,627] - [rest_client] [140096917202688] - INFO - Collection created dst_bucket->scope_1->coll_0 manifest:{'uid': '6'} [2021-02-12 23:35:05,656] - [rest_client] [140096917202688] - INFO - http://172.23.122.117:8091/pools/default/buckets/dst_bucket/scopes/scope_1/collections with params: name=coll_1 [2021-02-12 23:35:05,656] - [rest_client] [140096917202688] - INFO - Collection created dst_bucket->scope_1->coll_1 manifest:{'uid': '7'} [2021-02-12 23:35:05,685] - [rest_client] [140096917202688] - INFO - http://172.23.122.117:8091/pools/default/buckets/dst_bucket/scopes/scope_1/collections with params: name=coll_2 [2021-02-12 23:35:05,685] - [rest_client] [140096917202688] - INFO - Collection created dst_bucket->scope_1->coll_2 manifest:{'uid': '8'} [2021-02-12 23:35:05,715] - [rest_client] [140096917202688] - INFO - http://172.23.122.117:8091/pools/default/buckets/dst_bucket/scopes/scope_1/collections with params: name=coll_3 [2021-02-12 23:35:05,715] - [rest_client] [140096917202688] - INFO - Collection created dst_bucket->scope_1->coll_3 manifest:{'uid': '9'} [2021-02-12 23:35:05,747] - [rest_client] [140096917202688] - INFO - http://172.23.122.117:8091/pools/default/buckets/dst_bucket/scopes/scope_1/collections with params: name=coll_4 [2021-02-12 23:35:05,747] - [rest_client] [140096917202688] - INFO - Collection created dst_bucket->scope_1->coll_4 manifest:{'uid': 'a'} [2021-02-12 23:35:05,778] - [rest_client] [140096917202688] - INFO - http://172.23.122.117:8091/pools/default/buckets/src_bucket/scopes/scope_1/collections with params: name=coll_0 [2021-02-12 23:35:05,778] - [rest_client] [140096917202688] - INFO - Collection created src_bucket->scope_1->coll_0 manifest:{'uid': '6'} [2021-02-12 23:35:05,821] - [rest_client] [140096917202688] - INFO - http://172.23.122.117:8091/pools/default/buckets/src_bucket/scopes/scope_1/collections with params: name=coll_1 [2021-02-12 23:35:05,822] - [rest_client] [140096917202688] - INFO - Collection created src_bucket->scope_1->coll_1 manifest:{'uid': '7'} [2021-02-12 23:35:05,850] - [rest_client] [140096917202688] - INFO - http://172.23.122.117:8091/pools/default/buckets/src_bucket/scopes/scope_1/collections with params: name=coll_2 [2021-02-12 23:35:05,850] - [rest_client] [140096917202688] - INFO - Collection created src_bucket->scope_1->coll_2 manifest:{'uid': '8'} [2021-02-12 23:35:05,879] - [rest_client] [140096917202688] - INFO - http://172.23.122.117:8091/pools/default/buckets/src_bucket/scopes/scope_1/collections with params: name=coll_3 [2021-02-12 23:35:05,879] - [rest_client] [140096917202688] - INFO - Collection created src_bucket->scope_1->coll_3 manifest:{'uid': '9'} [2021-02-12 23:35:05,908] - [rest_client] [140096917202688] - INFO - http://172.23.122.117:8091/pools/default/buckets/src_bucket/scopes/scope_1/collections with params: name=coll_4 [2021-02-12 23:35:05,908] - [rest_client] [140096917202688] - INFO - Collection created src_bucket->scope_1->coll_4 manifest:{'uid': 'a'} [2021-02-12 23:35:07,158] - [eventing_base] [140096917202688] - INFO - saving function bucket_op [2021-02-12 23:35:08,351] - [eventing_base] [140096917202688] - INFO - saving function timers [2021-02-12 23:35:09,553] - [eventing_base] [140096917202688] - INFO - saving function sbm [2021-02-12 23:35:10,747] - [eventing_base] [140096917202688] - INFO - saving function curl [2021-02-12 23:35:11,954] - [eventing_base] [140096917202688] - INFO - saving function n1ql [2021-02-12 23:35:13,196] - [basetestcase] [140096917202688] - INFO - sleep for 5 secs. Waiting for bucket_op to deployed... ... [2021-02-12 23:35:18,233] - [basetestcase] [140096917202688] - INFO - sleep for 5 secs. Waiting for bucket_op to deployed... ... [2021-02-12 23:35:23,283] - [basetestcase] [140096917202688] - INFO - sleep for 5 secs. Waiting for bucket_op to deployed... ... [2021-02-12 23:35:28,333] - [basetestcase] [140096917202688] - INFO - sleep for 5 secs. Waiting for bucket_op to deployed... ... [2021-02-12 23:35:34,633] - [basetestcase] [140096917202688] - INFO - sleep for 5 secs. Waiting for timers to deployed... ... [2021-02-12 23:35:39,674] - [basetestcase] [140096917202688] - INFO - sleep for 5 secs. Waiting for timers to deployed... ... [2021-02-12 23:35:44,722] - [basetestcase] [140096917202688] - INFO - sleep for 5 secs. Waiting for timers to deployed... ... [2021-02-12 23:35:49,764] - [basetestcase] [140096917202688] - INFO - sleep for 5 secs. Waiting for timers to deployed... ... [2021-02-12 23:35:56,133] - [basetestcase] [140096917202688] - INFO - sleep for 5 secs. Waiting for sbm to deployed... ... [2021-02-12 23:36:01,175] - [basetestcase] [140096917202688] - INFO - sleep for 5 secs. Waiting for sbm to deployed... ... [2021-02-12 23:36:06,217] - [basetestcase] [140096917202688] - INFO - sleep for 5 secs. Waiting for sbm to deployed... ... [2021-02-12 23:36:11,256] - [basetestcase] [140096917202688] - INFO - sleep for 5 secs. Waiting for sbm to deployed... ... [2021-02-12 23:36:17,502] - [basetestcase] [140096917202688] - INFO - sleep for 5 secs. Waiting for curl to deployed... ... [2021-02-12 23:36:22,542] - [basetestcase] [140096917202688] - INFO - sleep for 5 secs. Waiting for curl to deployed... ... [2021-02-12 23:36:27,583] - [basetestcase] [140096917202688] - INFO - sleep for 5 secs. Waiting for curl to deployed... ... [2021-02-12 23:36:32,622] - [basetestcase] [140096917202688] - INFO - sleep for 5 secs. Waiting for curl to deployed... ... [2021-02-12 23:36:38,884] - [basetestcase] [140096917202688] - INFO - sleep for 5 secs. Waiting for n1ql to deployed... ... [2021-02-12 23:36:43,923] - [basetestcase] [140096917202688] - INFO - sleep for 5 secs. Waiting for n1ql to deployed... ... [2021-02-12 23:36:48,967] - [basetestcase] [140096917202688] - INFO - sleep for 5 secs. Waiting for n1ql to deployed... ... [2021-02-12 23:36:54,006] - [basetestcase] [140096917202688] - INFO - sleep for 5 secs. Waiting for n1ql to deployed... ... [2021-02-12 23:36:59,065] - [basetestcase] [140096917202688] - INFO - list of eventing nodes in cluster: [ip:172.23.122.117 port:8091 ssh_username:root, ip:172.23.122.90 port:8091 ssh_username:root, ip:172.23.123.62 port:8091 ssh_username:root] [2021-02-12 23:36:59,382] - [task] [140098712618752] - INFO - java -jar java_sdk_client/collections/target/javaclient/javaclient.jar -i 172.23.123.61 -u Administrator -p password -b src_bucket -s _default -c _default -n 40320 -pc 100 -pu 0 -pd 0 -l uniform -dsn 1 -dpx doc_ -dt Person -de 0 -ds 500 -ac False -st 0 -en 40319 -o False -sd False [2021-02-12 23:37:02,161] - [rest_client] [140096917202688] - INFO - rebalance reached >100% in 3.005222797393799 seconds [2021-02-12 23:37:10,976] - [task] [140098712618752] - INFO - java -jar java_sdk_client/collections/target/javaclient/javaclient.jar -i 172.23.123.61 -u Administrator -p password -b src_bucket -s scope_1 -c coll_1 -n 40320 -pc 100 -pu 0 -pd 0 -l uniform -dsn 1 -dpx doc_ -dt Person -de 0 -ds 500 -ac False -st 0 -en 40319 -o False -sd False [2021-02-12 23:37:21,998] - [rest_client] [140098712618752] - INFO - rebalance params : {'knownNodes': 'ns_1@172.23.122.117,ns_1@172.23.122.82,ns_1@172.23.122.90,ns_1@172.23.122.97,ns_1@172.23.123.61,ns_1@172.23.123.62', 'ejectedNodes': 'ns_1@172.23.122.117', 'user': 'Administrator', 'password': 'password'} [2021-02-12 23:37:22,029] - [rest_client] [140098712618752] - INFO - rebalance operation started [2021-02-12 23:37:22,032] - [rest_client] [140098712618752] - INFO - rebalance percentage : 0.00 % [2021-02-12 23:37:22,032] - [task] [140098712618752] - INFO - Rebalance - status: running, progress: 0.00% [2021-02-12 23:37:32,057] - [rest_client] [140098712618752] - INFO - rebalance percentage : 33.00 % [2021-02-12 23:37:32,057] - [task] [140098712618752] - INFO - Rebalance - status: running, progress: 33.00% [2021-02-12 23:37:42,085] - [task] [140098712618752] - INFO - Rebalance - status: none, progress: 100.00% [2021-02-12 23:37:42,102] - [task] [140098712618752] - INFO - rebalancing was completed with progress: 100% in 20.073195219039917 sec [2021-02-12 23:37:42,109] - [basetestcase] [140096917202688] - INFO - list of eventing nodes in cluster: [ip:172.23.122.90 port:8091 ssh_username:root, ip:172.23.123.62 port:8091 ssh_username:root] [2021-02-12 23:37:42,114] - [tuq_helper] [140096917202688] - INFO - RUN QUERY create primary index on dst_bucket.scope_1.coll_3 [2021-02-12 23:37:42,119] - [rest_client] [140096917202688] - INFO - query params : statement=create+primary+index+on+dst_bucket.scope_1.coll_3 [2021-02-12 23:37:44,265] - [tuq_helper] [140096917202688] - INFO - TOTAL ELAPSED TIME: 2.14379563s [2021-02-12 23:37:44,283] - [basetestcase] [140096917202688] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root] [2021-02-12 23:37:44,287] - [rest_client] [140096917202688] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_3 [2021-02-12 23:37:44,296] - [eventing_base] [140096917202688] - INFO - Final docs count... Current : 40320 Expected : 40320 [2021-02-12 23:37:44,307] - [basetestcase] [140096917202688] - INFO - list of eventing nodes in cluster: [ip:172.23.122.90 port:8091 ssh_username:root, ip:172.23.123.62 port:8091 ssh_username:root] [2021-02-12 23:37:44,311] - [tuq_helper] [140096917202688] - INFO - RUN QUERY create primary index on dst_bucket.scope_1.coll_4 [2021-02-12 23:37:44,314] - [rest_client] [140096917202688] - INFO - query params : statement=create+primary+index+on+dst_bucket.scope_1.coll_4 [2021-02-12 23:37:47,335] - [tuq_helper] [140096917202688] - INFO - TOTAL ELAPSED TIME: 3.019387739s [2021-02-12 23:37:47,362] - [basetestcase] [140096917202688] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root] [2021-02-12 23:37:47,366] - [rest_client] [140096917202688] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-12 23:37:47,376] - [eventing_base] [140096917202688] - INFO - Final docs count... Current : 40320 Expected : 40320 [2021-02-12 23:37:47,384] - [basetestcase] [140096917202688] - INFO - list of eventing nodes in cluster: [ip:172.23.122.90 port:8091 ssh_username:root, ip:172.23.123.62 port:8091 ssh_username:root] [2021-02-12 23:37:47,389] - [tuq_helper] [140096917202688] - INFO - RUN QUERY create primary index on src_bucket.scope_1.coll_1 [2021-02-12 23:37:47,393] - [rest_client] [140096917202688] - INFO - query params : statement=create+primary+index+on+src_bucket.scope_1.coll_1 [2021-02-12 23:37:49,821] - [tuq_helper] [140096917202688] - INFO - TOTAL ELAPSED TIME: 2.425782542s [2021-02-12 23:37:49,839] - [basetestcase] [140096917202688] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root] [2021-02-12 23:37:49,843] - [rest_client] [140096917202688] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.scope_1.coll_1 [2021-02-12 23:37:49,851] - [eventing_base] [140096917202688] - INFO - Final docs count... Current : 80640 Expected : 80640 [2021-02-12 23:37:49,859] - [basetestcase] [140096917202688] - INFO - list of eventing nodes in cluster: [ip:172.23.122.90 port:8091 ssh_username:root, ip:172.23.123.62 port:8091 ssh_username:root] [2021-02-12 23:37:49,889] - [rest_client] [140096917202688] - INFO - http://172.23.123.61:8091/pools/default/buckets/dst_bucket/scopes/scope_1/collections with params: name=coll_0 [2021-02-12 23:37:49,889] - [rest_client] [140096917202688] - INFO - Collection created dst_bucket->scope_1->coll_0 manifest:{'uid': 'e'} [2021-02-12 23:37:49,925] - [rest_client] [140096917202688] - INFO - http://172.23.123.61:8091/pools/default/buckets/dst_bucket/scopes/scope_1/collections with params: name=coll_1 [2021-02-12 23:37:49,925] - [rest_client] [140096917202688] - INFO - Collection created dst_bucket->scope_1->coll_1 manifest:{'uid': 'f'} [2021-02-12 23:37:49,981] - [rest_client] [140096917202688] - INFO - http://172.23.123.61:8091/pools/default/buckets/dst_bucket/scopes/scope_1/collections with params: name=coll_2 [2021-02-12 23:37:49,981] - [rest_client] [140096917202688] - INFO - Collection created dst_bucket->scope_1->coll_2 manifest:{'uid': '10'} [2021-02-12 23:37:50,131] - [rest_client] [140098712618752] - INFO - rebalance params : {'knownNodes': 'ns_1@172.23.122.82,ns_1@172.23.122.90,ns_1@172.23.122.97,ns_1@172.23.123.61,ns_1@172.23.123.62', 'ejectedNodes': 'ns_1@172.23.122.90', 'user': 'Administrator', 'password': 'password'} [2021-02-12 23:37:50,164] - [rest_client] [140098712618752] - INFO - rebalance operation started [2021-02-12 23:37:50,165] - [task] [140098712618752] - INFO - java -jar java_sdk_client/collections/target/javaclient/javaclient.jar -i 172.23.123.61 -u Administrator -p password -b src_bucket -s _default -c _default -n 40320 -pc 0 -pu 0 -pd 100 -l uniform -dsn 1 -dpx doc_ -dt Person -de 0 -ds 500 -ac False -st 0 -en 40319 -o False -sd False [2021-02-12 23:37:54,422] - [rest_client] [140098712618752] - INFO - rebalance percentage : 40.00 % [2021-02-12 23:37:54,422] - [task] [140098712618752] - INFO - Rebalance - status: running, progress: 40.00% [2021-02-12 23:37:54,434] - [task] [140098712618752] - INFO - java -jar java_sdk_client/collections/target/javaclient/javaclient.jar -i 172.23.123.61 -u Administrator -p password -b src_bucket -s scope_1 -c coll_1 -n 40320 -pc 0 -pu 0 -pd 100 -l uniform -dsn 1 -dpx doc_ -dt Person -de 0 -ds 500 -ac False -st 0 -en 40319 -o False -sd False [2021-02-12 23:38:01,413] - [rest_client] [140096917202688] - INFO - rebalance reached >100% in 3.0064778327941895 seconds [2021-02-12 23:38:05,421] - [rest_client] [140098712618752] - INFO - rebalance percentage : 41.00 % [2021-02-12 23:38:05,421] - [task] [140098712618752] - INFO - Rebalance - status: running, progress: 41.00% [2021-02-12 23:38:15,448] - [task] [140098712618752] - INFO - Rebalance - status: none, progress: 100.00% [2021-02-12 23:38:15,467] - [task] [140098712618752] - INFO - rebalancing was completed with progress: 100% in 25.30224323272705 sec [2021-02-12 23:38:15,477] - [basetestcase] [140096917202688] - INFO - list of eventing nodes in cluster: [ip:172.23.123.62 port:8091 ssh_username:root] [2021-02-12 23:38:15,482] - [tuq_helper] [140096917202688] - INFO - RUN QUERY create primary index on dst_bucket.scope_1.coll_3 [2021-02-12 23:38:15,486] - [rest_client] [140096917202688] - INFO - query params : statement=create+primary+index+on+dst_bucket.scope_1.coll_3 [2021-02-12 23:38:15,490] - [rest_client] [140096917202688] - ERROR - POST http://172.23.122.82:8093/query?statement=create+primary+index+on+dst_bucket.scope_1.coll_3 body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 500 reason: unknown b'{\n"requestID": "1d035d1c-b55a-4052-8bce-d90609ecae37",\n"signature": null,\n"results": [\n],\n"errors": [{"code":5000,"msg":"GSI CreatePrimaryIndex() - cause: Index #primary already exists."}],\n"status": "errors",\n"metrics": {"elapsedTime": "2.38304ms","executionTime": "2.326362ms","resultCount": 0,"resultSize": 0,"serviceLoad": 6,"errorCount": 1}\n}\n' auth: Administrator:password [2021-02-12 23:38:15,499] - [basetestcase] [140096917202688] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root] [2021-02-12 23:38:15,502] - [rest_client] [140096917202688] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_3 [2021-02-12 23:38:15,511] - [basetestcase] [140096917202688] - INFO - sleep for 30 secs. Waiting for handler code dst_bucket.scope_1.coll_3 to complete bucket operations... Current : 21826 Expected : 0 ... [2021-02-12 23:38:45,560] - [basetestcase] [140096917202688] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root] [2021-02-12 23:38:45,565] - [rest_client] [140096917202688] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_3 [2021-02-12 23:38:45,580] - [basetestcase] [140096917202688] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root] [2021-02-12 23:38:45,584] - [rest_client] [140096917202688] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_3 [2021-02-12 23:38:45,587] - [eventing_base] [140096917202688] - INFO - Final docs count... Current : 0 Expected : 0 [2021-02-12 23:38:45,594] - [basetestcase] [140096917202688] - INFO - list of eventing nodes in cluster: [ip:172.23.123.62 port:8091 ssh_username:root] [2021-02-12 23:38:45,598] - [tuq_helper] [140096917202688] - INFO - RUN QUERY create primary index on dst_bucket.scope_1.coll_4 [2021-02-12 23:38:45,602] - [rest_client] [140096917202688] - INFO - query params : statement=create+primary+index+on+dst_bucket.scope_1.coll_4 [2021-02-12 23:38:45,604] - [rest_client] [140096917202688] - ERROR - POST http://172.23.122.82:8093/query?statement=create+primary+index+on+dst_bucket.scope_1.coll_4 body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 500 reason: unknown b'{\n"requestID": "6da67eaa-8ce3-4775-ba95-7ef063014321",\n"signature": null,\n"results": [\n],\n"errors": [{"code":5000,"msg":"GSI CreatePrimaryIndex() - cause: Index #primary already exists."}],\n"status": "errors",\n"metrics": {"elapsedTime": "1.333411ms","executionTime": "1.292242ms","resultCount": 0,"resultSize": 0,"serviceLoad": 6,"errorCount": 1}\n}\n' auth: Administrator:password [2021-02-12 23:38:45,611] - [basetestcase] [140096917202688] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root] [2021-02-12 23:38:45,615] - [rest_client] [140096917202688] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+dst_bucket.scope_1.coll_4 [2021-02-12 23:38:45,618] - [eventing_base] [140096917202688] - INFO - Final docs count... Current : 0 Expected : 0 [2021-02-12 23:38:45,625] - [basetestcase] [140096917202688] - INFO - list of eventing nodes in cluster: [ip:172.23.123.62 port:8091 ssh_username:root] [2021-02-12 23:38:45,629] - [tuq_helper] [140096917202688] - INFO - RUN QUERY create primary index on src_bucket.scope_1.coll_1 [2021-02-12 23:38:45,632] - [rest_client] [140096917202688] - INFO - query params : statement=create+primary+index+on+src_bucket.scope_1.coll_1 [2021-02-12 23:38:45,634] - [rest_client] [140096917202688] - ERROR - POST http://172.23.122.82:8093/query?statement=create+primary+index+on+src_bucket.scope_1.coll_1 body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 500 reason: unknown b'{\n"requestID": "2e779b7e-63bb-4cd4-9435-0cad918f940a",\n"signature": null,\n"results": [\n],\n"errors": [{"code":5000,"msg":"GSI CreatePrimaryIndex() - cause: Index #primary already exists."}],\n"status": "errors",\n"metrics": {"elapsedTime": "1.105255ms","executionTime": "1.068839ms","resultCount": 0,"resultSize": 0,"serviceLoad": 6,"errorCount": 1}\n}\n' auth: Administrator:password [2021-02-12 23:38:45,641] - [basetestcase] [140096917202688] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root] [2021-02-12 23:38:45,645] - [rest_client] [140096917202688] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.scope_1.coll_1 [2021-02-12 23:38:45,655] - [basetestcase] [140096917202688] - INFO - sleep for 30 secs. Waiting for handler code src_bucket.scope_1.coll_1 to complete bucket operations... Current : 40392 Expected : 40320 ... [2021-02-12 23:39:15,670] - [basetestcase] [140096917202688] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root] [2021-02-12 23:39:15,675] - [rest_client] [140096917202688] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.scope_1.coll_1 [2021-02-12 23:39:15,692] - [basetestcase] [140096917202688] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root] [2021-02-12 23:39:15,696] - [rest_client] [140096917202688] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.scope_1.coll_1 [2021-02-12 23:39:15,699] - [basetestcase] [140096917202688] - INFO - sleep for 30 secs. Waiting for handler code src_bucket.scope_1.coll_1 to complete bucket operations... Current : 40392 Expected : 40320 ... [2021-02-12 23:39:45,746] - [basetestcase] [140096917202688] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root] [2021-02-12 23:39:45,753] - [rest_client] [140096917202688] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.scope_1.coll_1 [2021-02-12 23:39:45,770] - [basetestcase] [140096917202688] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root] [2021-02-12 23:39:45,774] - [rest_client] [140096917202688] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.scope_1.coll_1 [2021-02-12 23:39:45,777] - [basetestcase] [140096917202688] - INFO - sleep for 30 secs. Waiting for handler code src_bucket.scope_1.coll_1 to complete bucket operations... Current : 40392 Expected : 40320 ... [2021-02-12 23:40:15,819] - [basetestcase] [140096917202688] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root] [2021-02-12 23:40:15,825] - [rest_client] [140096917202688] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.scope_1.coll_1 [2021-02-12 23:40:15,841] - [basetestcase] [140096917202688] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root] [2021-02-12 23:40:15,845] - [rest_client] [140096917202688] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.scope_1.coll_1 [2021-02-12 23:40:15,848] - [basetestcase] [140096917202688] - INFO - sleep for 30 secs. Waiting for handler code src_bucket.scope_1.coll_1 to complete bucket operations... Current : 40392 Expected : 40320 ... [2021-02-12 23:40:45,893] - [basetestcase] [140096917202688] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root] [2021-02-12 23:40:45,899] - [rest_client] [140096917202688] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.scope_1.coll_1 [2021-02-12 23:40:45,915] - [basetestcase] [140096917202688] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root] [2021-02-12 23:40:45,920] - [rest_client] [140096917202688] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.scope_1.coll_1 [2021-02-12 23:40:45,922] - [basetestcase] [140096917202688] - INFO - sleep for 30 secs. Waiting for handler code src_bucket.scope_1.coll_1 to complete bucket operations... Current : 40392 Expected : 40320 ... [2021-02-12 23:41:15,961] - [basetestcase] [140096917202688] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root] [2021-02-12 23:41:15,967] - [rest_client] [140096917202688] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.scope_1.coll_1 [2021-02-12 23:41:15,985] - [basetestcase] [140096917202688] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root] [2021-02-12 23:41:15,990] - [rest_client] [140096917202688] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.scope_1.coll_1 [2021-02-12 23:41:15,993] - [basetestcase] [140096917202688] - INFO - sleep for 30 secs. Waiting for handler code src_bucket.scope_1.coll_1 to complete bucket operations... Current : 40392 Expected : 40320 ... [2021-02-12 23:41:46,037] - [basetestcase] [140096917202688] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root] [2021-02-12 23:41:46,043] - [rest_client] [140096917202688] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.scope_1.coll_1 [2021-02-12 23:41:46,059] - [basetestcase] [140096917202688] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root] [2021-02-12 23:41:46,063] - [rest_client] [140096917202688] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.scope_1.coll_1 [2021-02-12 23:41:46,066] - [basetestcase] [140096917202688] - INFO - sleep for 30 secs. Waiting for handler code src_bucket.scope_1.coll_1 to complete bucket operations... Current : 40392 Expected : 40320 ... [2021-02-12 23:42:16,108] - [basetestcase] [140096917202688] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root] [2021-02-12 23:42:16,113] - [rest_client] [140096917202688] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.scope_1.coll_1 [2021-02-12 23:42:16,127] - [basetestcase] [140096917202688] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root] [2021-02-12 23:42:16,131] - [rest_client] [140096917202688] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.scope_1.coll_1 [2021-02-12 23:42:16,134] - [basetestcase] [140096917202688] - INFO - sleep for 30 secs. Waiting for handler code src_bucket.scope_1.coll_1 to complete bucket operations... Current : 40392 Expected : 40320 ... [2021-02-12 23:42:46,149] - [basetestcase] [140096917202688] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root] [2021-02-12 23:42:46,155] - [rest_client] [140096917202688] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.scope_1.coll_1 [2021-02-12 23:42:46,170] - [basetestcase] [140096917202688] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root] [2021-02-12 23:42:46,174] - [rest_client] [140096917202688] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.scope_1.coll_1 [2021-02-12 23:42:46,177] - [basetestcase] [140096917202688] - INFO - sleep for 30 secs. Waiting for handler code src_bucket.scope_1.coll_1 to complete bucket operations... Current : 40392 Expected : 40320 ... [2021-02-12 23:43:16,216] - [basetestcase] [140096917202688] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root] [2021-02-12 23:43:16,222] - [rest_client] [140096917202688] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.scope_1.coll_1 [2021-02-12 23:43:16,238] - [basetestcase] [140096917202688] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root] [2021-02-12 23:43:16,242] - [rest_client] [140096917202688] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.scope_1.coll_1 [2021-02-12 23:43:16,244] - [basetestcase] [140096917202688] - INFO - sleep for 30 secs. Waiting for handler code src_bucket.scope_1.coll_1 to complete bucket operations... Current : 40392 Expected : 40320 ... [2021-02-12 23:43:46,263] - [basetestcase] [140096917202688] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root] [2021-02-12 23:43:46,268] - [rest_client] [140096917202688] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.scope_1.coll_1 [2021-02-12 23:43:46,283] - [basetestcase] [140096917202688] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root] [2021-02-12 23:43:46,288] - [rest_client] [140096917202688] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.scope_1.coll_1 [2021-02-12 23:43:46,291] - [basetestcase] [140096917202688] - INFO - sleep for 30 secs. Waiting for handler code src_bucket.scope_1.coll_1 to complete bucket operations... Current : 40392 Expected : 40320 ... [2021-02-12 23:44:16,336] - [basetestcase] [140096917202688] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root] [2021-02-12 23:44:16,342] - [rest_client] [140096917202688] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.scope_1.coll_1 [2021-02-12 23:44:16,355] - [eventing_base] [140096917202688] - INFO - Compacting bucket : dst_bucket [2021-02-12 23:44:16,360] - [rest_client] [140096917202688] - INFO - bucket compaction successful [2021-02-12 23:44:16,360] - [eventing_base] [140096917202688] - INFO - Compacting bucket : metadata [2021-02-12 23:44:16,364] - [rest_client] [140096917202688] - INFO - bucket compaction successful [2021-02-12 23:44:16,364] - [eventing_base] [140096917202688] - INFO - Compacting bucket : src_bucket [2021-02-12 23:44:16,372] - [rest_client] [140096917202688] - INFO - bucket compaction successful [2021-02-12 23:44:16,381] - [basetestcase] [140096917202688] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root] [2021-02-12 23:44:16,385] - [rest_client] [140096917202688] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.scope_1.coll_1 [2021-02-12 23:44:16,389] - [basetestcase] [140096917202688] - INFO - sleep for 30 secs. Waiting for handler code src_bucket.scope_1.coll_1 to complete bucket operations... Current : 40392 Expected : 40320 ... [2021-02-12 23:44:46,425] - [basetestcase] [140096917202688] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root] [2021-02-12 23:44:46,431] - [rest_client] [140096917202688] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.scope_1.coll_1 [2021-02-12 23:44:46,446] - [basetestcase] [140096917202688] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root] [2021-02-12 23:44:46,450] - [rest_client] [140096917202688] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.scope_1.coll_1 [2021-02-12 23:44:46,453] - [basetestcase] [140096917202688] - INFO - sleep for 30 secs. Waiting for handler code src_bucket.scope_1.coll_1 to complete bucket operations... Current : 40392 Expected : 40320 ... [2021-02-12 23:45:16,494] - [basetestcase] [140096917202688] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root] [2021-02-12 23:45:16,500] - [rest_client] [140096917202688] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.scope_1.coll_1 [2021-02-12 23:45:16,514] - [basetestcase] [140096917202688] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root] [2021-02-12 23:45:16,519] - [rest_client] [140096917202688] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.scope_1.coll_1 [2021-02-12 23:45:16,521] - [basetestcase] [140096917202688] - INFO - sleep for 30 secs. Waiting for handler code src_bucket.scope_1.coll_1 to complete bucket operations... Current : 40392 Expected : 40320 ... [2021-02-12 23:45:46,567] - [basetestcase] [140096917202688] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root] [2021-02-12 23:45:46,572] - [rest_client] [140096917202688] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.scope_1.coll_1 [2021-02-12 23:45:46,587] - [basetestcase] [140096917202688] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root] [2021-02-12 23:45:46,592] - [rest_client] [140096917202688] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.scope_1.coll_1 [2021-02-12 23:45:46,595] - [basetestcase] [140096917202688] - INFO - sleep for 30 secs. Waiting for handler code src_bucket.scope_1.coll_1 to complete bucket operations... Current : 40392 Expected : 40320 ... [2021-02-12 23:46:16,641] - [basetestcase] [140096917202688] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root] [2021-02-12 23:46:16,647] - [rest_client] [140096917202688] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.scope_1.coll_1 [2021-02-12 23:46:16,661] - [basetestcase] [140096917202688] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root] [2021-02-12 23:46:16,665] - [rest_client] [140096917202688] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.scope_1.coll_1 [2021-02-12 23:46:16,668] - [basetestcase] [140096917202688] - INFO - sleep for 30 secs. Waiting for handler code src_bucket.scope_1.coll_1 to complete bucket operations... Current : 40392 Expected : 40320 ... [2021-02-12 23:46:46,702] - [basetestcase] [140096917202688] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root] [2021-02-12 23:46:46,707] - [rest_client] [140096917202688] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.scope_1.coll_1 [2021-02-12 23:46:46,722] - [basetestcase] [140096917202688] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root] [2021-02-12 23:46:46,726] - [rest_client] [140096917202688] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.scope_1.coll_1 [2021-02-12 23:46:46,729] - [basetestcase] [140096917202688] - INFO - sleep for 30 secs. Waiting for handler code src_bucket.scope_1.coll_1 to complete bucket operations... Current : 40392 Expected : 40320 ... [2021-02-12 23:47:16,745] - [basetestcase] [140096917202688] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root] [2021-02-12 23:47:16,751] - [rest_client] [140096917202688] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.scope_1.coll_1 [2021-02-12 23:47:16,768] - [basetestcase] [140096917202688] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root] [2021-02-12 23:47:16,772] - [rest_client] [140096917202688] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.scope_1.coll_1 [2021-02-12 23:47:16,775] - [basetestcase] [140096917202688] - INFO - sleep for 30 secs. Waiting for handler code src_bucket.scope_1.coll_1 to complete bucket operations... Current : 40392 Expected : 40320 ... [2021-02-12 23:47:46,792] - [basetestcase] [140096917202688] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root] [2021-02-12 23:47:46,797] - [rest_client] [140096917202688] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.scope_1.coll_1 [2021-02-12 23:47:46,812] - [basetestcase] [140096917202688] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root] [2021-02-12 23:47:46,816] - [rest_client] [140096917202688] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.scope_1.coll_1 [2021-02-12 23:47:46,819] - [basetestcase] [140096917202688] - INFO - sleep for 30 secs. Waiting for handler code src_bucket.scope_1.coll_1 to complete bucket operations... Current : 40392 Expected : 40320 ... [2021-02-12 23:48:16,847] - [basetestcase] [140096917202688] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root] [2021-02-12 23:48:16,853] - [rest_client] [140096917202688] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.scope_1.coll_1 [2021-02-12 23:48:16,869] - [basetestcase] [140096917202688] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root] [2021-02-12 23:48:16,874] - [rest_client] [140096917202688] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.scope_1.coll_1 [2021-02-12 23:48:16,877] - [basetestcase] [140096917202688] - INFO - sleep for 30 secs. Waiting for handler code src_bucket.scope_1.coll_1 to complete bucket operations... Current : 40392 Expected : 40320 ... [2021-02-12 23:48:46,919] - [basetestcase] [140096917202688] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root] [2021-02-12 23:48:46,926] - [rest_client] [140096917202688] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.scope_1.coll_1 [2021-02-12 23:48:46,941] - [basetestcase] [140096917202688] - INFO - list of n1ql nodes in cluster: [ip:172.23.122.82 port:8091 ssh_username:root] [2021-02-12 23:48:46,946] - [rest_client] [140096917202688] - INFO - query params : statement=select+RAW%28count%28%2A%29%29+from+src_bucket.scope_1.coll_1 [2021-02-12 23:48:46,955] - [basetestcase] [140096917202688] - INFO - list of eventing nodes in cluster: [ip:172.23.123.62 port:8091 ssh_username:root] [2021-02-12 23:48:47,007] - [eventing_base] [140096917202688] - INFO - Stats for Node 172.23.123.62 is [ { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 6, "agg_messages_sent_to_worker": 105957, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_deletion": 103103, "dcp_deletion_sent_to_worker": 59453, "dcp_mutation": 154226, "dcp_mutation_sent_to_worker": 39746, "dcp_snapshot": 51017, "dcp_stream_req_counter": 1877, "dcp_streamend": 853, "dcp_streamreq": 1877, "dcp_system_event": 14500, "execution_stats": 803, "failure_stats": 803, "latency_stats": 803, "lcb_exception_stats": 803, "log_level": 1, "num_processed_events": 99199, "processed_events_size": 51761880, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 59453, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 39746, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 59453, "enqueued_dcp_mutation_msg_counter": 39746, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 38940, "lcb_retry_failure": 0, "messages_parsed": 105953, "num_processed_events": 99199, "on_delete_failure": 0, "on_delete_success": 39293, "on_update_failure": 37, "on_update_success": 20929, "processed_events_size": 51761880, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "64920": "2021-02-13T07:48:46Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 37, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "64920": "2021-02-13T07:48:46Z" }, "v8worker_events_lost": 0 }, "function_id": 1844693768, "function_name": "bucket_op", "gocb_creds_request_counter": 22, "internal_vb_distribution_stats": { "worker_bucket_op_0": "[0-1023]" }, "latency_percentile_stats": { "100": 2507100, "50": 400, "80": 900, "90": 1400, "95": 2500, "99": 9200 }, "lcb_creds_request_counter": 26806, "lcb_exception_stats": { "211": 37 }, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.123.62:8096", "start_vb": 0, "vb_count": 1024 } ], "vb_distribution_stats_from_metadata": { "172.23.123.62:8096": { "worker_bucket_op_0": "[0-1023]" } }, "worker_pids": { "worker_bucket_op_0": 64920 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 6, "agg_messages_sent_to_worker": 98789, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_deletion": 103877, "dcp_deletion_sent_to_worker": 60480, "dcp_mutation": 160112, "dcp_mutation_sent_to_worker": 31657, "dcp_snapshot": 51087, "dcp_stream_req_counter": 1877, "dcp_streamend": 853, "dcp_streamreq": 1877, "dcp_system_event": 9390, "execution_stats": 782, "failure_stats": 782, "latency_stats": 782, "lcb_exception_stats": 782, "log_level": 1, "num_processed_events": 92137, "processed_events_size": 44798884, "thr_count": 1, "thr_map": 1, "timer_events": 74236, "timer_responses_received": 58247, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 60480, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 31657, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 60480, "enqueued_dcp_mutation_msg_counter": 31657, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 33890, "lcb_retry_failure": 0, "messages_parsed": 98785, "num_processed_events": 92137, "on_delete_failure": 0, "on_delete_success": 40320, "on_update_failure": 0, "on_update_success": 17927, "processed_events_size": 44798884, "timer_callback_failure": 30, "timer_callback_success": 74206, "timer_cancel_counter": 0, "timer_create_counter": 58247, "timer_create_failure": 0, "timer_msg_counter": 74236, "timer_responses_sent": 0, "timestamp": { "64952": "2021-02-13T07:48:46Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 30, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "64952": "2021-02-13T07:48:46Z" }, "v8worker_events_lost": 0 }, "function_id": 364579283, "function_name": "timers", "gocb_creds_request_counter": 22, "internal_vb_distribution_stats": { "worker_timers_0": "[0-1023]" }, "latency_percentile_stats": { "100": 2507100, "50": 800, "80": 1200, "90": 1800, "95": 2400, "99": 5000 }, "lcb_creds_request_counter": 26806, "lcb_exception_stats": { "211": 30 }, "metastore_stats": { "metastore_delete_err": 0, "metastore_deletes": 0, "metastore_not_found": 0, "metastore_scan": 0, "metastore_scan_due": 0, "metastore_scan_err": 0, "metastore_set": 0, "metastore_set_err": 0 }, "planner_stats": [ { "host_name": "172.23.123.62:8096", "start_vb": 0, "vb_count": 1024 } ], "vb_distribution_stats_from_metadata": { "172.23.123.62:8096": { "worker_timers_0": "[0-1023]" } }, "worker_pids": { "worker_timers_0": 64952 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 3, "agg_messages_sent_to_worker": 67445, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_deletion": 68512, "dcp_deletion_sent_to_worker": 38731, "dcp_mutation": 70126, "dcp_mutation_sent_to_worker": 22186, "dcp_mutation_suppressed_counter": 28417, "dcp_snapshot": 50484, "dcp_stream_req_counter": 1878, "dcp_stream_req_err_counter": 1, "dcp_streamend": 853, "dcp_streamreq": 1877, "dcp_system_event": 6710, "execution_stats": 760, "failure_stats": 760, "latency_stats": 760, "lcb_exception_stats": 760, "log_level": 1, "num_processed_events": 60917, "processed_events_size": 30398048, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 38731, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 22186, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 38731, "enqueued_dcp_mutation_msg_counter": 22186, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 10195, "lcb_retry_failure": 0, "messages_parsed": 67441, "num_processed_events": 60917, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13461, "processed_events_size": 30398048, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "64985": "2021-02-13T07:48:46Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "64985": "2021-02-13T07:48:46Z" }, "v8worker_events_lost": 0 }, "function_id": 1218075759, "function_name": "sbm", "gocb_creds_request_counter": 22, "internal_vb_distribution_stats": { "worker_sbm_0": "[0-1023]" }, "latency_percentile_stats": { "100": 21600, "50": 100, "80": 500, "90": 900, "95": 1400, "99": 3100 }, "lcb_creds_request_counter": 26806, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.123.62:8096", "start_vb": 0, "vb_count": 1024 } ], "vb_distribution_stats_from_metadata": { "172.23.123.62:8096": { "worker_sbm_0": "[0-1023]" } }, "worker_pids": { "worker_sbm_0": 64985 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 5, "agg_messages_sent_to_worker": 82738, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_deletion": 99514, "dcp_deletion_sent_to_worker": 55746, "dcp_mutation": 105349, "dcp_mutation_sent_to_worker": 20557, "dcp_snapshot": 51735, "dcp_stream_req_counter": 1877, "dcp_streamend": 853, "dcp_streamreq": 1877, "dcp_system_event": 6910, "execution_stats": 739, "failure_stats": 739, "latency_stats": 739, "lcb_exception_stats": 739, "log_level": 1, "num_processed_events": 76303, "processed_events_size": 33573624, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 50537, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 50537, "dcp_delete_msg_counter": 55746, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 20557, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 55746, "enqueued_dcp_mutation_msg_counter": 20557, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 25766, "lcb_retry_failure": 0, "messages_parsed": 82734, "num_processed_events": 76303, "on_delete_failure": 0, "on_delete_success": 37114, "on_update_failure": 0, "on_update_success": 13423, "processed_events_size": 33573624, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "65060": "2021-02-13T07:48:46Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "65060": "2021-02-13T07:48:46Z" }, "v8worker_events_lost": 0 }, "function_id": 3750915646, "function_name": "curl", "gocb_creds_request_counter": 22, "internal_vb_distribution_stats": { "worker_curl_0": "[0-1023]" }, "latency_percentile_stats": { "100": 285100, "50": 1400, "80": 1900, "90": 2600, "95": 4000, "99": 11600 }, "lcb_creds_request_counter": 26806, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.123.62:8096", "start_vb": 0, "vb_count": 1024 } ], "vb_distribution_stats_from_metadata": { "172.23.123.62:8096": { "worker_curl_0": "[0-1023]" } }, "worker_pids": { "worker_curl_0": 65060 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 3, "agg_messages_sent_to_worker": 79248, "agg_queue_memory": 0, "agg_queue_memory_cap": 20971520, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_deletion": 95714, "dcp_deletion_sent_to_worker": 52205, "dcp_mutation": 104813, "dcp_mutation_sent_to_worker": 20714, "dcp_snapshot": 51167, "dcp_stream_req_counter": 1877, "dcp_streamend": 853, "dcp_streamreq": 1877, "dcp_system_event": 6830, "execution_stats": 718, "failure_stats": 718, "latency_stats": 718, "lcb_exception_stats": 718, "log_level": 1, "num_processed_events": 72919, "processed_events_size": 32751140, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 52205, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 20714, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 52205, "enqueued_dcp_mutation_msg_counter": 20714, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 24868, "lcb_retry_failure": 0, "messages_parsed": 79244, "num_processed_events": 72919, "on_delete_failure": 0, "on_delete_success": 34662, "on_update_failure": 0, "on_update_success": 13389, "processed_events_size": 32751140, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "65093": "2021-02-13T07:48:46Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "65093": "2021-02-13T07:48:46Z" }, "v8worker_events_lost": 0 }, "function_id": 4203222296, "function_name": "n1ql", "gocb_creds_request_counter": 22, "internal_vb_distribution_stats": { "worker_n1ql_0": "[0-1023]" }, "latency_percentile_stats": { "100": 188000, "50": 700, "80": 3100, "90": 4100, "95": 5400, "99": 10700 }, "lcb_creds_request_counter": 26806, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.123.62:8096", "start_vb": 0, "vb_count": 1024 } ], "vb_distribution_stats_from_metadata": { "172.23.123.62:8096": { "worker_n1ql_0": "[0-1023]" } }, "worker_pids": { "worker_n1ql_0": 65093 } } ] [2021-02-12 23:48:51,647] - [eventing_base] [140096917202688] - INFO - Stats for Node 172.23.123.62 is [ { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 6, "agg_messages_sent_to_worker": 105957, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_deletion": 103103, "dcp_deletion_sent_to_worker": 59453, "dcp_mutation": 154226, "dcp_mutation_sent_to_worker": 39746, "dcp_snapshot": 51017, "dcp_stream_req_counter": 1877, "dcp_streamend": 853, "dcp_streamreq": 1877, "dcp_system_event": 14500, "execution_stats": 803, "failure_stats": 803, "latency_stats": 803, "lcb_exception_stats": 803, "log_level": 1, "num_processed_events": 99199, "processed_events_size": 51761880, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 59453, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 39746, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 59453, "enqueued_dcp_mutation_msg_counter": 39746, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 38940, "lcb_retry_failure": 0, "messages_parsed": 105953, "num_processed_events": 99199, "on_delete_failure": 0, "on_delete_success": 39293, "on_update_failure": 37, "on_update_success": 20929, "processed_events_size": 51761880, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "64920": "2021-02-13T07:48:46Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 37, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "64920": "2021-02-13T07:48:46Z" }, "v8worker_events_lost": 0 }, "function_id": 1844693768, "function_name": "bucket_op", "gocb_creds_request_counter": 22, "internal_vb_distribution_stats": { "worker_bucket_op_0": "[0-1023]" }, "latency_percentile_stats": { "100": 2507100, "50": 400, "80": 900, "90": 1400, "95": 2500, "99": 9200 }, "lcb_creds_request_counter": 26806, "lcb_exception_stats": { "211": 37 }, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.123.62:8096", "start_vb": 0, "vb_count": 1024 } ], "vb_distribution_stats_from_metadata": { "172.23.123.62:8096": { "worker_bucket_op_0": "[0-1023]" } }, "worker_pids": { "worker_bucket_op_0": 64920 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 6, "agg_messages_sent_to_worker": 98789, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_deletion": 103877, "dcp_deletion_sent_to_worker": 60480, "dcp_mutation": 160112, "dcp_mutation_sent_to_worker": 31657, "dcp_snapshot": 51087, "dcp_stream_req_counter": 1877, "dcp_streamend": 853, "dcp_streamreq": 1877, "dcp_system_event": 9390, "execution_stats": 782, "failure_stats": 782, "latency_stats": 782, "lcb_exception_stats": 782, "log_level": 1, "num_processed_events": 92137, "processed_events_size": 44798884, "thr_count": 1, "thr_map": 1, "timer_events": 74236, "timer_responses_received": 58247, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 60480, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 31657, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 60480, "enqueued_dcp_mutation_msg_counter": 31657, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 33890, "lcb_retry_failure": 0, "messages_parsed": 98785, "num_processed_events": 92137, "on_delete_failure": 0, "on_delete_success": 40320, "on_update_failure": 0, "on_update_success": 17927, "processed_events_size": 44798884, "timer_callback_failure": 30, "timer_callback_success": 74206, "timer_cancel_counter": 0, "timer_create_counter": 58247, "timer_create_failure": 0, "timer_msg_counter": 74236, "timer_responses_sent": 0, "timestamp": { "64952": "2021-02-13T07:48:46Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 30, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "64952": "2021-02-13T07:48:46Z" }, "v8worker_events_lost": 0 }, "function_id": 364579283, "function_name": "timers", "gocb_creds_request_counter": 22, "internal_vb_distribution_stats": { "worker_timers_0": "[0-1023]" }, "latency_percentile_stats": { "100": 2507100, "50": 800, "80": 1200, "90": 1800, "95": 2400, "99": 5000 }, "lcb_creds_request_counter": 26806, "lcb_exception_stats": { "211": 30 }, "metastore_stats": { "metastore_delete_err": 0, "metastore_deletes": 0, "metastore_not_found": 0, "metastore_scan": 0, "metastore_scan_due": 0, "metastore_scan_err": 0, "metastore_set": 0, "metastore_set_err": 0 }, "planner_stats": [ { "host_name": "172.23.123.62:8096", "start_vb": 0, "vb_count": 1024 } ], "vb_distribution_stats_from_metadata": { "172.23.123.62:8096": { "worker_timers_0": "[0-1023]" } }, "worker_pids": { "worker_timers_0": 64952 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 3, "agg_messages_sent_to_worker": 67445, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_deletion": 68512, "dcp_deletion_sent_to_worker": 38731, "dcp_mutation": 70126, "dcp_mutation_sent_to_worker": 22186, "dcp_mutation_suppressed_counter": 28417, "dcp_snapshot": 50484, "dcp_stream_req_counter": 1878, "dcp_stream_req_err_counter": 1, "dcp_streamend": 853, "dcp_streamreq": 1877, "dcp_system_event": 6710, "execution_stats": 760, "failure_stats": 760, "latency_stats": 760, "lcb_exception_stats": 760, "log_level": 1, "num_processed_events": 60917, "processed_events_size": 30398048, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 38731, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 22186, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 38731, "enqueued_dcp_mutation_msg_counter": 22186, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 10195, "lcb_retry_failure": 0, "messages_parsed": 67441, "num_processed_events": 60917, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13461, "processed_events_size": 30398048, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "64985": "2021-02-13T07:48:46Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "64985": "2021-02-13T07:48:46Z" }, "v8worker_events_lost": 0 }, "function_id": 1218075759, "function_name": "sbm", "gocb_creds_request_counter": 22, "internal_vb_distribution_stats": { "worker_sbm_0": "[0-1023]" }, "latency_percentile_stats": { "100": 21600, "50": 100, "80": 500, "90": 900, "95": 1400, "99": 3100 }, "lcb_creds_request_counter": 26806, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.123.62:8096", "start_vb": 0, "vb_count": 1024 } ], "vb_distribution_stats_from_metadata": { "172.23.123.62:8096": { "worker_sbm_0": "[0-1023]" } }, "worker_pids": { "worker_sbm_0": 64985 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 5, "agg_messages_sent_to_worker": 82738, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_deletion": 99514, "dcp_deletion_sent_to_worker": 55746, "dcp_mutation": 105349, "dcp_mutation_sent_to_worker": 20557, "dcp_snapshot": 51735, "dcp_stream_req_counter": 1877, "dcp_streamend": 853, "dcp_streamreq": 1877, "dcp_system_event": 6910, "execution_stats": 739, "failure_stats": 739, "latency_stats": 739, "lcb_exception_stats": 739, "log_level": 1, "num_processed_events": 76303, "processed_events_size": 33573624, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 50537, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 50537, "dcp_delete_msg_counter": 55746, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 20557, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 55746, "enqueued_dcp_mutation_msg_counter": 20557, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 25766, "lcb_retry_failure": 0, "messages_parsed": 82734, "num_processed_events": 76303, "on_delete_failure": 0, "on_delete_success": 37114, "on_update_failure": 0, "on_update_success": 13423, "processed_events_size": 33573624, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "65060": "2021-02-13T07:48:46Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "65060": "2021-02-13T07:48:46Z" }, "v8worker_events_lost": 0 }, "function_id": 3750915646, "function_name": "curl", "gocb_creds_request_counter": 22, "internal_vb_distribution_stats": { "worker_curl_0": "[0-1023]" }, "latency_percentile_stats": { "100": 285100, "50": 1400, "80": 1900, "90": 2600, "95": 4000, "99": 11600 }, "lcb_creds_request_counter": 26806, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.123.62:8096", "start_vb": 0, "vb_count": 1024 } ], "vb_distribution_stats_from_metadata": { "172.23.123.62:8096": { "worker_curl_0": "[0-1023]" } }, "worker_pids": { "worker_curl_0": 65060 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 3, "agg_messages_sent_to_worker": 79248, "agg_queue_memory": 0, "agg_queue_memory_cap": 20971520, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_deletion": 95714, "dcp_deletion_sent_to_worker": 52205, "dcp_mutation": 104813, "dcp_mutation_sent_to_worker": 20714, "dcp_snapshot": 51167, "dcp_stream_req_counter": 1877, "dcp_streamend": 853, "dcp_streamreq": 1877, "dcp_system_event": 6830, "execution_stats": 718, "failure_stats": 718, "latency_stats": 718, "lcb_exception_stats": 718, "log_level": 1, "num_processed_events": 72919, "processed_events_size": 32751140, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 52205, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 20714, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 52205, "enqueued_dcp_mutation_msg_counter": 20714, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 24868, "lcb_retry_failure": 0, "messages_parsed": 79244, "num_processed_events": 72919, "on_delete_failure": 0, "on_delete_success": 34662, "on_update_failure": 0, "on_update_success": 13389, "processed_events_size": 32751140, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "65093": "2021-02-13T07:48:46Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "65093": "2021-02-13T07:48:46Z" }, "v8worker_events_lost": 0 }, "function_id": 4203222296, "function_name": "n1ql", "gocb_creds_request_counter": 22, "internal_vb_distribution_stats": { "worker_n1ql_0": "[0-1023]" }, "latency_percentile_stats": { "100": 188000, "50": 700, "80": 3100, "90": 4100, "95": 5400, "99": 10700 }, "lcb_creds_request_counter": 26806, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.123.62:8096", "start_vb": 0, "vb_count": 1024 } ], "vb_distribution_stats_from_metadata": { "172.23.123.62:8096": { "worker_n1ql_0": "[0-1023]" } }, "worker_pids": { "worker_n1ql_0": 65093 } } ] [2021-02-12 23:48:52,811] - [eventing_base] [140096917202688] - INFO - Execution stats update_success: 20929 update_failure: 37 delete_success: 39293 delete_failure: 0 [2021-02-12 23:48:52,826] - [basetestcase] [140096917202688] - INFO - list of eventing nodes in cluster: [ip:172.23.123.62 port:8091 ssh_username:root] [2021-02-12 23:48:52,874] - [eventing_base] [140096917202688] - INFO - Stats for Node 172.23.123.62 is [ { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 3, "agg_messages_sent_to_worker": 67475, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_deletion": 68512, "dcp_deletion_sent_to_worker": 38731, "dcp_mutation": 70126, "dcp_mutation_sent_to_worker": 22186, "dcp_mutation_suppressed_counter": 28417, "dcp_snapshot": 50484, "dcp_stream_req_counter": 1878, "dcp_stream_req_err_counter": 1, "dcp_streamend": 853, "dcp_streamreq": 1877, "dcp_system_event": 6710, "execution_stats": 766, "failure_stats": 766, "latency_stats": 766, "lcb_exception_stats": 766, "log_level": 1, "num_processed_events": 60917, "processed_events_size": 30398048, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 38731, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 22186, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 38731, "enqueued_dcp_mutation_msg_counter": 22186, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 10195, "lcb_retry_failure": 0, "messages_parsed": 67471, "num_processed_events": 60917, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 13461, "processed_events_size": 30398048, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "64985": "2021-02-13T07:48:52Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "64985": "2021-02-13T07:48:52Z" }, "v8worker_events_lost": 0 }, "function_id": 1218075759, "function_name": "sbm", "gocb_creds_request_counter": 22, "internal_vb_distribution_stats": { "worker_sbm_0": "[0-1023]" }, "latency_percentile_stats": { "100": 21600, "50": 100, "80": 500, "90": 900, "95": 1400, "99": 3100 }, "lcb_creds_request_counter": 26806, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.123.62:8096", "start_vb": 0, "vb_count": 1024 } ], "vb_distribution_stats_from_metadata": { "172.23.123.62:8096": { "worker_sbm_0": "[0-1023]" } }, "worker_pids": { "worker_sbm_0": 64985 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 5, "agg_messages_sent_to_worker": 82768, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_deletion": 99514, "dcp_deletion_sent_to_worker": 55746, "dcp_mutation": 105349, "dcp_mutation_sent_to_worker": 20557, "dcp_snapshot": 51735, "dcp_stream_req_counter": 1877, "dcp_streamend": 853, "dcp_streamreq": 1877, "dcp_system_event": 6910, "execution_stats": 745, "failure_stats": 745, "latency_stats": 745, "lcb_exception_stats": 745, "log_level": 1, "num_processed_events": 76303, "processed_events_size": 33573624, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 50537, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 50537, "dcp_delete_msg_counter": 55746, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 20557, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 55746, "enqueued_dcp_mutation_msg_counter": 20557, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 25766, "lcb_retry_failure": 0, "messages_parsed": 82764, "num_processed_events": 76303, "on_delete_failure": 0, "on_delete_success": 37114, "on_update_failure": 0, "on_update_success": 13423, "processed_events_size": 33573624, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "65060": "2021-02-13T07:48:52Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "65060": "2021-02-13T07:48:52Z" }, "v8worker_events_lost": 0 }, "function_id": 3750915646, "function_name": "curl", "gocb_creds_request_counter": 22, "internal_vb_distribution_stats": { "worker_curl_0": "[0-1023]" }, "latency_percentile_stats": { "100": 285100, "50": 1400, "80": 1900, "90": 2600, "95": 4000, "99": 11600 }, "lcb_creds_request_counter": 26806, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.123.62:8096", "start_vb": 0, "vb_count": 1024 } ], "vb_distribution_stats_from_metadata": { "172.23.123.62:8096": { "worker_curl_0": "[0-1023]" } }, "worker_pids": { "worker_curl_0": 65060 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 3, "agg_messages_sent_to_worker": 79273, "agg_queue_memory": 0, "agg_queue_memory_cap": 20971520, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_deletion": 95714, "dcp_deletion_sent_to_worker": 52205, "dcp_mutation": 104813, "dcp_mutation_sent_to_worker": 20714, "dcp_snapshot": 51167, "dcp_stream_req_counter": 1877, "dcp_streamend": 853, "dcp_streamreq": 1877, "dcp_system_event": 6830, "execution_stats": 723, "failure_stats": 723, "latency_stats": 723, "lcb_exception_stats": 723, "log_level": 1, "num_processed_events": 72919, "processed_events_size": 32751140, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 52205, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 20714, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 52205, "enqueued_dcp_mutation_msg_counter": 20714, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 24868, "lcb_retry_failure": 0, "messages_parsed": 79269, "num_processed_events": 72919, "on_delete_failure": 0, "on_delete_success": 34662, "on_update_failure": 0, "on_update_success": 13389, "processed_events_size": 32751140, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "65093": "2021-02-13T07:48:51Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "65093": "2021-02-13T07:48:51Z" }, "v8worker_events_lost": 0 }, "function_id": 4203222296, "function_name": "n1ql", "gocb_creds_request_counter": 22, "internal_vb_distribution_stats": { "worker_n1ql_0": "[0-1023]" }, "latency_percentile_stats": { "100": 188000, "50": 700, "80": 3100, "90": 4100, "95": 5400, "99": 10700 }, "lcb_creds_request_counter": 26806, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.123.62:8096", "start_vb": 0, "vb_count": 1024 } ], "vb_distribution_stats_from_metadata": { "172.23.123.62:8096": { "worker_n1ql_0": "[0-1023]" } }, "worker_pids": { "worker_n1ql_0": 65093 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 6, "agg_messages_sent_to_worker": 105987, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_deletion": 103103, "dcp_deletion_sent_to_worker": 59453, "dcp_mutation": 154226, "dcp_mutation_sent_to_worker": 39746, "dcp_snapshot": 51017, "dcp_stream_req_counter": 1877, "dcp_streamend": 853, "dcp_streamreq": 1877, "dcp_system_event": 14500, "execution_stats": 809, "failure_stats": 809, "latency_stats": 809, "lcb_exception_stats": 809, "log_level": 1, "num_processed_events": 99199, "processed_events_size": 51761880, "thr_count": 1, "thr_map": 1, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 59453, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 39746, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 59453, "enqueued_dcp_mutation_msg_counter": 39746, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 38940, "lcb_retry_failure": 0, "messages_parsed": 105983, "num_processed_events": 99199, "on_delete_failure": 0, "on_delete_success": 39293, "on_update_failure": 37, "on_update_success": 20929, "processed_events_size": 51761880, "timer_callback_failure": 0, "timer_callback_success": 0, "timer_cancel_counter": 0, "timer_create_counter": 0, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "64920": "2021-02-13T07:48:52Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 37, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "64920": "2021-02-13T07:48:52Z" }, "v8worker_events_lost": 0 }, "function_id": 1844693768, "function_name": "bucket_op", "gocb_creds_request_counter": 22, "internal_vb_distribution_stats": { "worker_bucket_op_0": "[0-1023]" }, "latency_percentile_stats": { "100": 2507100, "50": 400, "80": 900, "90": 1400, "95": 2500, "99": 9200 }, "lcb_creds_request_counter": 26806, "lcb_exception_stats": { "211": 37 }, "metastore_stats": {}, "planner_stats": [ { "host_name": "172.23.123.62:8096", "start_vb": 0, "vb_count": 1024 } ], "vb_distribution_stats_from_metadata": { "172.23.123.62:8096": { "worker_bucket_op_0": "[0-1023]" } }, "worker_pids": { "worker_bucket_op_0": 64920 } }, { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 6, "agg_messages_sent_to_worker": 98819, "agg_queue_memory": 0, "agg_queue_memory_cap": 53477376, "agg_queue_size": 0, "agg_queue_size_cap": 100000, "agg_timer_feedback_queue_cap": 500, "dcp_deletion": 103877, "dcp_deletion_sent_to_worker": 60480, "dcp_mutation": 160112, "dcp_mutation_sent_to_worker": 31657, "dcp_snapshot": 51087, "dcp_stream_req_counter": 1877, "dcp_streamend": 853, "dcp_streamreq": 1877, "dcp_system_event": 9390, "execution_stats": 788, "failure_stats": 788, "latency_stats": 788, "lcb_exception_stats": 788, "log_level": 1, "num_processed_events": 92137, "processed_events_size": 44798884, "thr_count": 1, "thr_map": 1, "timer_events": 74236, "timer_responses_received": 58247, "v8_init": 1, "v8_load": 1 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 0, "head": 0, "post": 0, "put": 0 }, "curl_success_count": 0, "dcp_delete_msg_counter": 60480, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 31657, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 60480, "enqueued_dcp_mutation_msg_counter": 31657, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 33890, "lcb_retry_failure": 0, "messages_parsed": 98815, "num_processed_events": 92137, "on_delete_failure": 0, "on_delete_success": 40320, "on_update_failure": 0, "on_update_success": 17927, "processed_events_size": 44798884, "timer_callback_failure": 30, "timer_callback_success": 74206, "timer_cancel_counter": 0, "timer_create_counter": 58247, "timer_create_failure": 0, "timer_msg_counter": 74236, "timer_responses_sent": 0, "timestamp": { "64952": "2021-02-13T07:48:52Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 30, "checkpoint_failure_count": 0, "curl_failure_count": 0, "curl_max_resp_size_exceeded": 0, "curl_non_200_response": 0, "curl_timeout_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "64952": "2021-02-13T07:48:52Z" }, "v8worker_events_lost": 0 }, "function_id": 364579283, "function_name": "timers", "gocb_creds_request_counter": 22, "internal_vb_distribution_stats": { "worker_timers_0": "[0-1023]" }, "latency_percentile_stats": { "100": 2507100, "50": 800, "80": 1200, "90": 1800, "95": 2400, "99": 5000 }, "lcb_creds_request_counter": 26806, "lcb_exception_stats": { "211": 30 }, "metastore_stats": { "metastore_delete_err": 0, "metastore_deletes": 0, "metastore_not_found": 0, "metastore_scan": 0, "metastore_scan_due": 0, "metastore_scan_err": 0, "metastore_set": 0, "metastore_set_err": 0 }, "planner_stats": [ { "host_name": "172.23.123.62:8096", "start_vb": 0, "vb_count": 1024 } ], "vb_distribution_stats_from_metadata": { "172.23.123.62:8096": { "worker_timers_0": "[0-1023]" } }, "worker_pids": { "worker_timers_0": 64952 } } ] [2021-02-12 23:48:52,887] - [basetestcase] [140096917202688] - INFO - list of eventing nodes in cluster: [ip:172.23.123.62 port:8091 ssh_username:root] [2021-02-12 23:48:52,889] - [remote_util] [140096917202688] - INFO - SSH Connecting to 172.23.123.62 with username:root, attempt#1 of 5 [2021-02-12 23:48:52,995] - [remote_util] [140096917202688] - INFO - SSH Connected to 172.23.123.62 as root [2021-02-12 23:48:53,277] - [remote_util] [140096917202688] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:48:53,580] - [remote_util] [140096917202688] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:48:53,588] - [rest_client] [140096917202688] - INFO - /diag/eval status on 172.23.123.62:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). [2021-02-12 23:48:53,588] - [remote_util] [140096917202688] - INFO - running command.raw on 172.23.123.62: zgrep "panic" "/opt/couchbase/var/lib/couchbase/logs"/eventing.log* | wc -l [2021-02-12 23:48:54,677] - [remote_util] [140096917202688] - INFO - command executed successfully with root [2021-02-12 23:48:54,677] - [remote_util] [140096917202688] - INFO - running command.raw on 172.23.123.62: ls "/opt/couchbase/var/lib/couchbase/logs"/../crash/| wc -l [2021-02-12 23:48:54,730] - [remote_util] [140096917202688] - INFO - command executed successfully with root [2021-02-12 23:48:54,776] - [basetestcase] [140096917202688] - INFO - ------- Cluster statistics ------- [2021-02-12 23:48:54,777] - [basetestcase] [140096917202688] - INFO - 172.23.122.82:8091 => {'services': ['n1ql'], 'cpu_utilization': 0.5277707966825835, 'mem_free': 3471962112, 'mem_total': 4201684992, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-02-12 23:48:54,777] - [basetestcase] [140096917202688] - INFO - 172.23.122.97:8091 => {'services': ['index'], 'cpu_utilization': 0.789406671759613, 'mem_free': 3278618624, 'mem_total': 4201684992, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-02-12 23:48:54,777] - [basetestcase] [140096917202688] - INFO - 172.23.123.61:8091 => {'services': ['kv'], 'cpu_utilization': 8.526124252664413, 'mem_free': 2741174272, 'mem_total': 4201672704, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-02-12 23:48:54,777] - [basetestcase] [140096917202688] - INFO - 172.23.123.62:8091 => {'services': ['eventing'], 'cpu_utilization': 5.581039755351682, 'mem_free': 3031793664, 'mem_total': 4201676800, 'swap_mem_used': 0, 'swap_mem_total': 3758092288} [2021-02-12 23:48:54,777] - [basetestcase] [140096917202688] - INFO - --- End of cluster statistics --- [2021-02-12 23:48:54,778] - [remote_util] [140096917202688] - INFO - SSH Connecting to 172.23.123.61 with username:root, attempt#1 of 5 [2021-02-12 23:48:54,887] - [remote_util] [140096917202688] - INFO - SSH Connected to 172.23.123.61 as root [2021-02-12 23:48:55,173] - [remote_util] [140096917202688] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:48:55,480] - [remote_util] [140096917202688] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:48:55,482] - [remote_util] [140096917202688] - INFO - SSH Connecting to 172.23.123.62 with username:root, attempt#1 of 5 [2021-02-12 23:48:55,587] - [remote_util] [140096917202688] - INFO - SSH Connected to 172.23.123.62 as root [2021-02-12 23:48:55,875] - [remote_util] [140096917202688] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:48:56,179] - [remote_util] [140096917202688] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:48:56,181] - [remote_util] [140096917202688] - INFO - SSH Connecting to 172.23.122.90 with username:root, attempt#1 of 5 [2021-02-12 23:48:56,282] - [remote_util] [140096917202688] - INFO - SSH Connected to 172.23.122.90 as root [2021-02-12 23:48:56,553] - [remote_util] [140096917202688] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:48:56,837] - [remote_util] [140096917202688] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:48:56,839] - [remote_util] [140096917202688] - INFO - SSH Connecting to 172.23.122.117 with username:root, attempt#1 of 5 [2021-02-12 23:48:56,941] - [remote_util] [140096917202688] - INFO - SSH Connected to 172.23.122.117 as root [2021-02-12 23:48:57,198] - [remote_util] [140096917202688] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:48:57,494] - [remote_util] [140096917202688] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:48:57,496] - [remote_util] [140096917202688] - INFO - SSH Connecting to 172.23.122.97 with username:root, attempt#1 of 5 [2021-02-12 23:48:57,597] - [remote_util] [140096917202688] - INFO - SSH Connected to 172.23.122.97 as root [2021-02-12 23:48:57,855] - [remote_util] [140096917202688] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:48:58,143] - [remote_util] [140096917202688] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:48:58,145] - [remote_util] [140096917202688] - INFO - SSH Connecting to 172.23.122.82 with username:root, attempt#1 of 5 [2021-02-12 23:48:58,246] - [remote_util] [140096917202688] - INFO - SSH Connected to 172.23.122.82 as root [2021-02-12 23:48:58,507] - [remote_util] [140096917202688] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:48:58,803] - [remote_util] [140096917202688] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:48:58,805] - [remote_util] [140096917202688] - INFO - SSH Connecting to 172.23.122.118 with username:root, attempt#1 of 5 [2021-02-12 23:48:58,904] - [remote_util] [140096917202688] - INFO - SSH Connected to 172.23.122.118 as root [2021-02-12 23:48:59,147] - [remote_util] [140096917202688] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:48:59,440] - [remote_util] [140096917202688] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:48:59,442] - [remote_util] [140096917202688] - INFO - SSH Connecting to 172.23.122.116 with username:root, attempt#1 of 5 [2021-02-12 23:48:59,543] - [remote_util] [140096917202688] - INFO - SSH Connected to 172.23.122.116 as root [2021-02-12 23:48:59,810] - [remote_util] [140096917202688] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:49:00,097] - [remote_util] [140096917202688] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:50:28,353] - [basetestcase] [140096917202688] - INFO - ============== basetestcase cleanup was started for test #10 test_eventing_rebalance_out_delete_recreate_collections ============== [2021-02-12 23:50:28,403] - [bucket_helper] [140096917202688] - INFO - deleting existing buckets ['dst_bucket', 'metadata', 'src_bucket'] on 172.23.123.61 [2021-02-12 23:50:29,214] - [bucket_helper] [140096917202688] - INFO - waiting for bucket deletion to complete.... [2021-02-12 23:50:29,233] - [rest_client] [140096917202688] - INFO - node 172.23.123.61 existing buckets : ['metadata', 'src_bucket'] [2021-02-12 23:50:29,234] - [bucket_helper] [140096917202688] - INFO - deleted bucket : dst_bucket from 172.23.123.61 [2021-02-12 23:50:29,832] - [bucket_helper] [140096917202688] - INFO - waiting for bucket deletion to complete.... [2021-02-12 23:50:29,841] - [rest_client] [140096917202688] - INFO - node 172.23.123.61 existing buckets : ['src_bucket'] [2021-02-12 23:50:29,841] - [bucket_helper] [140096917202688] - INFO - deleted bucket : metadata from 172.23.123.61 [2021-02-12 23:50:30,626] - [bucket_helper] [140096917202688] - INFO - waiting for bucket deletion to complete.... [2021-02-12 23:50:30,628] - [rest_client] [140096917202688] - INFO - node 172.23.123.61 existing buckets : [] [2021-02-12 23:50:30,628] - [bucket_helper] [140096917202688] - INFO - deleted bucket : src_bucket from 172.23.123.61 [2021-02-12 23:50:30,636] - [bucket_helper] [140096917202688] - INFO - Could not find any buckets for node 172.23.123.62, nothing to delete [2021-02-12 23:50:30,642] - [bucket_helper] [140096917202688] - INFO - Could not find any buckets for node 172.23.122.90, nothing to delete [2021-02-12 23:50:30,646] - [bucket_helper] [140096917202688] - INFO - Could not find any buckets for node 172.23.122.117, nothing to delete [2021-02-12 23:50:30,653] - [bucket_helper] [140096917202688] - INFO - Could not find any buckets for node 172.23.122.97, nothing to delete [2021-02-12 23:50:30,659] - [bucket_helper] [140096917202688] - INFO - Could not find any buckets for node 172.23.122.82, nothing to delete [2021-02-12 23:50:30,665] - [bucket_helper] [140096917202688] - INFO - Could not find any buckets for node 172.23.122.118, nothing to delete [2021-02-12 23:50:30,670] - [bucket_helper] [140096917202688] - INFO - Could not find any buckets for node 172.23.122.116, nothing to delete [2021-02-12 23:50:30,675] - [rest_client] [140096917202688] - INFO - -->is_ns_server_running? [2021-02-12 23:50:30,685] - [cluster_helper] [140096917202688] - INFO - rebalancing all nodes in order to remove nodes [2021-02-12 23:50:30,686] - [rest_client] [140096917202688] - INFO - rebalance params : {'knownNodes': 'ns_1@172.23.122.82,ns_1@172.23.122.97,ns_1@172.23.123.61,ns_1@172.23.123.62', 'ejectedNodes': 'ns_1@172.23.122.82,ns_1@172.23.122.97,ns_1@172.23.123.62', 'user': 'Administrator', 'password': 'password'} [2021-02-12 23:50:30,716] - [rest_client] [140096917202688] - INFO - rebalance operation started [2021-02-12 23:50:30,718] - [rest_client] [140096917202688] - INFO - rebalance percentage : 0.00 % [2021-02-12 23:50:40,722] - [rest_client] [140096917202688] - INFO - rebalance percentage : 50.00 % [2021-02-12 23:50:50,735] - [rest_client] [140096917202688] - INFO - rebalance percentage : 75.00 % [2021-02-12 23:51:10,761] - [rest_client] [140096917202688] - INFO - rebalance progress took 40.04 seconds [2021-02-12 23:51:10,761] - [rest_client] [140096917202688] - INFO - sleep for 10 seconds after rebalance... [2021-02-12 23:51:20,787] - [cluster_helper] [140096917202688] - INFO - removed all the nodes from cluster associated with ip:172.23.123.61 port:8091 ssh_username:root ? [('ns_1@172.23.122.82', 8091), ('ns_1@172.23.122.97', 8091), ('ns_1@172.23.123.62', 8091)] [2021-02-12 23:51:20,792] - [cluster_helper] [140096917202688] - INFO - waiting for ns_server @ 172.23.123.61:8091 [2021-02-12 23:51:20,792] - [rest_client] [140096917202688] - INFO - -->is_ns_server_running? [2021-02-12 23:51:20,796] - [cluster_helper] [140096917202688] - INFO - ns_server @ 172.23.123.61:8091 is running [2021-02-12 23:51:20,799] - [cluster_helper] [140096917202688] - INFO - waiting for ns_server @ 172.23.123.62:8091 [2021-02-12 23:51:20,799] - [rest_client] [140096917202688] - INFO - -->is_ns_server_running? [2021-02-12 23:51:20,802] - [cluster_helper] [140096917202688] - INFO - ns_server @ 172.23.123.62:8091 is running [2021-02-12 23:51:20,805] - [cluster_helper] [140096917202688] - INFO - waiting for ns_server @ 172.23.122.90:8091 [2021-02-12 23:51:20,805] - [rest_client] [140096917202688] - INFO - -->is_ns_server_running? [2021-02-12 23:51:20,808] - [cluster_helper] [140096917202688] - INFO - ns_server @ 172.23.122.90:8091 is running [2021-02-12 23:51:20,811] - [cluster_helper] [140096917202688] - INFO - waiting for ns_server @ 172.23.122.117:8091 [2021-02-12 23:51:20,812] - [rest_client] [140096917202688] - INFO - -->is_ns_server_running? [2021-02-12 23:51:20,814] - [cluster_helper] [140096917202688] - INFO - ns_server @ 172.23.122.117:8091 is running [2021-02-12 23:51:20,817] - [cluster_helper] [140096917202688] - INFO - waiting for ns_server @ 172.23.122.97:8091 [2021-02-12 23:51:20,817] - [rest_client] [140096917202688] - INFO - -->is_ns_server_running? [2021-02-12 23:51:20,819] - [cluster_helper] [140096917202688] - INFO - ns_server @ 172.23.122.97:8091 is running [2021-02-12 23:51:20,822] - [cluster_helper] [140096917202688] - INFO - waiting for ns_server @ 172.23.122.82:8091 [2021-02-12 23:51:20,822] - [rest_client] [140096917202688] - INFO - -->is_ns_server_running? [2021-02-12 23:51:20,825] - [cluster_helper] [140096917202688] - INFO - ns_server @ 172.23.122.82:8091 is running [2021-02-12 23:51:20,828] - [cluster_helper] [140096917202688] - INFO - waiting for ns_server @ 172.23.122.118:8091 [2021-02-12 23:51:20,828] - [rest_client] [140096917202688] - INFO - -->is_ns_server_running? [2021-02-12 23:51:20,831] - [cluster_helper] [140096917202688] - INFO - ns_server @ 172.23.122.118:8091 is running [2021-02-12 23:51:20,834] - [cluster_helper] [140096917202688] - INFO - waiting for ns_server @ 172.23.122.116:8091 [2021-02-12 23:51:20,834] - [rest_client] [140096917202688] - INFO - -->is_ns_server_running? [2021-02-12 23:51:20,837] - [cluster_helper] [140096917202688] - INFO - ns_server @ 172.23.122.116:8091 is running [2021-02-12 23:51:20,837] - [basetestcase] [140096917202688] - INFO - ============== basetestcase cleanup was finished for test #10 test_eventing_rebalance_out_delete_recreate_collections ============== [2021-02-12 23:51:20,837] - [basetestcase] [140096917202688] - INFO - closing all ssh connections [2021-02-12 23:51:20,840] - [basetestcase] [140096917202688] - INFO - closing all memcached connections [2021-02-12 23:51:20,841] - [remote_util] [140096917202688] - INFO - SSH Connecting to 172.23.123.61 with username:root, attempt#1 of 5 [2021-02-12 23:51:20,947] - [remote_util] [140096917202688] - INFO - SSH Connected to 172.23.123.61 as root [2021-02-12 23:51:21,211] - [remote_util] [140096917202688] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:51:21,506] - [remote_util] [140096917202688] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:51:21,515] - [remote_util] [140096917202688] - INFO - SSH Connecting to 172.23.123.61 with username:root, attempt#1 of 5 [2021-02-12 23:51:21,617] - [remote_util] [140096917202688] - INFO - SSH Connected to 172.23.123.61 as root [2021-02-12 23:51:21,878] - [remote_util] [140096917202688] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:51:22,173] - [remote_util] [140096917202688] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:51:22,193] - [remote_util] [140096917202688] - INFO - 172.23.123.61 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 23:51:22,246] - [remote_util] [140096917202688] - INFO - Running systemd command on this server [2021-02-12 23:51:22,246] - [remote_util] [140096917202688] - INFO - running command.raw on 172.23.123.61: systemctl stop couchbase-server.service [2021-02-12 23:51:24,058] - [remote_util] [140096917202688] - INFO - command executed successfully with root [2021-02-12 23:51:24,058] - [basetestcase] [140096917202688] - INFO - Couchbase stopped [2021-02-12 23:51:24,059] - [basetestcase] [140096917202688] - INFO - sleep for 5 secs. ... [2021-02-12 23:51:29,064] - [remote_util] [140096917202688] - INFO - running command.raw on 172.23.123.61: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-12 23:51:29,080] - [remote_util] [140096917202688] - INFO - command executed successfully with root [2021-02-12 23:51:29,080] - [remote_util] [140096917202688] - INFO - running command.raw on 172.23.123.61: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-12 23:51:29,135] - [remote_util] [140096917202688] - INFO - command executed successfully with root [2021-02-12 23:51:29,137] - [remote_util] [140096917202688] - INFO - SSH Connecting to 172.23.123.61 with username:root, attempt#1 of 5 [2021-02-12 23:51:29,239] - [remote_util] [140096917202688] - INFO - SSH Connected to 172.23.123.61 as root [2021-02-12 23:51:29,511] - [remote_util] [140096917202688] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:51:29,811] - [remote_util] [140096917202688] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:51:29,870] - [remote_util] [140096917202688] - INFO - 172.23.123.61 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 23:51:29,870] - [remote_util] [140096917202688] - INFO - Checking for process beam.smp on linux [2021-02-12 23:51:29,907] - [remote_util] [140096917202688] - INFO - Starting couchbase server [2021-02-12 23:51:30,003] - [remote_util] [140096917202688] - INFO - Running systemd command on this server [2021-02-12 23:51:30,003] - [remote_util] [140096917202688] - INFO - running command.raw on 172.23.123.61: systemctl start couchbase-server.service [2021-02-12 23:51:30,028] - [remote_util] [140096917202688] - INFO - command executed successfully with root [2021-02-12 23:51:30,028] - [remote_util] [140096917202688] - INFO - 172.23.123.61:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-12 23:51:35,031] - [remote_util] [140096917202688] - INFO - running command.raw on 172.23.123.61: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-12 23:51:35,055] - [remote_util] [140096917202688] - INFO - command executed successfully with root [2021-02-12 23:51:35,056] - [remote_util] [140096917202688] - INFO - Couchbase server status: [' Process: 23992 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-12 23:51:35,056] - [remote_util] [140096917202688] - INFO - Checking for process beam.smp on linux [2021-02-12 23:51:35,123] - [remote_util] [140096917202688] - INFO - process beam.smp is running on 172.23.123.61: with pid 24135 [2021-02-12 23:51:35,123] - [basetestcase] [140096917202688] - INFO - Couchbase started [2021-02-12 23:51:35,125] - [remote_util] [140096917202688] - INFO - SSH Connecting to 172.23.123.62 with username:root, attempt#1 of 5 [2021-02-12 23:51:35,231] - [remote_util] [140096917202688] - INFO - SSH Connected to 172.23.123.62 as root [2021-02-12 23:51:35,518] - [remote_util] [140096917202688] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:51:35,821] - [remote_util] [140096917202688] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:51:35,830] - [remote_util] [140096917202688] - INFO - SSH Connecting to 172.23.123.62 with username:root, attempt#1 of 5 [2021-02-12 23:51:35,929] - [remote_util] [140096917202688] - INFO - SSH Connected to 172.23.123.62 as root [2021-02-12 23:51:36,198] - [remote_util] [140096917202688] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:51:36,502] - [remote_util] [140096917202688] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:51:36,524] - [remote_util] [140096917202688] - INFO - 172.23.123.62 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 23:51:36,582] - [remote_util] [140096917202688] - INFO - Running systemd command on this server [2021-02-12 23:51:36,582] - [remote_util] [140096917202688] - INFO - running command.raw on 172.23.123.62: systemctl stop couchbase-server.service [2021-02-12 23:51:38,283] - [remote_util] [140096917202688] - INFO - command executed successfully with root [2021-02-12 23:51:38,283] - [basetestcase] [140096917202688] - INFO - Couchbase stopped [2021-02-12 23:51:38,284] - [basetestcase] [140096917202688] - INFO - sleep for 5 secs. ... [2021-02-12 23:51:43,289] - [remote_util] [140096917202688] - INFO - running command.raw on 172.23.123.62: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-12 23:51:43,320] - [remote_util] [140096917202688] - INFO - command executed successfully with root [2021-02-12 23:51:43,320] - [remote_util] [140096917202688] - INFO - running command.raw on 172.23.123.62: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-12 23:51:43,375] - [remote_util] [140096917202688] - INFO - command executed successfully with root [2021-02-12 23:51:43,377] - [remote_util] [140096917202688] - INFO - SSH Connecting to 172.23.123.62 with username:root, attempt#1 of 5 [2021-02-12 23:51:43,486] - [remote_util] [140096917202688] - INFO - SSH Connected to 172.23.123.62 as root [2021-02-12 23:51:43,780] - [remote_util] [140096917202688] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:51:44,084] - [remote_util] [140096917202688] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:51:44,145] - [remote_util] [140096917202688] - INFO - 172.23.123.62 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 23:51:44,145] - [remote_util] [140096917202688] - INFO - Checking for process beam.smp on linux [2021-02-12 23:51:44,185] - [remote_util] [140096917202688] - INFO - Starting couchbase server [2021-02-12 23:51:44,283] - [remote_util] [140096917202688] - INFO - Running systemd command on this server [2021-02-12 23:51:44,284] - [remote_util] [140096917202688] - INFO - running command.raw on 172.23.123.62: systemctl start couchbase-server.service [2021-02-12 23:51:44,308] - [remote_util] [140096917202688] - INFO - command executed successfully with root [2021-02-12 23:51:44,308] - [remote_util] [140096917202688] - INFO - 172.23.123.62:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-12 23:51:49,313] - [remote_util] [140096917202688] - INFO - running command.raw on 172.23.123.62: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-12 23:51:49,339] - [remote_util] [140096917202688] - INFO - command executed successfully with root [2021-02-12 23:51:49,340] - [remote_util] [140096917202688] - INFO - Couchbase server status: [' Process: 83039 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-12 23:51:49,340] - [remote_util] [140096917202688] - INFO - Checking for process beam.smp on linux [2021-02-12 23:51:49,407] - [remote_util] [140096917202688] - INFO - process beam.smp is running on 172.23.123.62: with pid 83181 [2021-02-12 23:51:49,407] - [basetestcase] [140096917202688] - INFO - Couchbase started [2021-02-12 23:51:49,409] - [remote_util] [140096917202688] - INFO - SSH Connecting to 172.23.122.90 with username:root, attempt#1 of 5 [2021-02-12 23:51:49,512] - [remote_util] [140096917202688] - INFO - SSH Connected to 172.23.122.90 as root [2021-02-12 23:51:49,797] - [remote_util] [140096917202688] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:51:50,098] - [remote_util] [140096917202688] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:51:50,106] - [remote_util] [140096917202688] - INFO - SSH Connecting to 172.23.122.90 with username:root, attempt#1 of 5 [2021-02-12 23:51:50,209] - [remote_util] [140096917202688] - INFO - SSH Connected to 172.23.122.90 as root [2021-02-12 23:51:50,490] - [remote_util] [140096917202688] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:51:50,787] - [remote_util] [140096917202688] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:51:50,806] - [remote_util] [140096917202688] - INFO - 172.23.122.90 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 23:51:50,859] - [remote_util] [140096917202688] - INFO - Running systemd command on this server [2021-02-12 23:51:50,859] - [remote_util] [140096917202688] - INFO - running command.raw on 172.23.122.90: systemctl stop couchbase-server.service [2021-02-12 23:51:52,687] - [remote_util] [140096917202688] - INFO - command executed successfully with root [2021-02-12 23:51:52,688] - [basetestcase] [140096917202688] - INFO - Couchbase stopped [2021-02-12 23:51:52,688] - [basetestcase] [140096917202688] - INFO - sleep for 5 secs. ... [2021-02-12 23:51:57,693] - [remote_util] [140096917202688] - INFO - running command.raw on 172.23.122.90: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-12 23:51:57,714] - [remote_util] [140096917202688] - INFO - command executed successfully with root [2021-02-12 23:51:57,714] - [remote_util] [140096917202688] - INFO - running command.raw on 172.23.122.90: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-12 23:51:57,768] - [remote_util] [140096917202688] - INFO - command executed successfully with root [2021-02-12 23:51:57,770] - [remote_util] [140096917202688] - INFO - SSH Connecting to 172.23.122.90 with username:root, attempt#1 of 5 [2021-02-12 23:51:57,874] - [remote_util] [140096917202688] - INFO - SSH Connected to 172.23.122.90 as root [2021-02-12 23:51:58,160] - [remote_util] [140096917202688] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:51:58,463] - [remote_util] [140096917202688] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:51:58,481] - [remote_util] [140096917202688] - INFO - 172.23.122.90 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 23:51:58,481] - [remote_util] [140096917202688] - INFO - Checking for process beam.smp on linux [2021-02-12 23:51:58,520] - [remote_util] [140096917202688] - INFO - Starting couchbase server [2021-02-12 23:51:58,615] - [remote_util] [140096917202688] - INFO - Running systemd command on this server [2021-02-12 23:51:58,615] - [remote_util] [140096917202688] - INFO - running command.raw on 172.23.122.90: systemctl start couchbase-server.service [2021-02-12 23:51:58,639] - [remote_util] [140096917202688] - INFO - command executed successfully with root [2021-02-12 23:51:58,639] - [remote_util] [140096917202688] - INFO - 172.23.122.90:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-12 23:52:03,644] - [remote_util] [140096917202688] - INFO - running command.raw on 172.23.122.90: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-12 23:52:03,674] - [remote_util] [140096917202688] - INFO - command executed successfully with root [2021-02-12 23:52:03,675] - [remote_util] [140096917202688] - INFO - Couchbase server status: [' Process: 33700 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-12 23:52:03,675] - [remote_util] [140096917202688] - INFO - Checking for process beam.smp on linux [2021-02-12 23:52:03,747] - [remote_util] [140096917202688] - INFO - process beam.smp is running on 172.23.122.90: with pid 33844 [2021-02-12 23:52:03,747] - [basetestcase] [140096917202688] - INFO - Couchbase started [2021-02-12 23:52:03,748] - [remote_util] [140096917202688] - INFO - SSH Connecting to 172.23.122.117 with username:root, attempt#1 of 5 [2021-02-12 23:52:03,852] - [remote_util] [140096917202688] - INFO - SSH Connected to 172.23.122.117 as root [2021-02-12 23:52:04,117] - [remote_util] [140096917202688] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:52:04,418] - [remote_util] [140096917202688] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:52:04,427] - [remote_util] [140096917202688] - INFO - SSH Connecting to 172.23.122.117 with username:root, attempt#1 of 5 [2021-02-12 23:52:04,529] - [remote_util] [140096917202688] - INFO - SSH Connected to 172.23.122.117 as root [2021-02-12 23:52:04,785] - [remote_util] [140096917202688] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:52:05,083] - [remote_util] [140096917202688] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:52:05,143] - [remote_util] [140096917202688] - INFO - 172.23.122.117 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 23:52:05,199] - [remote_util] [140096917202688] - INFO - Running systemd command on this server [2021-02-12 23:52:05,199] - [remote_util] [140096917202688] - INFO - running command.raw on 172.23.122.117: systemctl stop couchbase-server.service [2021-02-12 23:52:06,868] - [remote_util] [140096917202688] - INFO - command executed successfully with root [2021-02-12 23:52:06,868] - [basetestcase] [140096917202688] - INFO - Couchbase stopped [2021-02-12 23:52:06,869] - [basetestcase] [140096917202688] - INFO - sleep for 5 secs. ... [2021-02-12 23:52:11,874] - [remote_util] [140096917202688] - INFO - running command.raw on 172.23.122.117: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-12 23:52:11,892] - [remote_util] [140096917202688] - INFO - command executed successfully with root [2021-02-12 23:52:11,892] - [remote_util] [140096917202688] - INFO - running command.raw on 172.23.122.117: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-12 23:52:11,945] - [remote_util] [140096917202688] - INFO - command executed successfully with root [2021-02-12 23:52:11,947] - [remote_util] [140096917202688] - INFO - SSH Connecting to 172.23.122.117 with username:root, attempt#1 of 5 [2021-02-12 23:52:12,049] - [remote_util] [140096917202688] - INFO - SSH Connected to 172.23.122.117 as root [2021-02-12 23:52:12,309] - [remote_util] [140096917202688] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:52:12,603] - [remote_util] [140096917202688] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:52:12,661] - [remote_util] [140096917202688] - INFO - 172.23.122.117 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 23:52:12,661] - [remote_util] [140096917202688] - INFO - Checking for process beam.smp on linux [2021-02-12 23:52:12,696] - [remote_util] [140096917202688] - INFO - Starting couchbase server [2021-02-12 23:52:12,787] - [remote_util] [140096917202688] - INFO - Running systemd command on this server [2021-02-12 23:52:12,788] - [remote_util] [140096917202688] - INFO - running command.raw on 172.23.122.117: systemctl start couchbase-server.service [2021-02-12 23:52:12,810] - [remote_util] [140096917202688] - INFO - command executed successfully with root [2021-02-12 23:52:12,810] - [remote_util] [140096917202688] - INFO - 172.23.122.117:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-12 23:52:17,815] - [remote_util] [140096917202688] - INFO - running command.raw on 172.23.122.117: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-12 23:52:17,840] - [remote_util] [140096917202688] - INFO - command executed successfully with root [2021-02-12 23:52:17,840] - [remote_util] [140096917202688] - INFO - Couchbase server status: [' Process: 34309 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-12 23:52:17,841] - [remote_util] [140096917202688] - INFO - Checking for process beam.smp on linux [2021-02-12 23:52:17,906] - [remote_util] [140096917202688] - INFO - process beam.smp is running on 172.23.122.117: with pid 34453 [2021-02-12 23:52:17,906] - [basetestcase] [140096917202688] - INFO - Couchbase started [2021-02-12 23:52:17,908] - [remote_util] [140096917202688] - INFO - SSH Connecting to 172.23.122.97 with username:root, attempt#1 of 5 [2021-02-12 23:52:18,011] - [remote_util] [140096917202688] - INFO - SSH Connected to 172.23.122.97 as root [2021-02-12 23:52:18,273] - [remote_util] [140096917202688] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:52:18,570] - [remote_util] [140096917202688] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:52:18,578] - [remote_util] [140096917202688] - INFO - SSH Connecting to 172.23.122.97 with username:root, attempt#1 of 5 [2021-02-12 23:52:18,676] - [remote_util] [140096917202688] - INFO - SSH Connected to 172.23.122.97 as root [2021-02-12 23:52:18,935] - [remote_util] [140096917202688] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:52:19,233] - [remote_util] [140096917202688] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:52:19,253] - [remote_util] [140096917202688] - INFO - 172.23.122.97 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 23:52:19,306] - [remote_util] [140096917202688] - INFO - Running systemd command on this server [2021-02-12 23:52:19,306] - [remote_util] [140096917202688] - INFO - running command.raw on 172.23.122.97: systemctl stop couchbase-server.service [2021-02-12 23:52:21,070] - [remote_util] [140096917202688] - INFO - command executed successfully with root [2021-02-12 23:52:21,070] - [basetestcase] [140096917202688] - INFO - Couchbase stopped [2021-02-12 23:52:21,071] - [basetestcase] [140096917202688] - INFO - sleep for 5 secs. ... [2021-02-12 23:52:26,073] - [remote_util] [140096917202688] - INFO - running command.raw on 172.23.122.97: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-12 23:52:26,093] - [remote_util] [140096917202688] - INFO - command executed successfully with root [2021-02-12 23:52:26,093] - [remote_util] [140096917202688] - INFO - running command.raw on 172.23.122.97: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-12 23:52:26,148] - [remote_util] [140096917202688] - INFO - command executed successfully with root [2021-02-12 23:52:26,149] - [remote_util] [140096917202688] - INFO - SSH Connecting to 172.23.122.97 with username:root, attempt#1 of 5 [2021-02-12 23:52:26,256] - [remote_util] [140096917202688] - INFO - SSH Connected to 172.23.122.97 as root [2021-02-12 23:52:26,518] - [remote_util] [140096917202688] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:52:26,822] - [remote_util] [140096917202688] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:52:26,842] - [remote_util] [140096917202688] - INFO - 172.23.122.97 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 23:52:26,842] - [remote_util] [140096917202688] - INFO - Checking for process beam.smp on linux [2021-02-12 23:52:26,879] - [remote_util] [140096917202688] - INFO - Starting couchbase server [2021-02-12 23:52:26,975] - [remote_util] [140096917202688] - INFO - Running systemd command on this server [2021-02-12 23:52:26,975] - [remote_util] [140096917202688] - INFO - running command.raw on 172.23.122.97: systemctl start couchbase-server.service [2021-02-12 23:52:26,999] - [remote_util] [140096917202688] - INFO - command executed successfully with root [2021-02-12 23:52:26,999] - [remote_util] [140096917202688] - INFO - 172.23.122.97:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-12 23:52:32,002] - [remote_util] [140096917202688] - INFO - running command.raw on 172.23.122.97: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-12 23:52:32,029] - [remote_util] [140096917202688] - INFO - command executed successfully with root [2021-02-12 23:52:32,029] - [remote_util] [140096917202688] - INFO - Couchbase server status: [' Process: 16150 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-12 23:52:32,029] - [remote_util] [140096917202688] - INFO - Checking for process beam.smp on linux [2021-02-12 23:52:32,101] - [remote_util] [140096917202688] - INFO - process beam.smp is running on 172.23.122.97: with pid 16294 [2021-02-12 23:52:32,102] - [basetestcase] [140096917202688] - INFO - Couchbase started [2021-02-12 23:52:32,103] - [remote_util] [140096917202688] - INFO - SSH Connecting to 172.23.122.82 with username:root, attempt#1 of 5 [2021-02-12 23:52:32,205] - [remote_util] [140096917202688] - INFO - SSH Connected to 172.23.122.82 as root [2021-02-12 23:52:32,467] - [remote_util] [140096917202688] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:52:32,757] - [remote_util] [140096917202688] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:52:32,765] - [remote_util] [140096917202688] - INFO - SSH Connecting to 172.23.122.82 with username:root, attempt#1 of 5 [2021-02-12 23:52:32,861] - [remote_util] [140096917202688] - INFO - SSH Connected to 172.23.122.82 as root [2021-02-12 23:52:33,114] - [remote_util] [140096917202688] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:52:33,406] - [remote_util] [140096917202688] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:52:33,425] - [remote_util] [140096917202688] - INFO - 172.23.122.82 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 23:52:33,478] - [remote_util] [140096917202688] - INFO - Running systemd command on this server [2021-02-12 23:52:33,478] - [remote_util] [140096917202688] - INFO - running command.raw on 172.23.122.82: systemctl stop couchbase-server.service [2021-02-12 23:52:35,271] - [remote_util] [140096917202688] - INFO - command executed successfully with root [2021-02-12 23:52:35,271] - [basetestcase] [140096917202688] - INFO - Couchbase stopped [2021-02-12 23:52:35,272] - [basetestcase] [140096917202688] - INFO - sleep for 5 secs. ... [2021-02-12 23:52:40,277] - [remote_util] [140096917202688] - INFO - running command.raw on 172.23.122.82: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-12 23:52:40,293] - [remote_util] [140096917202688] - INFO - command executed successfully with root [2021-02-12 23:52:40,294] - [remote_util] [140096917202688] - INFO - running command.raw on 172.23.122.82: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-12 23:52:40,347] - [remote_util] [140096917202688] - INFO - command executed successfully with root [2021-02-12 23:52:40,348] - [remote_util] [140096917202688] - INFO - SSH Connecting to 172.23.122.82 with username:root, attempt#1 of 5 [2021-02-12 23:52:40,447] - [remote_util] [140096917202688] - INFO - SSH Connected to 172.23.122.82 as root [2021-02-12 23:52:40,710] - [remote_util] [140096917202688] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:52:41,006] - [remote_util] [140096917202688] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:52:41,025] - [remote_util] [140096917202688] - INFO - 172.23.122.82 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 23:52:41,025] - [remote_util] [140096917202688] - INFO - Checking for process beam.smp on linux [2021-02-12 23:52:41,063] - [remote_util] [140096917202688] - INFO - Starting couchbase server [2021-02-12 23:52:41,156] - [remote_util] [140096917202688] - INFO - Running systemd command on this server [2021-02-12 23:52:41,157] - [remote_util] [140096917202688] - INFO - running command.raw on 172.23.122.82: systemctl start couchbase-server.service [2021-02-12 23:52:41,180] - [remote_util] [140096917202688] - INFO - command executed successfully with root [2021-02-12 23:52:41,180] - [remote_util] [140096917202688] - INFO - 172.23.122.82:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-12 23:52:46,185] - [remote_util] [140096917202688] - INFO - running command.raw on 172.23.122.82: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-12 23:52:46,213] - [remote_util] [140096917202688] - INFO - command executed successfully with root [2021-02-12 23:52:46,213] - [remote_util] [140096917202688] - INFO - Couchbase server status: [' Process: 109592 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-12 23:52:46,213] - [remote_util] [140096917202688] - INFO - Checking for process beam.smp on linux [2021-02-12 23:52:46,281] - [remote_util] [140096917202688] - INFO - process beam.smp is running on 172.23.122.82: with pid 109738 [2021-02-12 23:52:46,281] - [basetestcase] [140096917202688] - INFO - Couchbase started [2021-02-12 23:52:46,282] - [remote_util] [140096917202688] - INFO - SSH Connecting to 172.23.122.118 with username:root, attempt#1 of 5 [2021-02-12 23:52:46,383] - [remote_util] [140096917202688] - INFO - SSH Connected to 172.23.122.118 as root [2021-02-12 23:52:46,625] - [remote_util] [140096917202688] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:52:46,913] - [remote_util] [140096917202688] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:52:46,921] - [remote_util] [140096917202688] - INFO - SSH Connecting to 172.23.122.118 with username:root, attempt#1 of 5 [2021-02-12 23:52:47,015] - [remote_util] [140096917202688] - INFO - SSH Connected to 172.23.122.118 as root [2021-02-12 23:52:47,258] - [remote_util] [140096917202688] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:52:47,551] - [remote_util] [140096917202688] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:52:47,610] - [remote_util] [140096917202688] - INFO - 172.23.122.118 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 23:52:47,661] - [remote_util] [140096917202688] - INFO - Running systemd command on this server [2021-02-12 23:52:47,662] - [remote_util] [140096917202688] - INFO - running command.raw on 172.23.122.118: systemctl stop couchbase-server.service [2021-02-12 23:52:49,488] - [remote_util] [140096917202688] - INFO - command executed successfully with root [2021-02-12 23:52:49,488] - [basetestcase] [140096917202688] - INFO - Couchbase stopped [2021-02-12 23:52:49,489] - [basetestcase] [140096917202688] - INFO - sleep for 5 secs. ... [2021-02-12 23:52:54,494] - [remote_util] [140096917202688] - INFO - running command.raw on 172.23.122.118: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-12 23:52:54,509] - [remote_util] [140096917202688] - INFO - command executed successfully with root [2021-02-12 23:52:54,509] - [remote_util] [140096917202688] - INFO - running command.raw on 172.23.122.118: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-12 23:52:54,563] - [remote_util] [140096917202688] - INFO - command executed successfully with root [2021-02-12 23:52:54,564] - [remote_util] [140096917202688] - INFO - SSH Connecting to 172.23.122.118 with username:root, attempt#1 of 5 [2021-02-12 23:52:54,662] - [remote_util] [140096917202688] - INFO - SSH Connected to 172.23.122.118 as root [2021-02-12 23:52:54,905] - [remote_util] [140096917202688] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:52:55,192] - [remote_util] [140096917202688] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:52:55,210] - [remote_util] [140096917202688] - INFO - 172.23.122.118 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 23:52:55,211] - [remote_util] [140096917202688] - INFO - Checking for process beam.smp on linux [2021-02-12 23:52:55,244] - [remote_util] [140096917202688] - INFO - Starting couchbase server [2021-02-12 23:52:55,333] - [remote_util] [140096917202688] - INFO - Running systemd command on this server [2021-02-12 23:52:55,333] - [remote_util] [140096917202688] - INFO - running command.raw on 172.23.122.118: systemctl start couchbase-server.service [2021-02-12 23:52:55,354] - [remote_util] [140096917202688] - INFO - command executed successfully with root [2021-02-12 23:52:55,354] - [remote_util] [140096917202688] - INFO - 172.23.122.118:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-12 23:53:00,359] - [remote_util] [140096917202688] - INFO - running command.raw on 172.23.122.118: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-12 23:53:00,384] - [remote_util] [140096917202688] - INFO - command executed successfully with root [2021-02-12 23:53:00,384] - [remote_util] [140096917202688] - INFO - Couchbase server status: [' Process: 3730 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-12 23:53:00,384] - [remote_util] [140096917202688] - INFO - Checking for process beam.smp on linux [2021-02-12 23:53:00,452] - [remote_util] [140096917202688] - INFO - process beam.smp is running on 172.23.122.118: with pid 3918 [2021-02-12 23:53:00,452] - [basetestcase] [140096917202688] - INFO - Couchbase started [2021-02-12 23:53:00,454] - [remote_util] [140096917202688] - INFO - SSH Connecting to 172.23.122.116 with username:root, attempt#1 of 5 [2021-02-12 23:53:00,557] - [remote_util] [140096917202688] - INFO - SSH Connected to 172.23.122.116 as root [2021-02-12 23:53:00,829] - [remote_util] [140096917202688] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:53:01,120] - [remote_util] [140096917202688] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:53:01,128] - [remote_util] [140096917202688] - INFO - SSH Connecting to 172.23.122.116 with username:root, attempt#1 of 5 [2021-02-12 23:53:01,223] - [remote_util] [140096917202688] - INFO - SSH Connected to 172.23.122.116 as root [2021-02-12 23:53:01,486] - [remote_util] [140096917202688] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:53:01,780] - [remote_util] [140096917202688] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:53:01,799] - [remote_util] [140096917202688] - INFO - 172.23.122.116 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 23:53:01,850] - [remote_util] [140096917202688] - INFO - Running systemd command on this server [2021-02-12 23:53:01,850] - [remote_util] [140096917202688] - INFO - running command.raw on 172.23.122.116: systemctl stop couchbase-server.service [2021-02-12 23:53:03,899] - [remote_util] [140096917202688] - INFO - command executed successfully with root [2021-02-12 23:53:03,900] - [basetestcase] [140096917202688] - INFO - Couchbase stopped [2021-02-12 23:53:03,900] - [basetestcase] [140096917202688] - INFO - sleep for 5 secs. ... [2021-02-12 23:53:08,905] - [remote_util] [140096917202688] - INFO - running command.raw on 172.23.122.116: rm -rf /opt/couchbase/var/lib/couchbase/data/* [2021-02-12 23:53:08,921] - [remote_util] [140096917202688] - INFO - command executed successfully with root [2021-02-12 23:53:08,921] - [remote_util] [140096917202688] - INFO - running command.raw on 172.23.122.116: rm -rf /opt/couchbase/var/lib/couchbase/config/* [2021-02-12 23:53:08,973] - [remote_util] [140096917202688] - INFO - command executed successfully with root [2021-02-12 23:53:08,975] - [remote_util] [140096917202688] - INFO - SSH Connecting to 172.23.122.116 with username:root, attempt#1 of 5 [2021-02-12 23:53:09,073] - [remote_util] [140096917202688] - INFO - SSH Connected to 172.23.122.116 as root [2021-02-12 23:53:09,341] - [remote_util] [140096917202688] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:53:09,634] - [remote_util] [140096917202688] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:53:09,653] - [remote_util] [140096917202688] - INFO - 172.23.122.116 **** The linux version file /opt/couchbase/ VERSION.txt exists [2021-02-12 23:53:09,653] - [remote_util] [140096917202688] - INFO - Checking for process beam.smp on linux [2021-02-12 23:53:09,690] - [remote_util] [140096917202688] - INFO - Starting couchbase server [2021-02-12 23:53:09,782] - [remote_util] [140096917202688] - INFO - Running systemd command on this server [2021-02-12 23:53:09,782] - [remote_util] [140096917202688] - INFO - running command.raw on 172.23.122.116: systemctl start couchbase-server.service [2021-02-12 23:53:09,805] - [remote_util] [140096917202688] - INFO - command executed successfully with root [2021-02-12 23:53:09,805] - [remote_util] [140096917202688] - INFO - 172.23.122.116:sleep for 5 secs. waiting for couchbase server to come up ... [2021-02-12 23:53:14,810] - [remote_util] [140096917202688] - INFO - running command.raw on 172.23.122.116: systemctl status couchbase-server.service | grep ExecStop=/opt/couchbase/bin/couchbase-server [2021-02-12 23:53:14,833] - [remote_util] [140096917202688] - INFO - command executed successfully with root [2021-02-12 23:53:14,833] - [remote_util] [140096917202688] - INFO - Couchbase server status: [' Process: 93354 ExecStop=/opt/couchbase/bin/couchbase-server -k (code=exited, status=0/SUCCESS)'] [2021-02-12 23:53:14,833] - [remote_util] [140096917202688] - INFO - Checking for process beam.smp on linux [2021-02-12 23:53:14,896] - [remote_util] [140096917202688] - INFO - process beam.smp is running on 172.23.122.116: with pid 93542 [2021-02-12 23:53:14,896] - [basetestcase] [140096917202688] - INFO - Couchbase started [2021-02-12 23:53:14,896] - [basetestcase] [140096917202688] - INFO - sleep for 10 secs. ... [2021-02-12 23:53:24,941] - [remote_util] [140096917202688] - INFO - SSH Connecting to 172.23.123.61 with username:root, attempt#1 of 5 [2021-02-12 23:53:24,943] - [remote_util] [140099567843072] - INFO - SSH Connecting to 172.23.123.62 with username:root, attempt#1 of 5 [2021-02-12 23:53:24,945] - [remote_util] [140099350165248] - INFO - SSH Connecting to 172.23.122.90 with username:root, attempt#1 of 5 [2021-02-12 23:53:24,947] - [remote_util] [140099324987136] - INFO - SSH Connecting to 172.23.122.117 with username:root, attempt#1 of 5 [2021-02-12 23:53:24,948] - [remote_util] [140099299809024] - INFO - SSH Connecting to 172.23.122.97 with username:root, attempt#1 of 5 [2021-02-12 23:53:24,950] - [remote_util] [140097183020800] - INFO - SSH Connecting to 172.23.122.82 with username:root, attempt#1 of 5 [2021-02-12 23:53:24,954] - [remote_util] [140097168602880] - INFO - SSH Connecting to 172.23.122.118 with username:root, attempt#1 of 5 [2021-02-12 23:53:24,956] - [remote_util] [140096900417280] - INFO - SSH Connecting to 172.23.122.116 with username:root, attempt#1 of 5 [2021-02-12 23:53:25,046] - [remote_util] [140096917202688] - INFO - SSH Connected to 172.23.123.61 as root [2021-02-12 23:53:25,049] - [remote_util] [140099350165248] - INFO - SSH Connected to 172.23.122.90 as root [2021-02-12 23:53:25,051] - [remote_util] [140099567843072] - INFO - SSH Connected to 172.23.123.62 as root [2021-02-12 23:53:25,052] - [remote_util] [140099324987136] - INFO - SSH Connected to 172.23.122.117 as root [2021-02-12 23:53:25,053] - [remote_util] [140099299809024] - INFO - SSH Connected to 172.23.122.97 as root [2021-02-12 23:53:25,056] - [remote_util] [140097183020800] - INFO - SSH Connected to 172.23.122.82 as root [2021-02-12 23:53:25,056] - [remote_util] [140097168602880] - INFO - SSH Connected to 172.23.122.118 as root [2021-02-12 23:53:25,057] - [remote_util] [140096900417280] - INFO - SSH Connected to 172.23.122.116 as root [2021-02-12 23:53:25,342] - [remote_util] [140096917202688] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:53:25,348] - [remote_util] [140097168602880] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:53:25,350] - [remote_util] [140099324987136] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:53:25,364] - [remote_util] [140099567843072] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:53:25,368] - [remote_util] [140099299809024] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:53:25,369] - [remote_util] [140097183020800] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:53:25,373] - [remote_util] [140099350165248] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:53:25,381] - [remote_util] [140096900417280] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-02-12 23:53:25,666] - [remote_util] [140099567843072] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:53:25,667] - [remote_util] [140099567843072] - INFO - running command.raw on 172.23.123.62: /opt/couchbase/bin/cbcollect_info 172.23.123.62-20210212-2353-diag.zip [2021-02-12 23:53:25,673] - [remote_util] [140099299809024] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:53:25,673] - [remote_util] [140099299809024] - INFO - running command.raw on 172.23.122.97: /opt/couchbase/bin/cbcollect_info 172.23.122.97-20210212-2353-diag.zip [2021-02-12 23:53:25,675] - [remote_util] [140097183020800] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:53:25,676] - [remote_util] [140099350165248] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:53:25,677] - [remote_util] [140099350165248] - INFO - running command.raw on 172.23.122.90: /opt/couchbase/bin/cbcollect_info 172.23.122.90-20210212-2353-diag.zip [2021-02-12 23:53:25,678] - [remote_util] [140097183020800] - INFO - running command.raw on 172.23.122.82: /opt/couchbase/bin/cbcollect_info 172.23.122.82-20210212-2353-diag.zip [2021-02-12 23:53:25,680] - [remote_util] [140096900417280] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:53:25,680] - [remote_util] [140096900417280] - INFO - running command.raw on 172.23.122.116: /opt/couchbase/bin/cbcollect_info 172.23.122.116-20210212-2353-diag.zip [2021-02-12 23:53:25,685] - [remote_util] [140096917202688] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:53:25,686] - [remote_util] [140096917202688] - INFO - running command.raw on 172.23.123.61: /opt/couchbase/bin/cbcollect_info 172.23.123.61-20210212-2353-diag.zip [2021-02-12 23:53:25,693] - [remote_util] [140097168602880] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:53:25,694] - [remote_util] [140097168602880] - INFO - running command.raw on 172.23.122.118: /opt/couchbase/bin/cbcollect_info 172.23.122.118-20210212-2353-diag.zip [2021-02-12 23:53:25,697] - [remote_util] [140099324987136] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-02-12 23:53:25,698] - [remote_util] [140099324987136] - INFO - running command.raw on 172.23.122.117: /opt/couchbase/bin/cbcollect_info 172.23.122.117-20210212-2353-diag.zip [2021-02-12 23:54:54,623] - [remote_util] [140096900417280] - INFO - command executed successfully with root [2021-02-12 23:54:54,871] - [remote_util] [140096900417280] - INFO - running command.raw on 172.23.122.116: rm -f /root/172.23.122.116-20210212-2353-diag.zip [2021-02-12 23:54:54,887] - [remote_util] [140096900417280] - INFO - command executed successfully with root [2021-02-12 23:55:03,805] - [remote_util] [140099324987136] - INFO - command executed successfully with root [2021-02-12 23:55:04,347] - [remote_util] [140099324987136] - INFO - running command.raw on 172.23.122.117: rm -f /root/172.23.122.117-20210212-2353-diag.zip [2021-02-12 23:55:04,366] - [remote_util] [140099324987136] - INFO - command executed successfully with root [2021-02-12 23:55:05,280] - [remote_util] [140097168602880] - INFO - command executed successfully with root [2021-02-12 23:55:05,733] - [remote_util] [140097168602880] - INFO - running command.raw on 172.23.122.118: rm -f /root/172.23.122.118-20210212-2353-diag.zip [2021-02-12 23:55:05,751] - [remote_util] [140097168602880] - INFO - command executed successfully with root [2021-02-12 23:55:07,728] - [remote_util] [140099567843072] - INFO - command executed successfully with root [2021-02-12 23:55:08,449] - [remote_util] [140099567843072] - INFO - running command.raw on 172.23.123.62: rm -f /root/172.23.123.62-20210212-2353-diag.zip [2021-02-12 23:55:08,471] - [remote_util] [140099567843072] - INFO - command executed successfully with root [2021-02-12 23:55:09,461] - [remote_util] [140097183020800] - INFO - command executed successfully with root [2021-02-12 23:55:10,013] - [remote_util] [140097183020800] - INFO - running command.raw on 172.23.122.82: rm -f /root/172.23.122.82-20210212-2353-diag.zip [2021-02-12 23:55:10,033] - [remote_util] [140097183020800] - INFO - command executed successfully with root [2021-02-12 23:55:10,984] - [remote_util] [140096917202688] - INFO - command executed successfully with root [2021-02-12 23:55:11,795] - [remote_util] [140096917202688] - INFO - running command.raw on 172.23.123.61: rm -f /root/172.23.123.61-20210212-2353-diag.zip [2021-02-12 23:55:11,818] - [remote_util] [140096917202688] - INFO - command executed successfully with root [2021-02-12 23:55:13,606] - [remote_util] [140099350165248] - INFO - command executed successfully with root [2021-02-12 23:55:14,414] - [remote_util] [140099350165248] - INFO - running command.raw on 172.23.122.90: rm -f /root/172.23.122.90-20210212-2353-diag.zip [2021-02-12 23:55:14,435] - [remote_util] [140099350165248] - INFO - command executed successfully with root [2021-02-12 23:55:16,851] - [remote_util] [140099299809024] - INFO - command executed successfully with root [2021-02-12 23:55:17,711] - [remote_util] [140099299809024] - INFO - running command.raw on 172.23.122.97: rm -f /root/172.23.122.97-20210212-2353-diag.zip [2021-02-12 23:55:17,733] - [remote_util] [140099299809024] - INFO - command executed successfully with root