[2021-04-21 03:46:44,752] - [remote_util] [140342217144064] - INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2021-04-21 03:46:44,896] - [remote_util] [140342217144064] - INFO - SSH Connected to 172.23.104.172 as nonroot [2021-04-21 03:46:45,183] - [remote_util] [140342217144064] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-04-21 03:46:45,559] - [remote_util] [140342217144064] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-04-21 03:46:45,567] - [rest_client] [140342217144064] - INFO - -->is_ns_server_running? [2021-04-21 03:46:45,583] - [rest_client] [140342217144064] - INFO - Node version in cluster 7.0.0-4960-enterprise [2021-04-21 03:46:45,591] - [rest_client] [140342217144064] - ERROR - GET http://172.23.104.172:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-04-21 03:46:45,592] - [rest_client] [140342217144064] - INFO - Node versions in cluster [] [2021-04-21 03:46:45,593] - [basetestcase] [140342217144064] - INFO - ============== basetestcase setup was started for test #4 collectinfo_test============== [2021-04-21 03:46:45,601] - [rest_client] [140342217144064] - ERROR - GET http://172.23.104.172:8091/pools/default/ body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-04-21 03:46:45,614] - [bucket_helper] [140342217144064] - INFO - Could not find any buckets for node 172.23.104.172, nothing to delete [2021-04-21 03:46:45,621] - [bucket_helper] [140342217144064] - INFO - Could not find any buckets for node 172.23.105.127, nothing to delete [2021-04-21 03:46:45,630] - [bucket_helper] [140342217144064] - INFO - Could not find any buckets for node 172.23.105.128, nothing to delete [2021-04-21 03:46:45,637] - [bucket_helper] [140342217144064] - INFO - Could not find any buckets for node 172.23.105.126, nothing to delete [2021-04-21 03:46:45,645] - [bucket_helper] [140342217144064] - INFO - Could not find any buckets for node 172.23.105.158, nothing to delete [2021-04-21 03:46:45,654] - [bucket_helper] [140342217144064] - INFO - Could not find any buckets for node 172.23.104.158, nothing to delete [2021-04-21 03:46:45,660] - [rest_client] [140342217144064] - INFO - -->is_ns_server_running? [2021-04-21 03:46:45,680] - [cluster_helper] [140342217144064] - INFO - waiting for ns_server @ 172.23.104.172:8091 [2021-04-21 03:46:45,680] - [rest_client] [140342217144064] - INFO - -->is_ns_server_running? [2021-04-21 03:46:45,686] - [cluster_helper] [140342217144064] - INFO - ns_server @ 172.23.104.172:8091 is running [2021-04-21 03:46:45,690] - [cluster_helper] [140342217144064] - INFO - waiting for ns_server @ 172.23.105.127:8091 [2021-04-21 03:46:45,690] - [rest_client] [140342217144064] - INFO - -->is_ns_server_running? [2021-04-21 03:46:45,694] - [cluster_helper] [140342217144064] - INFO - ns_server @ 172.23.105.127:8091 is running [2021-04-21 03:46:45,700] - [cluster_helper] [140342217144064] - INFO - waiting for ns_server @ 172.23.105.128:8091 [2021-04-21 03:46:45,700] - [rest_client] [140342217144064] - INFO - -->is_ns_server_running? [2021-04-21 03:46:45,705] - [cluster_helper] [140342217144064] - INFO - ns_server @ 172.23.105.128:8091 is running [2021-04-21 03:46:45,710] - [cluster_helper] [140342217144064] - INFO - waiting for ns_server @ 172.23.105.126:8091 [2021-04-21 03:46:45,711] - [rest_client] [140342217144064] - INFO - -->is_ns_server_running? [2021-04-21 03:46:45,715] - [cluster_helper] [140342217144064] - INFO - ns_server @ 172.23.105.126:8091 is running [2021-04-21 03:46:45,719] - [cluster_helper] [140342217144064] - INFO - waiting for ns_server @ 172.23.105.158:8091 [2021-04-21 03:46:45,719] - [rest_client] [140342217144064] - INFO - -->is_ns_server_running? [2021-04-21 03:46:45,723] - [cluster_helper] [140342217144064] - INFO - ns_server @ 172.23.105.158:8091 is running [2021-04-21 03:46:45,730] - [cluster_helper] [140342217144064] - INFO - waiting for ns_server @ 172.23.104.158:8091 [2021-04-21 03:46:45,730] - [rest_client] [140342217144064] - INFO - -->is_ns_server_running? [2021-04-21 03:46:45,736] - [cluster_helper] [140342217144064] - INFO - ns_server @ 172.23.104.158:8091 is running [2021-04-21 03:46:45,737] - [basetestcase] [140342217144064] - INFO - initializing cluster [2021-04-21 03:46:45,757] - [task] [140342821123840] - INFO - server: ip:172.23.104.172 port:8091 ssh_username:nonroot, nodes/self [2021-04-21 03:46:45,762] - [task] [140342821123840] - INFO - {'uptime': '2000', 'memoryTotal': 3947372544, 'memoryFree': 3176980480, 'mcdMemoryReserved': 3011, 'mcdMemoryAllocated': 3011, 'status': 'healthy', 'hostname': '172.23.104.172:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4960-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1592, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.104.172', 'ip': '172.23.104.172', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 3764, 'curr_items': 0} [2021-04-21 03:46:45,766] - [rest_client] [140342821123840] - ERROR - GET http://172.23.104.172:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-04-21 03:46:45,767] - [task] [140342821123840] - INFO - quota for index service will be 256 MB [2021-04-21 03:46:45,767] - [task] [140342821123840] - INFO - set index quota to node 172.23.104.172 [2021-04-21 03:46:45,767] - [rest_client] [140342821123840] - INFO - pools/default params : indexMemoryQuota=256 [2021-04-21 03:46:45,775] - [rest_client] [140342821123840] - INFO - pools/default params : memoryQuota=1761 [2021-04-21 03:46:45,782] - [rest_client] [140342821123840] - INFO - --> init_node_services(Administrator,password,172.23.104.172,8091,['kv', 'index', 'n1ql']) [2021-04-21 03:46:45,783] - [rest_client] [140342821123840] - INFO - /node/controller/setupServices params on 172.23.104.172: 8091:hostname=172.23.104.172&user=Administrator&password=password&services=kv%2Cindex%2Cn1ql [2021-04-21 03:46:45,814] - [rest_client] [140342821123840] - INFO - --> in init_cluster...Administrator,password,8091 [2021-04-21 03:46:45,814] - [rest_client] [140342821123840] - INFO - settings/web params on 172.23.104.172:8091:port=8091&username=Administrator&password=password [2021-04-21 03:46:45,950] - [rest_client] [140342821123840] - INFO - --> status:True [2021-04-21 03:46:45,953] - [remote_util] [140342821123840] - INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2021-04-21 03:46:46,100] - [remote_util] [140342821123840] - INFO - SSH Connected to 172.23.104.172 as nonroot [2021-04-21 03:46:46,382] - [remote_util] [140342821123840] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-04-21 03:46:46,876] - [remote_util] [140342821123840] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-04-21 03:46:46,891] - [remote_util] [140342821123840] - INFO - running command.raw on 172.23.104.172: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-04-21 03:46:46,970] - [remote_util] [140342821123840] - INFO - command executed successfully with nonroot [2021-04-21 03:46:46,971] - [remote_util] [140342821123840] - INFO - ['ok'] [2021-04-21 03:46:46,975] - [rest_client] [140342821123840] - INFO - /diag/eval status on 172.23.104.172:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-04-21 03:46:46,979] - [rest_client] [140342821123840] - INFO - /diag/eval status on 172.23.104.172:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-04-21 03:46:46,989] - [rest_client] [140342821123840] - INFO - settings/indexes params : storageMode=plasma [2021-04-21 03:46:47,019] - [task] [140342821123840] - INFO - server: ip:172.23.105.127 port:8091 ssh_username:nonroot, nodes/self [2021-04-21 03:46:47,023] - [task] [140342821123840] - INFO - {'uptime': '1980', 'memoryTotal': 4201627648, 'memoryFree': 3552358400, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.105.127:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4960-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.105.127', 'ip': '172.23.105.127', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4006, 'curr_items': 0} [2021-04-21 03:46:47,026] - [rest_client] [140342821123840] - ERROR - GET http://172.23.105.127:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-04-21 03:46:47,026] - [rest_client] [140342821123840] - INFO - pools/default params : memoryQuota=2147 [2021-04-21 03:46:47,031] - [rest_client] [140342821123840] - INFO - --> in init_cluster...Administrator,password,8091 [2021-04-21 03:46:47,031] - [rest_client] [140342821123840] - INFO - settings/web params on 172.23.105.127:8091:port=8091&username=Administrator&password=password [2021-04-21 03:46:47,092] - [rest_client] [140342821123840] - INFO - --> status:True [2021-04-21 03:46:47,095] - [remote_util] [140342821123840] - INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2021-04-21 03:46:47,200] - [remote_util] [140342821123840] - INFO - SSH Connected to 172.23.105.127 as nonroot [2021-04-21 03:46:47,481] - [remote_util] [140342821123840] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-04-21 03:46:47,811] - [remote_util] [140342821123840] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-04-21 03:46:47,822] - [remote_util] [140342821123840] - INFO - running command.raw on 172.23.105.127: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-04-21 03:46:47,879] - [remote_util] [140342821123840] - INFO - command executed successfully with nonroot [2021-04-21 03:46:47,879] - [remote_util] [140342821123840] - INFO - ['ok'] [2021-04-21 03:46:47,883] - [rest_client] [140342821123840] - INFO - /diag/eval status on 172.23.105.127:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-04-21 03:46:47,885] - [rest_client] [140342821123840] - INFO - /diag/eval status on 172.23.105.127:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-04-21 03:46:47,890] - [rest_client] [140342821123840] - INFO - settings/indexes params : storageMode=plasma [2021-04-21 03:46:47,910] - [task] [140342821123840] - INFO - server: ip:172.23.105.128 port:8091 ssh_username:nonroot, nodes/self [2021-04-21 03:46:47,915] - [task] [140342821123840] - INFO - {'uptime': '1980', 'memoryTotal': 4201627648, 'memoryFree': 3477561344, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.105.128:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4960-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1761, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.105.128', 'ip': '172.23.105.128', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4006, 'curr_items': 0} [2021-04-21 03:46:47,920] - [rest_client] [140342821123840] - INFO - pools/default params : memoryQuota=2147 [2021-04-21 03:46:47,927] - [rest_client] [140342821123840] - INFO - --> in init_cluster...Administrator,password,8091 [2021-04-21 03:46:47,927] - [rest_client] [140342821123840] - INFO - settings/web params on 172.23.105.128:8091:port=8091&username=Administrator&password=password [2021-04-21 03:46:47,984] - [rest_client] [140342821123840] - INFO - --> status:True [2021-04-21 03:46:47,987] - [remote_util] [140342821123840] - INFO - SSH Connecting to 172.23.105.128 with username:nonroot, attempt#1 of 5 [2021-04-21 03:46:48,089] - [remote_util] [140342821123840] - INFO - SSH Connected to 172.23.105.128 as nonroot [2021-04-21 03:46:48,380] - [remote_util] [140342821123840] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-04-21 03:46:48,711] - [remote_util] [140342821123840] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-04-21 03:46:48,724] - [remote_util] [140342821123840] - INFO - running command.raw on 172.23.105.128: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-04-21 03:46:48,777] - [remote_util] [140342821123840] - INFO - command executed successfully with nonroot [2021-04-21 03:46:48,777] - [remote_util] [140342821123840] - INFO - ['ok'] [2021-04-21 03:46:48,781] - [rest_client] [140342821123840] - INFO - /diag/eval status on 172.23.105.128:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-04-21 03:46:48,785] - [rest_client] [140342821123840] - INFO - /diag/eval status on 172.23.105.128:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-04-21 03:46:48,792] - [rest_client] [140342821123840] - INFO - settings/indexes params : storageMode=plasma [2021-04-21 03:46:48,816] - [task] [140342821123840] - INFO - server: ip:172.23.105.126 port:8091 ssh_username:nonroot, nodes/self [2021-04-21 03:46:48,820] - [task] [140342821123840] - INFO - {'uptime': '1984', 'memoryTotal': 4201627648, 'memoryFree': 3553685504, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.105.126:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4960-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.105.126', 'ip': '172.23.105.126', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4006, 'curr_items': 0} [2021-04-21 03:46:48,823] - [rest_client] [140342821123840] - ERROR - GET http://172.23.105.126:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-04-21 03:46:48,823] - [rest_client] [140342821123840] - INFO - pools/default params : memoryQuota=2147 [2021-04-21 03:46:48,828] - [rest_client] [140342821123840] - INFO - --> in init_cluster...Administrator,password,8091 [2021-04-21 03:46:48,828] - [rest_client] [140342821123840] - INFO - settings/web params on 172.23.105.126:8091:port=8091&username=Administrator&password=password [2021-04-21 03:46:48,888] - [rest_client] [140342821123840] - INFO - --> status:True [2021-04-21 03:46:48,891] - [remote_util] [140342821123840] - INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2021-04-21 03:46:48,994] - [remote_util] [140342821123840] - INFO - SSH Connected to 172.23.105.126 as nonroot [2021-04-21 03:46:49,305] - [remote_util] [140342821123840] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-04-21 03:46:49,630] - [remote_util] [140342821123840] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-04-21 03:46:49,642] - [remote_util] [140342821123840] - INFO - running command.raw on 172.23.105.126: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-04-21 03:46:49,698] - [remote_util] [140342821123840] - INFO - command executed successfully with nonroot [2021-04-21 03:46:49,698] - [remote_util] [140342821123840] - INFO - ['ok'] [2021-04-21 03:46:49,701] - [rest_client] [140342821123840] - INFO - /diag/eval status on 172.23.105.126:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-04-21 03:46:49,704] - [rest_client] [140342821123840] - INFO - /diag/eval status on 172.23.105.126:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-04-21 03:46:49,709] - [rest_client] [140342821123840] - INFO - settings/indexes params : storageMode=plasma [2021-04-21 03:46:49,727] - [task] [140342821123840] - INFO - server: ip:172.23.105.158 port:8091 ssh_username:nonroot, nodes/self [2021-04-21 03:46:49,732] - [task] [140342821123840] - INFO - {'uptime': '1983', 'memoryTotal': 4201631744, 'memoryFree': 3591376896, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.105.158:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4960-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.105.158', 'ip': '172.23.105.158', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4006, 'curr_items': 0} [2021-04-21 03:46:49,734] - [rest_client] [140342821123840] - ERROR - GET http://172.23.105.158:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-04-21 03:46:49,735] - [rest_client] [140342821123840] - INFO - pools/default params : memoryQuota=2147 [2021-04-21 03:46:49,740] - [rest_client] [140342821123840] - INFO - --> in init_cluster...Administrator,password,8091 [2021-04-21 03:46:49,740] - [rest_client] [140342821123840] - INFO - settings/web params on 172.23.105.158:8091:port=8091&username=Administrator&password=password [2021-04-21 03:46:49,805] - [rest_client] [140342821123840] - INFO - --> status:True [2021-04-21 03:46:49,808] - [remote_util] [140342821123840] - INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2021-04-21 03:46:49,912] - [remote_util] [140342821123840] - INFO - SSH Connected to 172.23.105.158 as nonroot [2021-04-21 03:46:50,204] - [remote_util] [140342821123840] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-04-21 03:46:50,530] - [remote_util] [140342821123840] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-04-21 03:46:50,541] - [remote_util] [140342821123840] - INFO - running command.raw on 172.23.105.158: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-04-21 03:46:50,597] - [remote_util] [140342821123840] - INFO - command executed successfully with nonroot [2021-04-21 03:46:50,597] - [remote_util] [140342821123840] - INFO - ['ok'] [2021-04-21 03:46:50,600] - [rest_client] [140342821123840] - INFO - /diag/eval status on 172.23.105.158:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-04-21 03:46:50,603] - [rest_client] [140342821123840] - INFO - /diag/eval status on 172.23.105.158:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-04-21 03:46:50,608] - [rest_client] [140342821123840] - INFO - settings/indexes params : storageMode=plasma [2021-04-21 03:46:50,627] - [task] [140342821123840] - INFO - server: ip:172.23.104.158 port:8091 ssh_username:nonroot, nodes/self [2021-04-21 03:46:50,633] - [task] [140342821123840] - INFO - {'uptime': '2003', 'memoryTotal': 4201676800, 'memoryFree': 3546501120, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.104.158:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'recoveryType': 'none', 'version': '7.0.0-4960-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.104.158', 'ip': '172.23.104.158', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} [2021-04-21 03:46:50,636] - [rest_client] [140342821123840] - ERROR - GET http://172.23.104.158:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2021-04-21 03:46:50,637] - [rest_client] [140342821123840] - INFO - pools/default params : memoryQuota=2147 [2021-04-21 03:46:50,644] - [rest_client] [140342821123840] - INFO - --> in init_cluster...Administrator,password,8091 [2021-04-21 03:46:50,645] - [rest_client] [140342821123840] - INFO - settings/web params on 172.23.104.158:8091:port=8091&username=Administrator&password=password [2021-04-21 03:46:50,781] - [rest_client] [140342821123840] - INFO - --> status:True [2021-04-21 03:46:50,784] - [remote_util] [140342821123840] - INFO - SSH Connecting to 172.23.104.158 with username:nonroot, attempt#1 of 5 [2021-04-21 03:46:50,961] - [remote_util] [140342821123840] - INFO - SSH Connected to 172.23.104.158 as nonroot [2021-04-21 03:46:51,463] - [remote_util] [140342821123840] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-04-21 03:46:51,897] - [remote_util] [140342821123840] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-04-21 03:46:51,914] - [remote_util] [140342821123840] - INFO - running command.raw on 172.23.104.158: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-04-21 03:46:51,983] - [remote_util] [140342821123840] - INFO - command executed successfully with nonroot [2021-04-21 03:46:51,984] - [remote_util] [140342821123840] - INFO - ['ok'] [2021-04-21 03:46:51,989] - [rest_client] [140342821123840] - INFO - /diag/eval status on 172.23.104.158:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-04-21 03:46:51,993] - [rest_client] [140342821123840] - INFO - /diag/eval status on 172.23.104.158:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2021-04-21 03:46:52,002] - [rest_client] [140342821123840] - INFO - settings/indexes params : storageMode=plasma [2021-04-21 03:46:52,038] - [basetestcase] [140342217144064] - INFO - **** add built-in 'cbadminbucket' user to node 172.23.104.172 **** [2021-04-21 03:46:52,057] - [rest_client] [140342217144064] - ERROR - DELETE http://172.23.104.172:8091/settings/rbac/users/local/cbadminbucket body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:password [2021-04-21 03:46:52,058] - [internal_user] [140342217144064] - INFO - Exception while deleting user. Exception is -b'"User was not found."' [2021-04-21 03:46:52,214] - [basetestcase] [140342217144064] - INFO - sleep for 5 secs. ... [2021-04-21 03:46:57,219] - [basetestcase] [140342217144064] - INFO - **** add 'admin' role to 'cbadminbucket' user **** [2021-04-21 03:46:57,257] - [basetestcase] [140342217144064] - INFO - done initializing cluster [2021-04-21 03:46:57,260] - [remote_util] [140342217144064] - INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2021-04-21 03:46:57,400] - [remote_util] [140342217144064] - INFO - SSH Connected to 172.23.104.172 as nonroot [2021-04-21 03:46:57,675] - [remote_util] [140342217144064] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-04-21 03:46:58,069] - [remote_util] [140342217144064] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-04-21 03:46:58,089] - [remote_util] [140342217144064] - INFO - running command.raw on 172.23.104.172: curl --silent --show-error http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2021-04-21 03:46:58,155] - [remote_util] [140342217144064] - INFO - command executed successfully with nonroot [2021-04-21 03:46:58,156] - [remote_util] [140342217144064] - INFO - ['ok'] [2021-04-21 03:46:58,156] - [basetestcase] [140342217144064] - INFO - Enabled diag/eval for non-local hosts from 172.23.104.172 [2021-04-21 03:46:59,064] - [rest_client] [140342821123840] - INFO - http://172.23.104.172:8091/pools/default/buckets with param: name=default&ramQuotaMB=1761&replicaNumber=1&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=couchstore [2021-04-21 03:46:59,096] - [rest_client] [140342821123840] - INFO - 0.03 seconds to create bucket default [2021-04-21 03:46:59,097] - [bucket_helper] [140342821123840] - INFO - waiting for memcached bucket : default in 172.23.104.172 to accept set ops [2021-04-21 03:47:00,008] - [data_helper] [140342821123840] - INFO - creating direct client 172.23.104.172:11210 default [2021-04-21 03:47:00,359] - [data_helper] [140342821123840] - INFO - creating direct client 172.23.104.172:11210 default [2021-04-21 03:47:00,561] - [task] [140342821123840] - WARNING - vbucket map not ready after try 0 [2021-04-21 03:47:00,561] - [bucket_helper] [140342821123840] - INFO - waiting for memcached bucket : default in 172.23.104.172 to accept set ops [2021-04-21 03:47:00,860] - [data_helper] [140342821123840] - INFO - creating direct client 172.23.104.172:11210 default [2021-04-21 03:47:01,156] - [data_helper] [140342821123840] - INFO - creating direct client 172.23.104.172:11210 default [2021-04-21 03:47:01,310] - [task] [140342821123840] - WARNING - vbucket map not ready after try 1 [2021-04-21 03:47:01,310] - [bucket_helper] [140342821123840] - INFO - waiting for memcached bucket : default in 172.23.104.172 to accept set ops [2021-04-21 03:47:01,611] - [data_helper] [140342821123840] - INFO - creating direct client 172.23.104.172:11210 default [2021-04-21 03:47:01,872] - [data_helper] [140342821123840] - INFO - creating direct client 172.23.104.172:11210 default [2021-04-21 03:47:02,022] - [task] [140342821123840] - WARNING - vbucket map not ready after try 2 [2021-04-21 03:47:02,023] - [bucket_helper] [140342821123840] - INFO - waiting for memcached bucket : default in 172.23.104.172 to accept set ops [2021-04-21 03:47:02,303] - [data_helper] [140342821123840] - INFO - creating direct client 172.23.104.172:11210 default [2021-04-21 03:47:02,588] - [data_helper] [140342821123840] - INFO - creating direct client 172.23.104.172:11210 default [2021-04-21 03:47:02,735] - [task] [140342821123840] - WARNING - vbucket map not ready after try 3 [2021-04-21 03:47:02,736] - [bucket_helper] [140342821123840] - INFO - waiting for memcached bucket : default in 172.23.104.172 to accept set ops [2021-04-21 03:47:02,993] - [data_helper] [140342821123840] - INFO - creating direct client 172.23.104.172:11210 default [2021-04-21 03:47:03,294] - [data_helper] [140342821123840] - INFO - creating direct client 172.23.104.172:11210 default [2021-04-21 03:47:03,452] - [task] [140342821123840] - WARNING - vbucket map not ready after try 4 [2021-04-21 03:47:03,452] - [bucket_helper] [140342821123840] - INFO - waiting for memcached bucket : default in 172.23.104.172 to accept set ops [2021-04-21 03:47:03,745] - [data_helper] [140342821123840] - INFO - creating direct client 172.23.104.172:11210 default [2021-04-21 03:47:04,002] - [data_helper] [140342821123840] - INFO - creating direct client 172.23.104.172:11210 default [2021-04-21 03:47:04,194] - [task] [140342821123840] - WARNING - vbucket map not ready after try 5 [2021-04-21 03:47:04,198] - [basetestcase] [140342217144064] - INFO - ============== basetestcase setup was finished for test #4 collectinfo_test ============== [2021-04-21 03:47:04,210] - [remote_util] [140342217144064] - INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2021-04-21 03:47:04,355] - [remote_util] [140342217144064] - INFO - SSH Connected to 172.23.104.172 as nonroot [2021-04-21 03:47:04,633] - [remote_util] [140342217144064] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-04-21 03:47:05,012] - [remote_util] [140342217144064] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-04-21 03:47:05,016] - [remote_util] [140342217144064] - INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2021-04-21 03:47:05,118] - [remote_util] [140342217144064] - INFO - SSH Connected to 172.23.105.127 as nonroot [2021-04-21 03:47:05,386] - [remote_util] [140342217144064] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-04-21 03:47:05,710] - [remote_util] [140342217144064] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-04-21 03:47:05,714] - [remote_util] [140342217144064] - INFO - SSH Connecting to 172.23.105.128 with username:nonroot, attempt#1 of 5 [2021-04-21 03:47:05,817] - [remote_util] [140342217144064] - INFO - SSH Connected to 172.23.105.128 as nonroot [2021-04-21 03:47:06,112] - [remote_util] [140342217144064] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-04-21 03:47:06,446] - [remote_util] [140342217144064] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-04-21 03:47:06,450] - [remote_util] [140342217144064] - INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2021-04-21 03:47:06,555] - [remote_util] [140342217144064] - INFO - SSH Connected to 172.23.105.126 as nonroot [2021-04-21 03:47:06,857] - [remote_util] [140342217144064] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-04-21 03:47:07,190] - [remote_util] [140342217144064] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-04-21 03:47:07,193] - [remote_util] [140342217144064] - INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2021-04-21 03:47:07,298] - [remote_util] [140342217144064] - INFO - SSH Connected to 172.23.105.158 as nonroot [2021-04-21 03:47:07,595] - [remote_util] [140342217144064] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-04-21 03:47:07,931] - [remote_util] [140342217144064] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-04-21 03:47:07,936] - [remote_util] [140342217144064] - INFO - SSH Connecting to 172.23.104.158 with username:nonroot, attempt#1 of 5 [2021-04-21 03:47:08,099] - [remote_util] [140342217144064] - INFO - SSH Connected to 172.23.104.158 as nonroot [2021-04-21 03:47:08,580] - [remote_util] [140342217144064] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-04-21 03:47:09,014] - [remote_util] [140342217144064] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-04-21 03:47:14,126] - [basetestcase] [140342217144064] - INFO - ------- Cluster statistics ------- [2021-04-21 03:47:14,127] - [basetestcase] [140342217144064] - INFO - 172.23.104.172:8091 => {'services': ['index', 'kv', 'n1ql'], 'cpu_utilization': 27.78499091145157, 'mem_free': 3039522816, 'mem_total': 3947372544, 'swap_mem_used': 195776512, 'swap_mem_total': 3758092288} [2021-04-21 03:47:14,127] - [basetestcase] [140342217144064] - INFO - --- End of cluster statistics --- [2021-04-21 03:47:14,130] - [remote_util] [140342217144064] - INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2021-04-21 03:47:14,269] - [remote_util] [140342217144064] - INFO - SSH Connected to 172.23.104.172 as nonroot [2021-04-21 03:47:14,560] - [remote_util] [140342217144064] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-04-21 03:47:14,944] - [remote_util] [140342217144064] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-04-21 03:47:14,962] - [remote_util] [140342217144064] - INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2021-04-21 03:47:15,098] - [remote_util] [140342217144064] - INFO - SSH Connected to 172.23.104.172 as nonroot [2021-04-21 03:47:15,380] - [remote_util] [140342217144064] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-04-21 03:47:15,777] - [remote_util] [140342217144064] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-04-21 03:47:15,792] - [rest_client] [140342217144064] - INFO - Node version in cluster 7.0.0-4960-enterprise [2021-04-21 03:47:15,822] - [remote_util] [140342217144064] - INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2021-04-21 03:47:15,961] - [remote_util] [140342217144064] - INFO - SSH Connected to 172.23.104.172 as nonroot [2021-04-21 03:47:16,253] - [remote_util] [140342217144064] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-04-21 03:47:16,641] - [remote_util] [140342217144064] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-04-21 03:47:16,781] - [cli_base] [140342217144064] - INFO - Test command with absolute path [2021-04-21 03:47:17,131] - [task] [140342804338432] - INFO - adding node 172.23.105.127:8091 to cluster [2021-04-21 03:47:17,132] - [rest_client] [140342804338432] - INFO - adding remote node @172.23.105.127:8091 to this cluster @172.23.104.172:8091 [2021-04-21 03:47:27,143] - [rest_client] [140342804338432] - INFO - rebalance progress took 10.01 seconds [2021-04-21 03:47:27,144] - [rest_client] [140342804338432] - INFO - sleep for 10 seconds after rebalance... [2021-04-21 03:47:40,166] - [task] [140342804338432] - INFO - adding node 172.23.105.128:8091 to cluster [2021-04-21 03:47:40,166] - [rest_client] [140342804338432] - INFO - adding remote node @172.23.105.128:8091 to this cluster @172.23.104.172:8091 [2021-04-21 03:47:50,205] - [rest_client] [140342804338432] - INFO - rebalance progress took 10.04 seconds [2021-04-21 03:47:50,206] - [rest_client] [140342804338432] - INFO - sleep for 10 seconds after rebalance... [2021-04-21 03:48:02,883] - [task] [140342804338432] - INFO - adding node 172.23.105.126:8091 to cluster [2021-04-21 03:48:02,883] - [rest_client] [140342804338432] - INFO - adding remote node @172.23.105.126:8091 to this cluster @172.23.104.172:8091 [2021-04-21 03:48:12,908] - [rest_client] [140342804338432] - INFO - rebalance progress took 10.02 seconds [2021-04-21 03:48:12,909] - [rest_client] [140342804338432] - INFO - sleep for 10 seconds after rebalance... [2021-04-21 03:48:25,415] - [task] [140342804338432] - INFO - adding node 172.23.105.158:8091 to cluster [2021-04-21 03:48:25,416] - [rest_client] [140342804338432] - INFO - adding remote node @172.23.105.158:8091 to this cluster @172.23.104.172:8091 [2021-04-21 03:48:35,434] - [rest_client] [140342804338432] - INFO - rebalance progress took 10.02 seconds [2021-04-21 03:48:35,434] - [rest_client] [140342804338432] - INFO - sleep for 10 seconds after rebalance... [2021-04-21 03:48:48,205] - [task] [140342804338432] - INFO - adding node 172.23.104.158:8091 to cluster [2021-04-21 03:48:48,206] - [rest_client] [140342804338432] - INFO - adding remote node @172.23.104.158:8091 to this cluster @172.23.104.172:8091 [2021-04-21 03:48:58,237] - [rest_client] [140342804338432] - INFO - rebalance progress took 10.03 seconds [2021-04-21 03:48:58,237] - [rest_client] [140342804338432] - INFO - sleep for 10 seconds after rebalance... [2021-04-21 03:49:14,302] - [rest_client] [140342804338432] - INFO - rebalance params : {'knownNodes': 'ns_1@172.23.104.158,ns_1@172.23.104.172,ns_1@172.23.105.126,ns_1@172.23.105.127,ns_1@172.23.105.128,ns_1@172.23.105.158', 'ejectedNodes': '', 'user': 'Administrator', 'password': 'password'} [2021-04-21 03:49:14,336] - [rest_client] [140342804338432] - INFO - rebalance operation started [2021-04-21 03:49:14,341] - [rest_client] [140342804338432] - INFO - rebalance percentage : 0.00 % [2021-04-21 03:49:14,342] - [task] [140342804338432] - INFO - Rebalance - status: running, progress: 0.00% [2021-04-21 03:49:24,389] - [rest_client] [140342804338432] - INFO - rebalance percentage : 17.00 % [2021-04-21 03:49:24,390] - [task] [140342804338432] - INFO - Rebalance - status: running, progress: 17.00% [2021-04-21 03:49:34,445] - [rest_client] [140342804338432] - INFO - rebalance percentage : 36.00 % [2021-04-21 03:49:34,446] - [task] [140342804338432] - INFO - Rebalance - status: running, progress: 36.00% [2021-04-21 03:49:44,516] - [rest_client] [140342804338432] - INFO - rebalance percentage : 76.00 % [2021-04-21 03:49:44,516] - [task] [140342804338432] - INFO - Rebalance - status: running, progress: 76.00% [2021-04-21 03:49:54,595] - [task] [140342804338432] - INFO - Rebalance - status: none, progress: 100.00% [2021-04-21 03:49:54,621] - [task] [140342804338432] - INFO - rebalancing was completed with progress: 100% in 40.284560441970825 sec [2021-04-21 03:49:54,653] - [basetestcase] [140342217144064] - INFO - **** add built-in 'default' user to node 172.23.104.172 **** [2021-04-21 03:49:54,710] - [rest_client] [140342217144064] - ERROR - DELETE http://172.23.104.172:8091/settings/rbac/users/local/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"User was not found."' auth: Administrator:password [2021-04-21 03:49:54,712] - [internal_user] [140342217144064] - INFO - Exception while deleting user. Exception is -b'"User was not found."' [2021-04-21 03:49:54,900] - [basetestcase] [140342217144064] - INFO - sleep for 5 secs. ... [2021-04-21 03:49:59,902] - [basetestcase] [140342217144064] - INFO - **** add 'admin' role to 'default' user **** [2021-04-21 03:50:00,067] - [data_helper] [140342217144064] - INFO - creating direct client 172.23.104.158:11210 default [2021-04-21 03:50:00,309] - [data_helper] [140342217144064] - INFO - creating direct client 172.23.104.172:11210 default [2021-04-21 03:50:00,460] - [data_helper] [140342217144064] - INFO - creating direct client 172.23.105.126:11210 default [2021-04-21 03:50:00,564] - [data_helper] [140342217144064] - INFO - creating direct client 172.23.105.127:11210 default [2021-04-21 03:50:00,709] - [data_helper] [140342217144064] - INFO - creating direct client 172.23.105.128:11210 default [2021-04-21 03:50:00,818] - [data_helper] [140342217144064] - INFO - creating direct client 172.23.105.158:11210 default [2021-04-21 03:50:01,952] - [data_helper] [140343509112576] - INFO - creating direct client 172.23.104.158:11210 default [2021-04-21 03:50:01,954] - [data_helper] [140343509112576] - INFO - creating direct client 172.23.104.158:11210 default [2021-04-21 03:50:01,957] - [data_helper] [140343509112576] - INFO - creating direct client 172.23.104.158:11210 default [2021-04-21 03:50:01,969] - [data_helper] [140343509112576] - INFO - creating direct client 172.23.104.158:11210 default [2021-04-21 03:50:02,173] - [data_helper] [140343509112576] - INFO - creating direct client 172.23.104.172:11210 default [2021-04-21 03:50:02,175] - [data_helper] [140343509112576] - INFO - creating direct client 172.23.104.172:11210 default [2021-04-21 03:50:02,177] - [data_helper] [140343509112576] - INFO - creating direct client 172.23.104.172:11210 default [2021-04-21 03:50:02,188] - [data_helper] [140343509112576] - INFO - creating direct client 172.23.104.172:11210 default [2021-04-21 03:50:02,343] - [data_helper] [140343509112576] - INFO - creating direct client 172.23.105.126:11210 default [2021-04-21 03:50:02,346] - [data_helper] [140343509112576] - INFO - creating direct client 172.23.105.126:11210 default [2021-04-21 03:50:02,349] - [data_helper] [140343509112576] - INFO - creating direct client 172.23.105.126:11210 default [2021-04-21 03:50:02,367] - [data_helper] [140343509112576] - INFO - creating direct client 172.23.105.126:11210 default [2021-04-21 03:50:02,507] - [data_helper] [140343509112576] - INFO - creating direct client 172.23.105.127:11210 default [2021-04-21 03:50:02,509] - [data_helper] [140343509112576] - INFO - creating direct client 172.23.105.127:11210 default [2021-04-21 03:50:02,511] - [data_helper] [140343509112576] - INFO - creating direct client 172.23.105.127:11210 default [2021-04-21 03:50:02,530] - [data_helper] [140343509112576] - INFO - creating direct client 172.23.105.127:11210 default [2021-04-21 03:50:02,629] - [data_helper] [140343509112576] - INFO - creating direct client 172.23.105.128:11210 default [2021-04-21 03:50:02,629] - [data_helper] [140343509112576] - INFO - creating direct client 172.23.105.128:11210 default [2021-04-21 03:50:02,632] - [data_helper] [140343509112576] - INFO - creating direct client 172.23.105.128:11210 default [2021-04-21 03:50:02,639] - [data_helper] [140343509112576] - INFO - creating direct client 172.23.105.128:11210 default [2021-04-21 03:50:02,749] - [data_helper] [140343509112576] - INFO - creating direct client 172.23.105.158:11210 default [2021-04-21 03:50:02,752] - [data_helper] [140343509112576] - INFO - creating direct client 172.23.105.158:11210 default [2021-04-21 03:50:02,761] - [data_helper] [140343509112576] - INFO - creating direct client 172.23.105.158:11210 default [2021-04-21 03:50:02,764] - [data_helper] [140343509112576] - INFO - creating direct client 172.23.105.158:11210 default [2021-04-21 03:50:03,369] - [rest_client] [140342217144064] - INFO - Node versions in cluster ['7.0.0-4960-enterprise', '7.0.0-4960-enterprise', '7.0.0-4960-enterprise', '7.0.0-4960-enterprise', '7.0.0-4960-enterprise', '7.0.0-4960-enterprise'] [2021-04-21 03:50:03,385] - [basetestcase] [140342217144064] - INFO - list of kv nodes in cluster: [ip:172.23.104.158 port:8091 ssh_username:nonroot, ip:172.23.104.172 port:8091 ssh_username:nonroot, ip:172.23.105.126 port:8091 ssh_username:nonroot, ip:172.23.105.127 port:8091 ssh_username:nonroot, ip:172.23.105.128 port:8091 ssh_username:nonroot, ip:172.23.105.158 port:8091 ssh_username:nonroot] [2021-04-21 03:50:03,718] - [data_helper] [140342804338432] - INFO - creating direct client 172.23.104.172:11210 default [2021-04-21 03:50:03,889] - [task] [140342804338432] - INFO - Saw ep_queue_size 0 == 0 expected on '172.23.104.172:8091',default bucket [2021-04-21 03:50:03,903] - [data_helper] [140342804338432] - INFO - creating direct client 172.23.105.127:11210 default [2021-04-21 03:50:04,022] - [task] [140342804338432] - INFO - Saw ep_queue_size 0 == 0 expected on '172.23.105.127:8091',default bucket [2021-04-21 03:50:04,034] - [data_helper] [140342804338432] - INFO - creating direct client 172.23.105.128:11210 default [2021-04-21 03:50:04,194] - [task] [140342804338432] - INFO - Saw ep_queue_size 0 == 0 expected on '172.23.105.128:8091',default bucket [2021-04-21 03:50:04,208] - [data_helper] [140342804338432] - INFO - creating direct client 172.23.105.126:11210 default [2021-04-21 03:50:04,331] - [task] [140342804338432] - INFO - Saw ep_queue_size 0 == 0 expected on '172.23.105.126:8091',default bucket [2021-04-21 03:50:04,344] - [data_helper] [140342804338432] - INFO - creating direct client 172.23.105.158:11210 default [2021-04-21 03:50:04,462] - [task] [140342804338432] - INFO - Saw ep_queue_size 0 == 0 expected on '172.23.105.158:8091',default bucket [2021-04-21 03:50:04,483] - [data_helper] [140342804338432] - INFO - creating direct client 172.23.104.158:11210 default [2021-04-21 03:50:04,712] - [task] [140342804338432] - INFO - Saw ep_queue_size 0 == 0 expected on '172.23.104.158:8091',default bucket [2021-04-21 03:50:04,816] - [rest_client] [140342217144064] - INFO - settings/autoFailover params : timeout=60&enabled=false&failoverOnDataDiskIssues%5Benabled%5D=false&maxCount=1&failoverServerGroup=false [2021-04-21 03:50:04,823] - [remote_util] [140342217144064] - INFO - running command.raw on 172.23.104.172: killall -9 memcached & killall -9 beam.smp [2021-04-21 03:50:04,914] - [remote_util] [140342217144064] - INFO - command executed with nonroot but got an error ['memcached: no process found'] ... [2021-04-21 03:50:04,914] - [remote_util] [140342217144064] - ERROR - memcached: no process found [2021-04-21 03:50:04,914] - [remote_util] [140342217144064] - INFO - running command.raw on 172.23.104.172: /home/nonroot/opt/couchbase/bin/cbcollect_info info.zip [2021-04-21 03:52:08,048] - [remote_util] [140342217144064] - INFO - command executed successfully with nonroot [2021-04-21 03:52:08,052] - [remote_util] [140342217144064] - INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2021-04-21 03:52:08,194] - [remote_util] [140342217144064] - INFO - SSH Connected to 172.23.104.172 as nonroot [2021-04-21 03:52:08,480] - [remote_util] [140342217144064] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-04-21 03:52:08,861] - [remote_util] [140342217144064] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-04-21 03:52:08,866] - [rest_client] [140342217144064] - ERROR - socket error while connecting to http://172.23.104.172:8091/nodes/self error [Errno 111] Connection refused [2021-04-21 03:52:11,871] - [rest_client] [140342217144064] - ERROR - socket error while connecting to http://172.23.104.172:8091/nodes/self error [Errno 111] Connection refused [2021-04-21 03:52:17,879] - [rest_client] [140342217144064] - ERROR - socket error while connecting to http://172.23.104.172:8091/nodes/self error [Errno 111] Connection refused [2021-04-21 03:55:18,026] - [rest_client] [140342217144064] - ERROR - Giving up due to [Errno 111] Connection refused! Tried http://172.23.104.172:8091/nodes/self connect 7 times. [2021-04-21 03:55:37,923] - [remote_util] [140342217144064] - INFO - SSH Connecting to 172.23.104.172 with username:nonroot, attempt#1 of 5 [2021-04-21 03:55:37,927] - [remote_util] [140343509112576] - INFO - SSH Connecting to 172.23.105.127 with username:nonroot, attempt#1 of 5 [2021-04-21 03:55:37,930] - [remote_util] [140343517505280] - INFO - SSH Connecting to 172.23.105.128 with username:nonroot, attempt#1 of 5 [2021-04-21 03:55:37,933] - [remote_util] [140342795945728] - INFO - SSH Connecting to 172.23.105.126 with username:nonroot, attempt#1 of 5 [2021-04-21 03:55:37,936] - [remote_util] [140342787553024] - INFO - SSH Connecting to 172.23.105.158 with username:nonroot, attempt#1 of 5 [2021-04-21 03:55:37,941] - [remote_util] [140342770767616] - INFO - SSH Connecting to 172.23.104.158 with username:nonroot, attempt#1 of 5 [2021-04-21 03:55:38,041] - [remote_util] [140343517505280] - INFO - SSH Connected to 172.23.105.128 as nonroot [2021-04-21 03:55:38,044] - [remote_util] [140343509112576] - INFO - SSH Connected to 172.23.105.127 as nonroot [2021-04-21 03:55:38,050] - [remote_util] [140342795945728] - INFO - SSH Connected to 172.23.105.126 as nonroot [2021-04-21 03:55:38,056] - [remote_util] [140342787553024] - INFO - SSH Connected to 172.23.105.158 as nonroot [2021-04-21 03:55:38,072] - [remote_util] [140342217144064] - INFO - SSH Connected to 172.23.104.172 as nonroot [2021-04-21 03:55:38,115] - [remote_util] [140342770767616] - INFO - SSH Connected to 172.23.104.158 as nonroot [2021-04-21 03:55:38,330] - [remote_util] [140343509112576] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-04-21 03:55:38,389] - [remote_util] [140343517505280] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-04-21 03:55:38,402] - [remote_util] [140342217144064] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-04-21 03:55:38,404] - [remote_util] [140342787553024] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-04-21 03:55:38,410] - [remote_util] [140342795945728] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-04-21 03:55:38,628] - [remote_util] [140342770767616] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2021-04-21 03:55:38,722] - [remote_util] [140343509112576] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-04-21 03:55:38,723] - [remote_util] [140343509112576] - INFO - running command.raw on 172.23.105.127: /home/nonroot/opt/couchbase/bin/cbcollect_info 172.23.105.127-20210421-0355-diag.zip [2021-04-21 03:55:38,748] - [remote_util] [140342787553024] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-04-21 03:55:38,752] - [remote_util] [140342787553024] - INFO - running command.raw on 172.23.105.158: /home/nonroot/opt/couchbase/bin/cbcollect_info 172.23.105.158-20210421-0355-diag.zip [2021-04-21 03:55:38,776] - [remote_util] [140343517505280] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-04-21 03:55:38,778] - [remote_util] [140343517505280] - INFO - running command.raw on 172.23.105.128: /home/nonroot/opt/couchbase/bin/cbcollect_info 172.23.105.128-20210421-0355-diag.zip [2021-04-21 03:55:38,801] - [remote_util] [140342795945728] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-04-21 03:55:38,802] - [remote_util] [140342795945728] - INFO - running command.raw on 172.23.105.126: /home/nonroot/opt/couchbase/bin/cbcollect_info 172.23.105.126-20210421-0355-diag.zip [2021-04-21 03:55:38,839] - [remote_util] [140342217144064] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-04-21 03:55:38,840] - [remote_util] [140342217144064] - INFO - running command.raw on 172.23.104.172: /home/nonroot/opt/couchbase/bin/cbcollect_info 172.23.104.172-20210421-0355-diag.zip [2021-04-21 03:55:39,070] - [remote_util] [140342770767616] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2021-04-21 03:55:39,071] - [remote_util] [140342770767616] - INFO - running command.raw on 172.23.104.158: /home/nonroot/opt/couchbase/bin/cbcollect_info 172.23.104.158-20210421-0355-diag.zip [2021-04-21 03:57:44,148] - [remote_util] [140342217144064] - INFO - command executed successfully with nonroot [2021-04-21 03:57:45,251] - [remote_util] [140342217144064] - INFO - running command.raw on 172.23.104.172: rm -f /home/nonroot/172.23.104.172-20210421-0355-diag.zip [2021-04-21 03:57:45,287] - [remote_util] [140342217144064] - INFO - command executed successfully with nonroot [2021-04-21 03:58:00,157] - [remote_util] [140342787553024] - INFO - command executed successfully with nonroot [2021-04-21 03:58:00,743] - [remote_util] [140342787553024] - INFO - running command.raw on 172.23.105.158: rm -f /home/nonroot/172.23.105.158-20210421-0355-diag.zip [2021-04-21 03:58:00,802] - [remote_util] [140342787553024] - INFO - command executed successfully with nonroot [2021-04-21 03:58:00,892] - [remote_util] [140343517505280] - INFO - command executed successfully with nonroot [2021-04-21 03:58:01,400] - [remote_util] [140343517505280] - INFO - running command.raw on 172.23.105.128: rm -f /home/nonroot/172.23.105.128-20210421-0355-diag.zip [2021-04-21 03:58:01,460] - [remote_util] [140343517505280] - INFO - command executed successfully with nonroot [2021-04-21 03:58:08,242] - [remote_util] [140343509112576] - INFO - command executed successfully with nonroot [2021-04-21 03:58:08,827] - [remote_util] [140343509112576] - INFO - running command.raw on 172.23.105.127: rm -f /home/nonroot/172.23.105.127-20210421-0355-diag.zip [2021-04-21 03:58:08,887] - [remote_util] [140343509112576] - INFO - command executed successfully with nonroot [2021-04-21 03:58:09,301] - [remote_util] [140342795945728] - INFO - command executed successfully with nonroot [2021-04-21 03:58:09,779] - [remote_util] [140342795945728] - INFO - running command.raw on 172.23.105.126: rm -f /home/nonroot/172.23.105.126-20210421-0355-diag.zip [2021-04-21 03:58:09,840] - [remote_util] [140342795945728] - INFO - command executed successfully with nonroot [2021-04-21 03:58:48,224] - [remote_util] [140342770767616] - INFO - command executed successfully with nonroot [2021-04-21 03:58:48,763] - [remote_util] [140342770767616] - INFO - running command.raw on 172.23.104.158: rm -f /home/nonroot/172.23.104.158-20210421-0355-diag.zip [2021-04-21 03:58:48,799] - [remote_util] [140342770767616] - INFO - command executed successfully with nonroot