[2020-09-30 03:28:16,868] - [eventing_base] [140493040056064] - INFO - Starting Test: test_advance_bucket_op [2020-09-30 03:28:16,870] - [remote_util] [140493040056064] - INFO - SSH Connecting to 172.23.123.108 with username:root, attempt#1 of 5 [2020-09-30 03:28:16,991] - [remote_util] [140493040056064] - INFO - SSH Connected to 172.23.123.108 as root [2020-09-30 03:28:17,320] - [remote_util] [140493040056064] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-09-30 03:28:17,617] - [remote_util] [140493040056064] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2020-09-30 03:28:17,622] - [rest_client] [140493040056064] - INFO - -->is_ns_server_running? [2020-09-30 03:28:17,639] - [rest_client] [140493040056064] - INFO - Node version in cluster 7.0.0-3239-enterprise [2020-09-30 03:28:17,654] - [rest_client] [140493040056064] - INFO - Node versions in cluster ['7.0.0-3239-enterprise', '7.0.0-3239-enterprise', '7.0.0-3239-enterprise', '7.0.0-3239-enterprise'] [2020-09-30 03:28:17,654] - [basetestcase] [140493040056064] - INFO - ============== basetestcase setup was started for test #12 test_advance_bucket_op============== [2020-09-30 03:28:17,700] - [bucket_helper] [140493040056064] - INFO - deleting existing buckets ['dst_bucket', 'dst_bucket1', 'metadata', 'src_bucket'] on 172.23.123.108 [2020-09-30 03:28:22,147] - [bucket_helper] [140493040056064] - INFO - waiting for bucket deletion to complete.... [2020-09-30 03:28:22,173] - [rest_client] [140493040056064] - INFO - node 172.23.123.108 existing buckets : ['dst_bucket1', 'metadata', 'src_bucket'] [2020-09-30 03:28:22,176] - [bucket_helper] [140493040056064] - INFO - deleted bucket : dst_bucket from 172.23.123.108 [2020-09-30 03:28:27,152] - [bucket_helper] [140493040056064] - INFO - waiting for bucket deletion to complete.... [2020-09-30 03:28:27,167] - [rest_client] [140493040056064] - INFO - node 172.23.123.108 existing buckets : ['metadata', 'src_bucket'] [2020-09-30 03:28:27,168] - [bucket_helper] [140493040056064] - INFO - deleted bucket : dst_bucket1 from 172.23.123.108 [2020-09-30 03:28:32,705] - [bucket_helper] [140493040056064] - INFO - waiting for bucket deletion to complete.... [2020-09-30 03:28:32,714] - [rest_client] [140493040056064] - INFO - node 172.23.123.108 existing buckets : ['src_bucket'] [2020-09-30 03:28:32,715] - [bucket_helper] [140493040056064] - INFO - deleted bucket : metadata from 172.23.123.108 [2020-09-30 03:28:35,802] - [bucket_helper] [140493040056064] - INFO - waiting for bucket deletion to complete.... [2020-09-30 03:28:35,805] - [rest_client] [140493040056064] - INFO - node 172.23.123.108 existing buckets : [] [2020-09-30 03:28:35,805] - [bucket_helper] [140493040056064] - INFO - deleted bucket : src_bucket from 172.23.123.108 [2020-09-30 03:28:35,815] - [bucket_helper] [140493040056064] - INFO - Could not find any buckets for node 172.23.123.105, nothing to delete [2020-09-30 03:28:35,822] - [bucket_helper] [140493040056064] - INFO - Could not find any buckets for node 172.23.123.107, nothing to delete [2020-09-30 03:28:35,828] - [bucket_helper] [140493040056064] - INFO - Could not find any buckets for node 172.23.123.104, nothing to delete [2020-09-30 03:28:35,833] - [rest_client] [140493040056064] - INFO - -->is_ns_server_running? [2020-09-30 03:28:35,843] - [cluster_helper] [140493040056064] - INFO - rebalancing all nodes in order to remove nodes [2020-09-30 03:28:35,845] - [rest_client] [140493040056064] - INFO - rebalance params : {'knownNodes': 'ns_1@172.23.123.104,ns_1@172.23.123.105,ns_1@172.23.123.107,ns_1@172.23.123.108', 'ejectedNodes': 'ns_1@172.23.123.104,ns_1@172.23.123.105,ns_1@172.23.123.107', 'user': 'Administrator', 'password': 'password'} [2020-09-30 03:28:35,850] - [rest_client] [140493040056064] - INFO - rebalance operation started [2020-09-30 03:28:35,851] - [rest_client] [140493040056064] - INFO - rebalance percentage : 0.00 % [2020-09-30 03:28:45,864] - [rest_client] [140493040056064] - INFO - rebalance percentage : 50.00 % [2020-09-30 03:28:55,877] - [rest_client] [140493040056064] - INFO - rebalance percentage : 75.00 % [2020-09-30 03:29:15,902] - [rest_client] [140493040056064] - INFO - rebalance progress took 40.05 seconds [2020-09-30 03:29:15,903] - [rest_client] [140493040056064] - INFO - sleep for 10 seconds after rebalance... [2020-09-30 03:29:25,930] - [cluster_helper] [140493040056064] - INFO - removed all the nodes from cluster associated with ip:172.23.123.108 port:8091 ssh_username:root ? [('ns_1@172.23.123.104', 8091), ('ns_1@172.23.123.105', 8091), ('ns_1@172.23.123.107', 8091)] [2020-09-30 03:29:25,935] - [cluster_helper] [140493040056064] - INFO - waiting for ns_server @ 172.23.123.108:8091 [2020-09-30 03:29:25,935] - [rest_client] [140493040056064] - INFO - -->is_ns_server_running? [2020-09-30 03:29:25,939] - [cluster_helper] [140493040056064] - INFO - ns_server @ 172.23.123.108:8091 is running [2020-09-30 03:29:25,942] - [cluster_helper] [140493040056064] - INFO - waiting for ns_server @ 172.23.123.105:8091 [2020-09-30 03:29:25,942] - [rest_client] [140493040056064] - INFO - -->is_ns_server_running? [2020-09-30 03:29:25,945] - [cluster_helper] [140493040056064] - INFO - ns_server @ 172.23.123.105:8091 is running [2020-09-30 03:29:25,948] - [cluster_helper] [140493040056064] - INFO - waiting for ns_server @ 172.23.123.107:8091 [2020-09-30 03:29:25,948] - [rest_client] [140493040056064] - INFO - -->is_ns_server_running? [2020-09-30 03:29:25,950] - [cluster_helper] [140493040056064] - INFO - ns_server @ 172.23.123.107:8091 is running [2020-09-30 03:29:25,953] - [cluster_helper] [140493040056064] - INFO - waiting for ns_server @ 172.23.123.104:8091 [2020-09-30 03:29:25,953] - [rest_client] [140493040056064] - INFO - -->is_ns_server_running? [2020-09-30 03:29:25,956] - [cluster_helper] [140493040056064] - INFO - ns_server @ 172.23.123.104:8091 is running [2020-09-30 03:29:25,956] - [basetestcase] [140493040056064] - INFO - initializing cluster [2020-09-30 03:29:26,946] - [task] [140489839859456] - INFO - server: ip:172.23.123.108 port:8091 ssh_username:root, nodes/self [2020-09-30 03:29:26,949] - [task] [140489839859456] - INFO - {'uptime': '14802', 'memoryTotal': 4201676800, 'memoryFree': 2729730048, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.123.108:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'version': '7.0.0-3239-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 900, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.123.108', 'ip': '172.23.123.108', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} [2020-09-30 03:29:26,953] - [rest_client] [140489839859456] - INFO - pools/default params : memoryQuota=2147 [2020-09-30 03:29:26,958] - [rest_client] [140489839859456] - INFO - --> init_node_services(Administrator,password,172.23.123.108,8091,['kv']) [2020-09-30 03:29:26,959] - [rest_client] [140489839859456] - INFO - /node/controller/setupServices params on 172.23.123.108: 8091:hostname=172.23.123.108&user=Administrator&password=password&services=kv [2020-09-30 03:29:26,961] - [rest_client] [140489839859456] - ERROR - POST http://172.23.123.108:8091//node/controller/setupServices body: hostname=172.23.123.108&user=Administrator&password=password&services=kv headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 400 reason: unknown b'["cannot change node services after cluster is provisioned"]' auth: Administrator:password [2020-09-30 03:29:26,962] - [rest_client] [140489839859456] - INFO - This node is already provisioned with services, we do not consider this as failure for test case [2020-09-30 03:29:26,962] - [rest_client] [140489839859456] - INFO - --> in init_cluster...Administrator,password,8091 [2020-09-30 03:29:26,962] - [rest_client] [140489839859456] - INFO - settings/web params on 172.23.123.108:8091:port=8091&username=Administrator&password=password [2020-09-30 03:29:27,010] - [rest_client] [140489839859456] - INFO - --> status:True [2020-09-30 03:29:27,010] - [remote_util] [140489839859456] - INFO - SSH Connecting to 172.23.123.108 with username:root, attempt#1 of 5 [2020-09-30 03:29:27,116] - [remote_util] [140489839859456] - INFO - SSH Connected to 172.23.123.108 as root [2020-09-30 03:29:27,382] - [remote_util] [140489839859456] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-09-30 03:29:27,668] - [remote_util] [140489839859456] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2020-09-30 03:29:27,677] - [remote_util] [140489839859456] - INFO - running command.raw on 172.23.123.108: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2020-09-30 03:29:27,738] - [remote_util] [140489839859456] - INFO - command executed with root but got an error [' % Total % Received % Xferd Average Speed Time Time Time Current', ' Dload Upload Total Spent Left Speed', '', ' 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0', '100 43 100 2 100 41 291 5966 --:--:-- --:--:-- --:--:-- 6833'] ... [2020-09-30 03:29:27,738] - [remote_util] [140489839859456] - INFO - ['ok'] [2020-09-30 03:29:27,740] - [rest_client] [140489839859456] - INFO - /diag/eval status on 172.23.123.108:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2020-09-30 03:29:27,742] - [rest_client] [140489839859456] - INFO - /diag/eval status on 172.23.123.108:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2020-09-30 03:29:27,748] - [rest_client] [140489839859456] - INFO - settings/indexes params : storageMode=plasma [2020-09-30 03:29:27,770] - [task] [140489839859456] - INFO - server: ip:172.23.123.105 port:8091 ssh_username:root, nodes/self [2020-09-30 03:29:27,773] - [task] [140489839859456] - INFO - {'uptime': '14799', 'memoryTotal': 4201672704, 'memoryFree': 3499593728, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.123.105:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'version': '7.0.0-3239-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.123.105', 'ip': '172.23.123.105', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} [2020-09-30 03:29:27,775] - [rest_client] [140489839859456] - ERROR - GET http://172.23.123.105:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2020-09-30 03:29:27,775] - [rest_client] [140489839859456] - INFO - pools/default params : memoryQuota=2147 [2020-09-30 03:29:27,778] - [rest_client] [140489839859456] - INFO - --> in init_cluster...Administrator,password,8091 [2020-09-30 03:29:27,779] - [rest_client] [140489839859456] - INFO - settings/web params on 172.23.123.105:8091:port=8091&username=Administrator&password=password [2020-09-30 03:29:27,831] - [rest_client] [140489839859456] - INFO - --> status:True [2020-09-30 03:29:27,832] - [remote_util] [140489839859456] - INFO - SSH Connecting to 172.23.123.105 with username:root, attempt#1 of 5 [2020-09-30 03:29:27,934] - [remote_util] [140489839859456] - INFO - SSH Connected to 172.23.123.105 as root [2020-09-30 03:29:28,205] - [remote_util] [140489839859456] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-09-30 03:29:28,497] - [remote_util] [140489839859456] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2020-09-30 03:29:28,503] - [remote_util] [140489839859456] - INFO - running command.raw on 172.23.123.105: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2020-09-30 03:29:28,560] - [remote_util] [140489839859456] - INFO - command executed with root but got an error [' % Total % Received % Xferd Average Speed Time Time Time Current', ' Dload Upload Total Spent Left Speed', '', ' 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0', '100 43 100 2 100 41 278 5707 --:--:-- --:--:-- --:--:-- 5857'] ... [2020-09-30 03:29:28,561] - [remote_util] [140489839859456] - INFO - ['ok'] [2020-09-30 03:29:28,563] - [rest_client] [140489839859456] - INFO - /diag/eval status on 172.23.123.105:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2020-09-30 03:29:28,564] - [rest_client] [140489839859456] - INFO - /diag/eval status on 172.23.123.105:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2020-09-30 03:29:28,568] - [rest_client] [140489839859456] - INFO - settings/indexes params : storageMode=plasma [2020-09-30 03:29:28,583] - [task] [140489839859456] - INFO - server: ip:172.23.123.107 port:8091 ssh_username:root, nodes/self [2020-09-30 03:29:28,586] - [task] [140489839859456] - INFO - {'uptime': '14803', 'memoryTotal': 4201672704, 'memoryFree': 3520483328, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.123.107:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'version': '7.0.0-3239-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.123.107', 'ip': '172.23.123.107', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} [2020-09-30 03:29:28,588] - [rest_client] [140489839859456] - ERROR - GET http://172.23.123.107:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2020-09-30 03:29:28,588] - [rest_client] [140489839859456] - INFO - pools/default params : memoryQuota=2147 [2020-09-30 03:29:28,591] - [rest_client] [140489839859456] - INFO - --> in init_cluster...Administrator,password,8091 [2020-09-30 03:29:28,591] - [rest_client] [140489839859456] - INFO - settings/web params on 172.23.123.107:8091:port=8091&username=Administrator&password=password [2020-09-30 03:29:28,642] - [rest_client] [140489839859456] - INFO - --> status:True [2020-09-30 03:29:28,643] - [remote_util] [140489839859456] - INFO - SSH Connecting to 172.23.123.107 with username:root, attempt#1 of 5 [2020-09-30 03:29:28,747] - [remote_util] [140489839859456] - INFO - SSH Connected to 172.23.123.107 as root [2020-09-30 03:29:29,021] - [remote_util] [140489839859456] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-09-30 03:29:29,307] - [remote_util] [140489839859456] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2020-09-30 03:29:29,315] - [remote_util] [140489839859456] - INFO - running command.raw on 172.23.123.107: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2020-09-30 03:29:29,372] - [remote_util] [140489839859456] - INFO - command executed with root but got an error [' % Total % Received % Xferd Average Speed Time Time Time Current', ' Dload Upload Total Spent Left Speed', '', ' 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0', '100 43 100 2 100 41 263 5408 --:--:-- --:--:-- --:--:-- 5857'] ... [2020-09-30 03:29:29,372] - [remote_util] [140489839859456] - INFO - ['ok'] [2020-09-30 03:29:29,375] - [rest_client] [140489839859456] - INFO - /diag/eval status on 172.23.123.107:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2020-09-30 03:29:29,377] - [rest_client] [140489839859456] - INFO - /diag/eval status on 172.23.123.107:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2020-09-30 03:29:29,380] - [rest_client] [140489839859456] - INFO - settings/indexes params : storageMode=plasma [2020-09-30 03:29:29,396] - [task] [140489839859456] - INFO - server: ip:172.23.123.104 port:8091 ssh_username:root, nodes/self [2020-09-30 03:29:29,398] - [task] [140489839859456] - INFO - {'uptime': '14804', 'memoryTotal': 4201672704, 'memoryFree': 3565629440, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.123.104:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'version': '7.0.0-3239-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.123.104', 'ip': '172.23.123.104', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} [2020-09-30 03:29:29,400] - [rest_client] [140489839859456] - ERROR - GET http://172.23.123.104:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2020-09-30 03:29:29,400] - [rest_client] [140489839859456] - INFO - pools/default params : memoryQuota=2147 [2020-09-30 03:29:29,403] - [rest_client] [140489839859456] - INFO - --> in init_cluster...Administrator,password,8091 [2020-09-30 03:29:29,403] - [rest_client] [140489839859456] - INFO - settings/web params on 172.23.123.104:8091:port=8091&username=Administrator&password=password [2020-09-30 03:29:29,454] - [rest_client] [140489839859456] - INFO - --> status:True [2020-09-30 03:29:29,455] - [remote_util] [140489839859456] - INFO - SSH Connecting to 172.23.123.104 with username:root, attempt#1 of 5 [2020-09-30 03:29:29,557] - [remote_util] [140489839859456] - INFO - SSH Connected to 172.23.123.104 as root [2020-09-30 03:29:29,816] - [remote_util] [140489839859456] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-09-30 03:29:30,104] - [remote_util] [140489839859456] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2020-09-30 03:29:30,111] - [remote_util] [140489839859456] - INFO - running command.raw on 172.23.123.104: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2020-09-30 03:29:30,168] - [remote_util] [140489839859456] - INFO - command executed with root but got an error [' % Total % Received % Xferd Average Speed Time Time Time Current', ' Dload Upload Total Spent Left Speed', '', ' 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0', '100 43 100 2 100 41 291 5981 --:--:-- --:--:-- --:--:-- 6833'] ... [2020-09-30 03:29:30,168] - [remote_util] [140489839859456] - INFO - ['ok'] [2020-09-30 03:29:30,170] - [rest_client] [140489839859456] - INFO - /diag/eval status on 172.23.123.104:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2020-09-30 03:29:30,172] - [rest_client] [140489839859456] - INFO - /diag/eval status on 172.23.123.104:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2020-09-30 03:29:30,178] - [rest_client] [140489839859456] - INFO - settings/indexes params : storageMode=plasma [2020-09-30 03:29:30,198] - [basetestcase] [140493040056064] - INFO - **** add built-in 'cbadminbucket' user to node 172.23.123.108 **** [2020-09-30 03:29:30,273] - [basetestcase] [140493040056064] - INFO - sleep for 5 secs. ... [2020-09-30 03:29:35,276] - [basetestcase] [140493040056064] - INFO - **** add 'admin' role to 'cbadminbucket' user **** [2020-09-30 03:29:35,288] - [basetestcase] [140493040056064] - INFO - done initializing cluster [2020-09-30 03:29:36,195] - [task] [140489839859456] - INFO - adding node 172.23.123.105:8091 to cluster [2020-09-30 03:29:36,196] - [rest_client] [140489839859456] - INFO - adding remote node @172.23.123.105:8091 to this cluster @172.23.123.108:8091 [2020-09-30 03:29:38,237] - [task] [140489839859456] - INFO - adding node 172.23.123.107:8091 to cluster [2020-09-30 03:29:38,237] - [rest_client] [140489839859456] - INFO - adding remote node @172.23.123.107:8091 to this cluster @172.23.123.108:8091 [2020-09-30 03:29:40,442] - [task] [140489839859456] - INFO - adding node 172.23.123.104:8091 to cluster [2020-09-30 03:29:40,442] - [rest_client] [140489839859456] - INFO - adding remote node @172.23.123.104:8091 to this cluster @172.23.123.108:8091 [2020-09-30 03:29:42,513] - [rest_client] [140489839859456] - INFO - rebalance params : {'knownNodes': 'ns_1@172.23.123.104,ns_1@172.23.123.105,ns_1@172.23.123.107,ns_1@172.23.123.108', 'ejectedNodes': '', 'user': 'Administrator', 'password': 'password'} [2020-09-30 03:29:42,517] - [rest_client] [140489839859456] - INFO - rebalance operation started [2020-09-30 03:29:42,520] - [rest_client] [140489839859456] - INFO - rebalance percentage : 0.00 % [2020-09-30 03:29:42,520] - [task] [140489839859456] - INFO - Rebalance - status: running, progress: 0.00% [2020-09-30 03:29:52,538] - [rest_client] [140489839859456] - INFO - rebalance percentage : 50.00 % [2020-09-30 03:29:52,538] - [task] [140489839859456] - INFO - Rebalance - status: running, progress: 50.00% [2020-09-30 03:30:02,557] - [task] [140489839859456] - INFO - Rebalance - status: none, progress: 100.00% [2020-09-30 03:30:02,563] - [task] [140489839859456] - INFO - rebalancing was completed with progress: 100% in 20.045892477035522 sec [2020-09-30 03:30:02,564] - [remote_util] [140493040056064] - INFO - SSH Connecting to 172.23.123.108 with username:root, attempt#1 of 5 [2020-09-30 03:30:02,662] - [remote_util] [140493040056064] - INFO - SSH Connected to 172.23.123.108 as root [2020-09-30 03:30:02,927] - [remote_util] [140493040056064] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-09-30 03:30:03,211] - [remote_util] [140493040056064] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2020-09-30 03:30:03,219] - [remote_util] [140493040056064] - INFO - running command.raw on 172.23.123.108: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2020-09-30 03:30:03,273] - [remote_util] [140493040056064] - INFO - command executed with root but got an error [' % Total % Received % Xferd Average Speed Time Time Time Current', ' Dload Upload Total Spent Left Speed', '', ' 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0', '100 43 100 2 100 41 290 5948 --:--:-- --:--:-- --:--:-- 6833'] ... [2020-09-30 03:30:03,273] - [remote_util] [140493040056064] - INFO - ['ok'] [2020-09-30 03:30:03,274] - [basetestcase] [140493040056064] - INFO - Enabled diag/eval for non-local hosts from 172.23.123.108 [2020-09-30 03:30:03,274] - [basetestcase] [140493040056064] - INFO - ============== basetestcase setup was finished for test #12 test_advance_bucket_op ============== [2020-09-30 03:30:03,291] - [basetestcase] [140493040056064] - INFO - ------- Cluster statistics ------- [2020-09-30 03:30:03,291] - [basetestcase] [140493040056064] - INFO - 172.23.123.104:8091 => {'services': ['n1ql'], 'cpu_utilization': 0.2506265664160401, 'mem_free': 3536117760, 'mem_total': 4201672704, 'swap_mem_used': 68681728, 'swap_mem_total': 3758092288} [2020-09-30 03:30:03,291] - [basetestcase] [140493040056064] - INFO - 172.23.123.105:8091 => {'services': ['eventing'], 'cpu_utilization': 2.5, 'mem_free': 3471761408, 'mem_total': 4201672704, 'swap_mem_used': 5767168, 'swap_mem_total': 3758092288} [2020-09-30 03:30:03,291] - [basetestcase] [140493040056064] - INFO - 172.23.123.107:8091 => {'services': ['index'], 'cpu_utilization': 0.2506265664160401, 'mem_free': 3458822144, 'mem_total': 4201672704, 'swap_mem_used': 3932160, 'swap_mem_total': 3758092288} [2020-09-30 03:30:03,291] - [basetestcase] [140493040056064] - INFO - 172.23.123.108:8091 => {'services': ['kv'], 'cpu_utilization': 0.2538071065989848, 'mem_free': 2702671872, 'mem_total': 4201676800, 'swap_mem_used': 78643200, 'swap_mem_total': 3758092288} [2020-09-30 03:30:03,292] - [basetestcase] [140493040056064] - INFO - --- End of cluster statistics --- [2020-09-30 03:30:03,292] - [remote_util] [140493040056064] - INFO - SSH Connecting to 172.23.123.108 with username:root, attempt#1 of 5 [2020-09-30 03:30:03,392] - [remote_util] [140493040056064] - INFO - SSH Connected to 172.23.123.108 as root [2020-09-30 03:30:03,652] - [remote_util] [140493040056064] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-09-30 03:30:03,939] - [remote_util] [140493040056064] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2020-09-30 03:30:09,710] - [basetestcase] [140493040056064] - INFO - list of n1ql nodes in cluster: [ip:172.23.123.104 port:8091 ssh_username:root] [2020-09-30 03:30:09,716] - [basetestcase] [140493040056064] - INFO - list of eventing nodes in cluster: [ip:172.23.123.105 port:8091 ssh_username:root] [2020-09-30 03:30:09,722] - [rest_client] [140493040056064] - INFO - settings/indexes params : storageMode=plasma [2020-09-30 03:30:09,727] - [eventing_base] [140493040056064] - INFO - Setting the min possible memory quota so that adding mode nodes to the cluster wouldn't be a problem. [2020-09-30 03:30:09,727] - [rest_client] [140493040056064] - INFO - pools/default params : memoryQuota=330 [2020-09-30 03:30:09,734] - [rest_client] [140493040056064] - INFO - pools/default params : indexMemoryQuota=256 [2020-09-30 03:30:09,742] - [rest_client] [140493040056064] - INFO - pools/default params : eventingMemoryQuota=512 [2020-09-30 03:30:09,762] - [basetestcase] [140493040056064] - INFO - list of n1ql nodes in cluster: [ip:172.23.123.104 port:8091 ssh_username:root] [2020-09-30 03:30:09,762] - [rest_client] [140493040056064] - INFO - pools/default params : memoryQuota=900 [2020-09-30 03:30:09,768] - [eventing_sanity] [140493040056064] - INFO - 200 [2020-09-30 03:30:09,869] - [rest_client] [140489839859456] - INFO - http://172.23.123.108:8091/pools/default/buckets with param: name=src_bucket&ramQuotaMB=200&authType=none&replicaNumber=0&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=magma [2020-09-30 03:30:09,874] - [rest_client] [140489839859456] - INFO - 0.01 seconds to create bucket src_bucket [2020-09-30 03:30:09,875] - [bucket_helper] [140489839859456] - INFO - waiting for memcached bucket : src_bucket in 172.23.123.108 to accept set ops [2020-09-30 03:30:10,905] - [data_helper] [140489839859456] - INFO - creating direct client 172.23.123.108:11210 src_bucket [2020-09-30 03:30:10,965] - [data_helper] [140489839859456] - INFO - creating direct client 172.23.123.108:11210 src_bucket [2020-09-30 03:30:10,998] - [task] [140489839859456] - WARNING - vbucket map not ready after try 0 [2020-09-30 03:30:10,998] - [bucket_helper] [140489839859456] - INFO - waiting for memcached bucket : src_bucket in 172.23.123.108 to accept set ops [2020-09-30 03:30:11,054] - [data_helper] [140489839859456] - INFO - creating direct client 172.23.123.108:11210 src_bucket [2020-09-30 03:30:11,115] - [data_helper] [140489839859456] - INFO - creating direct client 172.23.123.108:11210 src_bucket [2020-09-30 03:30:11,146] - [task] [140489839859456] - WARNING - vbucket map not ready after try 1 [2020-09-30 03:30:11,146] - [bucket_helper] [140489839859456] - INFO - waiting for memcached bucket : src_bucket in 172.23.123.108 to accept set ops [2020-09-30 03:30:11,199] - [data_helper] [140489839859456] - INFO - creating direct client 172.23.123.108:11210 src_bucket [2020-09-30 03:30:11,255] - [data_helper] [140489839859456] - INFO - creating direct client 172.23.123.108:11210 src_bucket [2020-09-30 03:30:11,287] - [task] [140489839859456] - WARNING - vbucket map not ready after try 2 [2020-09-30 03:30:11,287] - [bucket_helper] [140489839859456] - INFO - waiting for memcached bucket : src_bucket in 172.23.123.108 to accept set ops [2020-09-30 03:30:11,341] - [data_helper] [140489839859456] - INFO - creating direct client 172.23.123.108:11210 src_bucket [2020-09-30 03:30:11,399] - [data_helper] [140489839859456] - INFO - creating direct client 172.23.123.108:11210 src_bucket [2020-09-30 03:30:11,430] - [task] [140489839859456] - WARNING - vbucket map not ready after try 3 [2020-09-30 03:30:11,430] - [bucket_helper] [140489839859456] - INFO - waiting for memcached bucket : src_bucket in 172.23.123.108 to accept set ops [2020-09-30 03:30:11,938] - [data_helper] [140489839859456] - INFO - creating direct client 172.23.123.108:11210 src_bucket [2020-09-30 03:30:11,995] - [data_helper] [140489839859456] - INFO - creating direct client 172.23.123.108:11210 src_bucket [2020-09-30 03:30:12,032] - [task] [140489839859456] - WARNING - vbucket map not ready after try 4 [2020-09-30 03:30:12,032] - [bucket_helper] [140489839859456] - INFO - waiting for memcached bucket : src_bucket in 172.23.123.108 to accept set ops [2020-09-30 03:30:12,087] - [data_helper] [140489839859456] - INFO - creating direct client 172.23.123.108:11210 src_bucket [2020-09-30 03:30:12,142] - [data_helper] [140489839859456] - INFO - creating direct client 172.23.123.108:11210 src_bucket [2020-09-30 03:30:12,173] - [task] [140489839859456] - WARNING - vbucket map not ready after try 5 [2020-09-30 03:30:13,194] - [rest_client] [140489839859456] - INFO - http://172.23.123.108:8091/pools/default/buckets with param: name=dst_bucket&ramQuotaMB=200&authType=none&replicaNumber=0&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=magma [2020-09-30 03:30:13,200] - [rest_client] [140489839859456] - INFO - 0.01 seconds to create bucket dst_bucket [2020-09-30 03:30:13,201] - [bucket_helper] [140489839859456] - INFO - waiting for memcached bucket : dst_bucket in 172.23.123.108 to accept set ops [2020-09-30 03:30:13,797] - [data_helper] [140489839859456] - INFO - creating direct client 172.23.123.108:11210 dst_bucket [2020-09-30 03:30:13,858] - [data_helper] [140489839859456] - INFO - creating direct client 172.23.123.108:11210 dst_bucket [2020-09-30 03:30:13,893] - [task] [140489839859456] - WARNING - vbucket map not ready after try 0 [2020-09-30 03:30:13,893] - [bucket_helper] [140489839859456] - INFO - waiting for memcached bucket : dst_bucket in 172.23.123.108 to accept set ops [2020-09-30 03:30:13,961] - [data_helper] [140489839859456] - INFO - creating direct client 172.23.123.108:11210 dst_bucket [2020-09-30 03:30:14,021] - [data_helper] [140489839859456] - INFO - creating direct client 172.23.123.108:11210 dst_bucket [2020-09-30 03:30:14,059] - [task] [140489839859456] - WARNING - vbucket map not ready after try 1 [2020-09-30 03:30:14,059] - [bucket_helper] [140489839859456] - INFO - waiting for memcached bucket : dst_bucket in 172.23.123.108 to accept set ops [2020-09-30 03:30:14,568] - [data_helper] [140489839859456] - INFO - creating direct client 172.23.123.108:11210 dst_bucket [2020-09-30 03:30:14,628] - [data_helper] [140489839859456] - INFO - creating direct client 172.23.123.108:11210 dst_bucket [2020-09-30 03:30:14,661] - [task] [140489839859456] - WARNING - vbucket map not ready after try 2 [2020-09-30 03:30:14,661] - [bucket_helper] [140489839859456] - INFO - waiting for memcached bucket : dst_bucket in 172.23.123.108 to accept set ops [2020-09-30 03:30:14,718] - [data_helper] [140489839859456] - INFO - creating direct client 172.23.123.108:11210 dst_bucket [2020-09-30 03:30:14,775] - [data_helper] [140489839859456] - INFO - creating direct client 172.23.123.108:11210 dst_bucket [2020-09-30 03:30:14,807] - [task] [140489839859456] - WARNING - vbucket map not ready after try 3 [2020-09-30 03:30:14,807] - [bucket_helper] [140489839859456] - INFO - waiting for memcached bucket : dst_bucket in 172.23.123.108 to accept set ops [2020-09-30 03:30:14,863] - [data_helper] [140489839859456] - INFO - creating direct client 172.23.123.108:11210 dst_bucket [2020-09-30 03:30:14,924] - [data_helper] [140489839859456] - INFO - creating direct client 172.23.123.108:11210 dst_bucket [2020-09-30 03:30:14,965] - [task] [140489839859456] - WARNING - vbucket map not ready after try 4 [2020-09-30 03:30:14,965] - [bucket_helper] [140489839859456] - INFO - waiting for memcached bucket : dst_bucket in 172.23.123.108 to accept set ops [2020-09-30 03:30:15,020] - [data_helper] [140489839859456] - INFO - creating direct client 172.23.123.108:11210 dst_bucket [2020-09-30 03:30:15,077] - [data_helper] [140489839859456] - INFO - creating direct client 172.23.123.108:11210 dst_bucket [2020-09-30 03:30:15,110] - [task] [140489839859456] - WARNING - vbucket map not ready after try 5 [2020-09-30 03:30:16,133] - [rest_client] [140489839859456] - INFO - http://172.23.123.108:8091/pools/default/buckets with param: name=metadata&ramQuotaMB=200&authType=none&replicaNumber=0&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=magma [2020-09-30 03:30:16,139] - [rest_client] [140489839859456] - INFO - 0.01 seconds to create bucket metadata [2020-09-30 03:30:16,139] - [bucket_helper] [140489839859456] - INFO - waiting for memcached bucket : metadata in 172.23.123.108 to accept set ops [2020-09-30 03:30:16,738] - [data_helper] [140489839859456] - INFO - creating direct client 172.23.123.108:11210 metadata [2020-09-30 03:30:17,257] - [data_helper] [140489839859456] - INFO - creating direct client 172.23.123.108:11210 metadata [2020-09-30 03:30:17,295] - [task] [140489839859456] - WARNING - vbucket map not ready after try 0 [2020-09-30 03:30:17,296] - [bucket_helper] [140489839859456] - INFO - waiting for memcached bucket : metadata in 172.23.123.108 to accept set ops [2020-09-30 03:30:17,360] - [data_helper] [140489839859456] - INFO - creating direct client 172.23.123.108:11210 metadata [2020-09-30 03:30:17,423] - [data_helper] [140489839859456] - INFO - creating direct client 172.23.123.108:11210 metadata [2020-09-30 03:30:17,458] - [task] [140489839859456] - WARNING - vbucket map not ready after try 1 [2020-09-30 03:30:17,458] - [bucket_helper] [140489839859456] - INFO - waiting for memcached bucket : metadata in 172.23.123.108 to accept set ops [2020-09-30 03:30:17,519] - [data_helper] [140489839859456] - INFO - creating direct client 172.23.123.108:11210 metadata [2020-09-30 03:30:17,583] - [data_helper] [140489839859456] - INFO - creating direct client 172.23.123.108:11210 metadata [2020-09-30 03:30:17,616] - [task] [140489839859456] - WARNING - vbucket map not ready after try 2 [2020-09-30 03:30:17,617] - [bucket_helper] [140489839859456] - INFO - waiting for memcached bucket : metadata in 172.23.123.108 to accept set ops [2020-09-30 03:30:17,678] - [data_helper] [140489839859456] - INFO - creating direct client 172.23.123.108:11210 metadata [2020-09-30 03:30:17,736] - [data_helper] [140489839859456] - INFO - creating direct client 172.23.123.108:11210 metadata [2020-09-30 03:30:17,771] - [task] [140489839859456] - WARNING - vbucket map not ready after try 3 [2020-09-30 03:30:17,771] - [bucket_helper] [140489839859456] - INFO - waiting for memcached bucket : metadata in 172.23.123.108 to accept set ops [2020-09-30 03:30:17,831] - [data_helper] [140489839859456] - INFO - creating direct client 172.23.123.108:11210 metadata [2020-09-30 03:30:18,357] - [data_helper] [140489839859456] - INFO - creating direct client 172.23.123.108:11210 metadata [2020-09-30 03:30:18,392] - [task] [140489839859456] - WARNING - vbucket map not ready after try 4 [2020-09-30 03:30:18,392] - [bucket_helper] [140489839859456] - INFO - waiting for memcached bucket : metadata in 172.23.123.108 to accept set ops [2020-09-30 03:30:18,452] - [data_helper] [140489839859456] - INFO - creating direct client 172.23.123.108:11210 metadata [2020-09-30 03:30:18,513] - [data_helper] [140489839859456] - INFO - creating direct client 172.23.123.108:11210 metadata [2020-09-30 03:30:18,548] - [task] [140489839859456] - WARNING - vbucket map not ready after try 5 [2020-09-30 03:30:18,619] - [tuq_helper] [140493040056064] - INFO - RUN QUERY create primary index on src_bucket [2020-09-30 03:30:18,623] - [rest_client] [140493040056064] - INFO - query params : statement=create+primary+index+on+src_bucket [2020-09-30 03:30:20,659] - [tuq_helper] [140493040056064] - INFO - TOTAL ELAPSED TIME: 2.034331286s [2020-09-30 03:30:20,664] - [tuq_helper] [140493040056064] - INFO - RUN QUERY create primary index on dst_bucket [2020-09-30 03:30:20,669] - [rest_client] [140493040056064] - INFO - query params : statement=create+primary+index+on+dst_bucket [2020-09-30 03:30:22,548] - [tuq_helper] [140493040056064] - INFO - TOTAL ELAPSED TIME: 1.876292013s [2020-09-30 03:30:22,554] - [tuq_helper] [140493040056064] - INFO - RUN QUERY create primary index on metadata [2020-09-30 03:30:22,558] - [rest_client] [140493040056064] - INFO - query params : statement=create+primary+index+on+metadata [2020-09-30 03:30:24,433] - [tuq_helper] [140493040056064] - INFO - TOTAL ELAPSED TIME: 1.873724246s [2020-09-30 03:30:24,714] - [basetestcase] [140493040056064] - INFO - create 2016.0 to src_bucket documents... [2020-09-30 03:30:24,759] - [data_helper] [140493040056064] - INFO - creating direct client 172.23.123.108:11210 src_bucket [2020-09-30 03:30:26,964] - [basetestcase] [140493040056064] - INFO - LOAD IS FINISHED [2020-09-30 03:30:27,002] - [data_helper] [140493040056064] - INFO - dict:{'ip': '172.23.123.108', 'port': '8091', 'username': 'Administrator', 'password': 'password'} [2020-09-30 03:30:27,002] - [data_helper] [140493040056064] - INFO - creating direct client 172.23.123.108:11210 src_bucket [2020-09-30 03:30:27,040] - [cluster_helper] [140493040056064] - INFO - Setting flush param on server {'ip': '172.23.123.108', 'port': '8091', 'username': 'Administrator', 'password': 'password'}, exp_pager_stime to 3 on src_bucket [2020-09-30 03:30:27,040] - [mc_bin_client] [140493040056064] - INFO - setting param: exp_pager_stime 3 [2020-09-30 03:30:27,040] - [cluster_helper] [140493040056064] - INFO - Setting flush param on server {'ip': '172.23.123.108', 'port': '8091', 'username': 'Administrator', 'password': 'password'}, exp_pager_stime to 3, result: (2958850428, 0, b'') [2020-09-30 03:30:28,289] - [eventing_base] [140493040056064] - INFO - saving function b'{\n "code": 0,\n "info": {\n "status": "Stored function: \'Function_134103615_test_advance_bucket_op\' in metakv",\n "warnings": null\n }\n}' [2020-09-30 03:30:29,477] - [eventing_base] [140493040056064] - INFO - Deploying the following handler code : Function_134103615_test_advance_bucket_op with bindings: {'buckets': [{'alias': 'dst_bucket', 'bucket_name': 'dst_bucket', 'access': 'rw'}, {'alias': 'src_bucket', 'bucket_name': 'src_bucket'}], 'metadata_bucket': 'metadata', 'source_bucket': 'src_bucket', 'curl': [{'hostname': 'http://qa.sc.couchbase.com/', 'value': 'server', 'auth_type': 'no-auth', 'username': None, 'password': None, 'allow_cookies': False, 'validate_ssl_certificate': False}]} and settings: {'checkpoint_interval': 20000, 'cleanup_timers': False, 'dcp_stream_boundary': 'everything', 'deployment_status': False, 'description': 'Sample Description', 'log_level': 'INFO', 'skip_timer_threshold': 86400, 'sock_batch_size': 1, 'tick_duration': 5000, 'timer_processing_tick_interval': 500, 'timer_worker_pool_size': 3, 'worker_count': 3, 'processing_status': False, 'cpp_worker_thread_count': 1, 'execution_timeout': 20, 'data_chan_size': 10000, 'worker_queue_cap': 100000, 'use_memory_manager': True, 'deadline_timeout': 21, 'timer_storage_chan_size': 10000, 'dcp_gen_chan_size': 10000, 'language_compatibility': '6.5.0'} [2020-09-30 03:30:29,477] - [eventing_base] [140493040056064] - INFO - function OnUpdate(doc, meta) { var expiry = new Date(); expiry.setSeconds(expiry.getSeconds() + 300); var context = {docID : meta.id, random_text : "e6cZZGHuh0R7Aumoe6cZZGHuh0R7Aumoe6cZZGHuh0R7Aumoe6cZZGHuh0R7Aumoe6cZZGHuh0R7Aumoe6cZZGHuh0R7Aumoe6cZZGHuh0R7Aumoe6cZZGHuh0R7Aumoe6cZZGHuh0R7Aumoe6cZZGHuh0R7Aumoe6cZZGHuh0R7Aumoe6cZZGHuh0R7Aumoe6cZZGHuh0R7Aumoe6cZZGHuh07Aumoe6cZZGHuh07Aumoe6cZZGHuh07Aumoe6"}; createTimer(timerCallback, expiry, meta.id, context); } function OnDelete(meta) { var expiry = new Date(); expiry.setSeconds(expiry.getSeconds() + 300); var context = {docID : meta.id}; createTimer(NDtimerCallback, expiry, meta.id, context); } function NDtimerCallback(context) { var request = { path : 'job/test_suite_executor/api/json?tree=jobs[component]' }; try { var response = curl("GET", server, request); log('response body received from server:', response.body); log('response headers received from server:', response.headers); log('response status received from server:', response.status); var res= new Uint8Array(response.body); var meta={"id":context.docID}; var result = couchbase.delete(dst_bucket,meta); log(result); } catch (e) { log('error:', e); } } function timerCallback(context) { var request = { path : 'job/test_suite_executor/api/json?tree=jobs[component]' }; try { var response = curl("GET", server, request); log('response body received from server:', response.body); log('response headers received from server:', response.headers); log('response status received from server:', response.status); var res= new Uint8Array(response.body); var meta={"id":context.docID}; if(response.status == 200){ var result= couchbase.insert(dst_bucket,meta,response.body); log(result); } else{ var result= couchbase.insert(dst_bucket,meta,response.status); log(result); } } catch (e) { log('error:', e); } } [2020-09-30 03:30:30,581] - [eventing_base] [140493040056064] - INFO - deploy Application : b'' [2020-09-30 03:30:30,581] - [basetestcase] [140493040056064] - INFO - sleep for 5 secs. Waiting for Function_134103615_test_advance_bucket_op to deployed... ... [2020-09-30 03:30:35,686] - [basetestcase] [140493040056064] - INFO - sleep for 5 secs. Waiting for Function_134103615_test_advance_bucket_op to deployed... ... [2020-09-30 03:30:40,787] - [basetestcase] [140493040056064] - INFO - sleep for 5 secs. Waiting for Function_134103615_test_advance_bucket_op to deployed... ... [2020-09-30 03:30:45,892] - [basetestcase] [140493040056064] - INFO - sleep for 5 secs. Waiting for Function_134103615_test_advance_bucket_op to deployed... ... [2020-09-30 03:30:51,014] - [basetestcase] [140493040056064] - INFO - list of eventing nodes in cluster: [ip:172.23.123.105 port:8091 ssh_username:root] [2020-09-30 03:30:51,024] - [basetestcase] [140493040056064] - INFO - list of eventing nodes in cluster: [ip:172.23.123.105 port:8091 ssh_username:root] [2020-09-30 03:30:51,065] - [basetestcase] [140493040056064] - INFO - sleep for 30 secs. Waiting for handler code Function_134103615_test_advance_bucket_op to complete bucket operations... Current : 0 Expected : 2016 ... [2020-09-30 03:31:21,213] - [basetestcase] [140493040056064] - INFO - sleep for 30 secs. Waiting for handler code Function_134103615_test_advance_bucket_op to complete bucket operations... Current : 0 Expected : 2016 ... [2020-09-30 03:31:51,354] - [basetestcase] [140493040056064] - INFO - sleep for 30 secs. Waiting for handler code Function_134103615_test_advance_bucket_op to complete bucket operations... Current : 0 Expected : 2016 ... [2020-09-30 03:32:21,493] - [basetestcase] [140493040056064] - INFO - sleep for 30 secs. Waiting for handler code Function_134103615_test_advance_bucket_op to complete bucket operations... Current : 0 Expected : 2016 ... [2020-09-30 03:32:51,628] - [basetestcase] [140493040056064] - INFO - sleep for 30 secs. Waiting for handler code Function_134103615_test_advance_bucket_op to complete bucket operations... Current : 0 Expected : 2016 ... [2020-09-30 03:33:21,763] - [basetestcase] [140493040056064] - INFO - sleep for 30 secs. Waiting for handler code Function_134103615_test_advance_bucket_op to complete bucket operations... Current : 0 Expected : 2016 ... [2020-09-30 03:33:51,899] - [basetestcase] [140493040056064] - INFO - sleep for 30 secs. Waiting for handler code Function_134103615_test_advance_bucket_op to complete bucket operations... Current : 0 Expected : 2016 ... [2020-09-30 03:34:22,035] - [basetestcase] [140493040056064] - INFO - sleep for 30 secs. Waiting for handler code Function_134103615_test_advance_bucket_op to complete bucket operations... Current : 0 Expected : 2016 ... [2020-09-30 03:34:52,171] - [basetestcase] [140493040056064] - INFO - sleep for 30 secs. Waiting for handler code Function_134103615_test_advance_bucket_op to complete bucket operations... Current : 0 Expected : 2016 ... [2020-09-30 03:35:22,306] - [basetestcase] [140493040056064] - INFO - sleep for 30 secs. Waiting for handler code Function_134103615_test_advance_bucket_op to complete bucket operations... Current : 0 Expected : 2016 ... [2020-09-30 03:35:52,452] - [basetestcase] [140493040056064] - INFO - sleep for 30 secs. Waiting for handler code Function_134103615_test_advance_bucket_op to complete bucket operations... Current : 0 Expected : 2016 ... [2020-09-30 03:36:22,537] - [eventing_base] [140493040056064] - INFO - Compacting bucket : dst_bucket [2020-09-30 03:36:22,544] - [rest_client] [140493040056064] - INFO - bucket compaction successful [2020-09-30 03:36:22,544] - [eventing_base] [140493040056064] - INFO - Compacting bucket : metadata [2020-09-30 03:36:22,548] - [rest_client] [140493040056064] - INFO - bucket compaction successful [2020-09-30 03:36:22,548] - [eventing_base] [140493040056064] - INFO - Compacting bucket : src_bucket [2020-09-30 03:36:22,552] - [rest_client] [140493040056064] - INFO - bucket compaction successful [2020-09-30 03:36:22,658] - [eventing_base] [140493040056064] - INFO - Documents in source bucket : 0 [2020-09-30 03:36:22,658] - [eventing_base] [140493040056064] - INFO - Final docs count... Current : 2016 Expected : 2016 [2020-09-30 03:36:24,580] - [eventing_base] [140493040056064] - INFO - Stats for Node 172.23.123.105 is [ { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 8, "agg_messages_sent_to_worker": 10192, "agg_queue_memory": 0, "agg_queue_memory_cap": 106954752, "agg_queue_size": 0, "agg_queue_size_cap": 300000, "agg_timer_feedback_queue_cap": 1500, "dcp_expiration": 2016, "dcp_expiry_sent_to_worker": 2016, "dcp_mutation": 2016, "dcp_mutation_sent_to_worker": 2016, "dcp_snapshot": 2436, "dcp_stream_req_counter": 1024, "dcp_streamreq": 1024, "execution_stats": 1023, "failure_stats": 1023, "latency_stats": 1023, "lcb_exception_stats": 1023, "log_level": 3, "num_processed_events": 4032, "processed_events_size": 2005444, "thr_count": 3, "thr_map": 3, "timer_events": 2016, "timer_responses_received": 4032, "v8_init": 3, "v8_load": 3 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 2016, "head": 0, "post": 0, "put": 0 }, "dcp_delete_msg_counter": 2016, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 2016, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 2016, "enqueued_dcp_mutation_msg_counter": 2016, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 10180, "num_processed_events": 4032, "on_delete_failure": 0, "on_delete_success": 2016, "on_update_failure": 0, "on_update_success": 2016, "processed_events_size": 2005444, "timer_callback_failure": 0, "timer_cancel_counter": 0, "timer_create_counter": 4032, "timer_create_failure": 0, "timer_msg_counter": 2016, "timer_responses_sent": 0, "timestamp": { "81521": "2020-09-30T10:36:22Z", "81523": "2020-09-30T10:36:22Z", "81531": "2020-09-30T10:36:22Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_non_200_response": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "81521": "2020-09-30T10:36:22Z", "81523": "2020-09-30T10:36:22Z", "81531": "2020-09-30T10:36:22Z" }, "v8worker_events_lost": 0 }, "function_id": 3727121461, "function_name": "Function_134103615_test_advance_bucket_op", "gocb_creds_request_counter": 9, "internal_vb_distribution_stats": { "worker_Function_134103615_test_advance_bucket_op_0": "[0-341]", "worker_Function_134103615_test_advance_bucket_op_1": "[342-682]", "worker_Function_134103615_test_advance_bucket_op_2": "[683-1023]" }, "latency_percentile_stats": { "100": 113500, "50": 1200, "80": 1900, "90": 2600, "95": 3400, "99": 6700 }, "lcb_creds_request_counter": 18, "lcb_exception_stats": {}, "metastore_stats": { "metastore_delete_err": 0, "metastore_deletes": 0, "metastore_not_found": 0, "metastore_scan": 0, "metastore_scan_due": 0, "metastore_scan_err": 0, "metastore_set": 0, "metastore_set_err": 0 }, "planner_stats": [ { "host_name": "172.23.123.105:8096", "start_vb": 0, "vb_count": 1024 } ], "vb_distribution_stats_from_metadata": { "172.23.123.105:8096": { "worker_Function_134103615_test_advance_bucket_op_0": "[0-341]", "worker_Function_134103615_test_advance_bucket_op_1": "[342-682]", "worker_Function_134103615_test_advance_bucket_op_2": "[683-1023]" } }, "worker_pids": { "worker_Function_134103615_test_advance_bucket_op_0": 81523, "worker_Function_134103615_test_advance_bucket_op_1": 81531, "worker_Function_134103615_test_advance_bucket_op_2": 81521 } } ] [2020-09-30 03:36:24,742] - [basetestcase] [140493040056064] - INFO - list of eventing nodes in cluster: [ip:172.23.123.105 port:8091 ssh_username:root] [2020-09-30 03:36:24,754] - [basetestcase] [140493040056064] - INFO - list of eventing nodes in cluster: [ip:172.23.123.105 port:8091 ssh_username:root] [2020-09-30 03:36:24,805] - [basetestcase] [140493040056064] - INFO - sleep for 30 secs. Waiting for handler code Function_134103615_test_advance_bucket_op to complete bucket operations... Current : 2016 Expected : 0 ... [2020-09-30 03:36:54,940] - [basetestcase] [140493040056064] - INFO - sleep for 30 secs. Waiting for handler code Function_134103615_test_advance_bucket_op to complete bucket operations... Current : 2016 Expected : 0 ... [2020-09-30 03:37:25,075] - [basetestcase] [140493040056064] - INFO - sleep for 30 secs. Waiting for handler code Function_134103615_test_advance_bucket_op to complete bucket operations... Current : 2016 Expected : 0 ... [2020-09-30 03:37:55,206] - [basetestcase] [140493040056064] - INFO - sleep for 30 secs. Waiting for handler code Function_134103615_test_advance_bucket_op to complete bucket operations... Current : 2016 Expected : 0 ... [2020-09-30 03:38:25,339] - [basetestcase] [140493040056064] - INFO - sleep for 30 secs. Waiting for handler code Function_134103615_test_advance_bucket_op to complete bucket operations... Current : 2016 Expected : 0 ... [2020-09-30 03:38:55,471] - [basetestcase] [140493040056064] - INFO - sleep for 30 secs. Waiting for handler code Function_134103615_test_advance_bucket_op to complete bucket operations... Current : 2016 Expected : 0 ... [2020-09-30 03:39:25,609] - [basetestcase] [140493040056064] - INFO - sleep for 30 secs. Waiting for handler code Function_134103615_test_advance_bucket_op to complete bucket operations... Current : 2016 Expected : 0 ... [2020-09-30 03:39:55,741] - [basetestcase] [140493040056064] - INFO - sleep for 30 secs. Waiting for handler code Function_134103615_test_advance_bucket_op to complete bucket operations... Current : 2016 Expected : 0 ... [2020-09-30 03:40:25,878] - [basetestcase] [140493040056064] - INFO - sleep for 30 secs. Waiting for handler code Function_134103615_test_advance_bucket_op to complete bucket operations... Current : 2016 Expected : 0 ... [2020-09-30 03:40:56,078] - [eventing_base] [140493040056064] - INFO - Documents in source bucket : 0 [2020-09-30 03:40:56,078] - [eventing_base] [140493040056064] - INFO - Final docs count... Current : 0 Expected : 0 [2020-09-30 03:40:57,077] - [eventing_base] [140493040056064] - INFO - Stats for Node 172.23.123.105 is [ { "dcp_feed_boundary": "everything", "event_processing_stats": { "adhoc_timer_response_received": 8, "agg_messages_sent_to_worker": 14287, "agg_queue_memory": 0, "agg_queue_memory_cap": 106954752, "agg_queue_size": 0, "agg_queue_size_cap": 300000, "agg_timer_feedback_queue_cap": 1500, "dcp_expiration": 2016, "dcp_expiry_sent_to_worker": 2016, "dcp_mutation": 2016, "dcp_mutation_sent_to_worker": 2016, "dcp_snapshot": 2436, "dcp_stream_req_counter": 1024, "dcp_streamreq": 1024, "execution_stats": 1842, "failure_stats": 1842, "latency_stats": 1842, "lcb_exception_stats": 1842, "log_level": 3, "num_processed_events": 4032, "processed_events_size": 2005444, "thr_count": 3, "thr_map": 3, "timer_events": 4032, "timer_responses_received": 4032, "v8_init": 3, "v8_load": 3 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "curl": { "delete": 0, "get": 4032, "head": 0, "post": 0, "put": 0 }, "dcp_delete_msg_counter": 2016, "dcp_delete_parse_failure": 0, "dcp_mutation_msg_counter": 2016, "dcp_mutation_parse_failure": 0, "enqueued_dcp_delete_msg_counter": 2016, "enqueued_dcp_mutation_msg_counter": 2016, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "filtered_dcp_delete_counter": 0, "filtered_dcp_mutation_counter": 0, "lcb_retry_failure": 0, "messages_parsed": 14275, "num_processed_events": 4032, "on_delete_failure": 0, "on_delete_success": 2016, "on_update_failure": 0, "on_update_success": 2016, "processed_events_size": 2005444, "timer_callback_failure": 0, "timer_cancel_counter": 0, "timer_create_counter": 4032, "timer_create_failure": 0, "timer_msg_counter": 4032, "timer_responses_sent": 0, "timestamp": { "81521": "2020-09-30T10:40:55Z", "81523": "2020-09-30T10:40:55Z", "81531": "2020-09-30T10:40:55Z" }, "uv_msg_parse_failure": 0, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bkt_ops_cas_mismatch_count": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "curl_non_200_response": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_callback_missing_counter": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "81521": "2020-09-30T10:40:55Z", "81523": "2020-09-30T10:40:55Z", "81531": "2020-09-30T10:40:55Z" }, "v8worker_events_lost": 0 }, "function_id": 3727121461, "function_name": "Function_134103615_test_advance_bucket_op", "gocb_creds_request_counter": 9, "internal_vb_distribution_stats": { "worker_Function_134103615_test_advance_bucket_op_0": "[0-341]", "worker_Function_134103615_test_advance_bucket_op_1": "[342-682]", "worker_Function_134103615_test_advance_bucket_op_2": "[683-1023]" }, "latency_percentile_stats": { "100": 113500, "50": 1200, "80": 1900, "90": 2600, "95": 3400, "99": 6700 }, "lcb_creds_request_counter": 18, "lcb_exception_stats": {}, "metastore_stats": { "metastore_delete_err": 0, "metastore_deletes": 0, "metastore_not_found": 0, "metastore_scan": 0, "metastore_scan_due": 0, "metastore_scan_err": 0, "metastore_set": 0, "metastore_set_err": 0 }, "planner_stats": [ { "host_name": "172.23.123.105:8096", "start_vb": 0, "vb_count": 1024 } ], "vb_distribution_stats_from_metadata": { "172.23.123.105:8096": { "worker_Function_134103615_test_advance_bucket_op_0": "[0-341]", "worker_Function_134103615_test_advance_bucket_op_1": "[342-682]", "worker_Function_134103615_test_advance_bucket_op_2": "[683-1023]" } }, "worker_pids": { "worker_Function_134103615_test_advance_bucket_op_0": 81523, "worker_Function_134103615_test_advance_bucket_op_1": 81531, "worker_Function_134103615_test_advance_bucket_op_2": 81521 } } ] [2020-09-30 03:40:57,229] - [basetestcase] [140493040056064] - INFO - list of eventing nodes in cluster: [ip:172.23.123.105 port:8091 ssh_username:root] [2020-09-30 03:40:57,314] - [eventing_base] [140493040056064] - INFO - Undeploy Application : Function_134103615_test_advance_bucket_op [2020-09-30 03:40:57,314] - [basetestcase] [140493040056064] - INFO - sleep for 5 secs. Waiting for Function_134103615_test_advance_bucket_op to undeployed... ... [2020-09-30 03:41:02,475] - [basetestcase] [140493040056064] - INFO - sleep for 5 secs. Waiting for Function_134103615_test_advance_bucket_op to undeployed... ... [2020-09-30 03:41:07,586] - [basetestcase] [140493040056064] - INFO - sleep for 5 secs. Waiting for Function_134103615_test_advance_bucket_op to undeployed... ... [2020-09-30 03:41:12,694] - [basetestcase] [140493040056064] - INFO - sleep for 5 secs. Waiting for Function_134103615_test_advance_bucket_op to undeployed... ... [2020-09-30 03:41:17,798] - [basetestcase] [140493040056064] - INFO - sleep for 5 secs. Waiting for Function_134103615_test_advance_bucket_op to undeployed... ... [2020-09-30 03:41:22,904] - [basetestcase] [140493040056064] - INFO - sleep for 5 secs. Waiting for Function_134103615_test_advance_bucket_op to undeployed... ... [2020-09-30 03:41:28,011] - [basetestcase] [140493040056064] - INFO - sleep for 5 secs. Waiting for Function_134103615_test_advance_bucket_op to undeployed... ... [2020-09-30 03:41:33,139] - [basetestcase] [140493040056064] - INFO - sleep for 5 secs. Waiting for Function_134103615_test_advance_bucket_op to undeployed... ... [2020-09-30 03:41:38,251] - [basetestcase] [140493040056064] - INFO - sleep for 5 secs. Waiting for Function_134103615_test_advance_bucket_op to undeployed... ... [2020-09-30 03:41:43,365] - [basetestcase] [140493040056064] - INFO - sleep for 5 secs. Waiting for Function_134103615_test_advance_bucket_op to undeployed... ... [2020-09-30 03:41:48,487] - [basetestcase] [140493040056064] - INFO - sleep for 5 secs. Waiting for Function_134103615_test_advance_bucket_op to undeployed... ... [2020-09-30 03:41:53,603] - [basetestcase] [140493040056064] - INFO - sleep for 5 secs. Waiting for Function_134103615_test_advance_bucket_op to undeployed... ... [2020-09-30 03:41:58,714] - [basetestcase] [140493040056064] - INFO - sleep for 5 secs. Waiting for Function_134103615_test_advance_bucket_op to undeployed... ... [2020-09-30 03:42:03,829] - [basetestcase] [140493040056064] - INFO - sleep for 5 secs. Waiting for Function_134103615_test_advance_bucket_op to undeployed... ... [2020-09-30 03:42:08,942] - [basetestcase] [140493040056064] - INFO - sleep for 5 secs. Waiting for Function_134103615_test_advance_bucket_op to undeployed... ... [2020-09-30 03:42:14,053] - [basetestcase] [140493040056064] - INFO - sleep for 5 secs. Waiting for Function_134103615_test_advance_bucket_op to undeployed... ... [2020-09-30 03:42:19,180] - [basetestcase] [140493040056064] - INFO - sleep for 5 secs. Waiting for Function_134103615_test_advance_bucket_op to undeployed... ... [2020-09-30 03:42:24,293] - [basetestcase] [140493040056064] - INFO - sleep for 5 secs. Waiting for Function_134103615_test_advance_bucket_op to undeployed... ... [2020-09-30 03:42:29,404] - [basetestcase] [140493040056064] - INFO - sleep for 5 secs. Waiting for Function_134103615_test_advance_bucket_op to undeployed... ... [2020-09-30 03:42:34,524] - [basetestcase] [140493040056064] - INFO - sleep for 5 secs. Waiting for Function_134103615_test_advance_bucket_op to undeployed... ... [2020-09-30 03:42:39,636] - [basetestcase] [140493040056064] - INFO - sleep for 5 secs. Waiting for Function_134103615_test_advance_bucket_op to undeployed... ... [2020-09-30 03:42:44,746] - [basetestcase] [140493040056064] - INFO - sleep for 5 secs. Waiting for Function_134103615_test_advance_bucket_op to undeployed... ... [2020-09-30 03:42:49,853] - [basetestcase] [140493040056064] - INFO - sleep for 5 secs. Waiting for Function_134103615_test_advance_bucket_op to undeployed... ... [2020-09-30 03:42:54,965] - [basetestcase] [140493040056064] - INFO - sleep for 5 secs. Waiting for Function_134103615_test_advance_bucket_op to undeployed... ... [2020-09-30 03:43:00,074] - [basetestcase] [140493040056064] - INFO - sleep for 5 secs. Waiting for Function_134103615_test_advance_bucket_op to undeployed... ... [2020-09-30 03:43:05,196] - [basetestcase] [140493040056064] - INFO - sleep for 5 secs. Waiting for Function_134103615_test_advance_bucket_op to undeployed... ... [2020-09-30 03:43:10,317] - [basetestcase] [140493040056064] - INFO - sleep for 5 secs. Waiting for Function_134103615_test_advance_bucket_op to undeployed... ... [2020-09-30 03:43:15,422] - [basetestcase] [140493040056064] - INFO - sleep for 5 secs. Waiting for Function_134103615_test_advance_bucket_op to undeployed... ... [2020-09-30 03:43:20,534] - [basetestcase] [140493040056064] - INFO - sleep for 5 secs. Waiting for Function_134103615_test_advance_bucket_op to undeployed... ... [2020-09-30 03:43:25,654] - [basetestcase] [140493040056064] - INFO - sleep for 5 secs. Waiting for Function_134103615_test_advance_bucket_op to undeployed... ... [2020-09-30 03:43:30,768] - [basetestcase] [140493040056064] - INFO - sleep for 5 secs. Waiting for Function_134103615_test_advance_bucket_op to undeployed... ... [2020-09-30 03:43:35,880] - [basetestcase] [140493040056064] - INFO - sleep for 5 secs. Waiting for Function_134103615_test_advance_bucket_op to undeployed... ... [2020-09-30 03:43:40,990] - [basetestcase] [140493040056064] - INFO - sleep for 5 secs. Waiting for Function_134103615_test_advance_bucket_op to undeployed... ... [2020-09-30 03:43:46,104] - [basetestcase] [140493040056064] - INFO - sleep for 5 secs. Waiting for Function_134103615_test_advance_bucket_op to undeployed... ... [2020-09-30 03:43:51,227] - [basetestcase] [140493040056064] - INFO - sleep for 5 secs. Waiting for Function_134103615_test_advance_bucket_op to undeployed... ... [2020-09-30 03:43:56,350] - [basetestcase] [140493040056064] - INFO - sleep for 5 secs. Waiting for Function_134103615_test_advance_bucket_op to undeployed... ... [2020-09-30 03:44:01,457] - [basetestcase] [140493040056064] - INFO - sleep for 5 secs. Waiting for Function_134103615_test_advance_bucket_op to undeployed... ... [2020-09-30 03:44:06,568] - [basetestcase] [140493040056064] - INFO - sleep for 5 secs. Waiting for Function_134103615_test_advance_bucket_op to undeployed... ... [2020-09-30 03:44:11,679] - [basetestcase] [140493040056064] - INFO - sleep for 5 secs. Waiting for Function_134103615_test_advance_bucket_op to undeployed... ... [2020-09-30 03:44:16,800] - [basetestcase] [140493040056064] - INFO - sleep for 5 secs. Waiting for Function_134103615_test_advance_bucket_op to undeployed... ... [2020-09-30 03:44:21,903] - [basetestcase] [140493040056064] - INFO - sleep for 5 secs. Waiting for Function_134103615_test_advance_bucket_op to undeployed... ... [2020-09-30 03:44:27,010] - [basetestcase] [140493040056064] - INFO - sleep for 5 secs. Waiting for Function_134103615_test_advance_bucket_op to undeployed... ... [2020-09-30 03:44:32,119] - [basetestcase] [140493040056064] - INFO - sleep for 5 secs. Waiting for Function_134103615_test_advance_bucket_op to undeployed... ... [2020-09-30 03:44:37,233] - [basetestcase] [140493040056064] - INFO - sleep for 5 secs. Waiting for Function_134103615_test_advance_bucket_op to undeployed... ... [2020-09-30 03:44:42,360] - [basetestcase] [140493040056064] - INFO - sleep for 5 secs. Waiting for Function_134103615_test_advance_bucket_op to undeployed... ... [2020-09-30 03:44:47,461] - [basetestcase] [140493040056064] - INFO - sleep for 5 secs. Waiting for Function_134103615_test_advance_bucket_op to undeployed... ... [2020-09-30 03:44:52,566] - [basetestcase] [140493040056064] - INFO - sleep for 5 secs. Waiting for Function_134103615_test_advance_bucket_op to undeployed... ... [2020-09-30 03:44:57,677] - [basetestcase] [140493040056064] - INFO - sleep for 5 secs. Waiting for Function_134103615_test_advance_bucket_op to undeployed... ... [2020-09-30 03:45:02,782] - [basetestcase] [140493040056064] - INFO - sleep for 5 secs. Waiting for Function_134103615_test_advance_bucket_op to undeployed... ... [2020-09-30 03:45:07,887] - [basetestcase] [140493040056064] - INFO - sleep for 5 secs. Waiting for Function_134103615_test_advance_bucket_op to undeployed... ... [2020-09-30 03:45:12,994] - [basetestcase] [140493040056064] - INFO - sleep for 5 secs. Waiting for Function_134103615_test_advance_bucket_op to undeployed... ... [2020-09-30 03:45:18,100] - [basetestcase] [140493040056064] - INFO - sleep for 5 secs. Waiting for Function_134103615_test_advance_bucket_op to undeployed... ... [2020-09-30 03:45:23,207] - [basetestcase] [140493040056064] - INFO - sleep for 5 secs. Waiting for Function_134103615_test_advance_bucket_op to undeployed... ... [2020-09-30 03:45:28,331] - [basetestcase] [140493040056064] - INFO - sleep for 5 secs. Waiting for Function_134103615_test_advance_bucket_op to undeployed... ... [2020-09-30 03:45:33,442] - [basetestcase] [140493040056064] - INFO - sleep for 5 secs. Waiting for Function_134103615_test_advance_bucket_op to undeployed... ... [2020-09-30 03:45:38,559] - [basetestcase] [140493040056064] - INFO - sleep for 5 secs. Waiting for Function_134103615_test_advance_bucket_op to undeployed... ... [2020-09-30 03:45:43,663] - [basetestcase] [140493040056064] - INFO - sleep for 5 secs. Waiting for Function_134103615_test_advance_bucket_op to undeployed... ... [2020-09-30 03:45:48,770] - [basetestcase] [140493040056064] - INFO - sleep for 5 secs. Waiting for Function_134103615_test_advance_bucket_op to undeployed... ... [2020-09-30 03:45:53,880] - [basetestcase] [140493040056064] - INFO - sleep for 5 secs. Waiting for Function_134103615_test_advance_bucket_op to undeployed... ... [2020-09-30 03:45:58,978] - [basetestcase] [140493040056064] - INFO - sleep for 5 secs. Waiting for Function_134103615_test_advance_bucket_op to undeployed... ... [2020-09-30 03:46:04,081] - [basetestcase] [140493040056064] - INFO - sleep for 5 secs. Waiting for Function_134103615_test_advance_bucket_op to undeployed... ... [2020-09-30 03:46:09,186] - [basetestcase] [140493040056064] - INFO - sleep for 5 secs. Waiting for Function_134103615_test_advance_bucket_op to undeployed... ... [2020-09-30 03:46:14,292] - [basetestcase] [140493040056064] - INFO - sleep for 5 secs. Waiting for Function_134103615_test_advance_bucket_op to undeployed... ... [2020-09-30 03:46:19,418] - [basetestcase] [140493040056064] - INFO - sleep for 5 secs. Waiting for Function_134103615_test_advance_bucket_op to undeployed... ... [2020-09-30 03:46:24,518] - [basetestcase] [140493040056064] - INFO - sleep for 5 secs. Waiting for Function_134103615_test_advance_bucket_op to undeployed... ... [2020-09-30 03:46:29,623] - [basetestcase] [140493040056064] - INFO - sleep for 5 secs. Waiting for Function_134103615_test_advance_bucket_op to undeployed... ... [2020-09-30 03:46:34,732] - [basetestcase] [140493040056064] - INFO - sleep for 5 secs. Waiting for Function_134103615_test_advance_bucket_op to undeployed... ... [2020-09-30 03:46:39,839] - [basetestcase] [140493040056064] - INFO - sleep for 5 secs. Waiting for Function_134103615_test_advance_bucket_op to undeployed... ... [2020-09-30 03:46:44,941] - [basetestcase] [140493040056064] - INFO - sleep for 5 secs. Waiting for Function_134103615_test_advance_bucket_op to undeployed... ... [2020-09-30 03:46:50,050] - [basetestcase] [140493040056064] - INFO - sleep for 5 secs. Waiting for Function_134103615_test_advance_bucket_op to undeployed... ... [2020-09-30 03:46:55,155] - [basetestcase] [140493040056064] - INFO - sleep for 5 secs. Waiting for Function_134103615_test_advance_bucket_op to undeployed... ... [2020-09-30 03:47:00,275] - [basetestcase] [140493040056064] - INFO - sleep for 5 secs. Waiting for Function_134103615_test_advance_bucket_op to undeployed... ... [2020-09-30 03:47:05,385] - [basetestcase] [140493040056064] - INFO - sleep for 5 secs. Waiting for Function_134103615_test_advance_bucket_op to undeployed... ... [2020-09-30 03:47:10,508] - [basetestcase] [140493040056064] - INFO - sleep for 5 secs. Waiting for Function_134103615_test_advance_bucket_op to undeployed... ... [2020-09-30 03:47:15,617] - [basetestcase] [140493040056064] - INFO - sleep for 5 secs. Waiting for Function_134103615_test_advance_bucket_op to undeployed... ... [2020-09-30 03:47:20,728] - [basetestcase] [140493040056064] - INFO - sleep for 5 secs. Waiting for Function_134103615_test_advance_bucket_op to undeployed... ... [2020-09-30 03:47:25,838] - [basetestcase] [140493040056064] - INFO - sleep for 5 secs. Waiting for Function_134103615_test_advance_bucket_op to undeployed... ... [2020-09-30 03:47:30,948] - [basetestcase] [140493040056064] - INFO - sleep for 5 secs. Waiting for Function_134103615_test_advance_bucket_op to undeployed... ... [2020-09-30 03:47:36,075] - [basetestcase] [140493040056064] - INFO - sleep for 5 secs. Waiting for Function_134103615_test_advance_bucket_op to undeployed... ... [2020-09-30 03:47:41,183] - [basetestcase] [140493040056064] - INFO - sleep for 5 secs. Waiting for Function_134103615_test_advance_bucket_op to undeployed... ... [2020-09-30 03:47:46,286] - [basetestcase] [140493040056064] - INFO - sleep for 5 secs. Waiting for Function_134103615_test_advance_bucket_op to undeployed... ... [2020-09-30 03:47:51,387] - [basetestcase] [140493040056064] - INFO - sleep for 5 secs. Waiting for Function_134103615_test_advance_bucket_op to undeployed... ... [2020-09-30 03:47:56,513] - [basetestcase] [140493040056064] - INFO - sleep for 5 secs. Waiting for Function_134103615_test_advance_bucket_op to undeployed... ... [2020-09-30 03:48:01,624] - [basetestcase] [140493040056064] - INFO - sleep for 5 secs. Waiting for Function_134103615_test_advance_bucket_op to undeployed... ... [2020-09-30 03:48:06,733] - [basetestcase] [140493040056064] - INFO - sleep for 5 secs. Waiting for Function_134103615_test_advance_bucket_op to undeployed... ... [2020-09-30 03:48:11,835] - [basetestcase] [140493040056064] - INFO - sleep for 5 secs. Waiting for Function_134103615_test_advance_bucket_op to undeployed... ... [2020-09-30 03:48:16,942] - [basetestcase] [140493040056064] - INFO - sleep for 5 secs. Waiting for Function_134103615_test_advance_bucket_op to undeployed... ... [2020-09-30 03:48:22,044] - [basetestcase] [140493040056064] - INFO - sleep for 5 secs. Waiting for Function_134103615_test_advance_bucket_op to undeployed... ... [2020-09-30 03:48:27,148] - [basetestcase] [140493040056064] - INFO - sleep for 5 secs. Waiting for Function_134103615_test_advance_bucket_op to undeployed... ... [2020-09-30 03:48:32,257] - [basetestcase] [140493040056064] - INFO - sleep for 5 secs. Waiting for Function_134103615_test_advance_bucket_op to undeployed... ... [2020-09-30 03:48:37,361] - [basetestcase] [140493040056064] - INFO - sleep for 5 secs. Waiting for Function_134103615_test_advance_bucket_op to undeployed... ... [2020-09-30 03:48:42,467] - [basetestcase] [140493040056064] - INFO - sleep for 5 secs. Waiting for Function_134103615_test_advance_bucket_op to undeployed... ... [2020-09-30 03:48:47,600] - [basetestcase] [140493040056064] - INFO - sleep for 5 secs. Waiting for Function_134103615_test_advance_bucket_op to undeployed... ... [2020-09-30 03:48:52,699] - [basetestcase] [140493040056064] - INFO - sleep for 5 secs. Waiting for Function_134103615_test_advance_bucket_op to undeployed... ... [2020-09-30 03:48:57,805] - [basetestcase] [140493040056064] - INFO - sleep for 5 secs. Waiting for Function_134103615_test_advance_bucket_op to undeployed... ... [2020-09-30 03:49:02,910] - [basetestcase] [140493040056064] - INFO - sleep for 5 secs. Waiting for Function_134103615_test_advance_bucket_op to undeployed... ... [2020-09-30 03:49:08,022] - [basetestcase] [140493040056064] - INFO - sleep for 5 secs. Waiting for Function_134103615_test_advance_bucket_op to undeployed... ... [2020-09-30 03:49:13,136] - [basetestcase] [140493040056064] - INFO - sleep for 5 secs. Waiting for Function_134103615_test_advance_bucket_op to undeployed... ... [2020-09-30 03:49:18,239] - [basetestcase] [140493040056064] - INFO - sleep for 5 secs. Waiting for Function_134103615_test_advance_bucket_op to undeployed... ... [2020-09-30 03:49:23,348] - [basetestcase] [140493040056064] - INFO - sleep for 5 secs. Waiting for Function_134103615_test_advance_bucket_op to undeployed... ... [2020-09-30 03:49:28,448] - [basetestcase] [140493040056064] - INFO - sleep for 5 secs. Waiting for Function_134103615_test_advance_bucket_op to undeployed... ... [2020-09-30 03:49:33,557] - [basetestcase] [140493040056064] - INFO - sleep for 5 secs. Waiting for Function_134103615_test_advance_bucket_op to undeployed... ... [2020-09-30 03:49:38,682] - [basetestcase] [140493040056064] - INFO - sleep for 5 secs. Waiting for Function_134103615_test_advance_bucket_op to undeployed... ... [2020-09-30 03:49:43,780] - [basetestcase] [140493040056064] - INFO - sleep for 5 secs. Waiting for Function_134103615_test_advance_bucket_op to undeployed... ... [2020-09-30 03:49:48,899] - [basetestcase] [140493040056064] - INFO - sleep for 5 secs. Waiting for Function_134103615_test_advance_bucket_op to undeployed... ... [2020-09-30 03:49:54,006] - [basetestcase] [140493040056064] - INFO - sleep for 5 secs. Waiting for Function_134103615_test_advance_bucket_op to undeployed... ... [2020-09-30 03:49:59,105] - [basetestcase] [140493040056064] - INFO - sleep for 5 secs. Waiting for Function_134103615_test_advance_bucket_op to undeployed... ... [2020-09-30 03:50:04,229] - [basetestcase] [140493040056064] - INFO - sleep for 5 secs. Waiting for Function_134103615_test_advance_bucket_op to undeployed... ... [2020-09-30 03:50:09,333] - [basetestcase] [140493040056064] - INFO - sleep for 5 secs. Waiting for Function_134103615_test_advance_bucket_op to undeployed... ... [2020-09-30 03:50:14,440] - [basetestcase] [140493040056064] - INFO - sleep for 5 secs. Waiting for Function_134103615_test_advance_bucket_op to undeployed... ... [2020-09-30 03:50:19,545] - [basetestcase] [140493040056064] - INFO - sleep for 5 secs. Waiting for Function_134103615_test_advance_bucket_op to undeployed... ... [2020-09-30 03:50:24,673] - [basetestcase] [140493040056064] - INFO - sleep for 5 secs. Waiting for Function_134103615_test_advance_bucket_op to undeployed... ... [2020-09-30 03:50:29,778] - [basetestcase] [140493040056064] - INFO - sleep for 5 secs. Waiting for Function_134103615_test_advance_bucket_op to undeployed... ... [2020-09-30 03:50:34,886] - [basetestcase] [140493040056064] - INFO - sleep for 5 secs. Waiting for Function_134103615_test_advance_bucket_op to undeployed... ... [2020-09-30 03:50:39,986] - [basetestcase] [140493040056064] - INFO - sleep for 5 secs. Waiting for Function_134103615_test_advance_bucket_op to undeployed... ... [2020-09-30 03:50:45,090] - [basetestcase] [140493040056064] - INFO - sleep for 5 secs. Waiting for Function_134103615_test_advance_bucket_op to undeployed... ... [2020-09-30 03:50:50,195] - [basetestcase] [140493040056064] - INFO - sleep for 5 secs. Waiting for Function_134103615_test_advance_bucket_op to undeployed... ... [2020-09-30 03:50:55,299] - [basetestcase] [140493040056064] - INFO - sleep for 5 secs. Waiting for Function_134103615_test_advance_bucket_op to undeployed... ... [2020-09-30 03:51:00,400] - [basetestcase] [140493040056064] - INFO - sleep for 5 secs. Waiting for Function_134103615_test_advance_bucket_op to undeployed... ... [2020-09-30 03:51:05,509] - [basetestcase] [140493040056064] - INFO - sleep for 5 secs. Waiting for Function_134103615_test_advance_bucket_op to undeployed... ... [2020-09-30 03:51:10,614] - [basetestcase] [140493040056064] - INFO - sleep for 5 secs. Waiting for Function_134103615_test_advance_bucket_op to undeployed... ... [2020-09-30 03:51:15,759] - [basetestcase] [140493040056064] - INFO - list of eventing nodes in cluster: [ip:172.23.123.105 port:8091 ssh_username:root] [2020-09-30 03:51:15,761] - [remote_util] [140493040056064] - INFO - SSH Connecting to 172.23.123.105 with username:root, attempt#1 of 5 [2020-09-30 03:51:15,862] - [remote_util] [140493040056064] - INFO - SSH Connected to 172.23.123.105 as root [2020-09-30 03:51:16,135] - [remote_util] [140493040056064] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-09-30 03:51:16,429] - [remote_util] [140493040056064] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2020-09-30 03:51:16,437] - [rest_client] [140493040056064] - INFO - /diag/eval status on 172.23.123.105:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). [2020-09-30 03:51:16,437] - [remote_util] [140493040056064] - INFO - running command.raw on 172.23.123.105: zgrep "panic" "/opt/couchbase/var/lib/couchbase/logs"/eventing.log* | wc -l [2020-09-30 03:51:16,890] - [remote_util] [140493040056064] - INFO - command executed successfully with root [2020-09-30 03:51:16,891] - [remote_util] [140493040056064] - INFO - running command.raw on 172.23.123.105: ls "/opt/couchbase/var/lib/couchbase/logs"/../crash/| wc -l [2020-09-30 03:51:16,905] - [remote_util] [140493040056064] - INFO - command executed successfully with root [2020-09-30 03:51:17,586] - [eventing_base] [140493040056064] - INFO - Bucket dst_bucket DGM is 100 [2020-09-30 03:51:17,638] - [eventing_base] [140493040056064] - INFO - Bucket metadata DGM is 0 [2020-09-30 03:51:17,692] - [eventing_base] [140493040056064] - INFO - Bucket src_bucket DGM is 100 [2020-09-30 03:51:17,747] - [eventing_base] [140493040056064] - INFO - number of documents in metadata bucket 2033 [2020-09-30 03:51:17,752] - [remote_util] [140493330810688] - INFO - SSH Connecting to 172.23.123.108 with username:root, attempt#1 of 5 [2020-09-30 03:51:17,852] - [remote_util] [140493330810688] - INFO - SSH Connected to 172.23.123.108 as root [2020-09-30 03:51:18,134] - [remote_util] [140493330810688] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-09-30 03:51:18,432] - [remote_util] [140493330810688] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2020-09-30 03:51:18,432] - [remote_util] [140493330810688] - INFO - running command.raw on 172.23.123.108: /opt/couchbase/bin/cbcollect_info 172.23.123.108-20200930-0351-diag.zip [2020-09-30 03:54:11,697] - [remote_util] [140493330810688] - INFO - command executed successfully with root [2020-09-30 03:54:12,714] - [remote_util] [140493330810688] - INFO - running command.raw on 172.23.123.108: rm -f /root/172.23.123.108-20200930-0351-diag.zip [2020-09-30 03:54:12,741] - [remote_util] [140493330810688] - INFO - command executed successfully with root [2020-09-30 03:54:12,742] - [remote_util] [140493330810688] - INFO - SSH Connecting to 172.23.123.105 with username:root, attempt#1 of 5 [2020-09-30 03:54:12,843] - [remote_util] [140493330810688] - INFO - SSH Connected to 172.23.123.105 as root [2020-09-30 03:54:13,118] - [remote_util] [140493330810688] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-09-30 03:54:13,414] - [remote_util] [140493330810688] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2020-09-30 03:54:13,414] - [remote_util] [140493330810688] - INFO - running command.raw on 172.23.123.105: /opt/couchbase/bin/cbcollect_info 172.23.123.105-20200930-0354-diag.zip [2020-09-30 03:55:47,514] - [remote_util] [140493330810688] - INFO - command executed successfully with root [2020-09-30 03:55:48,081] - [remote_util] [140493330810688] - INFO - running command.raw on 172.23.123.105: rm -f /root/172.23.123.105-20200930-0354-diag.zip [2020-09-30 03:55:48,101] - [remote_util] [140493330810688] - INFO - command executed successfully with root [2020-09-30 03:55:48,102] - [remote_util] [140493330810688] - INFO - SSH Connecting to 172.23.123.107 with username:root, attempt#1 of 5 [2020-09-30 03:55:48,205] - [remote_util] [140493330810688] - INFO - SSH Connected to 172.23.123.107 as root [2020-09-30 03:55:48,486] - [remote_util] [140493330810688] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-09-30 03:55:48,775] - [remote_util] [140493330810688] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2020-09-30 03:55:48,775] - [remote_util] [140493330810688] - INFO - running command.raw on 172.23.123.107: /opt/couchbase/bin/cbcollect_info 172.23.123.107-20200930-0355-diag.zip [2020-09-30 03:57:55,053] - [remote_util] [140493330810688] - INFO - command executed successfully with root [2020-09-30 03:57:55,643] - [remote_util] [140493330810688] - INFO - running command.raw on 172.23.123.107: rm -f /root/172.23.123.107-20200930-0355-diag.zip [2020-09-30 03:57:55,663] - [remote_util] [140493330810688] - INFO - command executed successfully with root [2020-09-30 03:57:55,664] - [remote_util] [140493330810688] - INFO - SSH Connecting to 172.23.123.104 with username:root, attempt#1 of 5 [2020-09-30 03:57:55,765] - [remote_util] [140493330810688] - INFO - SSH Connected to 172.23.123.104 as root [2020-09-30 03:57:56,027] - [remote_util] [140493330810688] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-09-30 03:57:56,318] - [remote_util] [140493330810688] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2020-09-30 03:57:56,319] - [remote_util] [140493330810688] - INFO - running command.raw on 172.23.123.104: /opt/couchbase/bin/cbcollect_info 172.23.123.104-20200930-0357-diag.zip [2020-09-30 03:59:32,753] - [remote_util] [140493330810688] - INFO - command executed successfully with root [2020-09-30 03:59:33,238] - [remote_util] [140493330810688] - INFO - running command.raw on 172.23.123.104: rm -f /root/172.23.123.104-20200930-0357-diag.zip [2020-09-30 03:59:33,257] - [remote_util] [140493330810688] - INFO - command executed successfully with root [2020-09-30 03:59:33,673] - [eventing_base] [140493330810688] - INFO - Starting Test: suite_tearDown [2020-09-30 03:59:33,675] - [remote_util] [140493330810688] - INFO - SSH Connecting to 172.23.123.108 with username:root, attempt#1 of 5 [2020-09-30 03:59:33,782] - [remote_util] [140493330810688] - INFO - SSH Connected to 172.23.123.108 as root [2020-09-30 03:59:34,060] - [remote_util] [140493330810688] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-09-30 03:59:34,352] - [remote_util] [140493330810688] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2020-09-30 03:59:34,358] - [rest_client] [140493330810688] - INFO - -->is_ns_server_running? [2020-09-30 03:59:34,371] - [rest_client] [140493330810688] - INFO - Node version in cluster 7.0.0-3239-enterprise [2020-09-30 03:59:34,387] - [rest_client] [140493330810688] - INFO - Node versions in cluster ['7.0.0-3239-enterprise', '7.0.0-3239-enterprise', '7.0.0-3239-enterprise', '7.0.0-3239-enterprise'] [2020-09-30 03:59:34,387] - [basetestcase] [140493330810688] - INFO - ============== basetestcase setup was started for test #12 suite_tearDown============== [2020-09-30 03:59:34,422] - [bucket_helper] [140493330810688] - INFO - deleting existing buckets ['dst_bucket', 'metadata', 'src_bucket'] on 172.23.123.108 [2020-09-30 03:59:39,421] - [bucket_helper] [140493330810688] - INFO - waiting for bucket deletion to complete.... [2020-09-30 03:59:39,438] - [rest_client] [140493330810688] - INFO - node 172.23.123.108 existing buckets : ['metadata', 'src_bucket'] [2020-09-30 03:59:39,439] - [bucket_helper] [140493330810688] - INFO - deleted bucket : dst_bucket from 172.23.123.108 [2020-09-30 03:59:48,108] - [bucket_helper] [140493330810688] - INFO - waiting for bucket deletion to complete.... [2020-09-30 03:59:48,118] - [rest_client] [140493330810688] - INFO - node 172.23.123.108 existing buckets : ['src_bucket'] [2020-09-30 03:59:48,119] - [bucket_helper] [140493330810688] - INFO - deleted bucket : metadata from 172.23.123.108 [2020-09-30 03:59:52,569] - [bucket_helper] [140493330810688] - INFO - waiting for bucket deletion to complete.... [2020-09-30 03:59:52,571] - [rest_client] [140493330810688] - INFO - node 172.23.123.108 existing buckets : [] [2020-09-30 03:59:52,571] - [bucket_helper] [140493330810688] - INFO - deleted bucket : src_bucket from 172.23.123.108 [2020-09-30 03:59:52,579] - [bucket_helper] [140493330810688] - INFO - Could not find any buckets for node 172.23.123.105, nothing to delete [2020-09-30 03:59:52,586] - [bucket_helper] [140493330810688] - INFO - Could not find any buckets for node 172.23.123.107, nothing to delete [2020-09-30 03:59:52,592] - [bucket_helper] [140493330810688] - INFO - Could not find any buckets for node 172.23.123.104, nothing to delete [2020-09-30 03:59:52,597] - [rest_client] [140493330810688] - INFO - -->is_ns_server_running? [2020-09-30 03:59:52,608] - [cluster_helper] [140493330810688] - INFO - rebalancing all nodes in order to remove nodes [2020-09-30 03:59:52,610] - [rest_client] [140493330810688] - INFO - rebalance params : {'knownNodes': 'ns_1@172.23.123.104,ns_1@172.23.123.105,ns_1@172.23.123.107,ns_1@172.23.123.108', 'ejectedNodes': 'ns_1@172.23.123.104,ns_1@172.23.123.105,ns_1@172.23.123.107', 'user': 'Administrator', 'password': 'password'} [2020-09-30 03:59:52,614] - [rest_client] [140493330810688] - INFO - rebalance operation started [2020-09-30 03:59:52,618] - [rest_client] [140493330810688] - INFO - rebalance percentage : 0.00 % [2020-09-30 04:00:02,631] - [rest_client] [140493330810688] - INFO - rebalance percentage : 50.00 % [2020-09-30 04:00:12,644] - [rest_client] [140493330810688] - INFO - rebalance percentage : 75.00 % [2020-09-30 04:00:32,669] - [rest_client] [140493330810688] - INFO - rebalance progress took 40.05 seconds [2020-09-30 04:00:32,669] - [rest_client] [140493330810688] - INFO - sleep for 10 seconds after rebalance... [2020-09-30 04:00:42,697] - [cluster_helper] [140493330810688] - INFO - removed all the nodes from cluster associated with ip:172.23.123.108 port:8091 ssh_username:root ? [('ns_1@172.23.123.104', 8091), ('ns_1@172.23.123.105', 8091), ('ns_1@172.23.123.107', 8091)] [2020-09-30 04:00:42,701] - [cluster_helper] [140493330810688] - INFO - waiting for ns_server @ 172.23.123.108:8091 [2020-09-30 04:00:42,701] - [rest_client] [140493330810688] - INFO - -->is_ns_server_running? [2020-09-30 04:00:42,704] - [cluster_helper] [140493330810688] - INFO - ns_server @ 172.23.123.108:8091 is running [2020-09-30 04:00:42,707] - [cluster_helper] [140493330810688] - INFO - waiting for ns_server @ 172.23.123.105:8091 [2020-09-30 04:00:42,707] - [rest_client] [140493330810688] - INFO - -->is_ns_server_running? [2020-09-30 04:00:42,710] - [cluster_helper] [140493330810688] - INFO - ns_server @ 172.23.123.105:8091 is running [2020-09-30 04:00:42,713] - [cluster_helper] [140493330810688] - INFO - waiting for ns_server @ 172.23.123.107:8091 [2020-09-30 04:00:42,713] - [rest_client] [140493330810688] - INFO - -->is_ns_server_running? [2020-09-30 04:00:42,715] - [cluster_helper] [140493330810688] - INFO - ns_server @ 172.23.123.107:8091 is running [2020-09-30 04:00:42,718] - [cluster_helper] [140493330810688] - INFO - waiting for ns_server @ 172.23.123.104:8091 [2020-09-30 04:00:42,718] - [rest_client] [140493330810688] - INFO - -->is_ns_server_running? [2020-09-30 04:00:42,720] - [cluster_helper] [140493330810688] - INFO - ns_server @ 172.23.123.104:8091 is running [2020-09-30 04:00:42,721] - [basetestcase] [140493330810688] - INFO - initializing cluster [2020-09-30 04:00:42,748] - [task] [140493040056064] - INFO - server: ip:172.23.123.108 port:8091 ssh_username:root, nodes/self [2020-09-30 04:00:42,752] - [task] [140493040056064] - INFO - {'uptime': '16679', 'memoryTotal': 4201676800, 'memoryFree': 2666471424, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.123.108:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'version': '7.0.0-3239-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 900, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.123.108', 'ip': '172.23.123.108', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} [2020-09-30 04:00:42,756] - [rest_client] [140493040056064] - INFO - pools/default params : memoryQuota=2147 [2020-09-30 04:00:42,763] - [rest_client] [140493040056064] - INFO - --> init_node_services(Administrator,password,172.23.123.108,8091,['kv']) [2020-09-30 04:00:42,764] - [rest_client] [140493040056064] - INFO - /node/controller/setupServices params on 172.23.123.108: 8091:hostname=172.23.123.108&user=Administrator&password=password&services=kv [2020-09-30 04:00:42,765] - [rest_client] [140493040056064] - ERROR - POST http://172.23.123.108:8091//node/controller/setupServices body: hostname=172.23.123.108&user=Administrator&password=password&services=kv headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 400 reason: unknown b'["cannot change node services after cluster is provisioned"]' auth: Administrator:password [2020-09-30 04:00:42,766] - [rest_client] [140493040056064] - INFO - This node is already provisioned with services, we do not consider this as failure for test case [2020-09-30 04:00:42,766] - [rest_client] [140493040056064] - INFO - --> in init_cluster...Administrator,password,8091 [2020-09-30 04:00:42,766] - [rest_client] [140493040056064] - INFO - settings/web params on 172.23.123.108:8091:port=8091&username=Administrator&password=password [2020-09-30 04:00:42,816] - [rest_client] [140493040056064] - INFO - --> status:True [2020-09-30 04:00:42,817] - [remote_util] [140493040056064] - INFO - SSH Connecting to 172.23.123.108 with username:root, attempt#1 of 5 [2020-09-30 04:00:42,917] - [remote_util] [140493040056064] - INFO - SSH Connected to 172.23.123.108 as root [2020-09-30 04:00:43,185] - [remote_util] [140493040056064] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-09-30 04:00:43,473] - [remote_util] [140493040056064] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2020-09-30 04:00:43,482] - [remote_util] [140493040056064] - INFO - running command.raw on 172.23.123.108: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2020-09-30 04:00:43,541] - [remote_util] [140493040056064] - INFO - command executed with root but got an error [' % Total % Received % Xferd Average Speed Time Time Time Current', ' Dload Upload Total Spent Left Speed', '', ' 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0', '100 43 100 2 100 41 285 5859 --:--:-- --:--:-- --:--:-- 6833'] ... [2020-09-30 04:00:43,541] - [remote_util] [140493040056064] - INFO - ['ok'] [2020-09-30 04:00:43,544] - [rest_client] [140493040056064] - INFO - /diag/eval status on 172.23.123.108:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2020-09-30 04:00:43,545] - [rest_client] [140493040056064] - INFO - /diag/eval status on 172.23.123.108:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2020-09-30 04:00:43,550] - [rest_client] [140493040056064] - INFO - settings/indexes params : storageMode=plasma [2020-09-30 04:00:43,567] - [task] [140493040056064] - INFO - server: ip:172.23.123.105 port:8091 ssh_username:root, nodes/self [2020-09-30 04:00:43,570] - [task] [140493040056064] - INFO - {'uptime': '16676', 'memoryTotal': 4201672704, 'memoryFree': 3508350976, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.123.105:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'version': '7.0.0-3239-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.123.105', 'ip': '172.23.123.105', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} [2020-09-30 04:00:43,571] - [rest_client] [140493040056064] - ERROR - GET http://172.23.123.105:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2020-09-30 04:00:43,572] - [rest_client] [140493040056064] - INFO - pools/default params : memoryQuota=2147 [2020-09-30 04:00:43,575] - [rest_client] [140493040056064] - INFO - --> in init_cluster...Administrator,password,8091 [2020-09-30 04:00:43,575] - [rest_client] [140493040056064] - INFO - settings/web params on 172.23.123.105:8091:port=8091&username=Administrator&password=password [2020-09-30 04:00:43,628] - [rest_client] [140493040056064] - INFO - --> status:True [2020-09-30 04:00:43,629] - [remote_util] [140493040056064] - INFO - SSH Connecting to 172.23.123.105 with username:root, attempt#1 of 5 [2020-09-30 04:00:43,733] - [remote_util] [140493040056064] - INFO - SSH Connected to 172.23.123.105 as root [2020-09-30 04:00:44,003] - [remote_util] [140493040056064] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-09-30 04:00:44,296] - [remote_util] [140493040056064] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2020-09-30 04:00:44,302] - [remote_util] [140493040056064] - INFO - running command.raw on 172.23.123.105: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2020-09-30 04:00:44,359] - [remote_util] [140493040056064] - INFO - command executed with root but got an error [' % Total % Received % Xferd Average Speed Time Time Time Current', ' Dload Upload Total Spent Left Speed', '', ' 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0', '100 43 100 2 100 41 288 5904 --:--:-- --:--:-- --:--:-- 6833'] ... [2020-09-30 04:00:44,359] - [remote_util] [140493040056064] - INFO - ['ok'] [2020-09-30 04:00:44,361] - [rest_client] [140493040056064] - INFO - /diag/eval status on 172.23.123.105:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2020-09-30 04:00:44,362] - [rest_client] [140493040056064] - INFO - /diag/eval status on 172.23.123.105:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2020-09-30 04:00:44,366] - [rest_client] [140493040056064] - INFO - settings/indexes params : storageMode=plasma [2020-09-30 04:00:44,379] - [task] [140493040056064] - INFO - server: ip:172.23.123.107 port:8091 ssh_username:root, nodes/self [2020-09-30 04:00:44,382] - [task] [140493040056064] - INFO - {'uptime': '16675', 'memoryTotal': 4201672704, 'memoryFree': 3523489792, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.123.107:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'version': '7.0.0-3239-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.123.107', 'ip': '172.23.123.107', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} [2020-09-30 04:00:44,383] - [rest_client] [140493040056064] - ERROR - GET http://172.23.123.107:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2020-09-30 04:00:44,384] - [rest_client] [140493040056064] - INFO - pools/default params : memoryQuota=2147 [2020-09-30 04:00:44,387] - [rest_client] [140493040056064] - INFO - --> in init_cluster...Administrator,password,8091 [2020-09-30 04:00:44,387] - [rest_client] [140493040056064] - INFO - settings/web params on 172.23.123.107:8091:port=8091&username=Administrator&password=password [2020-09-30 04:00:44,436] - [rest_client] [140493040056064] - INFO - --> status:True [2020-09-30 04:00:44,437] - [remote_util] [140493040056064] - INFO - SSH Connecting to 172.23.123.107 with username:root, attempt#1 of 5 [2020-09-30 04:00:44,538] - [remote_util] [140493040056064] - INFO - SSH Connected to 172.23.123.107 as root [2020-09-30 04:00:44,814] - [remote_util] [140493040056064] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-09-30 04:00:45,096] - [remote_util] [140493040056064] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2020-09-30 04:00:45,102] - [remote_util] [140493040056064] - INFO - running command.raw on 172.23.123.107: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2020-09-30 04:00:45,158] - [remote_util] [140493040056064] - INFO - command executed with root but got an error [' % Total % Received % Xferd Average Speed Time Time Time Current', ' Dload Upload Total Spent Left Speed', '', ' 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0', '100 43 100 2 100 41 303 6212 --:--:-- --:--:-- --:--:-- 6833'] ... [2020-09-30 04:00:45,158] - [remote_util] [140493040056064] - INFO - ['ok'] [2020-09-30 04:00:45,160] - [rest_client] [140493040056064] - INFO - /diag/eval status on 172.23.123.107:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2020-09-30 04:00:45,163] - [rest_client] [140493040056064] - INFO - /diag/eval status on 172.23.123.107:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2020-09-30 04:00:45,167] - [rest_client] [140493040056064] - INFO - settings/indexes params : storageMode=plasma [2020-09-30 04:00:45,179] - [task] [140493040056064] - INFO - server: ip:172.23.123.104 port:8091 ssh_username:root, nodes/self [2020-09-30 04:00:45,182] - [task] [140493040056064] - INFO - {'uptime': '16681', 'memoryTotal': 4201672704, 'memoryFree': 3571081216, 'mcdMemoryReserved': 3205, 'mcdMemoryAllocated': 3205, 'status': 'healthy', 'hostname': '172.23.123.104:8091', 'clusterCompatibility': 458752, 'clusterMembership': 'active', 'version': '7.0.0-3239-enterprise', 'os': 'x86_64-unknown-linux-gnu', 'ports': [], 'availableStorage': [], 'storage': [], 'memoryQuota': 1738, 'moxi': 11211, 'memcached': 11210, 'id': 'ns_1@172.23.123.104', 'ip': '172.23.123.104', 'rest_username': '', 'rest_password': '', 'port': '8091', 'services': ['kv'], 'storageTotalRam': 4007, 'curr_items': 0} [2020-09-30 04:00:45,183] - [rest_client] [140493040056064] - ERROR - GET http://172.23.123.104:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==', 'Accept': '*/*'} error: 404 reason: unknown b'"unknown pool"' auth: Administrator:password [2020-09-30 04:00:45,183] - [rest_client] [140493040056064] - INFO - pools/default params : memoryQuota=2147 [2020-09-30 04:00:45,186] - [rest_client] [140493040056064] - INFO - --> in init_cluster...Administrator,password,8091 [2020-09-30 04:00:45,186] - [rest_client] [140493040056064] - INFO - settings/web params on 172.23.123.104:8091:port=8091&username=Administrator&password=password [2020-09-30 04:00:45,236] - [rest_client] [140493040056064] - INFO - --> status:True [2020-09-30 04:00:45,236] - [remote_util] [140493040056064] - INFO - SSH Connecting to 172.23.123.104 with username:root, attempt#1 of 5 [2020-09-30 04:00:45,338] - [remote_util] [140493040056064] - INFO - SSH Connected to 172.23.123.104 as root [2020-09-30 04:00:45,595] - [remote_util] [140493040056064] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-09-30 04:00:45,883] - [remote_util] [140493040056064] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2020-09-30 04:00:45,890] - [remote_util] [140493040056064] - INFO - running command.raw on 172.23.123.104: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2020-09-30 04:00:45,945] - [remote_util] [140493040056064] - INFO - command executed with root but got an error [' % Total % Received % Xferd Average Speed Time Time Time Current', ' Dload Upload Total Spent Left Speed', '', ' 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0', '100 43 100 2 100 41 297 6095 --:--:-- --:--:-- --:--:-- 6833'] ... [2020-09-30 04:00:45,945] - [remote_util] [140493040056064] - INFO - ['ok'] [2020-09-30 04:00:45,947] - [rest_client] [140493040056064] - INFO - /diag/eval status on 172.23.123.104:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2020-09-30 04:00:45,949] - [rest_client] [140493040056064] - INFO - /diag/eval status on 172.23.123.104:8091: True content: [7,0] command: cluster_compat_mode:get_compat_version(). [2020-09-30 04:00:45,953] - [rest_client] [140493040056064] - INFO - settings/indexes params : storageMode=plasma [2020-09-30 04:00:45,975] - [basetestcase] [140493330810688] - INFO - **** add built-in 'cbadminbucket' user to node 172.23.123.108 **** [2020-09-30 04:00:46,050] - [basetestcase] [140493330810688] - INFO - sleep for 5 secs. ... [2020-09-30 04:00:51,055] - [basetestcase] [140493330810688] - INFO - **** add 'admin' role to 'cbadminbucket' user **** [2020-09-30 04:00:51,068] - [basetestcase] [140493330810688] - INFO - done initializing cluster [2020-09-30 04:00:51,973] - [task] [140493040056064] - INFO - adding node 172.23.123.105:8091 to cluster [2020-09-30 04:00:51,973] - [rest_client] [140493040056064] - INFO - adding remote node @172.23.123.105:8091 to this cluster @172.23.123.108:8091 [2020-09-30 04:00:54,062] - [task] [140493040056064] - INFO - adding node 172.23.123.107:8091 to cluster [2020-09-30 04:00:54,062] - [rest_client] [140493040056064] - INFO - adding remote node @172.23.123.107:8091 to this cluster @172.23.123.108:8091 [2020-09-30 04:00:56,420] - [task] [140493040056064] - INFO - adding node 172.23.123.104:8091 to cluster [2020-09-30 04:00:56,421] - [rest_client] [140493040056064] - INFO - adding remote node @172.23.123.104:8091 to this cluster @172.23.123.108:8091 [2020-09-30 04:00:58,452] - [rest_client] [140493040056064] - INFO - rebalance params : {'knownNodes': 'ns_1@172.23.123.104,ns_1@172.23.123.105,ns_1@172.23.123.107,ns_1@172.23.123.108', 'ejectedNodes': '', 'user': 'Administrator', 'password': 'password'} [2020-09-30 04:00:58,458] - [rest_client] [140493040056064] - INFO - rebalance operation started [2020-09-30 04:00:58,462] - [rest_client] [140493040056064] - INFO - rebalance percentage : 0.00 % [2020-09-30 04:00:58,462] - [task] [140493040056064] - INFO - Rebalance - status: running, progress: 0.00% [2020-09-30 04:01:08,485] - [rest_client] [140493040056064] - INFO - rebalance percentage : 50.00 % [2020-09-30 04:01:08,485] - [task] [140493040056064] - INFO - Rebalance - status: running, progress: 50.00% [2020-09-30 04:01:18,504] - [task] [140493040056064] - INFO - Rebalance - status: none, progress: 100.00% [2020-09-30 04:01:18,512] - [task] [140493040056064] - INFO - rebalancing was completed with progress: 100% in 20.0540771484375 sec [2020-09-30 04:01:18,514] - [remote_util] [140493330810688] - INFO - SSH Connecting to 172.23.123.108 with username:root, attempt#1 of 5 [2020-09-30 04:01:18,612] - [remote_util] [140493330810688] - INFO - SSH Connected to 172.23.123.108 as root [2020-09-30 04:01:18,881] - [remote_util] [140493330810688] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-09-30 04:01:19,168] - [remote_util] [140493330810688] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2020-09-30 04:01:19,176] - [remote_util] [140493330810688] - INFO - running command.raw on 172.23.123.108: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' [2020-09-30 04:01:19,232] - [remote_util] [140493330810688] - INFO - command executed with root but got an error [' % Total % Received % Xferd Average Speed Time Time Time Current', ' Dload Upload Total Spent Left Speed', '', ' 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0', '100 43 100 2 100 41 261 5352 --:--:-- --:--:-- --:--:-- 5857'] ... [2020-09-30 04:01:19,232] - [remote_util] [140493330810688] - INFO - ['ok'] [2020-09-30 04:01:19,232] - [basetestcase] [140493330810688] - INFO - Enabled diag/eval for non-local hosts from 172.23.123.108 [2020-09-30 04:01:19,232] - [basetestcase] [140493330810688] - INFO - ============== basetestcase setup was finished for test #12 suite_tearDown ============== [2020-09-30 04:01:19,250] - [basetestcase] [140493330810688] - INFO - ------- Cluster statistics ------- [2020-09-30 04:01:19,250] - [basetestcase] [140493330810688] - INFO - 172.23.123.104:8091 => {'services': ['n1ql'], 'cpu_utilization': 0.5012531328320802, 'mem_free': 3522297856, 'mem_total': 4201672704, 'swap_mem_used': 68681728, 'swap_mem_total': 3758092288} [2020-09-30 04:01:19,250] - [basetestcase] [140493330810688] - INFO - 172.23.123.105:8091 => {'services': ['eventing'], 'cpu_utilization': 2.261306532663316, 'mem_free': 3473604608, 'mem_total': 4201672704, 'swap_mem_used': 5767168, 'swap_mem_total': 3758092288} [2020-09-30 04:01:19,250] - [basetestcase] [140493330810688] - INFO - 172.23.123.107:8091 => {'services': ['index'], 'cpu_utilization': 0.5, 'mem_free': 3474935808, 'mem_total': 4201672704, 'swap_mem_used': 3932160, 'swap_mem_total': 3758092288} [2020-09-30 04:01:19,251] - [basetestcase] [140493330810688] - INFO - 172.23.123.108:8091 => {'services': ['kv'], 'cpu_utilization': 0.2538071065989848, 'mem_free': 2657509376, 'mem_total': 4201676800, 'swap_mem_used': 79429632, 'swap_mem_total': 3758092288} [2020-09-30 04:01:19,251] - [basetestcase] [140493330810688] - INFO - --- End of cluster statistics --- [2020-09-30 04:01:19,251] - [remote_util] [140493330810688] - INFO - SSH Connecting to 172.23.123.108 with username:root, attempt#1 of 5 [2020-09-30 04:01:19,350] - [remote_util] [140493330810688] - INFO - SSH Connected to 172.23.123.108 as root [2020-09-30 04:01:19,614] - [remote_util] [140493330810688] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-09-30 04:01:19,901] - [remote_util] [140493330810688] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2020-09-30 04:01:25,803] - [basetestcase] [140493330810688] - INFO - list of n1ql nodes in cluster: [ip:172.23.123.104 port:8091 ssh_username:root] [2020-09-30 04:01:25,811] - [basetestcase] [140493330810688] - INFO - list of eventing nodes in cluster: [ip:172.23.123.105 port:8091 ssh_username:root] [2020-09-30 04:01:25,817] - [rest_client] [140493330810688] - INFO - settings/indexes params : storageMode=plasma [2020-09-30 04:01:25,822] - [eventing_base] [140493330810688] - INFO - Setting the min possible memory quota so that adding mode nodes to the cluster wouldn't be a problem. [2020-09-30 04:01:25,823] - [rest_client] [140493330810688] - INFO - pools/default params : memoryQuota=330 [2020-09-30 04:01:25,829] - [rest_client] [140493330810688] - INFO - pools/default params : indexMemoryQuota=256 [2020-09-30 04:01:25,835] - [rest_client] [140493330810688] - INFO - pools/default params : eventingMemoryQuota=512 [2020-09-30 04:01:25,860] - [basetestcase] [140493330810688] - INFO - list of n1ql nodes in cluster: [ip:172.23.123.104 port:8091 ssh_username:root] [2020-09-30 04:01:25,861] - [rest_client] [140493330810688] - INFO - pools/default params : memoryQuota=900 [2020-09-30 04:01:25,867] - [eventing_sanity] [140493330810688] - INFO - 200 [2020-09-30 04:01:26,565] - [rest_client] [140493040056064] - INFO - http://172.23.123.108:8091/pools/default/buckets with param: name=src_bucket&ramQuotaMB=200&authType=none&replicaNumber=0&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=magma [2020-09-30 04:01:26,570] - [rest_client] [140493040056064] - INFO - 0.00 seconds to create bucket src_bucket [2020-09-30 04:01:26,570] - [bucket_helper] [140493040056064] - INFO - waiting for memcached bucket : src_bucket in 172.23.123.108 to accept set ops [2020-09-30 04:01:27,152] - [data_helper] [140493040056064] - INFO - creating direct client 172.23.123.108:11210 src_bucket [2020-09-30 04:01:27,210] - [data_helper] [140493040056064] - INFO - creating direct client 172.23.123.108:11210 src_bucket [2020-09-30 04:01:27,243] - [task] [140493040056064] - WARNING - vbucket map not ready after try 0 [2020-09-30 04:01:27,243] - [bucket_helper] [140493040056064] - INFO - waiting for memcached bucket : src_bucket in 172.23.123.108 to accept set ops [2020-09-30 04:01:27,321] - [data_helper] [140493040056064] - INFO - creating direct client 172.23.123.108:11210 src_bucket [2020-09-30 04:01:27,383] - [data_helper] [140493040056064] - INFO - creating direct client 172.23.123.108:11210 src_bucket [2020-09-30 04:01:27,416] - [task] [140493040056064] - WARNING - vbucket map not ready after try 1 [2020-09-30 04:01:27,416] - [bucket_helper] [140493040056064] - INFO - waiting for memcached bucket : src_bucket in 172.23.123.108 to accept set ops [2020-09-30 04:01:27,470] - [data_helper] [140493040056064] - INFO - creating direct client 172.23.123.108:11210 src_bucket [2020-09-30 04:01:27,528] - [data_helper] [140493040056064] - INFO - creating direct client 172.23.123.108:11210 src_bucket [2020-09-30 04:01:28,014] - [task] [140493040056064] - WARNING - vbucket map not ready after try 2 [2020-09-30 04:01:28,014] - [bucket_helper] [140493040056064] - INFO - waiting for memcached bucket : src_bucket in 172.23.123.108 to accept set ops [2020-09-30 04:01:28,066] - [data_helper] [140493040056064] - INFO - creating direct client 172.23.123.108:11210 src_bucket [2020-09-30 04:01:28,119] - [data_helper] [140493040056064] - INFO - creating direct client 172.23.123.108:11210 src_bucket [2020-09-30 04:01:28,148] - [task] [140493040056064] - WARNING - vbucket map not ready after try 3 [2020-09-30 04:01:28,148] - [bucket_helper] [140493040056064] - INFO - waiting for memcached bucket : src_bucket in 172.23.123.108 to accept set ops [2020-09-30 04:01:28,199] - [data_helper] [140493040056064] - INFO - creating direct client 172.23.123.108:11210 src_bucket [2020-09-30 04:01:28,253] - [data_helper] [140493040056064] - INFO - creating direct client 172.23.123.108:11210 src_bucket [2020-09-30 04:01:28,283] - [task] [140493040056064] - WARNING - vbucket map not ready after try 4 [2020-09-30 04:01:28,283] - [bucket_helper] [140493040056064] - INFO - waiting for memcached bucket : src_bucket in 172.23.123.108 to accept set ops [2020-09-30 04:01:28,340] - [data_helper] [140493040056064] - INFO - creating direct client 172.23.123.108:11210 src_bucket [2020-09-30 04:01:28,394] - [data_helper] [140493040056064] - INFO - creating direct client 172.23.123.108:11210 src_bucket [2020-09-30 04:01:28,424] - [task] [140493040056064] - WARNING - vbucket map not ready after try 5 [2020-09-30 04:01:29,445] - [rest_client] [140493040056064] - INFO - http://172.23.123.108:8091/pools/default/buckets with param: name=dst_bucket&ramQuotaMB=200&authType=none&replicaNumber=0&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=magma [2020-09-30 04:01:29,452] - [rest_client] [140493040056064] - INFO - 0.01 seconds to create bucket dst_bucket [2020-09-30 04:01:29,452] - [bucket_helper] [140493040056064] - INFO - waiting for memcached bucket : dst_bucket in 172.23.123.108 to accept set ops [2020-09-30 04:01:29,631] - [data_helper] [140493040056064] - INFO - creating direct client 172.23.123.108:11210 dst_bucket [2020-09-30 04:01:30,168] - [data_helper] [140493040056064] - INFO - creating direct client 172.23.123.108:11210 dst_bucket [2020-09-30 04:01:30,213] - [task] [140493040056064] - WARNING - vbucket map not ready after try 0 [2020-09-30 04:01:30,213] - [bucket_helper] [140493040056064] - INFO - waiting for memcached bucket : dst_bucket in 172.23.123.108 to accept set ops [2020-09-30 04:01:30,273] - [data_helper] [140493040056064] - INFO - creating direct client 172.23.123.108:11210 dst_bucket [2020-09-30 04:01:30,341] - [data_helper] [140493040056064] - INFO - creating direct client 172.23.123.108:11210 dst_bucket [2020-09-30 04:01:30,377] - [task] [140493040056064] - WARNING - vbucket map not ready after try 1 [2020-09-30 04:01:30,377] - [bucket_helper] [140493040056064] - INFO - waiting for memcached bucket : dst_bucket in 172.23.123.108 to accept set ops [2020-09-30 04:01:30,435] - [data_helper] [140493040056064] - INFO - creating direct client 172.23.123.108:11210 dst_bucket [2020-09-30 04:01:30,495] - [data_helper] [140493040056064] - INFO - creating direct client 172.23.123.108:11210 dst_bucket [2020-09-30 04:01:30,530] - [task] [140493040056064] - WARNING - vbucket map not ready after try 2 [2020-09-30 04:01:30,530] - [bucket_helper] [140493040056064] - INFO - waiting for memcached bucket : dst_bucket in 172.23.123.108 to accept set ops [2020-09-30 04:01:30,588] - [data_helper] [140493040056064] - INFO - creating direct client 172.23.123.108:11210 dst_bucket [2020-09-30 04:01:30,646] - [data_helper] [140493040056064] - INFO - creating direct client 172.23.123.108:11210 dst_bucket [2020-09-30 04:01:30,677] - [task] [140493040056064] - WARNING - vbucket map not ready after try 3 [2020-09-30 04:01:30,677] - [bucket_helper] [140493040056064] - INFO - waiting for memcached bucket : dst_bucket in 172.23.123.108 to accept set ops [2020-09-30 04:01:30,735] - [data_helper] [140493040056064] - INFO - creating direct client 172.23.123.108:11210 dst_bucket [2020-09-30 04:01:30,794] - [data_helper] [140493040056064] - INFO - creating direct client 172.23.123.108:11210 dst_bucket [2020-09-30 04:01:30,826] - [task] [140493040056064] - WARNING - vbucket map not ready after try 4 [2020-09-30 04:01:30,826] - [bucket_helper] [140493040056064] - INFO - waiting for memcached bucket : dst_bucket in 172.23.123.108 to accept set ops [2020-09-30 04:01:31,342] - [data_helper] [140493040056064] - INFO - creating direct client 172.23.123.108:11210 dst_bucket [2020-09-30 04:01:31,401] - [data_helper] [140493040056064] - INFO - creating direct client 172.23.123.108:11210 dst_bucket [2020-09-30 04:01:31,434] - [task] [140493040056064] - WARNING - vbucket map not ready after try 5 [2020-09-30 04:01:32,456] - [rest_client] [140493040056064] - INFO - http://172.23.123.108:8091/pools/default/buckets with param: name=metadata&ramQuotaMB=200&authType=none&replicaNumber=0&bucketType=membase&replicaIndex=1&threadsNumber=3&flushEnabled=1&evictionPolicy=valueOnly&compressionMode=passive&storageBackend=magma [2020-09-30 04:01:32,465] - [rest_client] [140493040056064] - INFO - 0.01 seconds to create bucket metadata [2020-09-30 04:01:32,465] - [bucket_helper] [140493040056064] - INFO - waiting for memcached bucket : metadata in 172.23.123.108 to accept set ops [2020-09-30 04:01:33,067] - [data_helper] [140493040056064] - INFO - creating direct client 172.23.123.108:11210 metadata [2020-09-30 04:01:33,130] - [data_helper] [140493040056064] - INFO - creating direct client 172.23.123.108:11210 metadata [2020-09-30 04:01:33,166] - [task] [140493040056064] - WARNING - vbucket map not ready after try 0 [2020-09-30 04:01:33,166] - [bucket_helper] [140493040056064] - INFO - waiting for memcached bucket : metadata in 172.23.123.108 to accept set ops [2020-09-30 04:01:33,228] - [data_helper] [140493040056064] - INFO - creating direct client 172.23.123.108:11210 metadata [2020-09-30 04:01:33,306] - [data_helper] [140493040056064] - INFO - creating direct client 172.23.123.108:11210 metadata [2020-09-30 04:01:33,356] - [task] [140493040056064] - WARNING - vbucket map not ready after try 1 [2020-09-30 04:01:33,356] - [bucket_helper] [140493040056064] - INFO - waiting for memcached bucket : metadata in 172.23.123.108 to accept set ops [2020-09-30 04:01:33,428] - [data_helper] [140493040056064] - INFO - creating direct client 172.23.123.108:11210 metadata [2020-09-30 04:01:33,496] - [data_helper] [140493040056064] - INFO - creating direct client 172.23.123.108:11210 metadata [2020-09-30 04:01:33,536] - [task] [140493040056064] - WARNING - vbucket map not ready after try 2 [2020-09-30 04:01:33,536] - [bucket_helper] [140493040056064] - INFO - waiting for memcached bucket : metadata in 172.23.123.108 to accept set ops [2020-09-30 04:01:34,054] - [data_helper] [140493040056064] - INFO - creating direct client 172.23.123.108:11210 metadata [2020-09-30 04:01:34,116] - [data_helper] [140493040056064] - INFO - creating direct client 172.23.123.108:11210 metadata [2020-09-30 04:01:34,151] - [task] [140493040056064] - WARNING - vbucket map not ready after try 3 [2020-09-30 04:01:34,151] - [bucket_helper] [140493040056064] - INFO - waiting for memcached bucket : metadata in 172.23.123.108 to accept set ops [2020-09-30 04:01:34,210] - [data_helper] [140493040056064] - INFO - creating direct client 172.23.123.108:11210 metadata [2020-09-30 04:01:34,270] - [data_helper] [140493040056064] - INFO - creating direct client 172.23.123.108:11210 metadata [2020-09-30 04:01:34,311] - [task] [140493040056064] - WARNING - vbucket map not ready after try 4 [2020-09-30 04:01:34,311] - [bucket_helper] [140493040056064] - INFO - waiting for memcached bucket : metadata in 172.23.123.108 to accept set ops [2020-09-30 04:01:34,373] - [data_helper] [140493040056064] - INFO - creating direct client 172.23.123.108:11210 metadata [2020-09-30 04:01:34,435] - [data_helper] [140493040056064] - INFO - creating direct client 172.23.123.108:11210 metadata [2020-09-30 04:01:34,470] - [task] [140493040056064] - WARNING - vbucket map not ready after try 5 [2020-09-30 04:01:34,543] - [tuq_helper] [140493330810688] - INFO - RUN QUERY create primary index on src_bucket [2020-09-30 04:01:34,547] - [rest_client] [140493330810688] - INFO - query params : statement=create+primary+index+on+src_bucket [2020-09-30 04:01:36,557] - [tuq_helper] [140493330810688] - INFO - TOTAL ELAPSED TIME: 2.007870382s [2020-09-30 04:01:36,562] - [tuq_helper] [140493330810688] - INFO - RUN QUERY create primary index on dst_bucket [2020-09-30 04:01:36,567] - [rest_client] [140493330810688] - INFO - query params : statement=create+primary+index+on+dst_bucket [2020-09-30 04:01:38,446] - [tuq_helper] [140493330810688] - INFO - TOTAL ELAPSED TIME: 1.876482247s [2020-09-30 04:01:38,450] - [tuq_helper] [140493330810688] - INFO - RUN QUERY create primary index on metadata [2020-09-30 04:01:38,455] - [rest_client] [140493330810688] - INFO - query params : statement=create+primary+index+on+metadata [2020-09-30 04:01:40,342] - [tuq_helper] [140493330810688] - INFO - TOTAL ELAPSED TIME: 1.885997963s [2020-09-30 04:01:40,363] - [basetestcase] [140493330810688] - INFO - list of eventing nodes in cluster: [ip:172.23.123.105 port:8091 ssh_username:root] [2020-09-30 04:01:40,364] - [remote_util] [140493330810688] - INFO - SSH Connecting to 172.23.123.105 with username:root, attempt#1 of 5 [2020-09-30 04:01:40,467] - [remote_util] [140493330810688] - INFO - SSH Connected to 172.23.123.105 as root [2020-09-30 04:01:40,742] - [remote_util] [140493330810688] - INFO - os_distro: CentOS, os_version: centos 7, is_linux_distro: True [2020-09-30 04:01:41,035] - [remote_util] [140493330810688] - INFO - extract_remote_info-->distribution_type: CentOS, distribution_version: centos 7 [2020-09-30 04:01:41,041] - [rest_client] [140493330810688] - INFO - /diag/eval status on 172.23.123.105:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). [2020-09-30 04:01:41,041] - [remote_util] [140493330810688] - INFO - running command.raw on 172.23.123.105: zgrep "panic" "/opt/couchbase/var/lib/couchbase/logs"/eventing.log* | wc -l [2020-09-30 04:01:41,485] - [remote_util] [140493330810688] - INFO - command executed successfully with root [2020-09-30 04:01:41,485] - [remote_util] [140493330810688] - INFO - running command.raw on 172.23.123.105: ls "/opt/couchbase/var/lib/couchbase/logs"/../crash/| wc -l [2020-09-30 04:01:41,540] - [remote_util] [140493330810688] - INFO - command executed successfully with root [2020-09-30 04:01:41,604] - [eventing_base] [140493330810688] - INFO - Bucket dst_bucket DGM is 100 [2020-09-30 04:01:41,643] - [eventing_base] [140493330810688] - INFO - Bucket metadata DGM is 100 [2020-09-30 04:01:41,687] - [eventing_base] [140493330810688] - INFO - Bucket src_bucket DGM is 100 [2020-09-30 04:01:41,718] - [eventing_base] [140493330810688] - INFO - number of documents in metadata bucket 0 [2020-09-30 04:01:41,731] - [basetestcase] [140493330810688] - INFO - ------- Cluster statistics ------- [2020-09-30 04:01:41,731] - [basetestcase] [140493330810688] - INFO - 172.23.123.104:8091 => {'services': ['n1ql'], 'cpu_utilization': 2.512562814070352, 'mem_free': 3436904448, 'mem_total': 4201672704, 'swap_mem_used': 68681728, 'swap_mem_total': 3758092288} [2020-09-30 04:01:41,732] - [basetestcase] [140493330810688] - INFO - 172.23.123.105:8091 => {'services': ['eventing'], 'cpu_utilization': 4.260651629072682, 'mem_free': 3411230720, 'mem_total': 4201672704, 'swap_mem_used': 5767168, 'swap_mem_total': 3758092288} [2020-09-30 04:01:41,732] - [basetestcase] [140493330810688] - INFO - 172.23.123.107:8091 => {'services': ['index'], 'cpu_utilization': 22.16494845360825, 'mem_free': 3085873152, 'mem_total': 4201672704, 'swap_mem_used': 3932160, 'swap_mem_total': 3758092288} [2020-09-30 04:01:41,732] - [basetestcase] [140493330810688] - INFO - 172.23.123.108:8091 => {'services': ['kv'], 'cpu_utilization': 45.85635359116022, 'mem_free': 2333937664, 'mem_total': 4201676800, 'swap_mem_used': 79429632, 'swap_mem_total': 3758092288} [2020-09-30 04:01:41,732] - [basetestcase] [140493330810688] - INFO - --- End of cluster statistics --- [2020-09-30 04:01:41,732] - [basetestcase] [140493330810688] - WARNING - CLEANUP WAS SKIPPED [2020-09-30 04:01:41,732] - [basetestcase] [140493330810688] - INFO - closing all ssh connections [2020-09-30 04:01:41,733] - [basetestcase] [140493330810688] - INFO - closing all memcached connections