Balakumarans-MacBook-Pro:testrunner balakumaran.g$ ./testrunner -i b/temp_br_centos6.ini -t eventing.eventing_tools.EventingTools.test_eventing_lifecycle_with_couchbase_cli,nodes_init=4,services_init=kv-eventing-index-n1ql,dataset=default,groups=simple,reset_services=True,skip_cleanup=True Global Test input params: {'cluster_name': 'temp_br_centos6', 'ini': 'b/temp_br_centos6.ini', 'num_nodes': 4} Logs will be stored at /Users/balakumaran.g/git/temp/testrunner/logs/testrunner-18-Sep-18_14-11-55/test_1 ./testrunner -i b/temp_br_centos6.ini -p -t eventing.eventing_tools.EventingTools.test_eventing_lifecycle_with_couchbase_cli,nodes_init=4,services_init=kv-eventing-index-n1ql,dataset=default,groups=simple,reset_services=True,skip_cleanup=True Test Input params: {'cluster_name': 'temp_br_centos6', 'logs_folder': '/Users/balakumaran.g/git/temp/testrunner/logs/testrunner-18-Sep-18_14-11-55/test_1', 'reset_services': 'True', 'dataset': 'default', 'skip_cleanup': 'True', 'services_init': 'kv-eventing-index-n1ql', 'ini': 'b/temp_br_centos6.ini', 'groups': 'simple', 'case_number': 1, 'num_nodes': 4, 'nodes_init': '4'} Run before suite setup for eventing.eventing_tools.EventingTools.test_eventing_lifecycle_with_couchbase_cli test_eventing_lifecycle_with_couchbase_cli (eventing.eventing_tools.EventingTools) ... 2018-09-18 14:11:55,511 - root - INFO - Starting Test: test_eventing_lifecycle_with_couchbase_cli 2018-09-18 14:11:55,512 - root - INFO - connecting to 10.111.170.101 with username:root password:couchbase ssh_key: /Library/Python/2.7/site-packages/paramiko-2.0.2-py2.7.egg/paramiko/rsakey.py:119: CryptographyDeprecationWarning: signer and verifier have been deprecated. Please use sign and verify instead. /Library/Python/2.7/site-packages/paramiko-2.0.2-py2.7.egg/paramiko/rsakey.py:99: CryptographyDeprecationWarning: signer and verifier have been deprecated. Please use sign and verify instead. 2018-09-18 14:11:55,728 - root - INFO - Connected to 10.111.170.101 2018-09-18 14:11:56,139 - root - INFO - Node version in cluster 6.0.0-1643-enterprise 2018-09-18 14:11:56,164 - root - INFO - Node versions in cluster [u'6.0.0-1643-enterprise', u'6.0.0-1643-enterprise', u'6.0.0-1643-enterprise', u'6.0.0-1643-enterprise'] 2018-09-18 14:11:56,164 - root - INFO - ============== basetestcase setup was started for test #1 test_eventing_lifecycle_with_couchbase_cli============== 2018-09-18 14:11:56,202 - root - INFO - deleting existing buckets [u'dst_bucket', u'metadata', u'src_bucket'] on 10.111.170.101 2018-09-18 14:11:56,203 - root - INFO - remove bucket dst_bucket ... 2018-09-18 14:11:56,944 - root - INFO - deleted bucket : dst_bucket from 10.111.170.101 2018-09-18 14:11:56,944 - root - INFO - waiting for bucket deletion to complete.... 2018-09-18 14:11:56,951 - root - INFO - node 10.111.170.101 existing buckets : [u'metadata', u'src_bucket'] 2018-09-18 14:11:56,952 - root - INFO - remove bucket metadata ... 2018-09-18 14:11:57,585 - root - INFO - deleted bucket : metadata from 10.111.170.101 2018-09-18 14:11:57,586 - root - INFO - waiting for bucket deletion to complete.... 2018-09-18 14:11:57,591 - root - INFO - node 10.111.170.101 existing buckets : [u'src_bucket'] 2018-09-18 14:11:57,591 - root - INFO - remove bucket src_bucket ... 2018-09-18 14:11:59,099 - root - INFO - deleted bucket : src_bucket from 10.111.170.101 2018-09-18 14:11:59,099 - root - INFO - waiting for bucket deletion to complete.... 2018-09-18 14:11:59,104 - root - INFO - node 10.111.170.101 existing buckets : [] 2018-09-18 14:11:59,104 - root - INFO - sleep 2 seconds to make sure all buckets ([u'dst_bucket', u'metadata', u'src_bucket']) were deleted completely. 2018-09-18 14:12:01,156 - root - INFO - rebalancing all nodes in order to remove nodes 2018-09-18 14:12:01,160 - root - INFO - rebalance params : {'password': 'password', 'ejectedNodes': u'ns_1@10.111.170.102,ns_1@10.111.170.104,ns_1@10.111.170.103', 'user': 'Administrator', 'knownNodes': u'ns_1@10.111.170.102,ns_1@10.111.170.101,ns_1@10.111.170.104,ns_1@10.111.170.103'} 2018-09-18 14:12:01,167 - root - INFO - rebalance operation started 2018-09-18 14:12:01,174 - root - INFO - rebalance percentage : 0.00 % 2018-09-18 14:12:11,183 - root - INFO - rebalance percentage : 66.00 % 2018-09-18 14:12:21,187 - root - INFO - rebalance percentage : 100.00 % 2018-09-18 14:12:31,194 - root - INFO - rebalance progress took 30.03 seconds 2018-09-18 14:12:31,195 - root - INFO - sleep for 10 seconds after rebalance... 2018-09-18 14:12:41,217 - root - INFO - removed all the nodes from cluster associated with ip:10.111.170.101 port:8091 ssh_username:root ? [(u'ns_1@10.111.170.102', 8091), (u'ns_1@10.111.170.104', 8091), (u'ns_1@10.111.170.103', 8091)] 2018-09-18 14:12:41,217 - root - INFO - sleep for 10 secs. ... 2018-09-18 14:12:51,226 - root - INFO - waiting for ns_server @ 10.111.170.101:8091 2018-09-18 14:12:51,230 - root - INFO - ns_server @ 10.111.170.101:8091 is running 2018-09-18 14:12:51,234 - root - INFO - waiting for ns_server @ 10.111.170.102:8091 2018-09-18 14:12:51,237 - root - INFO - ns_server @ 10.111.170.102:8091 is running 2018-09-18 14:12:51,241 - root - INFO - waiting for ns_server @ 10.111.170.103:8091 2018-09-18 14:12:51,244 - root - INFO - ns_server @ 10.111.170.103:8091 is running 2018-09-18 14:12:51,249 - root - INFO - waiting for ns_server @ 10.111.170.104:8091 2018-09-18 14:12:51,252 - root - INFO - ns_server @ 10.111.170.104:8091 is running 2018-09-18 14:12:51,277 - root - INFO - cannot find service node eventing in cluster 2018-09-18 14:12:51,277 - root - WARNING - CLEANUP WAS SKIPPED Cluster instance shutdown with force 2018-09-18 14:12:51,286 - root - INFO - initializing cluster 2018-09-18 14:12:51,287 - root - INFO - connecting to 10.111.170.101 with username:root password:couchbase ssh_key: 2018-09-18 14:12:51,385 - root - INFO - Connected to 10.111.170.101 2018-09-18 14:12:51,755 - root - INFO - connecting to 10.111.170.101 with username:root password:couchbase ssh_key: 2018-09-18 14:12:51,851 - root - INFO - Connected to 10.111.170.101 2018-09-18 14:12:52,366 - root - INFO - 10.111.170.101 **** The version file /opt/couchbase/ VERSION.txt exists 2018-09-18 14:12:52,400 - root - INFO - running command.raw on 10.111.170.101: /etc/init.d/couchbase-server stop 2018-09-18 14:12:55,362 - root - INFO - command executed successfully 2018-09-18 14:12:55,363 - root - INFO - Stopping couchbase-server 2018-09-18 14:12:55,363 - root - INFO - [ OK ] 2018-09-18 14:12:55,363 - root - INFO - 2018-09-18 14:12:55,363 - root - INFO - Couchbase stopped 2018-09-18 14:12:55,438 - root - INFO - running command.raw on 10.111.170.101: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2018-09-18 14:12:55,448 - root - INFO - command executed successfully 2018-09-18 14:12:55,448 - root - INFO - running command.raw on 10.111.170.101: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2018-09-18 14:12:55,524 - root - INFO - command executed successfully 2018-09-18 14:12:55,526 - root - INFO - connecting to 10.111.170.101 with username:root password:couchbase ssh_key: 2018-09-18 14:12:55,622 - root - INFO - Connected to 10.111.170.101 2018-09-18 14:12:56,092 - root - INFO - 10.111.170.101 **** The version file /opt/couchbase/ VERSION.txt exists 2018-09-18 14:12:56,135 - root - INFO - running command.raw on 10.111.170.101: /etc/init.d/couchbase-server start 2018-09-18 14:12:58,349 - root - INFO - command executed successfully 2018-09-18 14:12:58,350 - root - INFO - Starting couchbase-server 2018-09-18 14:12:58,350 - root - INFO - [ OK ] 2018-09-18 14:12:58,350 - root - INFO - Couchbase started 2018-09-18 14:12:58,472 - root - INFO - connecting to 10.111.170.102 with username:root password:couchbase ssh_key: 2018-09-18 14:12:58,568 - root - INFO - Connected to 10.111.170.102 2018-09-18 14:12:58,960 - root - INFO - connecting to 10.111.170.102 with username:root password:couchbase ssh_key: 2018-09-18 14:12:59,057 - root - INFO - Connected to 10.111.170.102 2018-09-18 14:12:59,513 - root - INFO - 10.111.170.102 **** The version file /opt/couchbase/ VERSION.txt exists 2018-09-18 14:12:59,547 - root - INFO - running command.raw on 10.111.170.102: /etc/init.d/couchbase-server stop 2018-09-18 14:13:02,103 - root - INFO - command executed successfully 2018-09-18 14:13:02,103 - root - INFO - Stopping couchbase-server 2018-09-18 14:13:02,103 - root - INFO - [ OK ] 2018-09-18 14:13:02,103 - root - INFO - 2018-09-18 14:13:02,103 - root - INFO - Couchbase stopped 2018-09-18 14:13:02,174 - root - INFO - running command.raw on 10.111.170.102: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2018-09-18 14:13:02,182 - root - INFO - command executed successfully 2018-09-18 14:13:02,183 - root - INFO - running command.raw on 10.111.170.102: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2018-09-18 14:13:02,230 - root - INFO - command executed successfully 2018-09-18 14:13:02,231 - root - INFO - connecting to 10.111.170.102 with username:root password:couchbase ssh_key: 2018-09-18 14:13:02,332 - root - INFO - Connected to 10.111.170.102 2018-09-18 14:13:02,817 - root - INFO - 10.111.170.102 **** The version file /opt/couchbase/ VERSION.txt exists 2018-09-18 14:13:02,853 - root - INFO - running command.raw on 10.111.170.102: /etc/init.d/couchbase-server start 2018-09-18 14:13:05,051 - root - INFO - command executed successfully 2018-09-18 14:13:05,051 - root - INFO - Starting couchbase-server 2018-09-18 14:13:05,052 - root - INFO - [ OK ] 2018-09-18 14:13:05,052 - root - INFO - Couchbase started 2018-09-18 14:13:05,177 - root - INFO - connecting to 10.111.170.103 with username:root password:couchbase ssh_key: 2018-09-18 14:13:05,276 - root - INFO - Connected to 10.111.170.103 2018-09-18 14:13:05,669 - root - INFO - connecting to 10.111.170.103 with username:root password:couchbase ssh_key: 2018-09-18 14:13:05,766 - root - INFO - Connected to 10.111.170.103 2018-09-18 14:13:06,219 - root - INFO - 10.111.170.103 **** The version file /opt/couchbase/ VERSION.txt exists 2018-09-18 14:13:06,257 - root - INFO - running command.raw on 10.111.170.103: /etc/init.d/couchbase-server stop 2018-09-18 14:13:08,822 - root - INFO - command executed successfully 2018-09-18 14:13:08,822 - root - INFO - Stopping couchbase-server 2018-09-18 14:13:08,822 - root - INFO - [ OK ] 2018-09-18 14:13:08,822 - root - INFO - 2018-09-18 14:13:08,823 - root - INFO - Couchbase stopped 2018-09-18 14:13:08,948 - root - INFO - running command.raw on 10.111.170.103: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2018-09-18 14:13:08,956 - root - INFO - command executed successfully 2018-09-18 14:13:08,957 - root - INFO - running command.raw on 10.111.170.103: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2018-09-18 14:13:09,038 - root - INFO - command executed successfully 2018-09-18 14:13:09,039 - root - INFO - connecting to 10.111.170.103 with username:root password:couchbase ssh_key: 2018-09-18 14:13:09,125 - root - INFO - Connected to 10.111.170.103 2018-09-18 14:13:09,644 - root - INFO - 10.111.170.103 **** The version file /opt/couchbase/ VERSION.txt exists 2018-09-18 14:13:09,685 - root - INFO - running command.raw on 10.111.170.103: /etc/init.d/couchbase-server start 2018-09-18 14:13:11,900 - root - INFO - command executed successfully 2018-09-18 14:13:11,901 - root - INFO - Starting couchbase-server 2018-09-18 14:13:11,901 - root - INFO - [ OK ] 2018-09-18 14:13:11,901 - root - INFO - Couchbase started 2018-09-18 14:13:12,032 - root - INFO - connecting to 10.111.170.104 with username:root password:couchbase ssh_key: 2018-09-18 14:13:12,132 - root - INFO - Connected to 10.111.170.104 2018-09-18 14:13:12,589 - root - INFO - connecting to 10.111.170.104 with username:root password:couchbase ssh_key: 2018-09-18 14:13:12,688 - root - INFO - Connected to 10.111.170.104 2018-09-18 14:13:13,121 - root - INFO - 10.111.170.104 **** The version file /opt/couchbase/ VERSION.txt exists 2018-09-18 14:13:13,161 - root - INFO - running command.raw on 10.111.170.104: /etc/init.d/couchbase-server stop 2018-09-18 14:13:15,654 - root - INFO - command executed successfully 2018-09-18 14:13:15,654 - root - INFO - Stopping couchbase-server 2018-09-18 14:13:15,655 - root - INFO - [ OK ] 2018-09-18 14:13:15,655 - root - INFO - 2018-09-18 14:13:15,655 - root - INFO - Couchbase stopped 2018-09-18 14:13:15,777 - root - INFO - running command.raw on 10.111.170.104: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2018-09-18 14:13:15,789 - root - INFO - command executed successfully 2018-09-18 14:13:15,790 - root - INFO - running command.raw on 10.111.170.104: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2018-09-18 14:13:15,869 - root - INFO - command executed successfully 2018-09-18 14:13:15,872 - root - INFO - connecting to 10.111.170.104 with username:root password:couchbase ssh_key: 2018-09-18 14:13:15,968 - root - INFO - Connected to 10.111.170.104 2018-09-18 14:13:16,391 - root - INFO - 10.111.170.104 **** The version file /opt/couchbase/ VERSION.txt exists 2018-09-18 14:13:16,429 - root - INFO - running command.raw on 10.111.170.104: /etc/init.d/couchbase-server start 2018-09-18 14:13:18,646 - root - INFO - command executed successfully 2018-09-18 14:13:18,647 - root - INFO - Starting couchbase-server 2018-09-18 14:13:18,647 - root - INFO - [ OK ] 2018-09-18 14:13:18,647 - root - INFO - Couchbase started 2018-09-18 14:13:18,775 - root - INFO - sleep for 10 secs. ... 2018-09-18 14:13:29,444 - root - INFO - server: ip:10.111.170.101 port:8091 ssh_username:root, nodes/self: {'ip': u'10.111.170.101', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@10.111.170.101', 'uptime': u'28', 'mcdMemoryReserved': 797, 'storageTotalRam': 996, 'hostname': u'10.111.170.101:8091', 'storage': [], 'moxi': 11211, 'port': u'8091', 'version': u'6.0.0-1643-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 393216, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 548765696, 'memoryTotal': 1044779008, 'memoryQuota': 256, 'mcdMemoryAllocated': 797, 'os': u'x86_64-unknown-linux-gnu', 'ports': []} 2018-09-18 14:13:29,445 - root - INFO - pools/default params : memoryQuota=531 2018-09-18 14:13:29,455 - root - INFO - connecting to 10.111.170.101 with username:root password:couchbase ssh_key: 2018-09-18 14:13:29,536 - root - INFO - Connected to 10.111.170.101 2018-09-18 14:13:29,952 - root - ERROR - GET http://10.111.170.101:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 404 reason: unknown "unknown pool" auth: Administrator:password 2018-09-18 14:13:29,960 - root - INFO - Enabling diag/eval on non-local hosts is available only post 5.5.2 or 6.0 releases 2018-09-18 14:13:29,963 - root - ERROR - POST http://10.111.170.101:8091/diag/eval/ body: cluster_compat_mode:get_compat_version(). headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 400 reason: status: 400, content: API is accessible from localhost only API is accessible from localhost only auth: Administrator:password 2018-09-18 14:13:29,963 - root - INFO - /diag/eval status on 10.111.170.101:8091: False content: API is accessible from localhost only command: cluster_compat_mode:get_compat_version(). 2018-09-18 14:13:29,963 - root - INFO - /node/controller/setupServices params on 10.111.170.101: 8091:services=kv&password=password&hostname=10.111.170.101&user=Administrator 2018-09-18 14:13:29,970 - root - INFO - settings/web params on 10.111.170.101:8091:username=Administrator&password=password&port=8091 2018-09-18 14:13:30,105 - root - INFO - server: ip:10.111.170.102 port:8091 ssh_username:root, nodes/self: {'ip': u'10.111.170.102', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@10.111.170.102', 'uptime': u'23', 'mcdMemoryReserved': 797, 'storageTotalRam': 996, 'hostname': u'10.111.170.102:8091', 'storage': [], 'moxi': 11211, 'port': u'8091', 'version': u'6.0.0-1643-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 393216, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 578207744, 'memoryTotal': 1044779008, 'memoryQuota': 256, 'mcdMemoryAllocated': 797, 'os': u'x86_64-unknown-linux-gnu', 'ports': []} 2018-09-18 14:13:30,105 - root - INFO - pools/default params : memoryQuota=531 2018-09-18 14:13:30,113 - root - INFO - connecting to 10.111.170.102 with username:root password:couchbase ssh_key: 2018-09-18 14:13:30,197 - root - INFO - Connected to 10.111.170.102 2018-09-18 14:13:30,677 - root - ERROR - GET http://10.111.170.102:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 404 reason: unknown "unknown pool" auth: Administrator:password 2018-09-18 14:13:30,677 - root - INFO - Enabling diag/eval on non-local hosts is available only post 5.5.2 or 6.0 releases 2018-09-18 14:13:30,680 - root - ERROR - POST http://10.111.170.102:8091/diag/eval/ body: cluster_compat_mode:get_compat_version(). headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 400 reason: status: 400, content: API is accessible from localhost only API is accessible from localhost only auth: Administrator:password 2018-09-18 14:13:30,680 - root - INFO - /diag/eval status on 10.111.170.102:8091: False content: API is accessible from localhost only command: cluster_compat_mode:get_compat_version(). 2018-09-18 14:13:30,681 - root - INFO - settings/web params on 10.111.170.102:8091:username=Administrator&password=password&port=8091 2018-09-18 14:13:30,806 - root - INFO - server: ip:10.111.170.103 port:8091 ssh_username:root, nodes/self: {'ip': u'10.111.170.103', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@10.111.170.103', 'uptime': u'13', 'mcdMemoryReserved': 797, 'storageTotalRam': 996, 'hostname': u'10.111.170.103:8091', 'storage': [], 'moxi': 11211, 'port': u'8091', 'version': u'6.0.0-1643-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 393216, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 557838336, 'memoryTotal': 1044779008, 'memoryQuota': 256, 'mcdMemoryAllocated': 797, 'os': u'x86_64-unknown-linux-gnu', 'ports': []} 2018-09-18 14:13:30,806 - root - INFO - pools/default params : memoryQuota=531 2018-09-18 14:13:30,813 - root - INFO - connecting to 10.111.170.103 with username:root password:couchbase ssh_key: 2018-09-18 14:13:30,896 - root - INFO - Connected to 10.111.170.103 2018-09-18 14:13:31,254 - root - ERROR - GET http://10.111.170.103:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 404 reason: unknown "unknown pool" auth: Administrator:password 2018-09-18 14:13:31,254 - root - INFO - Enabling diag/eval on non-local hosts is available only post 5.5.2 or 6.0 releases 2018-09-18 14:13:31,257 - root - ERROR - POST http://10.111.170.103:8091/diag/eval/ body: cluster_compat_mode:get_compat_version(). headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 400 reason: status: 400, content: API is accessible from localhost only API is accessible from localhost only auth: Administrator:password 2018-09-18 14:13:31,257 - root - INFO - /diag/eval status on 10.111.170.103:8091: False content: API is accessible from localhost only command: cluster_compat_mode:get_compat_version(). 2018-09-18 14:13:31,257 - root - INFO - settings/web params on 10.111.170.103:8091:username=Administrator&password=password&port=8091 2018-09-18 14:13:31,381 - root - INFO - server: ip:10.111.170.104 port:8091 ssh_username:root, nodes/self: {'ip': u'10.111.170.104', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@10.111.170.104', 'uptime': u'8', 'mcdMemoryReserved': 797, 'storageTotalRam': 996, 'hostname': u'10.111.170.104:8091', 'storage': [], 'moxi': 11211, 'port': u'8091', 'version': u'6.0.0-1643-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 393216, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 573087744, 'memoryTotal': 1044779008, 'memoryQuota': 256, 'mcdMemoryAllocated': 797, 'os': u'x86_64-unknown-linux-gnu', 'ports': []} 2018-09-18 14:13:31,382 - root - INFO - pools/default params : memoryQuota=531 2018-09-18 14:13:31,391 - root - INFO - connecting to 10.111.170.104 with username:root password:couchbase ssh_key: 2018-09-18 14:13:31,471 - root - INFO - Connected to 10.111.170.104 2018-09-18 14:13:31,858 - root - ERROR - GET http://10.111.170.104:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 404 reason: unknown "unknown pool" auth: Administrator:password 2018-09-18 14:13:31,859 - root - INFO - Enabling diag/eval on non-local hosts is available only post 5.5.2 or 6.0 releases 2018-09-18 14:13:31,862 - root - ERROR - POST http://10.111.170.104:8091/diag/eval/ body: cluster_compat_mode:get_compat_version(). headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 400 reason: status: 400, content: API is accessible from localhost only API is accessible from localhost only auth: Administrator:password 2018-09-18 14:13:31,862 - root - INFO - /diag/eval status on 10.111.170.104:8091: False content: API is accessible from localhost only command: cluster_compat_mode:get_compat_version(). 2018-09-18 14:13:31,863 - root - INFO - settings/web params on 10.111.170.104:8091:username=Administrator&password=password&port=8091 2018-09-18 14:13:32,046 - root - INFO - **** add built-in 'cbadminbucket' user to node 10.111.170.101 **** 2018-09-18 14:13:32,061 - root - ERROR - DELETE http://10.111.170.101:8091/settings/rbac/users/local/cbadminbucket body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 404 reason: unknown "User was not found." auth: Administrator:password 2018-09-18 14:13:32,068 - root - INFO - Exception while deleting user. Exception is -"User was not found." 2018-09-18 14:13:32,109 - root - INFO - sleep for 10 secs. ... 2018-09-18 14:13:42,113 - root - INFO - **** add 'admin' role to 'cbadminbucket' user **** 2018-09-18 14:13:42,136 - root - INFO - sleep for 10 secs. ... 2018-09-18 14:13:52,140 - root - INFO - done initializing cluster 2018-09-18 14:13:53,115 - root - INFO - adding node 10.111.170.102:8091 to cluster 2018-09-18 14:13:53,115 - root - INFO - adding remote node @10.111.170.102:8091 to this cluster @10.111.170.101:8091 2018-09-18 14:13:55,142 - root - INFO - adding node 10.111.170.103:8091 to cluster 2018-09-18 14:13:55,142 - root - INFO - adding remote node @10.111.170.103:8091 to this cluster @10.111.170.101:8091 2018-09-18 14:13:57,184 - root - INFO - adding node 10.111.170.104:8091 to cluster 2018-09-18 14:13:57,185 - root - INFO - adding remote node @10.111.170.104:8091 to this cluster @10.111.170.101:8091 2018-09-18 14:13:59,294 - root - INFO - rebalance params : {'password': 'password', 'ejectedNodes': '', 'user': 'Administrator', 'knownNodes': u'ns_1@10.111.170.102,ns_1@10.111.170.101,ns_1@10.111.170.104,ns_1@10.111.170.103'} 2018-09-18 14:13:59,310 - root - INFO - rebalance operation started 2018-09-18 14:13:59,318 - root - INFO - rebalance percentage : 0.00 % 2018-09-18 14:13:59,318 - root - INFO - Rebalance - status: running, progress: 0.00% 2018-09-18 14:14:09,368 - root - INFO - rebalance percentage : 66.00 % 2018-09-18 14:14:09,368 - root - INFO - Rebalance - status: running, progress: 66.00% 2018-09-18 14:14:19,408 - root - INFO - Rebalance - status: none, progress: 100.00% 2018-09-18 14:14:19,415 - root - INFO - rebalancing was completed with progress: 100% in 20.1038079262 sec 2018-09-18 14:14:19,416 - root - INFO - connecting to 10.111.170.101 with username:root password:couchbase ssh_key: 2018-09-18 14:14:19,513 - root - INFO - Connected to 10.111.170.101 2018-09-18 14:14:19,931 - root - INFO - running command.raw on 10.111.170.101: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2018-09-18 14:14:20,000 - root - INFO - command executed successfully 2018-09-18 14:14:20,001 - root - INFO - Enabled diag/eval for non-local hosts from 10.111.170.101 2018-09-18 14:14:20,128 - root - INFO - ============== basetestcase setup was finished for test #1 test_eventing_lifecycle_with_couchbase_cli ============== 2018-09-18 14:14:20,137 - root - INFO - sleep for 10 secs. ... 2018-09-18 14:14:30,143 - root - INFO - connecting to 10.111.170.101 with username:root password:couchbase ssh_key: 2018-09-18 14:14:30,242 - root - INFO - Connected to 10.111.170.101 2018-09-18 14:14:41,381 - root - INFO - list of n1ql nodes in cluster: [ip:10.111.170.104 port:8091 ssh_username:root] 2018-09-18 14:14:41,391 - root - INFO - list of eventing nodes in cluster: [ip:10.111.170.102 port:8091 ssh_username:root] 2018-09-18 14:14:41,398 - root - INFO - settings/indexes params : storageMode=plasma 2018-09-18 14:14:41,407 - root - INFO - Setting the min possible memory quota so that adding mode nodes to the cluster wouldn't be a problem. 2018-09-18 14:14:41,408 - root - INFO - pools/default params : memoryQuota=330 2018-09-18 14:14:41,417 - root - INFO - pools/default params : indexMemoryQuota=256 2018-09-18 14:14:41,505 - root - INFO - pools/default params : memoryQuota=500 2018-09-18 14:14:41,518 - root - INFO - 100 2018-09-18 14:14:41,705 - root - INFO - http://10.111.170.101:8091/pools/default/buckets with param: bucketType=membase&threadsNumber=3&authType=none&compressionMode=passive&replicaIndex=1&name=src_bucket&evictionPolicy=valueOnly&flushEnabled=1&replicaNumber=1&ramQuotaMB=100 2018-09-18 14:14:41,717 - root - INFO - 0.01 seconds to create bucket src_bucket 2018-09-18 14:14:41,718 - root - INFO - waiting for memcached bucket : src_bucket in 10.111.170.101 to accept set ops 2018-09-18 14:14:43,447 - root - INFO - creating direct client 10.111.170.101:11210 src_bucket 2018-09-18 14:14:43,547 - root - INFO - creating direct client 10.111.170.101:11210 src_bucket 2018-09-18 14:14:43,652 - root - INFO - bucket 'src_bucket' was created with per node RAM quota: 100 2018-09-18 14:14:44,682 - root - INFO - http://10.111.170.101:8091/pools/default/buckets with param: bucketType=membase&threadsNumber=3&authType=none&compressionMode=passive&replicaIndex=1&name=dst_bucket&evictionPolicy=valueOnly&flushEnabled=1&replicaNumber=1&ramQuotaMB=100 2018-09-18 14:14:44,695 - root - INFO - 0.01 seconds to create bucket dst_bucket 2018-09-18 14:14:44,696 - root - INFO - waiting for memcached bucket : dst_bucket in 10.111.170.101 to accept set ops 2018-09-18 14:14:45,977 - root - INFO - creating direct client 10.111.170.101:11210 dst_bucket 2018-09-18 14:14:46,199 - root - INFO - creating direct client 10.111.170.101:11210 dst_bucket 2018-09-18 14:14:46,279 - root - INFO - bucket 'dst_bucket' was created with per node RAM quota: 100 2018-09-18 14:14:47,306 - root - INFO - http://10.111.170.101:8091/pools/default/buckets with param: bucketType=membase&threadsNumber=3&authType=none&compressionMode=passive&replicaIndex=1&name=metadata&evictionPolicy=valueOnly&flushEnabled=1&replicaNumber=1&ramQuotaMB=100 2018-09-18 14:14:47,318 - root - INFO - 0.01 seconds to create bucket metadata 2018-09-18 14:14:47,319 - root - INFO - waiting for memcached bucket : metadata in 10.111.170.101 to accept set ops 2018-09-18 14:14:48,600 - root - INFO - creating direct client 10.111.170.101:11210 metadata 2018-09-18 14:14:48,813 - root - INFO - creating direct client 10.111.170.101:11210 metadata 2018-09-18 14:14:48,890 - root - INFO - bucket 'metadata' was created with per node RAM quota: 100 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 52 100 20 100 32 11954 19127 --:--:-- --:--:-- --:--:-- 32000 2018-09-18 14:14:49,196 - root - INFO - connecting to 10.111.170.101 with username:root password:couchbase ssh_key: 2018-09-18 14:14:49,279 - root - INFO - Connected to 10.111.170.101 2018-09-18 14:14:49,730 - root - INFO - connecting to 10.111.170.103 with username:root password:couchbase ssh_key: 2018-09-18 14:14:49,857 - root - INFO - Connected to 10.111.170.103 2018-09-18 14:14:50,249 - root - INFO - running command.raw on 10.111.170.103: nproc 2018-09-18 14:14:50,331 - root - INFO - command executed successfully 2018-09-18 14:14:50,831 - root - INFO - create 2016 to src_bucket documents... 2018-09-18 14:14:50,861 - root - INFO - creating direct client 10.111.170.101:11210 src_bucket 2018-09-18 14:14:54,856 - root - INFO - LOAD IS FINISHED 2018-09-18 14:14:54,872 - root - INFO - connecting to 10.111.170.101 with username:root password:couchbase ssh_key: 2018-09-18 14:14:54,960 - root - INFO - Connected to 10.111.170.101 2018-09-18 14:14:55,424 - root - INFO - list of eventing nodes in cluster: [ip:10.111.170.102 port:8091 ssh_username:root] 2018-09-18 14:14:55,426 - root - INFO - connecting to 10.111.170.102 with username:root password:couchbase ssh_key: 2018-09-18 14:14:55,518 - root - INFO - Connected to 10.111.170.102 2018-09-18 14:14:55,952 - root - INFO - running command.raw on 10.111.170.102: echo '[{"appname":"Function_396275055_test_export_function","id":0,"depcfg":{"buckets":[{"alias":"dst_bucket","bucket_name":"dst_bucket"}],"metadata_bucket":"metadata","source_bucket":"src_bucket"},"appcode":"function OnUpdate(doc, meta) {\n dst_bucket[meta.id] = doc;\n}\nfunction OnDelete(meta) {\n}","settings":{"app_log_max_files":10,"app_log_max_size":10485760,"checkpoint_interval":10000,"cleanup_timers":false,"cpp_worker_thread_count":2,"curl_timeout":500,"dcp_stream_boundary":"everything","deadline_timeout":3,"deployment_status":true,"description":"","enable_recursive_mutation":false,"execution_timeout":1,"fuzz_offset":0,"lcb_inst_capacity":5,"log_level":"INFO","processing_status":true,"skip_timer_threshold":86400,"sock_batch_size":100,"tick_duration":60000,"timer_processing_tick_interval":500,"vb_ownership_giveup_routine_count":3,"vb_ownership_takeover_routine_count":3,"worker_count":3,"worker_queue_cap":100000,"xattr_doc_timer_entry_prune_threshold":100}}]' > /root/Function_396275055_test_export_function.json 2018-09-18 14:14:56,033 - root - INFO - command executed successfully 2018-09-18 14:14:56,035 - root - INFO - connecting to 10.111.170.102 with username:root password:couchbase ssh_key: 2018-09-18 14:14:56,134 - root - INFO - Connected to 10.111.170.102 2018-09-18 14:14:56,554 - root - INFO - /opt/couchbase/bin//couchbase-cli eventing-function-setup -c 10.111.170.102 -u Administrator -p password --import --name Function_396275055_test_export_function --file Function_396275055_test_export_function.json 2018-09-18 14:14:56,555 - root - INFO - running command.raw on 10.111.170.102: /opt/couchbase/bin//couchbase-cli eventing-function-setup -c 10.111.170.102 -u Administrator -p password --import --name Function_396275055_test_export_function --file Function_396275055_test_export_function.json 2018-09-18 14:14:57,962 - root - INFO - command executed successfully 2018-09-18 14:14:57,963 - root - INFO - couchbase-cli event-setup function import succeeded : ['SUCCESS: Events imported'] 2018-09-18 14:14:57,967 - root - INFO - connecting to 10.111.170.102 with username:root password:couchbase ssh_key: 2018-09-18 14:14:58,080 - root - INFO - Connected to 10.111.170.102 2018-09-18 14:14:58,471 - root - INFO - /opt/couchbase/bin//couchbase-cli eventing-function-setup -c 10.111.170.102 -u Administrator -p password --deploy --name Function_396275055_test_export_function 2018-09-18 14:14:58,471 - root - INFO - running command.raw on 10.111.170.102: /opt/couchbase/bin//couchbase-cli eventing-function-setup -c 10.111.170.102 -u Administrator -p password --deploy --name Function_396275055_test_export_function 2018-09-18 14:14:58,780 - root - INFO - command executed successfully 2018-09-18 14:14:58,781 - root - INFO - couchbase-cli event-setup function deploy succeeded : ['SUCCESS: Request to deploy the function was accepted'] 2018-09-18 14:14:58,795 - root - INFO - list of eventing nodes in cluster: [ip:10.111.170.102 port:8091 ssh_username:root] 2018-09-18 14:14:58,810 - root - INFO - list of eventing nodes in cluster: [ip:10.111.170.102 port:8091 ssh_username:root] 2018-09-18 14:14:58,823 - root - INFO - sleep for 30 secs. Waiting for handler code to complete bucket operations... Current : 0 Expected : 2016 ... 2018-09-18 14:15:28,849 - root - INFO - Final docs count... Current : 2016 Expected : 2016 2018-09-18 14:15:29,004 - root - INFO - Stats for Node 10.111.170.102 is [ { "event_processing_stats": { "adhoc_timer_response_received": 2, "agg_messages_sent_to_worker": 2349, "agg_queue_memory": 0, "agg_queue_memory_cap": 132120576, "agg_queue_size": 0, "agg_queue_size_cap": 300000, "agg_timer_feedback_queue_cap": 1500, "agg_timer_feedback_queue_size": 0, "dcp_mutation": 2016, "dcp_mutation_sent_to_worker": 2016, "dcp_snapshot": 8, "dcp_stream_req_counter": 135, "dcp_streamreq": 135, "execution_stats": 12, "failure_stats": 12, "latency_stats": 12, "lcb_exception_stats": 12, "log_level": 3, "thr_count": 3, "thr_map": 3, "v8_init": 3, "v8_load": 3 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "agg_queue_memory": 0, "agg_queue_size": 0, "dcp_delete_msg_counter": 0, "dcp_mutation_msg_counter": 2016, "enqueued_dcp_delete_msg_counter": 0, "enqueued_dcp_mutation_msg_counter": 2016, "enqueued_timer_msg_counter": 0, "feedback_queue_size": 0, "lcb_retry_failure": 0, "messages_parsed": 2340, "on_delete_failure": 0, "on_delete_success": 0, "on_update_failure": 0, "on_update_success": 2016, "timer_create_failure": 0, "timer_msg_counter": 0, "timer_responses_sent": 0, "timestamp": { "32755": "2018-09-18T08:45:28Z", "32756": "2018-09-18T08:45:28Z", "32757": "2018-09-18T08:45:28Z" }, "uv_try_write_failure_counter": 0 }, "failure_stats": { "app_worker_setting_events_lost": 0, "bucket_op_exception_count": 0, "checkpoint_failure_count": 0, "dcp_events_lost": 0, "debugger_events_lost": 0, "delete_events_lost": 0, "mutation_events_lost": 0, "n1ql_op_exception_count": 0, "timeout_count": 0, "timer_alarm_delete_failure": 0, "timer_context_delete_failure": 0, "timer_context_size_exceeded_counter": 0, "timer_events_lost": 0, "timestamp": { "32755": "2018-09-18T08:45:28Z", "32756": "2018-09-18T08:45:28Z", "32757": "2018-09-18T08:45:28Z" }, "v8worker_events_lost": 0 }, "function_name": "Function_396275055_test_export_function", "gocb_creds_request_counter": 8, "internal_vb_distribution_stats": { "worker_Function_396275055_test_export_function_0": "[0-44]", "worker_Function_396275055_test_export_function_1": "[45-89]", "worker_Function_396275055_test_export_function_2": "[90-134]" }, "latency_percentile_stats": { "100": 26000, "50": 2000, "80": 3000, "90": 4000, "95": 6000, "99": 11000 }, "lcb_creds_request_counter": 12, "lcb_exception_stats": {}, "metastore_stats": {}, "planner_stats": [ { "host_name": "10.111.170.102:8096", "start_vb": 0, "vb_count": 135 } ], "vb_distribution_stats_from_metadata": { "10.111.170.102:8096": { "worker_Function_396275055_test_export_function_0": "[0-44]", "worker_Function_396275055_test_export_function_1": "[45-89]", "worker_Function_396275055_test_export_function_2": "[90-134]" } }, "worker_pids": { "worker_Function_396275055_test_export_function_0": 32756, "worker_Function_396275055_test_export_function_1": 32755, "worker_Function_396275055_test_export_function_2": 32757 } } ] 2018-09-18 14:15:29,032 - root - INFO - connecting to 10.111.170.102 with username:root password:couchbase ssh_key: 2018-09-18 14:15:29,129 - root - INFO - Connected to 10.111.170.102 2018-09-18 14:15:29,561 - root - INFO - /opt/couchbase/bin//couchbase-cli eventing-function-setup -c 10.111.170.102 -u Administrator -p password --list --name Function_396275055_test_export_function 2018-09-18 14:15:29,562 - root - INFO - running command.raw on 10.111.170.102: /opt/couchbase/bin//couchbase-cli eventing-function-setup -c 10.111.170.102 -u Administrator -p password --list --name Function_396275055_test_export_function 2018-09-18 14:15:29,789 - root - INFO - command executed successfully 2018-09-18 14:15:29,790 - root - INFO - couchbase-cli event-setup function list succeeded : ['Function_396275055_test_export_function', ' Status: Deployed', ' Source Bucket: src_bucket', ' Metadata Bucket: metadata'] 2018-09-18 14:15:29,793 - root - INFO - connecting to 10.111.170.102 with username:root password:couchbase ssh_key: 2018-09-18 14:15:29,892 - root - INFO - Connected to 10.111.170.102 2018-09-18 14:15:30,229 - root - INFO - /opt/couchbase/bin//couchbase-cli eventing-function-setup -c 10.111.170.102 -u Administrator -p password --export --name Function_396275055_test_export_function --file Function_396275055_test_export_function2.json 2018-09-18 14:15:30,229 - root - INFO - running command.raw on 10.111.170.102: /opt/couchbase/bin//couchbase-cli eventing-function-setup -c 10.111.170.102 -u Administrator -p password --export --name Function_396275055_test_export_function --file Function_396275055_test_export_function2.json 2018-09-18 14:15:30,475 - root - INFO - command executed successfully 2018-09-18 14:15:30,475 - root - INFO - couchbase-cli event-setup function export succeeded : ['SUCCESS: Function exported to: Function_396275055_test_export_function2.json'] 2018-09-18 14:15:30,492 - root - INFO - connecting to 10.111.170.102 with username:root password:couchbase ssh_key: 2018-09-18 14:15:30,588 - root - INFO - Connected to 10.111.170.102 2018-09-18 14:15:30,996 - root - INFO - /opt/couchbase/bin//couchbase-cli eventing-function-setup -c 10.111.170.102 -u Administrator -p password --export-all --file export_all.json 2018-09-18 14:15:30,996 - root - INFO - running command.raw on 10.111.170.102: /opt/couchbase/bin//couchbase-cli eventing-function-setup -c 10.111.170.102 -u Administrator -p password --export-all --file export_all.json 2018-09-18 14:15:31,231 - root - INFO - command executed successfully 2018-09-18 14:15:31,231 - root - INFO - couchbase-cli event-setup function export-all succeeded : ['SUCCESS: All functions exported to: export_all.json'] 2018-09-18 14:15:31,251 - root - INFO - connecting to 10.111.170.102 with username:root password:couchbase ssh_key: 2018-09-18 14:15:31,331 - root - INFO - Connected to 10.111.170.102 2018-09-18 14:15:31,741 - root - INFO - /opt/couchbase/bin//couchbase-cli eventing-function-setup -c 10.111.170.102 -u Administrator -p password --undeploy --name Function_396275055_test_export_function 2018-09-18 14:15:31,742 - root - INFO - running command.raw on 10.111.170.102: /opt/couchbase/bin//couchbase-cli eventing-function-setup -c 10.111.170.102 -u Administrator -p password --undeploy --name Function_396275055_test_export_function 2018-09-18 14:15:32,053 - root - INFO - command executed successfully 2018-09-18 14:15:32,054 - root - INFO - couchbase-cli event-setup function undeploy succeeded : ['SUCCESS: Request to undeploy the function was accepted'] 2018-09-18 14:15:32,056 - root - INFO - sleep for 120 secs. ... 2018-09-18 14:17:32,060 - root - INFO - connecting to 10.111.170.102 with username:root password:couchbase ssh_key: 2018-09-18 14:17:32,178 - root - INFO - Connected to 10.111.170.102 2018-09-18 14:17:32,544 - root - INFO - /opt/couchbase/bin//couchbase-cli eventing-function-setup -c 10.111.170.102 -u Administrator -p password --delete --name Function_396275055_test_export_function 2018-09-18 14:17:32,544 - root - INFO - running command.raw on 10.111.170.102: /opt/couchbase/bin//couchbase-cli eventing-function-setup -c 10.111.170.102 -u Administrator -p password --delete --name Function_396275055_test_export_function 2018-09-18 14:17:32,798 - root - INFO - command executed successfully 2018-09-18 14:17:32,799 - root - INFO - couchbase-cli event-setup function delete succeeded : ['SUCCESS: Request to delete the function was accepted'] 2018-09-18 14:17:32,956 - root - INFO - list of eventing nodes in cluster: [ip:10.111.170.102 port:8091 ssh_username:root] 2018-09-18 14:17:32,957 - root - INFO - connecting to 10.111.170.102 with username:root password:couchbase ssh_key: 2018-09-18 14:17:33,057 - root - INFO - Connected to 10.111.170.102 2018-09-18 14:17:33,431 - root - INFO - /diag/eval status on 10.111.170.102:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). 2018-09-18 14:17:33,432 - root - INFO - running command.raw on 10.111.170.102: zgrep "panic" "/opt/couchbase/var/lib/couchbase/logs"/eventing.log* | wc -l 2018-09-18 14:17:33,932 - root - INFO - command executed successfully 2018-09-18 14:17:33,933 - root - INFO - running command.raw on 10.111.170.102: ls "/opt/couchbase/var/lib/couchbase/logs"/../crash/| wc -l 2018-09-18 14:17:34,018 - root - INFO - command executed successfully 2018-09-18 14:17:34,020 - root - WARNING - CLEANUP WAS SKIPPED Cluster instance shutdown with force ok ---------------------------------------------------------------------- Ran 1 test in 338.522s OK summary so far suite eventing.eventing_tools.EventingTools , pass 1 , fail 0 testrunner logs, diags and results are available under /Users/balakumaran.g/git/temp/testrunner/logs/testrunner-18-Sep-18_14-11-55/test_1 Run after suite setup for eventing.eventing_tools.EventingTools.test_eventing_lifecycle_with_couchbase_cli Thread Thread-75 was not properly terminated, will be terminated now. Thread Cluster_Thread was not properly terminated, will be terminated now. Exception OSError: OSError(3, 'No such process') in ignored Balakumarans-MacBook-Pro:testrunner balakumaran.g$