Balakumarans-MacBook-Pro:testrunner balakumaran.g$ ./testrunner -i b/temp_centos6.ini -t eventing.eventing_recovery.EventingRecovery.test_killing_eventing_consumer_when_eventing_is_processing_mutations,nodes_init=4,services_init=kv-eventing-index-n1ql,dataset=default,groups=simple,reset_services=True,skip_cleanup=True,doc-per-day=2 Global Test input params: {'cluster_name': 'temp_centos6', 'ini': 'b/temp_centos6.ini', 'num_nodes': 4} Logs will be stored at /Users/balakumaran.g/git/temp/testrunner/logs/testrunner-18-May-09_14-08-55/test_1 ./testrunner -i b/temp_centos6.ini -p -t eventing.eventing_recovery.EventingRecovery.test_killing_eventing_consumer_when_eventing_is_processing_mutations,nodes_init=4,services_init=kv-eventing-index-n1ql,dataset=default,groups=simple,reset_services=True,skip_cleanup=True,doc-per-day=2 Test Input params: {'cluster_name': 'temp_centos6', 'doc-per-day': '2', 'logs_folder': '/Users/balakumaran.g/git/temp/testrunner/logs/testrunner-18-May-09_14-08-55/test_1', 'reset_services': 'True', 'dataset': 'default', 'skip_cleanup': 'True', 'services_init': 'kv-eventing-index-n1ql', 'ini': 'b/temp_centos6.ini', 'groups': 'simple', 'case_number': 1, 'num_nodes': 4, 'nodes_init': '4'} Run before suite setup for eventing.eventing_recovery.EventingRecovery.test_killing_eventing_consumer_when_eventing_is_processing_mutations test_killing_eventing_consumer_when_eventing_is_processing_mutations (eventing.eventing_recovery.EventingRecovery) ... 2018-05-09 14:08:55 | INFO | MainProcess | test_thread | [eventing_base.setUp] Starting Test: test_killing_eventing_consumer_when_eventing_is_processing_mutations 2018-05-09 14:08:55 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 10.111.170.101 with username:root password:couchbase ssh_key: 2018-05-09 14:08:55 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 10.111.170.101 2018-05-09 14:08:56 | INFO | MainProcess | test_thread | [rest_client.get_nodes_version] Node version in cluster 5.5.0-2689-enterprise 2018-05-09 14:08:56 | INFO | MainProcess | test_thread | [rest_client.get_nodes_versions] Node versions in cluster [u'5.5.0-2689-enterprise'] 2018-05-09 14:08:56 | INFO | MainProcess | test_thread | [basetestcase.setUp] ============== basetestcase setup was started for test #1 test_killing_eventing_consumer_when_eventing_is_processing_mutations============== 2018-05-09 14:08:56 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets [] on 10.111.170.101 2018-05-09 14:08:56 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] sleep 2 seconds to make sure all buckets were deleted completely. 2018-05-09 14:08:58 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets [] on 10.111.170.102 2018-05-09 14:08:58 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] sleep 2 seconds to make sure all buckets were deleted completely. 2018-05-09 14:09:00 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets [] on 10.111.170.103 2018-05-09 14:09:00 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] sleep 2 seconds to make sure all buckets were deleted completely. 2018-05-09 14:09:02 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets [] on 10.111.170.104 2018-05-09 14:09:02 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] sleep 2 seconds to make sure all buckets were deleted completely. 2018-05-09 14:09:04 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 10 secs. ... 2018-05-09 14:09:14 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 10.111.170.101:8091 2018-05-09 14:09:14 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 10.111.170.101:8091 is running 2018-05-09 14:09:14 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 10.111.170.102:8091 2018-05-09 14:09:14 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 10.111.170.102:8091 is running 2018-05-09 14:09:14 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 10.111.170.103:8091 2018-05-09 14:09:14 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 10.111.170.103:8091 is running 2018-05-09 14:09:14 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 10.111.170.104:8091 2018-05-09 14:09:14 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 10.111.170.104:8091 is running 2018-05-09 14:09:14 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] cannot find service node eventing in cluster 2018-05-09 14:09:14 | WARNING | MainProcess | test_thread | [basetestcase.tearDown] CLEANUP WAS SKIPPED Cluster instance shutdown with force 2018-05-09 14:09:14 | INFO | MainProcess | test_thread | [basetestcase.setUp] initializing cluster 2018-05-09 14:09:14 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 10.111.170.101 with username:root password:couchbase ssh_key: 2018-05-09 14:09:14 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 10.111.170.101 2018-05-09 14:09:14 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 10.111.170.101 with username:root password:couchbase ssh_key: 2018-05-09 14:09:14 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 10.111.170.101 2018-05-09 14:09:15 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 10.111.170.101 **** The version file /opt/couchbase/ VERSION.txt exists 2018-05-09 14:09:15 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.111.170.101: /etc/init.d/couchbase-server stop 2018-05-09 14:09:17 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2018-05-09 14:09:17 | INFO | MainProcess | test_thread | [remote_util.log_command_output] Stopping couchbase-server 2018-05-09 14:09:17 | INFO | MainProcess | test_thread | [r[ OK ]il.log_command_output] 2018-05-09 14:09:17 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 2018-05-09 14:09:17 | INFO | MainProcess | test_thread | [basetestcase.stop_server] Couchbase stopped 2018-05-09 14:09:18 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.111.170.101: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2018-05-09 14:09:18 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2018-05-09 14:09:18 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.111.170.101: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2018-05-09 14:09:18 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2018-05-09 14:09:18 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 10.111.170.101 with username:root password:couchbase ssh_key: 2018-05-09 14:09:18 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 10.111.170.101 2018-05-09 14:09:18 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 10.111.170.101 **** The version file /opt/couchbase/ VERSION.txt exists 2018-05-09 14:09:18 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.111.170.101: /etc/init.d/couchbase-server start 2018-05-09 14:09:20 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2018-05-09 14:09:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] Starting couchbase-server 2018-05-09 14:09:20 | INFO | MainProcess | test_thread | [remote_util.log_command_output] [ OK ] 2018-05-09 14:09:20 | INFO | MainProcess | test_thread | [basetestcase.start_server] Couchbase started 2018-05-09 14:09:20 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 10.111.170.102 with username:root password:couchbase ssh_key: 2018-05-09 14:09:21 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 10.111.170.102 2018-05-09 14:09:21 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 10.111.170.102 with username:root password:couchbase ssh_key: 2018-05-09 14:09:21 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 10.111.170.102 2018-05-09 14:09:22 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 10.111.170.102 **** The version file /opt/couchbase/ VERSION.txt exists 2018-05-09 14:09:22 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.111.170.102: /etc/init.d/couchbase-server stop 2018-05-09 14:09:24 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2018-05-09 14:09:24 | INFO | MainProcess | test_thread | [remote_util.log_command_output] Stopping couchbase-server 2018-05-09 14:09:24 | INFO | MainProcess | test_thread | [r[ OK ]il.log_command_output] 2018-05-09 14:09:24 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 2018-05-09 14:09:24 | INFO | MainProcess | test_thread | [basetestcase.stop_server] Couchbase stopped 2018-05-09 14:09:24 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.111.170.102: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2018-05-09 14:09:24 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2018-05-09 14:09:24 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.111.170.102: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2018-05-09 14:09:24 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2018-05-09 14:09:24 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 10.111.170.102 with username:root password:couchbase ssh_key: 2018-05-09 14:09:24 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 10.111.170.102 2018-05-09 14:09:25 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 10.111.170.102 **** The version file /opt/couchbase/ VERSION.txt exists 2018-05-09 14:09:25 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.111.170.102: /etc/init.d/couchbase-server start 2018-05-09 14:09:27 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2018-05-09 14:09:27 | INFO | MainProcess | test_thread | [remote_util.log_command_output] Starting couchbase-server 2018-05-09 14:09:27 | INFO | MainProcess | test_thread | [remote_util.log_command_output] [ OK ] 2018-05-09 14:09:27 | INFO | MainProcess | test_thread | [basetestcase.start_server] Couchbase started 2018-05-09 14:09:27 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 10.111.170.103 with username:root password:couchbase ssh_key: 2018-05-09 14:09:27 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 10.111.170.103 2018-05-09 14:09:28 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 10.111.170.103 with username:root password:couchbase ssh_key: 2018-05-09 14:09:28 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 10.111.170.103 2018-05-09 14:09:28 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 10.111.170.103 **** The version file /opt/couchbase/ VERSION.txt exists 2018-05-09 14:09:28 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.111.170.103: /etc/init.d/couchbase-server stop 2018-05-09 14:09:31 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2018-05-09 14:09:31 | INFO | MainProcess | test_thread | [remote_util.log_command_output] Stopping couchbase-server 2018-05-09 14:09:31 | INFO | MainProcess | test_thread | [r[ OK ]il.log_command_output] 2018-05-09 14:09:31 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 2018-05-09 14:09:31 | INFO | MainProcess | test_thread | [basetestcase.stop_server] Couchbase stopped 2018-05-09 14:09:31 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.111.170.103: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2018-05-09 14:09:31 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2018-05-09 14:09:31 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.111.170.103: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2018-05-09 14:09:31 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2018-05-09 14:09:31 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 10.111.170.103 with username:root password:couchbase ssh_key: 2018-05-09 14:09:31 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 10.111.170.103 2018-05-09 14:09:31 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 10.111.170.103 **** The version file /opt/couchbase/ VERSION.txt exists 2018-05-09 14:09:31 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.111.170.103: /etc/init.d/couchbase-server start 2018-05-09 14:09:34 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2018-05-09 14:09:34 | INFO | MainProcess | test_thread | [remote_util.log_command_output] Starting couchbase-server 2018-05-09 14:09:34 | INFO | MainProcess | test_thread | [remote_util.log_command_output] [ OK ] 2018-05-09 14:09:34 | INFO | MainProcess | test_thread | [basetestcase.start_server] Couchbase started 2018-05-09 14:09:34 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 10.111.170.104 with username:root password:couchbase ssh_key: 2018-05-09 14:09:34 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 10.111.170.104 2018-05-09 14:09:34 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 10.111.170.104 with username:root password:couchbase ssh_key: 2018-05-09 14:09:34 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 10.111.170.104 2018-05-09 14:09:35 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 10.111.170.104 **** The version file /opt/couchbase/ VERSION.txt exists 2018-05-09 14:09:35 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.111.170.104: /etc/init.d/couchbase-server stop 2018-05-09 14:09:37 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2018-05-09 14:09:37 | INFO | MainProcess | test_thread | [remote_util.log_command_output] Stopping couchbase-server 2018-05-09 14:09:37 | INFO | MainProcess | test_thread | [r[ OK ]il.log_command_output] 2018-05-09 14:09:37 | INFO | MainProcess | test_thread | [remote_util.log_command_output] 2018-05-09 14:09:37 | INFO | MainProcess | test_thread | [basetestcase.stop_server] Couchbase stopped 2018-05-09 14:09:38 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.111.170.104: rm -rf /opt/couchbase/var/lib/couchbase/data/* 2018-05-09 14:09:38 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2018-05-09 14:09:38 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.111.170.104: rm -rf /opt/couchbase/var/lib/couchbase/config/* 2018-05-09 14:09:38 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2018-05-09 14:09:38 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 10.111.170.104 with username:root password:couchbase ssh_key: 2018-05-09 14:09:38 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 10.111.170.104 2018-05-09 14:09:38 | INFO | MainProcess | test_thread | [remote_util.is_couchbase_installed] 10.111.170.104 **** The version file /opt/couchbase/ VERSION.txt exists 2018-05-09 14:09:38 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.111.170.104: /etc/init.d/couchbase-server start 2018-05-09 14:09:40 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2018-05-09 14:09:40 | INFO | MainProcess | test_thread | [remote_util.log_command_output] Starting couchbase-server 2018-05-09 14:09:40 | INFO | MainProcess | test_thread | [remote_util.log_command_output] [ OK ] 2018-05-09 14:09:40 | INFO | MainProcess | test_thread | [basetestcase.start_server] Couchbase started 2018-05-09 14:09:40 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 10 secs. ... 2018-05-09 14:09:51 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:10.111.170.101 port:8091 ssh_username:root, nodes/self: {'ip': u'127.0.0.1', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@127.0.0.1', 'uptime': u'29', 'mcdMemoryReserved': 797, 'storageTotalRam': 996, 'hostname': u'10.111.170.101:8091', 'storage': [], 'moxi': 11211, 'port': u'8091', 'version': u'5.5.0-2689-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 327685, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 581849088, 'memoryTotal': 1044779008, 'memoryQuota': 256, 'mcdMemoryAllocated': 797, 'os': u'x86_64-unknown-linux-gnu', 'ports': []} 2018-05-09 14:09:51 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=531 2018-05-09 14:09:51 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2018-05-09 14:09:51 | INFO | MainProcess | Cluster_Thread | [rest_client.init_node_services] /node/controller/setupServices params on 10.111.170.101: 8091:services=kv&password=password&hostname=10.111.170.101&user=Administrator 2018-05-09 14:09:51 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 10.111.170.101:8091:username=Administrator&password=password&port=8091 2018-05-09 14:09:51 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:10.111.170.102 port:8091 ssh_username:root, nodes/self: {'ip': u'127.0.0.1', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@127.0.0.1', 'uptime': u'18', 'mcdMemoryReserved': 797, 'storageTotalRam': 996, 'hostname': u'10.111.170.102:8091', 'storage': [], 'moxi': 11211, 'port': u'8091', 'version': u'5.5.0-2689-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 327685, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 583720960, 'memoryTotal': 1044779008, 'memoryQuota': 256, 'mcdMemoryAllocated': 797, 'os': u'x86_64-unknown-linux-gnu', 'ports': []} 2018-05-09 14:09:51 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=531 2018-05-09 14:09:51 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2018-05-09 14:09:51 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 10.111.170.102:8091:username=Administrator&password=password&port=8091 2018-05-09 14:09:51 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:10.111.170.103 port:8091 ssh_username:root, nodes/self: {'ip': u'127.0.0.1', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@127.0.0.1', 'uptime': u'13', 'mcdMemoryReserved': 797, 'storageTotalRam': 996, 'hostname': u'10.111.170.103:8091', 'storage': [], 'moxi': 11211, 'port': u'8091', 'version': u'5.5.0-2689-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 327685, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 574038016, 'memoryTotal': 1044779008, 'memoryQuota': 256, 'mcdMemoryAllocated': 797, 'os': u'x86_64-unknown-linux-gnu', 'ports': []} 2018-05-09 14:09:51 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=531 2018-05-09 14:09:51 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2018-05-09 14:09:51 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 10.111.170.103:8091:username=Administrator&password=password&port=8091 2018-05-09 14:09:51 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:10.111.170.104 port:8091 ssh_username:root, nodes/self: {'ip': u'127.0.0.1', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@127.0.0.1', 'uptime': u'8', 'mcdMemoryReserved': 797, 'storageTotalRam': 996, 'hostname': u'10.111.170.104:8091', 'storage': [], 'moxi': 11211, 'port': u'8091', 'version': u'5.5.0-2689-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 327685, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 569151488, 'memoryTotal': 1044779008, 'memoryQuota': 256, 'mcdMemoryAllocated': 797, 'os': u'x86_64-unknown-linux-gnu', 'ports': []} 2018-05-09 14:09:51 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=531 2018-05-09 14:09:51 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2018-05-09 14:09:51 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 10.111.170.104:8091:username=Administrator&password=password&port=8091 2018-05-09 14:09:51 | INFO | MainProcess | test_thread | [basetestcase.add_built_in_server_user] **** add built-in 'cbadminbucket' user to node 10.111.170.101 **** 2018-05-09 14:09:51 | ERROR | MainProcess | test_thread | [rest_client._http_request] DELETE http://10.111.170.101:8091/settings/rbac/users/local/cbadminbucket body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 404 reason: unknown "User was not found." auth: Administrator:password 2018-05-09 14:09:51 | INFO | MainProcess | test_thread | [internal_user.delete_user] Exception while deleting user. Exception is -"User was not found." 2018-05-09 14:09:51 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 10 secs. ... 2018-05-09 14:10:01 | INFO | MainProcess | test_thread | [basetestcase.add_built_in_server_user] **** add 'admin' role to 'cbadminbucket' user **** 2018-05-09 14:10:01 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 10 secs. ... 2018-05-09 14:10:11 | INFO | MainProcess | test_thread | [basetestcase.setUp] done initializing cluster 2018-05-09 14:10:12 | INFO | MainProcess | Cluster_Thread | [task.add_nodes] adding node 10.111.170.102:8091 to cluster 2018-05-09 14:10:12 | INFO | MainProcess | Cluster_Thread | [rest_client.add_node] adding remote node @10.111.170.102:8091 to this cluster @10.111.170.101:8091 2018-05-09 14:10:15 | INFO | MainProcess | Cluster_Thread | [task.add_nodes] adding node 10.111.170.103:8091 to cluster 2018-05-09 14:10:15 | INFO | MainProcess | Cluster_Thread | [rest_client.add_node] adding remote node @10.111.170.103:8091 to this cluster @10.111.170.101:8091 2018-05-09 14:10:18 | INFO | MainProcess | Cluster_Thread | [task.add_nodes] adding node 10.111.170.104:8091 to cluster 2018-05-09 14:10:18 | INFO | MainProcess | Cluster_Thread | [rest_client.add_node] adding remote node @10.111.170.104:8091 to this cluster @10.111.170.101:8091 2018-05-09 14:10:20 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance params : {'password': 'password', 'ejectedNodes': '', 'user': 'Administrator', 'knownNodes': u'ns_1@10.111.170.102,ns_1@10.111.170.101,ns_1@10.111.170.104,ns_1@10.111.170.103'} 2018-05-09 14:10:20 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance operation started 2018-05-09 14:10:20 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 0.00 % 2018-05-09 14:10:20 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 0.00% 2018-05-09 14:10:30 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 33.00 % 2018-05-09 14:10:30 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 33.00% 2018-05-09 14:10:40 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: none, progress: 100.00% 2018-05-09 14:10:40 | INFO | MainProcess | Cluster_Thread | [task.check] rebalancing was completed with progress: 100% in 20.1323449612 sec 2018-05-09 14:10:40 | INFO | MainProcess | test_thread | [basetestcase.setUp] ============== basetestcase setup was finished for test #1 test_killing_eventing_consumer_when_eventing_is_processing_mutations ============== 2018-05-09 14:10:40 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 10 secs. ... 2018-05-09 14:10:50 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 10.111.170.101 with username:root password:couchbase ssh_key: 2018-05-09 14:10:50 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 10.111.170.101 2018-05-09 14:10:51 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of n1ql nodes in cluster: [ip:10.111.170.104 port:8091 ssh_username:root] 2018-05-09 14:10:51 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of eventing nodes in cluster: [ip:10.111.170.102 port:8091 ssh_username:root] 2018-05-09 14:10:51 | INFO | MainProcess | test_thread | [eventing_base.setUp] Setting the min possible memory quota so that adding mode nodes to the cluster wouldn't be a problem. 2018-05-09 14:10:51 | INFO | MainProcess | test_thread | [rest_client.set_service_memoryQuota] pools/default params : memoryQuota=330 2018-05-09 14:10:51 | INFO | MainProcess | test_thread | [rest_client.set_service_memoryQuota] pools/default params : indexMemoryQuota=256 2018-05-09 14:10:51 | INFO | MainProcess | test_thread | [eventing_recovery.setUp] 100 2018-05-09 14:10:52 | INFO | MainProcess | Cluster_Thread | [rest_client.get_nodes_versions] Node versions in cluster [u'5.5.0-2689-enterprise', u'5.5.0-2689-enterprise', u'5.5.0-2689-enterprise', u'5.5.0-2689-enterprise'] 2018-05-09 14:10:52 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] http://10.111.170.101:8091/pools/default/buckets with param: bucketType=membase&threadsNumber=3&authType=none&compressionMode=passive&replicaIndex=1&name=src_bucket&evictionPolicy=valueOnly&flushEnabled=1&replicaNumber=1&ramQuotaMB=100 2018-05-09 14:10:52 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] 0.01 seconds to create bucket src_bucket 2018-05-09 14:10:52 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : src_bucket in 10.111.170.101 to accept set ops 2018-05-09 14:10:54 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.111.170.101:11210 src_bucket 2018-05-09 14:10:54 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.111.170.101:11210 src_bucket 2018-05-09 14:10:56 | INFO | MainProcess | Cluster_Thread | [task.check] bucket 'src_bucket' was created with per node RAM quota: 100 2018-05-09 14:10:57 | INFO | MainProcess | Cluster_Thread | [rest_client.get_nodes_versions] Node versions in cluster [u'5.5.0-2689-enterprise', u'5.5.0-2689-enterprise', u'5.5.0-2689-enterprise', u'5.5.0-2689-enterprise'] 2018-05-09 14:10:57 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] http://10.111.170.101:8091/pools/default/buckets with param: bucketType=membase&threadsNumber=3&authType=none&compressionMode=passive&replicaIndex=1&name=dst_bucket&evictionPolicy=valueOnly&flushEnabled=1&replicaNumber=1&ramQuotaMB=100 2018-05-09 14:10:57 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] 0.01 seconds to create bucket dst_bucket 2018-05-09 14:10:57 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : dst_bucket in 10.111.170.101 to accept set ops 2018-05-09 14:10:58 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.111.170.101:11210 dst_bucket 2018-05-09 14:10:59 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.111.170.101:11210 dst_bucket 2018-05-09 14:11:01 | INFO | MainProcess | Cluster_Thread | [task.check] bucket 'dst_bucket' was created with per node RAM quota: 100 2018-05-09 14:11:02 | INFO | MainProcess | Cluster_Thread | [rest_client.get_nodes_versions] Node versions in cluster [u'5.5.0-2689-enterprise', u'5.5.0-2689-enterprise', u'5.5.0-2689-enterprise', u'5.5.0-2689-enterprise'] 2018-05-09 14:11:02 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] http://10.111.170.101:8091/pools/default/buckets with param: bucketType=membase&threadsNumber=3&authType=none&compressionMode=passive&replicaIndex=1&name=metadata&evictionPolicy=valueOnly&flushEnabled=1&replicaNumber=1&ramQuotaMB=100 2018-05-09 14:11:02 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] 0.01 seconds to create bucket metadata 2018-05-09 14:11:02 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : metadata in 10.111.170.101 to accept set ops 2018-05-09 14:11:03 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.111.170.101:11210 metadata 2018-05-09 14:11:04 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 10.111.170.101:11210 metadata 2018-05-09 14:11:06 | INFO | MainProcess | Cluster_Thread | [task.check] bucket 'metadata' was created with per node RAM quota: 100 2018-05-09 14:11:06 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of eventing nodes in cluster: [ip:10.111.170.102 port:8091 ssh_username:root] 2018-05-09 14:11:06 | INFO | MainProcess | test_thread | [eventing_base.deploy_function] Deploying the following handler code : Function_538873170_test_killing_eventing_consumer_when_eventing_is_processing_mutations 2018-05-09 14:11:06 | INFO | MainProcess | test_thread | [eventing_base.deploy_function] function OnUpdate(doc, meta) { var doc_id = meta.id; log('creating document for : ', doc); dst_bucket[doc_id] = {'doc_id' : doc_id}; // SET operation } // This is intentionally left blank function OnDelete(meta) { log('deleting document', meta.id); delete dst_bucket[meta.id]; // DELETE operation } 2018-05-09 14:11:08 | INFO | MainProcess | test_thread | [eventing_base.deploy_function] deploy Application : 2018-05-09 14:11:08 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs. Waiting for eventing node to come out of bootstrap state... ... 2018-05-09 14:11:38 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs. Waiting for eventing node to come out of bootstrap state... ... 2018-05-09 14:12:08 | INFO | MainProcess | test_thread | [basetestcase.load] create 4032 to src_bucket documents... 2018-05-09 14:12:09 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 10.111.170.101:11210 src_bucket 2018-05-09 14:12:32 | INFO | MainProcess | test_thread | [basetestcase.load] LOAD IS FINISHED 2018-05-09 14:12:32 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 10.111.170.102 with username:root password:couchbase ssh_key: 2018-05-09 14:12:32 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 10.111.170.102 2018-05-09 14:12:32 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.111.170.102: killall -9 eventing-consumer 2018-05-09 14:12:32 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2018-05-09 14:12:33 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of eventing nodes in cluster: [ip:10.111.170.102 port:8091 ssh_username:root] 2018-05-09 14:12:33 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of eventing nodes in cluster: [ip:10.111.170.102 port:8091 ssh_username:root] 2018-05-09 14:12:33 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs. Waiting for handler code to complete all bucket operations... ... 2018-05-09 14:13:03 | INFO | MainProcess | test_thread | [eventing_base.verify_eventing_results] Stats for Node 10.111.170.102 is [ { "event_processing_stats": { "AGG_MESSAGES_SENT_TO_WORKER": 2923, "DCP_MUTATION": 2902, "DCP_MUTATION_SENT_TO_WORKER": 2902, "DCP_SNAPSHOT": 715, "DCP_STREAMREQ": 734, "HANDLER_CODE": 3, "LOG_LEVEL": 3, "SOURCE_MAP": 3, "THR_COUNT": 3, "THR_MAP": 3, "V8_INIT": 3, "V8_LOAD": 3 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "timestamp": {} }, "failure_stats": { "timestamp": {} }, "function_name": "Function_538873170_test_killing_eventing_consumer_when_eventing_is_processing_mutations", "internal_vb_distribution_stats": { "worker_Function_538873170_test_killing_eventing_consumer_when_eventing_is_processing_mutations_0": "[0-2, 4-7, 10-13, 15-18, 20-21, 24, 26, 28-36, 38-48, 51-54, 56-57, 61-62, 65-68, 70-71, 73-79, 81-82, 84-85, 88, 90, 93-102, 104-111, 119-122, 124, 126, 128-134, 137, 140-143, 145-147, 150, 152, 155-157, 160, 162, 164-168, 170-171, 173-175, 177-183, 185, 188-190, 192-196, 200-210, 213-215, 217-218, 220-222, 224-229, 231, 233-234, 236-238, 240-243, 246-248, 251-253, 255-265, 267-273, 275-280, 282, 284-287, 290, 293-295, 297, 300-304, 308, 310-323, 325-327, 329-332, 334, 336-337, 339-341]", "worker_Function_538873170_test_killing_eventing_consumer_when_eventing_is_processing_mutations_1": "[342-345, 349-352, 354-363, 368-370, 372-373, 376, 378-379, 381-391, 393-394, 396, 398-399, 401-402, 404-406, 408, 410-416, 418-428, 431-432, 435-445, 448-451, 453-454, 457-466, 470-477, 480-481, 484-485, 487-491, 494-495, 498-499, 504-507, 509, 511, 514-520, 523-537, 539-542, 545, 547, 550, 552-555, 558-565, 567, 570, 573, 575-585, 591-593, 595-598, 600-604, 606-608, 610, 612-615, 617-619, 621-624, 627-635, 637, 639-641, 643-644, 646, 649, 652-657, 659-667, 670-673, 675-676, 678, 680, 682]", "worker_Function_538873170_test_killing_eventing_consumer_when_eventing_is_processing_mutations_2": "[683, 686-691, 693-694, 696-703, 705-712, 714, 716-717, 719-720, 722, 726-729, 732-735, 737-739, 741-742, 744-751, 753-757, 759-760, 762-765, 768-770, 772-775, 777, 779, 781-785, 787, 789-790, 793, 795-797, 799-801, 804, 807-811, 813, 817, 819, 823-825, 827-829, 834-835, 837-839, 841-844, 847, 849, 851-852, 854-860, 862, 864-865, 867-871, 873, 876-878, 880-881, 883-884, 886-890, 892-895, 897, 899, 901, 904-905, 907-909, 911-913, 915-916, 918-924, 926-928, 930-934, 937, 940-942, 944-945, 947-959, 961-962, 964-984, 986-987, 990-992, 995-999, 1001-1002, 1004, 1006-1009, 1011-1012, 1014-1016, 1018-1019, 1022]" }, "lcb_exception_stats": {}, "planner_stats": [ { "host_name": "10.111.170.102:8096", "start_vb": 0, "vb_count": 1024 } ], "vb_distribution_stats_from_metadata": { "10.111.170.102:8096": { "worker_Function_538873170_test_killing_eventing_consumer_when_eventing_is_processing_mutations_0": "[0-341]", "worker_Function_538873170_test_killing_eventing_consumer_when_eventing_is_processing_mutations_1": "[342-682]", "worker_Function_538873170_test_killing_eventing_consumer_when_eventing_is_processing_mutations_2": "[683-1023]" } }, "worker_pids": { "worker_Function_538873170_test_killing_eventing_consumer_when_eventing_is_processing_mutations_0": 9567, "worker_Function_538873170_test_killing_eventing_consumer_when_eventing_is_processing_mutations_1": 9572, "worker_Function_538873170_test_killing_eventing_consumer_when_eventing_is_processing_mutations_2": 9573 } } ] 2018-05-09 14:13:04 | INFO | MainProcess | test_thread | [basetestcase.load] delete 4032 to src_bucket documents... 2018-05-09 14:13:04 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 10.111.170.101:11210 src_bucket 2018-05-09 14:13:20 | INFO | MainProcess | test_thread | [basetestcase.load] LOAD IS FINISHED 2018-05-09 14:13:20 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 10.111.170.102 with username:root password:couchbase ssh_key: 2018-05-09 14:13:20 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 10.111.170.102 2018-05-09 14:13:21 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.111.170.102: killall -9 eventing-consumer 2018-05-09 14:13:21 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2018-05-09 14:13:21 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of eventing nodes in cluster: [ip:10.111.170.102 port:8091 ssh_username:root] 2018-05-09 14:13:21 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of eventing nodes in cluster: [ip:10.111.170.102 port:8091 ssh_username:root] 2018-05-09 14:13:21 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs. Waiting for handler code to complete all bucket operations... ... 2018-05-09 14:13:51 | INFO | MainProcess | test_thread | [eventing_base.verify_eventing_results] Stats for Node 10.111.170.102 is [ { "event_processing_stats": { "AGG_MESSAGES_SENT_TO_WORKER": 2570, "DCP_DELETION": 2549, "DCP_DELETION_SENT_TO_WORKER": 2549, "DCP_SNAPSHOT": 634, "DCP_STREAMREQ": 649, "HANDLER_CODE": 3, "LOG_LEVEL": 3, "SOURCE_MAP": 3, "THR_COUNT": 3, "THR_MAP": 3, "V8_INIT": 3, "V8_LOAD": 3 }, "events_remaining": { "dcp_backlog": 0 }, "execution_stats": { "timestamp": {} }, "failure_stats": { "timestamp": {} }, "function_name": "Function_538873170_test_killing_eventing_consumer_when_eventing_is_processing_mutations", "internal_vb_distribution_stats": { "worker_Function_538873170_test_killing_eventing_consumer_when_eventing_is_processing_mutations_0": "[0-1, 8, 10, 12-18, 23, 25, 27-28, 30-31, 34-35, 37-46, 48-51, 56, 58-59, 61-63, 65, 67-68, 70-71, 73-77, 79-80, 83-84, 88-89, 93, 96, 98, 100, 102-105, 108, 110, 112, 114-117, 121, 123-127, 131, 133-137, 139-140, 142, 145, 147, 149, 151-160, 162-163, 167, 172-174, 176, 178-180, 182-184, 186-187, 189-192, 196-199, 201-206, 208-209, 212-216, 218, 223, 226-239, 241-243, 246-249, 251-252, 254, 260-265, 267-270, 272-274, 276-278, 280-283, 286, 288-292, 294-301, 305-306, 309, 311, 313-316, 318, 320-325, 328-329, 334-338]", "worker_Function_538873170_test_killing_eventing_consumer_when_eventing_is_processing_mutations_1": "[343, 346, 349-357, 360-361, 366, 368-371, 376-377, 380-382, 384-385, 387-389, 391-392, 394-395, 397-398, 401-407, 411, 413-414, 416, 420-423, 426-429, 431, 433, 436-437, 439-440, 442-445, 450-453, 457-458, 460-461, 463, 465-472, 474-476, 478, 480-484, 486-487, 491-495, 497-501, 503-505, 507-509, 512, 514, 517-521, 523-526, 528-529, 531, 534-535, 537, 539, 541-542, 545-546, 548, 552, 555-556, 559, 565, 567, 569-570, 572, 575-580, 584, 586-588, 590-593, 595-597, 599, 601, 603-608, 610-624, 627, 629-630, 632, 634-635, 637-638, 640-641, 643, 645-650, 652-656, 658-661, 664, 666, 668, 671-676, 679, 681-682]", "worker_Function_538873170_test_killing_eventing_consumer_when_eventing_is_processing_mutations_2": "[683, 686-688, 690, 692-693, 696-697, 699-701, 704, 710, 712, 714, 716, 718-720, 724, 726-728, 730-731, 733, 735-736, 739, 741-743, 745, 747-751, 753-756, 758-763, 767, 770, 772-773, 776-777, 779-780, 782-787, 789-790, 792, 796, 798-799, 802-805, 808, 810-811, 813-815, 818-822, 826-827, 831-832, 834-835, 837-842, 844, 849, 851-854, 856-861, 864, 866, 868, 870-872, 874-875, 878-884, 886, 888-889, 893-895, 897-901, 905-908, 910-911, 913-918, 920-922, 924-925, 927-929, 931-932, 934, 937, 939-941, 944-945, 947-948, 951-955, 959-961, 963-978, 980-984, 987, 989-990, 992-995, 997, 1000, 1002-1003, 1006-1008, 1010-1012, 1015, 1017-1018, 1021, 1023]" }, "lcb_exception_stats": {}, "planner_stats": [ { "host_name": "10.111.170.102:8096", "start_vb": 0, "vb_count": 1024 } ], "vb_distribution_stats_from_metadata": { "10.111.170.102:8096": { "worker_Function_538873170_test_killing_eventing_consumer_when_eventing_is_processing_mutations_0": "[0-341]", "worker_Function_538873170_test_killing_eventing_consumer_when_eventing_is_processing_mutations_1": "[342-682]", "worker_Function_538873170_test_killing_eventing_consumer_when_eventing_is_processing_mutations_2": "[683-1023]" } }, "worker_pids": { "worker_Function_538873170_test_killing_eventing_consumer_when_eventing_is_processing_mutations_0": 9652, "worker_Function_538873170_test_killing_eventing_consumer_when_eventing_is_processing_mutations_1": 9651, "worker_Function_538873170_test_killing_eventing_consumer_when_eventing_is_processing_mutations_2": 9650 } } ] 2018-05-09 14:13:52 | INFO | MainProcess | test_thread | [eventing_base.undeploy_function] Undeploy Application : Function_538873170_test_killing_eventing_consumer_when_eventing_is_processing_mutations 2018-05-09 14:13:52 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 30 secs. Waiting for undeployment of function... ... 2018-05-09 14:14:22 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. ... 2018-05-09 14:14:27 | INFO | MainProcess | test_thread | [eventing_base.delete_function] Delete Application : Function_538873170_test_killing_eventing_consumer_when_eventing_is_processing_mutations 2018-05-09 14:14:27 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 60 secs. ... 2018-05-09 14:15:27 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of eventing nodes in cluster: [ip:10.111.170.102 port:8091 ssh_username:root] 2018-05-09 14:15:27 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 10.111.170.102 with username:root password:couchbase ssh_key: 2018-05-09 14:15:27 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 10.111.170.102 2018-05-09 14:15:28 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.111.170.102: ps -ef | grep eventing-consumer | grep -v grep | wc -l 2018-05-09 14:15:28 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2018-05-09 14:15:28 | INFO | MainProcess | test_thread | [eventing_base.check_if_eventing_consumers_are_cleaned_up] Node : 10.111.170.102 , eventing_consumer processes running : 0 2018-05-09 14:15:28 | INFO | MainProcess | test_thread | [basetestcase.get_nodes_from_services_map] list of eventing nodes in cluster: [ip:10.111.170.102 port:8091 ssh_username:root] 2018-05-09 14:15:28 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 10.111.170.102 with username:root password:couchbase ssh_key: 2018-05-09 14:15:28 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 10.111.170.102 2018-05-09 14:15:28 | INFO | MainProcess | test_thread | [rest_client.diag_eval] /diag/eval status on 10.111.170.102:8091: True content: "/opt/couchbase/var/lib/couchbase/logs" command: filename:absname(element(2, application:get_env(ns_server,error_logger_mf_dir))). 2018-05-09 14:15:28 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.111.170.102: zgrep "panic" "/opt/couchbase/var/lib/couchbase/logs"/eventing.log* | wc -l 2018-05-09 14:15:28 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2018-05-09 14:15:28 | INFO | MainProcess | test_thread | [eventing_base.check_eventing_logs_for_panic] ===== PANIC OBSERVED IN EVENTING LOGS ON SERVER 10.111.170.102===== 2018-05-09 14:15:28 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.111.170.102: zgrep "panic" "/opt/couchbase/var/lib/couchbase/logs"/eventing.log* 2018-05-09 14:15:28 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2018-05-09 14:15:28 | INFO | MainProcess | test_thread | [eventing_base.check_eventing_logs_for_panic] ['panic(0xc4b120, 0xc420016080)', '\t/home/couchbase/.cbdepscache/exploded/x86_64/go-1.7.6/go/src/runtime/panic.go:458 +0x243', 'panic(0xc4b120, 0xc420016080)', '\t/home/couchbase/.cbdepscache/exploded/x86_64/go-1.7.6/go/src/runtime/panic.go:458 +0x243', 'panic(0xc4b120, 0xc420016080)', '\t/home/couchbase/.cbdepscache/exploded/x86_64/go-1.7.6/go/src/runtime/panic.go:458 +0x243', 'panic(0xc4b120, 0xc420016080)', '\t/home/couchbase/.cbdepscache/exploded/x86_64/go-1.7.6/go/src/runtime/panic.go:458 +0x243', 'panic(0xc4b120, 0xc420016080)', '\t/home/couchbase/.cbdepscache/exploded/x86_64/go-1.7.6/go/src/runtime/panic.go:458 +0x243'] 2018-05-09 14:15:28 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 10.111.170.102: ls "/opt/couchbase/var/lib/couchbase/logs"/../crash/| wc -l 2018-05-09 14:15:29 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2018-05-09 14:15:29 | WARNING | MainProcess | test_thread | [basetestcase.tearDown] CLEANUP WAS SKIPPED Cluster instance shutdown with force ok ---------------------------------------------------------------------- Ran 1 test in 393.617s OK summary so far suite eventing.eventing_recovery.EventingRecovery , pass 1 , fail 0 testrunner logs, diags and results are available under /Users/balakumaran.g/git/temp/testrunner/logs/testrunner-18-May-09_14-08-55/test_1 Run after suite setup for eventing.eventing_recovery.EventingRecovery.test_killing_eventing_consumer_when_eventing_is_processing_mutations Thread Thread-19 was not properly terminated, will be terminated now. Thread Thread-7 was not properly terminated, will be terminated now. Thread Thread-55 was not properly terminated, will be terminated now. Thread Thread-31 was not properly terminated, will be terminated now. Thread Thread-69 was not properly terminated, will be terminated now. Thread Cluster_Thread was not properly terminated, will be terminated now. Thread Thread-43 was not properly terminated, will be terminated now. Exception OSError: OSError(3, 'No such process') in ignored Balakumarans-MacBook-Pro:testrunner balakumaran.g$