Started by user Ashwin G [EnvInject] - Loading node environment variables. Building remotely on slv-s11101 (P0) in workspace /data/workspace/oel6-4node-rebalance_in_testrunner Wiping out workspace first. Cloning the remote Git repository Cloning repository https://github.com/couchbase/testrunner.git > /usr/bin/git init /data/workspace/oel6-4node-rebalance_in_testrunner # timeout=10 Fetching upstream changes from https://github.com/couchbase/testrunner.git > /usr/bin/git --version # timeout=10 > /usr/bin/git fetch --tags --progress https://github.com/couchbase/testrunner.git +refs/heads/*:refs/remotes/origin/* > /usr/bin/git config remote.origin.url https://github.com/couchbase/testrunner.git # timeout=10 > /usr/bin/git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > /usr/bin/git config remote.origin.url https://github.com/couchbase/testrunner.git # timeout=10 Fetching upstream changes from https://github.com/couchbase/testrunner.git > /usr/bin/git fetch --tags --progress https://github.com/couchbase/testrunner.git +refs/heads/*:refs/remotes/origin/* > /usr/bin/git rev-parse origin/master^{commit} # timeout=10 Checking out Revision 74807aa85fc090aafd3121d64c395c67b1549f72 (origin/master) > /usr/bin/git config core.sparsecheckout # timeout=10 > /usr/bin/git checkout -f 74807aa85fc090aafd3121d64c395c67b1549f72 > /usr/bin/git rev-list 74807aa85fc090aafd3121d64c395c67b1549f72 # timeout=10 > /usr/bin/git tag -a -f -m Jenkins Build #46 jenkins-oel6-4node-rebalance_in_testrunner-46 # timeout=10 No emails were triggered. [oel6-4node-rebalance_in_testrunner] $ /bin/bash /tmp/jenkins2771279399079165691.sh test_progress_add_back_after_failover (rebalance.rebalance_progress.RebalanceProgressTests) ... filename: /tmp/test_run.conf Global Test input params: {'cluster_name': '5-centos-nodes-jython', 'conf_file': '/tmp/test_run.conf', 'get-cbcollect-info': 'True', 'ini': '/tmp/5-centos-nodes-jython.ini', 'num_nodes': 5, 'spec': 'test_run'} Logs will be stored at /data/workspace/oel6-4node-rebalance_in_testrunner/logs/testrunner-19-Nov-26_21-48-10/test_1 ./testrunner -i /tmp/5-centos-nodes-jython.ini -p get-cbcollect-info=True -t rebalance.rebalance_progress.RebalanceProgressTests.test_progress_add_back_after_failover,nodes_init=4,nodes_out=1,GROUP=P1,blob_generator=false Test Input params: {'logs_folder': '/data/workspace/oel6-4node-rebalance_in_testrunner/logs/testrunner-19-Nov-26_21-48-10/test_1', 'GROUP': 'P1', 'case_number': 1, 'conf_file': '/tmp/test_run.conf', 'spec': 'test_run', 'cluster_name': '5-centos-nodes-jython', 'num_nodes': 5, 'ini': '/tmp/5-centos-nodes-jython.ini', 'nodes_out': '1', 'get-cbcollect-info': 'True', 'blob_generator': 'false', 'nodes_init': '4'} Run before suite setup for rebalance.rebalance_progress.RebalanceProgressTests.test_progress_add_back_after_failover 2019-11-26 21:48:10 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 172.23.105.155 with username:root 2019-11-26 21:48:10 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 172.23.105.155 2019-11-26 21:48:11 | INFO | MainProcess | test_thread | [rest_client.get_nodes_version] Node version in cluster 6.5.0-4874-enterprise 2019-11-26 21:48:11 | INFO | MainProcess | test_thread | [rest_client.get_nodes_versions] Node versions in cluster [u'6.5.0-4874-enterprise'] 2019-11-26 21:48:11 | INFO | MainProcess | test_thread | [basetestcase.setUp] ============== basetestcase setup was started for test #1 test_progress_add_back_after_failover============== 2019-11-26 21:48:11 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.105.155:8091 2019-11-26 21:48:11 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.105.155:8091 is running 2019-11-26 21:48:11 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.105.159:8091 2019-11-26 21:48:11 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.105.159:8091 is running 2019-11-26 21:48:11 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.105.205:8091 2019-11-26 21:48:11 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.105.205:8091 is running 2019-11-26 21:48:11 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.105.206:8091 2019-11-26 21:48:11 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.105.206:8091 is running 2019-11-26 21:48:11 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.105.210:8091 2019-11-26 21:48:11 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.105.210:8091 is running 2019-11-26 21:48:11 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] ------- Cluster statistics ------- 2019-11-26 21:48:11 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 172.23.105.155:8091 => {'swap_mem_used': 0, 'cpu_utilization': 0.2512562814070352, 'mem_free': 3461308416, 'swap_mem_total': 3758092288, 'mem_total': 4201840640, 'services': [u'fts', u'index', u'kv', u'n1ql']} 2019-11-26 21:48:11 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] --- End of cluster statistics --- 2019-11-26 21:48:11 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ============== basetestcase cleanup was started for test #1 test_progress_add_back_after_failover ============== 2019-11-26 21:48:11 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.105.155:8091 2019-11-26 21:48:11 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.105.155:8091 is running 2019-11-26 21:48:11 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.105.159:8091 2019-11-26 21:48:11 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.105.159:8091 is running 2019-11-26 21:48:11 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.105.205:8091 2019-11-26 21:48:11 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.105.205:8091 is running 2019-11-26 21:48:11 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.105.206:8091 2019-11-26 21:48:11 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.105.206:8091 is running 2019-11-26 21:48:11 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 172.23.105.210:8091 2019-11-26 21:48:11 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 172.23.105.210:8091 is running 2019-11-26 21:48:11 | INFO | MainProcess | test_thread | [basetestcase.tearDown] ============== basetestcase cleanup was finished for test #1 test_progress_add_back_after_failover ============== Cluster instance shutdown with force 2019-11-26 21:48:11 | INFO | MainProcess | test_thread | [basetestcase.setUp] initializing cluster 2019-11-26 21:48:12 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:172.23.105.155 port:8091 ssh_username:root, nodes/self: {'ip': u'172.23.105.155', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@172.23.105.155', 'uptime': u'178', 'mcdMemoryReserved': 3205, 'storageTotalRam': 4007, 'hostname': u'172.23.105.155:8091', 'storage': [], 'moxi': 11211, 'port': u'8091', 'version': u'6.5.0-4874-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 393221, 'curr_items': 0, 'services': [u'fts', u'index', u'kv', u'n1ql'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 3458363392, 'memoryTotal': 4201840640, 'memoryQuota': 1379, 'mcdMemoryAllocated': 3205, 'os': u'x86_64-unknown-linux-gnu', 'ports': []} 2019-11-26 21:48:12 | INFO | MainProcess | Cluster_Thread | [task.execute] quota for index service will be 256 MB 2019-11-26 21:48:12 | INFO | MainProcess | Cluster_Thread | [task.execute] set index quota to node 172.23.105.155 2019-11-26 21:48:12 | INFO | MainProcess | Cluster_Thread | [rest_client.set_service_memoryQuota] pools/default params : indexMemoryQuota=256 2019-11-26 21:48:12 | INFO | MainProcess | Cluster_Thread | [task.execute] quota for fts service will be 512 MB 2019-11-26 21:48:12 | INFO | MainProcess | Cluster_Thread | [task.execute] set both index and fts quota at node 172.23.105.155 2019-11-26 21:48:12 | INFO | MainProcess | Cluster_Thread | [rest_client.set_service_memoryQuota] pools/default params : ftsMemoryQuota=512 2019-11-26 21:48:12 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=1379 2019-11-26 21:48:12 | INFO | MainProcess | Cluster_Thread | [rest_client.init_node_services] /node/controller/setupServices params on 172.23.105.155:8091:services=kv%2Cn1ql%2Cindex%2Cfts&password=password&user=Administrator&hostname=172.23.105.155 2019-11-26 21:48:12 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] POST http://172.23.105.155:8091//node/controller/setupServices body: services=kv%2Cn1ql%2Cindex%2Cfts&password=password&user=Administrator&hostname=172.23.105.155 headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 400 reason: unknown ["cannot change node services after cluster is provisioned"] auth: Administrator:password 2019-11-26 21:48:12 | INFO | MainProcess | Cluster_Thread | [rest_client.init_node_services] This node is already provisioned with services, we do not consider this as failure for test case 2019-11-26 21:48:12 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.105.155:8091:username=Administrator&password=password&port=8091 2019-11-26 21:48:12 | INFO | MainProcess | Cluster_Thread | [remote_util.__init__] connecting to 172.23.105.155 with username:root 2019-11-26 21:48:13 | INFO | MainProcess | Cluster_Thread | [remote_util.__init__] Connected to 172.23.105.155 2019-11-26 21:48:13 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.105.155: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2019-11-26 21:48:14 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully 2019-11-26 21:48:14 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.105.155:8091: True content: [6,5] command: cluster_compat_mode:get_compat_version(). 2019-11-26 21:48:14 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.105.155:8091: True content: [6,5] command: cluster_compat_mode:get_compat_version(). 2019-11-26 21:48:14 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2019-11-26 21:48:14 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:172.23.105.159 port:8091 ssh_username:root, nodes/self: {'ip': u'172.23.105.159', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@172.23.105.159', 'uptime': u'176', 'mcdMemoryReserved': 3205, 'storageTotalRam': 4007, 'hostname': u'172.23.105.159:8091', 'storage': [], 'moxi': 11211, 'port': u'8091', 'version': u'6.5.0-4874-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 393221, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 3583762432, 'memoryTotal': 4201844736, 'memoryQuota': 1738, 'mcdMemoryAllocated': 3205, 'os': u'x86_64-unknown-linux-gnu', 'ports': []} 2019-11-26 21:48:14 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.105.159:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 404 reason: unknown "unknown pool" auth: Administrator:password 2019-11-26 21:48:14 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=2147 2019-11-26 21:48:14 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.105.159:8091:username=Administrator&password=password&port=8091 2019-11-26 21:48:14 | INFO | MainProcess | Cluster_Thread | [remote_util.__init__] connecting to 172.23.105.159 with username:root 2019-11-26 21:48:14 | INFO | MainProcess | Cluster_Thread | [remote_util.__init__] Connected to 172.23.105.159 2019-11-26 21:48:15 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.105.159: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2019-11-26 21:48:15 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully 2019-11-26 21:48:15 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.105.159:8091: True content: [6,5] command: cluster_compat_mode:get_compat_version(). 2019-11-26 21:48:15 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.105.159:8091: True content: [6,5] command: cluster_compat_mode:get_compat_version(). 2019-11-26 21:48:15 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2019-11-26 21:48:15 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:172.23.105.205 port:8091 ssh_username:root, nodes/self: {'ip': u'172.23.105.205', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@172.23.105.205', 'uptime': u'181', 'mcdMemoryReserved': 3205, 'storageTotalRam': 4007, 'hostname': u'172.23.105.205:8091', 'storage': [], 'moxi': 11211, 'port': u'8091', 'version': u'6.5.0-4874-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 393221, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 3651211264, 'memoryTotal': 4201840640, 'memoryQuota': 1738, 'mcdMemoryAllocated': 3205, 'os': u'x86_64-unknown-linux-gnu', 'ports': []} 2019-11-26 21:48:15 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.105.205:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 404 reason: unknown "unknown pool" auth: Administrator:password 2019-11-26 21:48:15 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=2147 2019-11-26 21:48:15 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.105.205:8091:username=Administrator&password=password&port=8091 2019-11-26 21:48:15 | INFO | MainProcess | Cluster_Thread | [remote_util.__init__] connecting to 172.23.105.205 with username:root 2019-11-26 21:48:15 | INFO | MainProcess | Cluster_Thread | [remote_util.__init__] Connected to 172.23.105.205 2019-11-26 21:48:16 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.105.205: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2019-11-26 21:48:16 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully 2019-11-26 21:48:16 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.105.205:8091: True content: [6,5] command: cluster_compat_mode:get_compat_version(). 2019-11-26 21:48:16 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.105.205:8091: True content: [6,5] command: cluster_compat_mode:get_compat_version(). 2019-11-26 21:48:16 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2019-11-26 21:48:16 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:172.23.105.206 port:8091 ssh_username:root, nodes/self: {'ip': u'172.23.105.206', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@172.23.105.206', 'uptime': u'186', 'mcdMemoryReserved': 3205, 'storageTotalRam': 4007, 'hostname': u'172.23.105.206:8091', 'storage': [], 'moxi': 11211, 'port': u'8091', 'version': u'6.5.0-4874-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 393221, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 3634081792, 'memoryTotal': 4201844736, 'memoryQuota': 1738, 'mcdMemoryAllocated': 3205, 'os': u'x86_64-unknown-linux-gnu', 'ports': []} 2019-11-26 21:48:16 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.105.206:8091/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 404 reason: unknown "unknown pool" auth: Administrator:password 2019-11-26 21:48:16 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=2147 2019-11-26 21:48:16 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.105.206:8091:username=Administrator&password=password&port=8091 2019-11-26 21:48:16 | INFO | MainProcess | Cluster_Thread | [remote_util.__init__] connecting to 172.23.105.206 with username:root 2019-11-26 21:48:16 | INFO | MainProcess | Cluster_Thread | [remote_util.__init__] Connected to 172.23.105.206 2019-11-26 21:48:17 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.105.206: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2019-11-26 21:48:17 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully 2019-11-26 21:48:17 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.105.206:8091: True content: [6,5] command: cluster_compat_mode:get_compat_version(). 2019-11-26 21:48:17 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.105.206:8091: True content: [6,5] command: cluster_compat_mode:get_compat_version(). 2019-11-26 21:48:17 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2019-11-26 21:48:17 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:172.23.105.210 port:8091 ssh_username:root, nodes/self: {'ip': u'172.23.105.210', 'availableStorage': [], 'rest_username': '', 'id': u'ns_1@cb.local', 'uptime': u'192', 'mcdMemoryReserved': 3205, 'storageTotalRam': 4007, 'hostname': u'172.23.105.210:8091', 'storage': [], 'moxi': 11211, 'port': u'8091', 'version': u'6.5.0-4874-enterprise', 'memcached': 11210, 'status': u'healthy', 'clusterCompatibility': 393221, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 3592093696, 'memoryTotal': 4201840640, 'memoryQuota': 2147, 'mcdMemoryAllocated': 3205, 'os': u'x86_64-unknown-linux-gnu', 'ports': []} 2019-11-26 21:48:17 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=2147 2019-11-26 21:48:17 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 172.23.105.210:8091:username=Administrator&password=password&port=8091 2019-11-26 21:48:17 | INFO | MainProcess | Cluster_Thread | [remote_util.__init__] connecting to 172.23.105.210 with username:root 2019-11-26 21:48:18 | INFO | MainProcess | Cluster_Thread | [remote_util.__init__] Connected to 172.23.105.210 2019-11-26 21:48:18 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] running command.raw on 172.23.105.210: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2019-11-26 21:48:18 | INFO | MainProcess | Cluster_Thread | [remote_util.execute_command_raw] command executed successfully 2019-11-26 21:48:18 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.105.210:8091: True content: [6,5] command: cluster_compat_mode:get_compat_version(). 2019-11-26 21:48:18 | INFO | MainProcess | Cluster_Thread | [rest_client.diag_eval] /diag/eval status on 172.23.105.210:8091: True content: [6,5] command: cluster_compat_mode:get_compat_version(). 2019-11-26 21:48:18 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma 2019-11-26 21:48:19 | INFO | MainProcess | test_thread | [basetestcase.add_built_in_server_user] **** add built-in 'cbadminbucket' user to node 172.23.105.155 **** 2019-11-26 21:48:19 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 15 secs. ... 2019-11-26 21:48:34 | INFO | MainProcess | test_thread | [basetestcase.add_built_in_server_user] **** add 'admin' role to 'cbadminbucket' user **** 2019-11-26 21:48:34 | INFO | MainProcess | test_thread | [basetestcase.setUp] done initializing cluster 2019-11-26 21:48:35 | INFO | MainProcess | Cluster_Thread | [task.add_nodes] adding node 172.23.105.159:8091 to cluster 2019-11-26 21:48:35 | INFO | MainProcess | Cluster_Thread | [rest_client.add_node] adding remote node @172.23.105.159:8091 to this cluster @172.23.105.155:8091 2019-11-26 21:48:36 | INFO | MainProcess | Cluster_Thread | [task.add_nodes] adding node 172.23.105.205:8091 to cluster 2019-11-26 21:48:36 | INFO | MainProcess | Cluster_Thread | [rest_client.add_node] adding remote node @172.23.105.205:8091 to this cluster @172.23.105.155:8091 2019-11-26 21:48:38 | INFO | MainProcess | Cluster_Thread | [task.add_nodes] adding node 172.23.105.206:8091 to cluster 2019-11-26 21:48:38 | INFO | MainProcess | Cluster_Thread | [rest_client.add_node] adding remote node @172.23.105.206:8091 to this cluster @172.23.105.155:8091 2019-11-26 21:48:40 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance params : {'password': 'password', 'ejectedNodes': '', 'user': 'Administrator', 'knownNodes': u'ns_1@172.23.105.155,ns_1@172.23.105.206,ns_1@172.23.105.159,ns_1@172.23.105.205'} 2019-11-26 21:48:40 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance operation started 2019-11-26 21:48:40 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 0.00 % 2019-11-26 21:48:40 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 0.00% 2019-11-26 21:48:50 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: none, progress: 100.00% 2019-11-26 21:48:50 | INFO | MainProcess | Cluster_Thread | [task.check] rebalancing was completed with progress: 100% in 10.0378010273 sec 2019-11-26 21:48:50 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 172.23.105.155 with username:root 2019-11-26 21:48:51 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 172.23.105.155 2019-11-26 21:48:51 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] running command.raw on 172.23.105.155: curl http://Administrator:password@localhost:8091/diag/eval -X POST -d 'ns_config:set(allow_nonlocal_eval, true).' 2019-11-26 21:48:52 | INFO | MainProcess | test_thread | [remote_util.execute_command_raw] command executed successfully 2019-11-26 21:48:52 | INFO | MainProcess | test_thread | [basetestcase.enable_diag_eval_on_non_local_hosts] Enabled diag/eval for non-local hosts from 172.23.105.155 2019-11-26 21:48:52 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] http://172.23.105.155:8091/pools/default/buckets with param: bucketType=membase&evictionPolicy=valueOnly&threadsNumber=3&ramQuotaMB=1379&authType=sasl&name=default&flushEnabled=1&replicaNumber=1&compressionMode=passive&saslPassword=None&replicaIndex=1 2019-11-26 21:48:52 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] 0.01 seconds to create bucket default 2019-11-26 21:48:52 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : default in 172.23.105.155 to accept set ops 2019-11-26 21:48:53 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.105.155:11210 default 2019-11-26 21:48:53 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.105.159:11210 default 2019-11-26 21:48:53 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.105.205:11210 default 2019-11-26 21:48:53 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.105.206:11210 default 2019-11-26 21:48:53 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.105.155:11210 default 2019-11-26 21:48:53 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.105.159:11210 default 2019-11-26 21:48:53 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.105.205:11210 default 2019-11-26 21:48:53 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.105.206:11210 default 2019-11-26 21:48:53 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 0 2019-11-26 21:48:53 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : default in 172.23.105.155 to accept set ops 2019-11-26 21:48:53 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.105.155:11210 default 2019-11-26 21:48:54 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.105.159:11210 default 2019-11-26 21:48:54 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.105.205:11210 default 2019-11-26 21:48:54 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.105.206:11210 default 2019-11-26 21:48:54 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.105.155:11210 default 2019-11-26 21:48:54 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.105.159:11210 default 2019-11-26 21:48:54 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.105.205:11210 default 2019-11-26 21:48:54 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.105.206:11210 default 2019-11-26 21:48:54 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 1 2019-11-26 21:48:54 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : default in 172.23.105.155 to accept set ops 2019-11-26 21:48:54 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.105.155:11210 default 2019-11-26 21:48:54 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.105.159:11210 default 2019-11-26 21:48:54 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.105.205:11210 default 2019-11-26 21:48:55 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.105.206:11210 default 2019-11-26 21:48:55 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.105.155:11210 default 2019-11-26 21:48:55 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.105.159:11210 default 2019-11-26 21:48:55 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.105.205:11210 default 2019-11-26 21:48:55 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.105.206:11210 default 2019-11-26 21:48:55 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 2 2019-11-26 21:48:55 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : default in 172.23.105.155 to accept set ops 2019-11-26 21:48:55 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.105.155:11210 default 2019-11-26 21:48:55 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.105.159:11210 default 2019-11-26 21:48:55 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.105.205:11210 default 2019-11-26 21:48:55 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.105.206:11210 default 2019-11-26 21:48:55 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.105.155:11210 default 2019-11-26 21:48:55 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.105.159:11210 default 2019-11-26 21:48:56 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.105.205:11210 default 2019-11-26 21:48:56 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.105.206:11210 default 2019-11-26 21:48:56 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 3 2019-11-26 21:48:56 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : default in 172.23.105.155 to accept set ops 2019-11-26 21:48:56 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.105.155:11210 default 2019-11-26 21:48:56 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.105.159:11210 default 2019-11-26 21:48:56 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.105.205:11210 default 2019-11-26 21:48:56 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.105.206:11210 default 2019-11-26 21:48:56 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.105.155:11210 default 2019-11-26 21:48:56 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.105.159:11210 default 2019-11-26 21:48:56 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.105.205:11210 default 2019-11-26 21:48:56 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.105.206:11210 default 2019-11-26 21:48:56 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 4 2019-11-26 21:48:56 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : default in 172.23.105.155 to accept set ops 2019-11-26 21:48:57 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.105.155:11210 default 2019-11-26 21:48:57 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.105.159:11210 default 2019-11-26 21:48:57 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.105.205:11210 default 2019-11-26 21:48:57 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.105.206:11210 default 2019-11-26 21:48:57 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.105.155:11210 default 2019-11-26 21:48:57 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.105.159:11210 default 2019-11-26 21:48:57 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.105.205:11210 default 2019-11-26 21:48:57 | INFO | MainProcess | Cluster_Thread | [data_helper.direct_client] creating direct client 172.23.105.206:11210 default 2019-11-26 21:48:57 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 5 2019-11-26 21:48:57 | INFO | MainProcess | test_thread | [basetestcase.setUp] ============== basetestcase setup was finished for test #1 test_progress_add_back_after_failover ============== 2019-11-26 21:48:57 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] ------- Cluster statistics ------- 2019-11-26 21:48:57 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 172.23.105.155:8091 => {'swap_mem_used': 0, 'cpu_utilization': 7.360406091370558, 'mem_free': 3397148672, 'swap_mem_total': 3758092288, 'mem_total': 4201840640, 'services': [u'fts', u'index', u'kv', u'n1ql']} 2019-11-26 21:48:57 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 172.23.105.206:8091 => {'swap_mem_used': 0, 'cpu_utilization': 20.30456852791878, 'mem_free': 3564408832, 'swap_mem_total': 3758092288, 'mem_total': 4201844736, 'services': [u'kv']} 2019-11-26 21:48:57 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 172.23.105.159:8091 => {'swap_mem_used': 0, 'cpu_utilization': 3.526448362720403, 'mem_free': 3514220544, 'swap_mem_total': 3758092288, 'mem_total': 4201844736, 'services': [u'kv']} 2019-11-26 21:48:57 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 172.23.105.205:8091 => {'swap_mem_used': 0, 'cpu_utilization': 6.801007556675063, 'mem_free': 3606728704, 'swap_mem_total': 3758092288, 'mem_total': 4201840640, 'services': [u'kv']} 2019-11-26 21:48:57 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] --- End of cluster statistics --- 2019-11-26 21:48:57 | INFO | MainProcess | test_thread | [basetestcase._load_doc_data_all_buckets] create 1000 documents... 2019-11-26 21:48:57 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.105.155:11210 default 2019-11-26 21:48:57 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.105.159:11210 default 2019-11-26 21:48:57 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.105.205:11210 default 2019-11-26 21:48:58 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.105.206:11210 default 2019-11-26 21:48:58 | INFO | Process-3 | load_gen_task | [data_helper.direct_client] creating direct client 172.23.105.155:11210 default 2019-11-26 21:48:58 | INFO | Process-4 | load_gen_task | [data_helper.direct_client] creating direct client 172.23.105.155:11210 default 2019-11-26 21:48:58 | INFO | Process-5 | load_gen_task | [data_helper.direct_client] creating direct client 172.23.105.155:11210 default 2019-11-26 21:48:58 | INFO | Process-2 | load_gen_task | [data_helper.direct_client] creating direct client 172.23.105.155:11210 default 2019-11-26 21:48:58 | INFO | Process-3 | load_gen_task | [data_helper.direct_client] creating direct client 172.23.105.159:11210 default 2019-11-26 21:48:58 | INFO | Process-4 | load_gen_task | [data_helper.direct_client] creating direct client 172.23.105.159:11210 default 2019-11-26 21:48:58 | INFO | Process-5 | load_gen_task | [data_helper.direct_client] creating direct client 172.23.105.159:11210 default 2019-11-26 21:48:58 | INFO | Process-2 | load_gen_task | [data_helper.direct_client] creating direct client 172.23.105.159:11210 default 2019-11-26 21:48:59 | INFO | Process-3 | load_gen_task | [data_helper.direct_client] creating direct client 172.23.105.205:11210 default 2019-11-26 21:48:59 | INFO | Process-4 | load_gen_task | [data_helper.direct_client] creating direct client 172.23.105.205:11210 default 2019-11-26 21:48:59 | INFO | Process-2 | load_gen_task | [data_helper.direct_client] creating direct client 172.23.105.205:11210 default 2019-11-26 21:48:59 | INFO | Process-5 | load_gen_task | [data_helper.direct_client] creating direct client 172.23.105.205:11210 default 2019-11-26 21:48:59 | INFO | Process-3 | load_gen_task | [data_helper.direct_client] creating direct client 172.23.105.206:11210 default 2019-11-26 21:48:59 | INFO | Process-4 | load_gen_task | [data_helper.direct_client] creating direct client 172.23.105.206:11210 default 2019-11-26 21:48:59 | INFO | Process-2 | load_gen_task | [data_helper.direct_client] creating direct client 172.23.105.206:11210 default 2019-11-26 21:48:59 | INFO | Process-5 | load_gen_task | [data_helper.direct_client] creating direct client 172.23.105.206:11210 default 2019-11-26 21:48:59 | INFO | MainProcess | test_thread | [basetestcase.check_snap_start_corruption] Servers to check bucket-seqno: [ip:172.23.105.155 port:8091 ssh_username:root, ip:172.23.105.159 port:8091 ssh_username:root, ip:172.23.105.205 port:8091 ssh_username:root, ip:172.23.105.206 port:8091 ssh_username:root] 2019-11-26 21:48:59 | INFO | MainProcess | test_thread | [basetestcase._record_vbuckets] Record vbucket for the bucket default 2019-11-26 21:48:59 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.105.155:11210 default 2019-11-26 21:48:59 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.105.159:11210 default 2019-11-26 21:48:59 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.105.205:11210 default 2019-11-26 21:49:00 | INFO | MainProcess | test_thread | [data_helper.direct_client] creating direct client 172.23.105.206:11210 default 2019-11-26 21:49:00 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] GET http://172.23.105.155:8092//default/_design/default_view body: headers: {'Content-Type': 'application/json', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n'} error: 404 reason: not_found {"error":"not_found","reason":"missing"} auth: Administrator:password 2019-11-26 21:49:00 | INFO | MainProcess | Cluster_Thread | [rest_client._query] index query url: http://172.23.105.155:8092/default/_design/default_view/_view/default_view0?stale=ok 2019-11-26 21:49:06 | INFO | MainProcess | Cluster_Thread | [task.check] view : default_view0 was created successfully in ddoc: default_view 2019-11-26 21:49:06 | INFO | MainProcess | Cluster_Thread | [rest_client._query] index query url: http://172.23.105.155:8092/default/_design/default_view/_view/default_view1?stale=ok 2019-11-26 21:49:06 | INFO | MainProcess | Cluster_Thread | [task.check] view : default_view1 was created successfully in ddoc: default_view 2019-11-26 21:49:06 | INFO | MainProcess | Cluster_Thread | [rest_client._query] index query url: http://172.23.105.155:8092/default/_design/default_view/_view/default_view2?stale=ok 2019-11-26 21:49:06 | INFO | MainProcess | Cluster_Thread | [task.check] view : default_view2 was created successfully in ddoc: default_view 2019-11-26 21:49:07 | INFO | MainProcess | Cluster_Thread | [rest_client._query] index query url: http://172.23.105.155:8092/default/_design/default_view/_view/default_view0?stale=false&limit=1000 2019-11-26 21:49:08 | INFO | MainProcess | Cluster_Thread | [rest_client._query] index query url: http://172.23.105.155:8092/default/_design/default_view/_view/default_view1?stale=false&limit=1000 2019-11-26 21:49:09 | INFO | MainProcess | Cluster_Thread | [rest_client._query] index query url: http://172.23.105.155:8092/default/_design/default_view/_view/default_view2?stale=false&limit=1000 2019-11-26 21:49:10 | INFO | MainProcess | Cluster_Thread | [task._failover_nodes] Failing over 172.23.105.206:8091 with graceful=False 2019-11-26 21:49:10 | INFO | MainProcess | Cluster_Thread | [rest_client.fail_over] fail_over node ns_1@172.23.105.206 successful 2019-11-26 21:49:10 | INFO | MainProcess | Cluster_Thread | [task.execute] 0 seconds sleep after failover, for nodes to go pending.... 2019-11-26 21:49:10 | INFO | MainProcess | test_thread | [rest_client.add_back_node] add_back_node ns_1@172.23.105.206 successful 2019-11-26 21:49:10 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 5 secs. wait for rebalance start ... 2019-11-26 21:49:11 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance params : {'password': 'password', 'ejectedNodes': '', 'user': 'Administrator', 'knownNodes': u'ns_1@172.23.105.155,ns_1@172.23.105.206,ns_1@172.23.105.159,ns_1@172.23.105.205'} 2019-11-26 21:49:11 | INFO | MainProcess | Cluster_Thread | [rest_client.rebalance] rebalance operation started 2019-11-26 21:49:11 | INFO | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] rebalance percentage : 0.00 % 2019-11-26 21:49:11 | INFO | MainProcess | Cluster_Thread | [task.check] Rebalance - status: running, progress: 0.00% 2019-11-26 21:49:15 | INFO | MainProcess | test_thread | [rebalance_progress._check_stats] Checked stat: {'buckets_count': 1, 'bucket': u'default', u'172.23.105.159': {u'outgoing': {u'activeVBucketsLeft': 85, u'docsTotal': 162, u'docsTransferred': 0, u'replicaVBucketsLeft': 0}, u'ingoing': {u'activeVBucketsLeft': 0, u'docsTotal': 0, u'docsTransferred': 0, u'replicaVBucketsLeft': 85}}, u'172.23.105.155': {u'outgoing': {u'activeVBucketsLeft': 85, u'docsTotal': 168, u'docsTransferred': 0, u'replicaVBucketsLeft': 0}, u'ingoing': {u'activeVBucketsLeft': 0, u'docsTotal': 0, u'docsTransferred': 0, u'replicaVBucketsLeft': 85}}, u'172.23.105.205': {u'outgoing': {u'activeVBucketsLeft': 86, u'docsTotal': 156, u'docsTransferred': 0, u'replicaVBucketsLeft': 0}, u'ingoing': {u'activeVBucketsLeft': 0, u'docsTotal': 0, u'docsTransferred': 0, u'replicaVBucketsLeft': 86}}, u'172.23.105.206': {u'outgoing': {u'activeVBucketsLeft': 0, u'docsTotal': 0, u'docsTransferred': 0, u'replicaVBucketsLeft': 0}, u'ingoing': {u'activeVBucketsLeft': 256, u'docsTotal': 486, u'docsTransferred': 0, u'replicaVBucketsLeft': 256}}} 2019-11-26 21:49:15 | INFO | MainProcess | test_thread | [rebalance_progress._check_stats] Checked stat: {'buckets_count': 1, 'bucket': u'default', u'172.23.105.159': {u'outgoing': {u'activeVBucketsLeft': 85, u'docsTotal': 162, u'docsTransferred': 0, u'replicaVBucketsLeft': 0}, u'ingoing': {u'activeVBucketsLeft': 0, u'docsTotal': 0, u'docsTransferred': 0, u'replicaVBucketsLeft': 85}}, u'172.23.105.155': {u'outgoing': {u'activeVBucketsLeft': 85, u'docsTotal': 168, u'docsTransferred': 0, u'replicaVBucketsLeft': 0}, u'ingoing': {u'activeVBucketsLeft': 0, u'docsTotal': 0, u'docsTransferred': 0, u'replicaVBucketsLeft': 85}}, u'172.23.105.205': {u'outgoing': {u'activeVBucketsLeft': 86, u'docsTotal': 156, u'docsTransferred': 0, u'replicaVBucketsLeft': 0}, u'ingoing': {u'activeVBucketsLeft': 0, u'docsTotal': 0, u'docsTransferred': 0, u'replicaVBucketsLeft': 86}}, u'172.23.105.206': {u'outgoing': {u'activeVBucketsLeft': 0, u'docsTotal': 0, u'docsTransferred': 0, u'replicaVBucketsLeft': 0}, u'ingoing': {u'activeVBucketsLeft': 256, u'docsTotal': 486, u'docsTransferred': 0, u'replicaVBucketsLeft': 256}}} 2019-11-26 21:49:15 | INFO | MainProcess | test_thread | [rebalance_progress._check_stats] Checked stat: {'buckets_count': 1, 'bucket': u'default', u'172.23.105.159': {u'outgoing': {u'activeVBucketsLeft': 85, u'docsTotal': 162, u'docsTransferred': 0, u'replicaVBucketsLeft': 0}, u'ingoing': {u'activeVBucketsLeft': 0, u'docsTotal': 0, u'docsTransferred': 0, u'replicaVBucketsLeft': 85}}, u'172.23.105.155': {u'outgoing': {u'activeVBucketsLeft': 85, u'docsTotal': 168, u'docsTransferred': 0, u'replicaVBucketsLeft': 0}, u'ingoing': {u'activeVBucketsLeft': 0, u'docsTotal': 0, u'docsTransferred': 0, u'replicaVBucketsLeft': 85}}, u'172.23.105.205': {u'outgoing': {u'activeVBucketsLeft': 86, u'docsTotal': 156, u'docsTransferred': 0, u'replicaVBucketsLeft': 0}, u'ingoing': {u'activeVBucketsLeft': 0, u'docsTotal': 0, u'docsTransferred': 0, u'replicaVBucketsLeft': 86}}, u'172.23.105.206': {u'outgoing': {u'activeVBucketsLeft': 0, u'docsTotal': 0, u'docsTransferred': 0, u'replicaVBucketsLeft': 0}, u'ingoing': {u'activeVBucketsLeft': 256, u'docsTotal': 486, u'docsTransferred': 0, u'replicaVBucketsLeft': 256}}} 2019-11-26 21:49:15 | INFO | MainProcess | test_thread | [rebalance_progress._check_stats] Checked stat: {'buckets_count': 1, 'bucket': u'default', u'172.23.105.159': {u'outgoing': {u'activeVBucketsLeft': 85, u'docsTotal': 162, u'docsTransferred': 0, u'replicaVBucketsLeft': 0}, u'ingoing': {u'activeVBucketsLeft': 0, u'docsTotal': 0, u'docsTransferred': 0, u'replicaVBucketsLeft': 85}}, u'172.23.105.155': {u'outgoing': {u'activeVBucketsLeft': 85, u'docsTotal': 168, u'docsTransferred': 0, u'replicaVBucketsLeft': 0}, u'ingoing': {u'activeVBucketsLeft': 0, u'docsTotal': 0, u'docsTransferred': 0, u'replicaVBucketsLeft': 85}}, u'172.23.105.205': {u'outgoing': {u'activeVBucketsLeft': 86, u'docsTotal': 156, u'docsTransferred': 0, u'replicaVBucketsLeft': 0}, u'ingoing': {u'activeVBucketsLeft': 0, u'docsTotal': 0, u'docsTransferred': 0, u'replicaVBucketsLeft': 86}}, u'172.23.105.206': {u'outgoing': {u'activeVBucketsLeft': 0, u'docsTotal': 0, u'docsTransferred': 0, u'replicaVBucketsLeft': 0}, u'ingoing': {u'activeVBucketsLeft': 256, u'docsTotal': 486, u'docsTransferred': 0, u'replicaVBucketsLeft': 256}}} 2019-11-26 21:49:16 | INFO | MainProcess | test_thread | [rebalance_progress.test_progress_add_back_after_failover] Got empty progress 2019-11-26 21:49:21 | ERROR | MainProcess | Cluster_Thread | [rest_client._rebalance_status_and_progress] {u'status': u'none', u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.'} - rebalance failed 2019-11-26 21:49:21 | INFO | MainProcess | Cluster_Thread | [rest_client.print_UI_logs] Latest logs from UI on 172.23.105.155: 2019-11-26 21:49:21 | ERROR | MainProcess | Cluster_Thread | [rest_client.print_UI_logs] {u'node': u'ns_1@172.23.105.155', u'code': 0, u'text': u'Rebalance exited with reason {mover_crashed,\n {unexpected_exit,\n {\'EXIT\',<0.10650.1>,\n {{error,\n {badrpc,\n {\'EXIT\',\n {{{{badmatch,{error,dcp_conn_closed}},\n [{couch_set_view_group,\n process_monitor_partition_update,4,\n [{file,\n "/home/couchbase/jenkins/workspace/couchbase-server-unix/couchdb/src/couch_set_view/src/couch_set_view_group.erl"},\n {line,3725}]},\n {couch_set_view_group,handle_call,3,\n [{file,\n "/home/couchbase/jenkins/workspace/couchbase-server-unix/couchdb/src/couch_set_view/src/couch_set_view_group.erl"},\n {line,934}]},\n {gen_server,try_handle_call,4,\n [{file,"gen_server.erl"},{line,636}]},\n {gen_server,handle_msg,6,\n [{file,"gen_server.erl"},{line,665}]},\n {proc_lib,init_p_do_apply,3,\n [{file,"proc_lib.erl"},{line,247}]}]},\n {gen_server,call,\n [<12939.514.0>,\n {monitor_partition_update,1020,\n #Ref<12939.176846531.2340945921.119382>,\n <12939.564.0>},\n infinity]}},\n {gen_server,call,\n [\'capi_set_view_manager-default\',\n {wait_index_updated,1020},\n infinity]}}}}},\n {gen_server,call,\n [{\'janitor_agent-default\',\n \'ns_1@172.23.105.206\'},\n {if_rebalance,<0.10461.1>,\n {wait_index_updated,1020}},\n infinity]}}}}}.\nRebalance Operation Id = f8d81e60511090035968902b399f19b6', u'shortText': u'message', u'serverTime': u'2019-11-26T21:49:15.940Z', u'module': u'ns_orchestrator', u'tstamp': 1574833755940, u'type': u'critical'} 2019-11-26 21:49:21 | ERROR | MainProcess | Cluster_Thread | [rest_client.print_UI_logs] {u'node': u'ns_1@172.23.105.155', u'code': 0, u'text': u'Worker <0.10489.1> (for action {move,{1020,\n [\'ns_1@172.23.105.205\',undefined],\n [\'ns_1@172.23.105.206\',\n \'ns_1@172.23.105.205\'],\n []}}) exited with reason {unexpected_exit,\n {\'EXIT\',\n <0.10650.1>,\n {{error,\n {badrpc,\n {\'EXIT\',\n {{{{badmatch,\n {error,\n dcp_conn_closed}},\n [{couch_set_view_group,\n process_monitor_partition_update,\n 4,\n [{file,\n "/home/couchbase/jenkins/workspace/couchbase-server-unix/couchdb/src/couch_set_view/src/couch_set_view_group.erl"},\n {line,\n 3725}]},\n {couch_set_view_group,\n handle_call,\n 3,\n [{file,\n "/home/couchbase/jenkins/workspace/couchbase-server-unix/couchdb/src/couch_set_view/src/couch_set_view_group.erl"},\n {line,\n 934}]},\n {gen_server,\n try_handle_call,\n 4,\n [{file,\n "gen_server.erl"},\n {line,\n 636}]},\n {gen_server,\n handle_msg,\n 6,\n [{file,\n "gen_server.erl"},\n {line,\n 665}]},\n {proc_lib,\n init_p_do_apply,\n 3,\n [{file,\n "proc_lib.erl"},\n {line,\n 247}]}]},\n {gen_server,\n call,\n [<12939.514.0>,\n {monitor_partition_update,\n 1020,\n #Ref<12939.176846531.2340945921.119382>,\n <12939.564.0>},\n infinity]}},\n {gen_server,\n call,\n [\'capi_set_view_manager-default\',\n {wait_index_updated,\n 1020},\n infinity]}}}}},\n {gen_server,\n call,\n [{\'janitor_agent-default\',\n \'ns_1@172.23.105.206\'},\n {if_rebalance,\n <0.10461.1>,\n {wait_index_updated,\n 1020}},\n infinity]}}}}', u'shortText': u'message', u'serverTime': u'2019-11-26T21:49:15.936Z', u'module': u'ns_vbucket_mover', u'tstamp': 1574833755936, u'type': u'critical'} 2019-11-26 21:49:21 | ERROR | MainProcess | Cluster_Thread | [rest_client.print_UI_logs] {u'node': u'ns_1@172.23.105.155', u'code': 0, u'text': u'Bucket "default" rebalance does not seem to be swap rebalance', u'shortText': u'message', u'serverTime': u'2019-11-26T21:49:12.965Z', u'module': u'ns_vbucket_mover', u'tstamp': 1574833752965, u'type': u'info'} 2019-11-26 21:49:21 | ERROR | MainProcess | Cluster_Thread | [rest_client.print_UI_logs] {u'node': u'ns_1@172.23.105.155', u'code': 0, u'text': u'Started rebalancing bucket default', u'shortText': u'message', u'serverTime': u'2019-11-26T21:49:11.880Z', u'module': u'ns_rebalancer', u'tstamp': 1574833751880, u'type': u'info'} 2019-11-26 21:49:21 | ERROR | MainProcess | Cluster_Thread | [rest_client.print_UI_logs] {u'node': u'ns_1@172.23.105.206', u'code': 0, u'text': u'Deleting old data files of bucket "default"', u'shortText': u'message', u'serverTime': u'2019-11-26T21:49:11.812Z', u'module': u'ns_storage_conf', u'tstamp': 1574833751812, u'type': u'info'} 2019-11-26 21:49:21 | ERROR | MainProcess | Cluster_Thread | [rest_client.print_UI_logs] {u'node': u'ns_1@172.23.105.155', u'code': 0, u'text': u"Starting rebalance, KeepNodes = ['ns_1@172.23.105.155','ns_1@172.23.105.206',\n 'ns_1@172.23.105.159','ns_1@172.23.105.205'], EjectNodes = [], Failed over and being ejected nodes = []; no delta recovery nodes; Operation Id = f8d81e60511090035968902b399f19b6", u'shortText': u'message', u'serverTime': u'2019-11-26T21:49:11.711Z', u'module': u'ns_orchestrator', u'tstamp': 1574833751711, u'type': u'info'} 2019-11-26 21:49:21 | ERROR | MainProcess | Cluster_Thread | [rest_client.print_UI_logs] {u'node': u'ns_1@172.23.105.155', u'code': 0, u'text': u'Failover completed successfully.\nRebalance Operation Id = de1a6d32db9156b48985f3bbb7446905', u'shortText': u'message', u'serverTime': u'2019-11-26T21:49:10.686Z', u'module': u'ns_orchestrator', u'tstamp': 1574833750686, u'type': u'info'} 2019-11-26 21:49:21 | ERROR | MainProcess | Cluster_Thread | [rest_client.print_UI_logs] {u'node': u'ns_1@172.23.105.155', u'code': 0, u'text': u"Deactivating failed over nodes ['ns_1@172.23.105.206']", u'shortText': u'message', u'serverTime': u'2019-11-26T21:49:10.681Z', u'module': u'failover', u'tstamp': 1574833750681, u'type': u'info'} 2019-11-26 21:49:21 | ERROR | MainProcess | Cluster_Thread | [rest_client.print_UI_logs] {u'node': u'ns_1@172.23.105.155', u'code': 0, u'text': u"Failed over ['ns_1@172.23.105.206']: ok", u'shortText': u'message', u'serverTime': u'2019-11-26T21:49:10.669Z', u'module': u'failover', u'tstamp': 1574833750669, u'type': u'info'} 2019-11-26 21:49:21 | ERROR | MainProcess | Cluster_Thread | [rest_client.print_UI_logs] {u'node': u'ns_1@172.23.105.206', u'code': 0, u'text': u'Shutting down bucket "default" on \'ns_1@172.23.105.206\' for deletion', u'shortText': u'message', u'serverTime': u'2019-11-26T21:49:10.588Z', u'module': u'ns_memcached', u'tstamp': 1574833750588, u'type': u'info'} ERROR [('/usr/lib64/python2.7/threading.py', 785, '__bootstrap', 'self.__bootstrap_inner()'), ('/usr/lib64/python2.7/threading.py', 812, '__bootstrap_inner', 'self.run()'), ('lib/tasks/taskmanager.py', 31, 'run', 'task.step(self)'), ('lib/tasks/task.py', 82, 'step', 'self.check(task_manager)'), ('lib/tasks/task.py', 544, 'check', 'self.set_exception(ex)'), ('lib/tasks/future.py', 264, 'set_exception', 'print traceback.extract_stack()')] Tue Nov 26 21:49:21 2019 [('/usr/lib64/python2.7/threading.py', 785, '__bootstrap', 'self.__bootstrap_inner()'), ('/usr/lib64/python2.7/threading.py', 812, '__bootstrap_inner', 'self.run()'), ('testrunner.py', 297, 'run', '**self._Thread__kwargs)'), ('/usr/lib64/python2.7/unittest/runner.py', 151, 'run', 'test(result)'), ('/usr/lib64/python2.7/unittest/case.py', 433, '__call__', 'return self.run(*args, **kwds)'), ('/usr/lib64/python2.7/unittest/case.py', 369, 'run', 'testMethod()'), ('pytests/rebalance/rebalance_progress.py', 139, 'test_progress_add_back_after_failover', 'rebalance.result()'), ('lib/tasks/future.py', 160, 'result', 'return self.__get_result()'), ('lib/tasks/future.py', 111, '__get_result', 'print traceback.extract_stack()')] 2019-11-26 21:49:21 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] ------- Cluster statistics ------- 2019-11-26 21:49:21 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 172.23.105.155:8091 => {'swap_mem_used': 0, 'cpu_utilization': 1.269035532994924, 'mem_free': 3365724160, 'swap_mem_total': 3758092288, 'mem_total': 4201840640, 'services': [u'fts', u'index', u'kv', u'n1ql']} 2019-11-26 21:49:21 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 172.23.105.206:8091 => {'swap_mem_used': 0, 'cpu_utilization': 1.00250626566416, 'mem_free': 3529691136, 'swap_mem_total': 3758092288, 'mem_total': 4201844736, 'services': [u'kv']} 2019-11-26 21:49:21 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 172.23.105.159:8091 => {'swap_mem_used': 0, 'cpu_utilization': 11.58690176322418, 'mem_free': 3470462976, 'swap_mem_total': 3758092288, 'mem_total': 4201844736, 'services': [u'kv']} 2019-11-26 21:49:21 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] 172.23.105.205:8091 => {'swap_mem_used': 0, 'cpu_utilization': 7.556675062972293, 'mem_free': 3522973696, 'swap_mem_total': 3758092288, 'mem_total': 4201840640, 'services': [u'kv']} 2019-11-26 21:49:21 | INFO | MainProcess | test_thread | [basetestcase.print_cluster_stats] --- End of cluster statistics --- 2019-11-26 21:49:21 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 172.23.105.155 with username:root 2019-11-26 21:49:22 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 172.23.105.155 2019-11-26 21:49:23 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 172.23.105.159 with username:root 2019-11-26 21:49:23 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 172.23.105.159 2019-11-26 21:49:24 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 172.23.105.205 with username:root 2019-11-26 21:49:24 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 172.23.105.205 2019-11-26 21:49:25 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 172.23.105.206 with username:root 2019-11-26 21:49:25 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 172.23.105.206 2019-11-26 21:49:26 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 172.23.105.210 with username:root 2019-11-26 21:49:26 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 172.23.105.210 ====================================================================== ERROR: test_progress_add_back_after_failover (rebalance.rebalance_progress.RebalanceProgressTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "pytests/rebalance/rebalance_progress.py", line 139, in test_progress_add_back_after_failover rebalance.result() File "lib/tasks/future.py", line 160, in result return self.__get_result() File "lib/tasks/future.py", line 112, in __get_result raise self._exception RebalanceFailedException: Rebalance Failed: {u'status': u'none', u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.'} - rebalance failed ---------------------------------------------------------------------- Ran 1 test in 77.390s FAILED (errors=1) Cluster instance shutdown with force grabbing cbcollect from 172.23.105.155 2019-11-26 21:49:27 | INFO | MainProcess | MainThread | [remote_util.__init__] connecting to 172.23.105.155 with username:root 2019-11-26 21:49:28 | INFO | MainProcess | MainThread | [remote_util.__init__] Connected to 172.23.105.155 Collecting logs from 172.23.105.155 2019-11-26 21:49:28 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.105.155: /opt/couchbase/bin/cbcollect_info 172.23.105.155-20191126-2149-diag.zip 2019-11-26 21:51:43 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully Downloading zipped logs from 172.23.105.155 2019-11-26 21:51:44 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.105.155: rm -f /root/172.23.105.155-20191126-2149-diag.zip 2019-11-26 21:51:44 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully grabbing cbcollect from 172.23.105.159 2019-11-26 21:51:44 | INFO | MainProcess | MainThread | [remote_util.__init__] connecting to 172.23.105.159 with username:root 2019-11-26 21:51:44 | INFO | MainProcess | MainThread | [remote_util.__init__] Connected to 172.23.105.159 Collecting logs from 172.23.105.159 2019-11-26 21:51:45 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.105.159: /opt/couchbase/bin/cbcollect_info 172.23.105.159-20191126-2151-diag.zip 2019-11-26 21:53:16 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully Downloading zipped logs from 172.23.105.159 2019-11-26 21:53:17 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.105.159: rm -f /root/172.23.105.159-20191126-2151-diag.zip 2019-11-26 21:53:17 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully grabbing cbcollect from 172.23.105.205 2019-11-26 21:53:17 | INFO | MainProcess | MainThread | [remote_util.__init__] connecting to 172.23.105.205 with username:root 2019-11-26 21:53:17 | INFO | MainProcess | MainThread | [remote_util.__init__] Connected to 172.23.105.205 Collecting logs from 172.23.105.205 2019-11-26 21:53:18 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.105.205: /opt/couchbase/bin/cbcollect_info 172.23.105.205-20191126-2153-diag.zip 2019-11-26 21:54:49 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully Downloading zipped logs from 172.23.105.205 2019-11-26 21:54:49 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.105.205: rm -f /root/172.23.105.205-20191126-2153-diag.zip 2019-11-26 21:54:50 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully grabbing cbcollect from 172.23.105.206 2019-11-26 21:54:50 | INFO | MainProcess | MainThread | [remote_util.__init__] connecting to 172.23.105.206 with username:root 2019-11-26 21:54:50 | INFO | MainProcess | MainThread | [remote_util.__init__] Connected to 172.23.105.206 Collecting logs from 172.23.105.206 2019-11-26 21:54:51 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.105.206: /opt/couchbase/bin/cbcollect_info 172.23.105.206-20191126-2154-diag.zip 2019-11-26 21:56:05 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully Downloading zipped logs from 172.23.105.206 2019-11-26 21:56:05 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.105.206: rm -f /root/172.23.105.206-20191126-2154-diag.zip 2019-11-26 21:56:05 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully grabbing cbcollect from 172.23.105.210 2019-11-26 21:56:05 | INFO | MainProcess | MainThread | [remote_util.__init__] connecting to 172.23.105.210 with username:root 2019-11-26 21:56:06 | INFO | MainProcess | MainThread | [remote_util.__init__] Connected to 172.23.105.210 Collecting logs from 172.23.105.210 2019-11-26 21:56:06 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.105.210: /opt/couchbase/bin/cbcollect_info 172.23.105.210-20191126-2156-diag.zip 2019-11-26 21:57:19 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully Downloading zipped logs from 172.23.105.210 2019-11-26 21:57:19 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] running command.raw on 172.23.105.210: rm -f /root/172.23.105.210-20191126-2156-diag.zip 2019-11-26 21:57:19 | INFO | MainProcess | MainThread | [remote_util.execute_command_raw] command executed successfully summary so far suite rebalance.rebalance_progress.RebalanceProgressTests , pass 0 , fail 1 failures so far... rebalance.rebalance_progress.RebalanceProgressTests.test_progress_add_back_after_failover testrunner logs, diags and results are available under /data/workspace/oel6-4node-rebalance_in_testrunner/logs/testrunner-19-Nov-26_21-48-10/test_1 Run after suite setup for rebalance.rebalance_progress.RebalanceProgressTests.test_progress_add_back_after_failover Archiving artifacts Recording test results Build step 'Publish JUnit test result report' changed build result to UNSTABLE [description-setter] Could not determine description. [BFA] Scanning build for known causes... [BFA] No failure causes found [BFA] Done. 0s Notifying upstream projects of job completion Email was triggered for: Unstable (Test Failures) Sending email for trigger: Unstable (Test Failures) Request made to compress build log Sending email to: ashwin.govindarajulu@couchbase.com