Test Blocker Description
Observed on latest spock manifest bucke warmup is taking a long time and tests are timing out.
make test-viewmerge
|
echo "Running view merge tests with single node cluster"
|
Running view merge tests with single node cluster
|
scripts/start_cluster_and_run_tests.sh b/resources/dev-single-node.ini conf/view-conf/py-viewmerge.conf
|
~/dev/spock/testrunner ~/dev/spock/testrunner
|
make[1]: Entering directory '/home/ubuntu/dev/spock/ns_server'
|
cd build && make --no-print-directory ns_dataclean
|
Built target ns_dataclean
|
make[1]: Leaving directory '/home/ubuntu/dev/spock/ns_server'
|
make[1]: Entering directory '/home/ubuntu/dev/spock/ns_server'
|
cd build && make --no-print-directory all
|
[ 0%] Built target all_images
|
[ 0%] Building Go target vbmap using Go 1.8.1
|
-- Executing: /home/ubuntu/.cbdepscache/exploded/x86_64/go-1.8.1/go/bin/go install -pkgdir /home/ubuntu/dev/spock/build/gopkg/go-1.8.1 -tags="" -gcflags="" -ldflags="" vbmap
|
[ 0%] Built target vbmap
|
[ 0%] Building Go target ns_gozip using Go 1.8.1
|
-- Executing: /home/ubuntu/.cbdepscache/exploded/x86_64/go-1.8.1/go/bin/go install -pkgdir /home/ubuntu/dev/spock/build/gopkg/go-1.8.1 -tags="" -gcflags="" -ldflags="" gozip
|
[ 0%] Built target ns_gozip
|
[ 0%] Building Go target ns_goport using Go 1.8.1
|
-- Executing: /home/ubuntu/.cbdepscache/exploded/x86_64/go-1.8.1/go/bin/go install -pkgdir /home/ubuntu/dev/spock/build/gopkg/go-1.8.1 -tags="" -gcflags="" -ldflags="" goport
|
[ 0%] Built target ns_goport
|
[ 0%] Building Go target ns_generate_cert using Go 1.8.1
|
-- Executing: /home/ubuntu/.cbdepscache/exploded/x86_64/go-1.8.1/go/bin/go install -pkgdir /home/ubuntu/dev/spock/build/gopkg/go-1.8.1 -tags="" -gcflags="" -ldflags="" generate_cert
|
[ 0%] Built target ns_generate_cert
|
[100%] Building Go target ns_godu using Go 1.8.1
|
-- Executing: /home/ubuntu/.cbdepscache/exploded/x86_64/go-1.8.1/go/bin/go install -pkgdir /home/ubuntu/dev/spock/build/gopkg/go-1.8.1 -tags="" -gcflags="" -ldflags="" godu
|
[100%] Built target ns_godu
|
[100%] Building Go target ns_minify using Go 1.8.1
|
-- Executing: /home/ubuntu/.cbdepscache/exploded/x86_64/go-1.8.1/go/bin/go install -pkgdir /home/ubuntu/dev/spock/build/gopkg/go-1.8.1 -tags="" -gcflags="" -ldflags="" minify
|
[100%] Built target ns_minify
|
-- Building app.min.js and index.min.html ...
|
-- Building classic-app.min.js and classic-index.min.html ...
|
[100%] Built target minify
|
==> meck (compile)
|
[100%] Built target meck
|
[100%] Built target ns_cfg
|
[100%] Built target ale-cfg
|
==> ale (compile)
|
[100%] Built target ale
|
==> triq (compile)
|
[100%] Built target triq
|
==> ns_server (compile)
|
[100%] Built target ns_server
|
==> gen_smtp (compile)
|
[100%] Built target gen_smtp
|
==> ns_babysitter (compile)
|
[100%] Built target ns_babysitter
|
==> ns_ssl_proxy (compile)
|
[100%] Built target ns_ssl_proxy
|
==> ns_couchdb (compile)
|
[100%] Built target ns_couchdb
|
[100%] Building Go target ns_gosecrets using Go 1.8.1
|
-- Executing: /home/ubuntu/.cbdepscache/exploded/x86_64/go-1.8.1/go/bin/go install -pkgdir /home/ubuntu/dev/spock/build/gopkg/go-1.8.1 -tags="" -gcflags="" -ldflags="" gosecrets
|
[100%] Built target ns_gosecrets
|
make[1]: Leaving directory '/home/ubuntu/dev/spock/ns_server'
|
~/dev/spock/testrunner
|
filename: conf/view-conf/py-viewmerge.conf
|
prefix: view.viewmergetests.ViewMergingTests
|
Global Test input params:
|
{'cluster_name': 'dev-single-node',
|
'conf_file': 'conf/view-conf/py-viewmerge.conf',
|
'ini': 'b/resources/dev-single-node.ini',
|
'makefile': 'True',
|
'num_nodes': 1,
|
'spec': 'py-viewmerge'}
|
Logs will be stored at /home/ubuntu/dev/spock/testrunner/logs/testrunner-17-Jun-14_16-51-54/test_1
|
|
|
./testrunner -i b/resources/dev-single-node.ini makefile=True -t view.viewmergetests.ViewMergingTests.test_empty_vbuckets,first_case=true
|
|
|
Test Input params:
|
{'cluster_name': 'dev-single-node', 'logs_folder': '/home/ubuntu/dev/spock/testrunner/logs/testrunner-17-Jun-14_16-51-54/test_1', 'ini': 'b/resources/dev-single-node.ini', 'case_number': 1, 'num_nodes': 1, 'makefile': 'True', 'spec': 'py-viewmerge', 'conf_file': 'conf/view-conf/py-viewmerge.conf', 'first_case': 'true'}
|
Run before suite setup for view.viewmergetests.ViewMergingTests.test_empty_vbuckets
|
2017-06-14 16:51:54 | WARNING | MainProcess | MainThread | [data_helper.<module>] Can not import concurrent module. Data for each server will be loaded/retrieved sequentially
|
test_empty_vbuckets (view.viewmergetests.ViewMergingTests) ... 2017-06-14 16:51:54 | INFO | MainProcess | test_thread | [remote_util.__init__] connecting to 127.0.0.1 with username:Administrator password:asdasd ssh_key:
|
2017-06-14 16:51:54 | INFO | MainProcess | test_thread | [remote_util.__init__] Connected to 127.0.0.1
|
2017-06-14 16:51:54 | ERROR | MainProcess | test_thread | [rest_client._http_request] socket error while connecting to http://127.0.0.1:9000/nodes/self error [Errno 111] Connection refused
|
2017-06-14 16:51:57 | ERROR | MainProcess | test_thread | [rest_client._http_request] socket error while connecting to http://127.0.0.1:9000/nodes/self error [Errno 111] Connection refused
|
2017-06-14 16:52:00 | INFO | MainProcess | test_thread | [rest_client.get_nodes_version] Node version in cluster 5.0.0-0000-community
|
2017-06-14 16:52:00 | ERROR | MainProcess | test_thread | [rest_client._http_request] GET http://127.0.0.1:9000/pools/default body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=\n'} error: 404 reason: unknown "unknown pool" auth: Administrator:asdasd
|
2017-06-14 16:52:00 | INFO | MainProcess | test_thread | [rest_client.get_nodes_versions] Node versions in cluster []
|
2017-06-14 16:52:00 | INFO | MainProcess | test_thread | [basetestcase.setUp] ============== basetestcase setup was started for test #1 test_empty_vbuckets==============
|
2017-06-14 16:52:00 | ERROR | MainProcess | test_thread | [rest_client._http_request] GET http://127.0.0.1:9000/pools/default/ body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=\n'} error: 404 reason: unknown "unknown pool" auth: Administrator:asdasd
|
2017-06-14 16:52:00 | INFO | MainProcess | test_thread | [bucket_helper.delete_all_buckets_or_assert] deleting existing buckets [] on 127.0.0.1
|
2017-06-14 16:52:00 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 10 secs. ...
|
2017-06-14 16:52:10 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] waiting for ns_server @ 127.0.0.1:9000
|
2017-06-14 16:52:10 | INFO | MainProcess | test_thread | [cluster_helper.wait_for_ns_servers_or_assert] ns_server @ 127.0.0.1:9000 is running
|
Cluster instance shutdown with force
|
2017-06-14 16:52:10 | INFO | MainProcess | test_thread | [basetestcase.setUp] initializing cluster
|
2017-06-14 16:52:11 | INFO | MainProcess | Cluster_Thread | [task.execute] server: ip:127.0.0.1 port:9000 ssh_username:Administrator, nodes/self: {'ip': u'127.0.0.1', 'availableStorage': [], 'rest_username': '', 'id': u'n_0@127.0.0.1', 'uptime': u'13', 'mcdMemoryReserved': 6385, 'storageTotalRam': 7982, 'hostname': u'127.0.0.1:9000', 'storage': [<membase.api.rest_client.NodeDataStorage object at 0x7fc47723d910>], 'moxi': 12001, 'port': u'9000', 'version': u'5.0.0-0000-community', 'memcached': 12000, 'status': u'healthy', 'clusterCompatibility': 327680, 'curr_items': 0, 'services': [u'kv'], 'rest_password': '', 'clusterMembership': u'active', 'memoryFree': 7781576704, 'memoryTotal': 8370143232, 'memoryQuota': 4789, 'mcdMemoryAllocated': 6385, 'os': u'x86_64-unknown-linux-gnu', 'ports': []}
|
2017-06-14 16:52:11 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster_memoryQuota] pools/default params : memoryQuota=4256
|
2017-06-14 16:52:11 | INFO | MainProcess | Cluster_Thread | [rest_client.set_indexer_storage_mode] settings/indexes params : storageMode=plasma
|
2017-06-14 16:52:11 | INFO | MainProcess | Cluster_Thread | [rest_client.init_cluster] settings/web params on 127.0.0.1:9000:username=Administrator&password=asdasd&port=9000
|
2017-06-14 16:52:11 | INFO | MainProcess | test_thread | [rest_client.get_nodes_versions] Node versions in cluster [u'5.0.0-0000-community']
|
2017-06-14 16:52:11 | INFO | MainProcess | test_thread | [basetestcase.add_built_in_server_user] **** add built-in 'cbadminbucket' user to node 127.0.0.1 ****
|
2017-06-14 16:52:11 | INFO | MainProcess | test_thread | [rest_client.get_nodes_versions] Node versions in cluster [u'5.0.0-0000-community']
|
2017-06-14 16:52:11 | ERROR | MainProcess | test_thread | [rest_client._http_request] DELETE http://127.0.0.1:9000/settings/rbac/users/local/cbadminbucket body: headers: {'Content-Type': 'application/x-www-form-urlencoded', 'Accept': '*/*', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjphc2Rhc2Q=\n'} error: 404 reason: unknown "User was not found." auth: Administrator:asdasd
|
2017-06-14 16:52:11 | INFO | MainProcess | test_thread | [internal_user.delete_user] Exception while deleting user. Exception is -"User was not found."
|
2017-06-14 16:52:11 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 10 secs. ...
|
2017-06-14 16:52:21 | INFO | MainProcess | test_thread | [basetestcase.add_built_in_server_user] **** add 'admin' role to 'cbadminbucket' user ****
|
2017-06-14 16:52:21 | INFO | MainProcess | test_thread | [rest_client.get_nodes_versions] Node versions in cluster [u'5.0.0-0000-community']
|
2017-06-14 16:52:21 | INFO | MainProcess | test_thread | [basetestcase.sleep] sleep for 10 secs. ...
|
2017-06-14 16:52:31 | INFO | MainProcess | test_thread | [basetestcase.setUp] done initializing cluster
|
2017-06-14 16:52:32 | INFO | MainProcess | Cluster_Thread | [rest_client.get_nodes_versions] Node versions in cluster [u'5.0.0-0000-community']
|
2017-06-14 16:52:32 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] http://127.0.0.1:9000/pools/default/buckets with param: bucketType=membase&threadsNumber=3&authType=sasl&replicaIndex=1&name=default&saslPassword=None&evictionPolicy=valueOnly&flushEnabled=1&replicaNumber=1&ramQuotaMB=4256
|
2017-06-14 16:52:32 | INFO | MainProcess | Cluster_Thread | [rest_client.create_bucket] 0.00 seconds to create bucket default
|
2017-06-14 16:52:32 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : default in 127.0.0.1 to accept set ops
|
2017-06-14 16:53:33 | INFO | MainProcess | Cluster_Thread | [rest_client.vbucket_map_ready] vbucket map is not ready for bucket default after waiting 60 seconds
|
2017-06-14 16:54:33 | INFO | MainProcess | Cluster_Thread | [rest_client.vbucket_map_ready] vbucket map is not ready for bucket default after waiting 60 seconds
|
2017-06-14 16:54:33 | ERROR | MainProcess | Cluster_Thread | [task.check] Unexpected error: vbucket map is not ready for bucket default
|
2017-06-14 16:54:33 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 0
|
2017-06-14 16:54:33 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : default in 127.0.0.1 to accept set ops
|
2017-06-14 16:55:33 | INFO | MainProcess | Cluster_Thread | [rest_client.vbucket_map_ready] vbucket map is not ready for bucket default after waiting 60 seconds
|
2017-06-14 16:56:34 | INFO | MainProcess | Cluster_Thread | [rest_client.vbucket_map_ready] vbucket map is not ready for bucket default after waiting 60 seconds
|
2017-06-14 16:56:34 | ERROR | MainProcess | Cluster_Thread | [task.check] Unexpected error: vbucket map is not ready for bucket default
|
2017-06-14 16:56:34 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 1
|
2017-06-14 16:56:34 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : default in 127.0.0.1 to accept set ops
|
2017-06-14 16:57:34 | INFO | MainProcess | Cluster_Thread | [rest_client.vbucket_map_ready] vbucket map is not ready for bucket default after waiting 60 seconds
|
2017-06-14 16:58:35 | INFO | MainProcess | Cluster_Thread | [rest_client.vbucket_map_ready] vbucket map is not ready for bucket default after waiting 60 seconds
|
2017-06-14 16:58:35 | ERROR | MainProcess | Cluster_Thread | [task.check] Unexpected error: vbucket map is not ready for bucket default
|
2017-06-14 16:58:35 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 2
|
2017-06-14 16:58:35 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : default in 127.0.0.1 to accept set ops
|
2017-06-14 16:59:35 | INFO | MainProcess | Cluster_Thread | [rest_client.vbucket_map_ready] vbucket map is not ready for bucket default after waiting 60 seconds
|
2017-06-14 17:00:35 | INFO | MainProcess | Cluster_Thread | [rest_client.vbucket_map_ready] vbucket map is not ready for bucket default after waiting 60 seconds
|
2017-06-14 17:00:35 | ERROR | MainProcess | Cluster_Thread | [task.check] Unexpected error: vbucket map is not ready for bucket default
|
2017-06-14 17:00:35 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 3
|
2017-06-14 17:00:35 | INFO | MainProcess | Cluster_Thread | [bucket_helper.wait_for_memcached] waiting for memcached bucket : default in 127.0.0.1 to accept set ops
|
2017-06-14 17:01:36 | INFO | MainProcess | Cluster_Thread | [rest_client.vbucket_map_ready] vbucket map is not ready for bucket default after waiting 60 seconds
|
Traceback (most recent call last):
|
File "pytests/basetestcase.py", line 325, in setUp
|
self._bucket_creation()
|
File "pytests/basetestcase.py", line 554, in _bucket_creation
|
self.cluster.create_default_bucket(default_params)
|
File "lib/couchbase_helper/cluster.py", line 263, in create_default_bucket
|
return _task.result(timeout)
|
File "lib/tasks/future.py", line 163, in result
|
raise TimeoutError()
|
TimeoutError
|
[('/usr/lib/python2.7/threading.py', 774, '__bootstrap', 'self.__bootstrap_inner()'), ('/usr/lib/python2.7/threading.py', 801, '__bootstrap_inner', 'self.run()'), ('./testrunner.py', 297, 'run', '**self._Thread__kwargs)'), ('/usr/lib/python2.7/unittest/runner.py', 151, 'run', 'test(result)'), ('/usr/lib/python2.7/unittest/case.py', 393, '__call__', 'return self.run(*args, **kwds)'), ('/usr/lib/python2.7/unittest/case.py', 320, 'run', 'self.setUp()'), ('pytests/view/viewmergetests.py', 29, 'setUp', 'super(ViewMergingTests, self).setUp()'), ('pytests/basetestcase.py', 325, 'setUp', 'self._bucket_creation()'), ('pytests/basetestcase.py', 554, '_bucket_creation', 'self.cluster.create_default_bucket(default_params)'), ('lib/couchbase_helper/cluster.py', 263, 'create_default_bucket', 'return _task.result(timeout)'), ('lib/tasks/future.py', 162, 'result', 'self.set_exception(TimeoutError())'), ('lib/tasks/future.py', 264, 'set_exception', 'print traceback.extract_stack()')]
|
Wed Jun 14 17:02:31 2017
|
Cluster instance shutdown with force
|
2017-06-14 17:02:31 | ERROR | MainProcess | test_thread | [viewmergetests.setUp] SETUP WAS FAILED. ALL TESTS WILL BE SKIPPED
|
ERROR
|
|
|
======================================================================
|
ERROR: test_empty_vbuckets (view.viewmergetests.ViewMergingTests)
|
----------------------------------------------------------------------
|
Traceback (most recent call last):
|
File "pytests/view/viewmergetests.py", line 50, in setUp
|
self.fail(ex)
|
AssertionError
|
|
|
----------------------------------------------------------------------
|
Ran 1 test in 637.259s
|
|
|
FAILED (errors=1)
|
summary so far suite view.viewmergetests.ViewMergingTests , pass 0 , fail 1
|
failures so far...
|
view.viewmergetests.ViewMergingTests.test_empty_vbuckets
|
testrunner logs, diags and results are available under /home/ubuntu/dev/spock/testrunner/logs/testrunner-17-Jun-14_16-51-54/test_1
|
test fails, all of the following tests will be skipped!!!
|
Run after suite setup for view.viewmergetests.ViewMergingTests.test_empty_vbuckets
|
view.viewmergetests.ViewMergingTests.test_empty_vbuckets fail
|
2017-06-14 17:02:36 | INFO | MainProcess | Cluster_Thread | [rest_client.vbucket_map_ready] vbucket map is not ready for bucket default after waiting 60 seconds
|
2017-06-14 17:02:36 | ERROR | MainProcess | Cluster_Thread | [task.check] Unexpected error: vbucket map is not ready for bucket default
|
2017-06-14 17:02:36 | WARNING | MainProcess | Cluster_Thread | [task.check] vbucket map not ready after try 4
|
scripts/start_cluster_and_run_tests.sh: line 81: 29823 Terminated COUCHBASE_NUM_VBUCKETS=64 python ./cluster_run --nodes=$servers_count &> $wd/cluster_run.log (wd: ~/dev/spock/ns_server)
|
Makefile:41: recipe for target 'test-viewmerge' failed
|
make: *** [test-viewmerge] Error 1
|
