I have seen this issue multiple times in past but now I am able to reproduce it with this test consistently:
|
python testrunner.py -i rqg1.ini -c bucketflusherror.conf -p gsi_type=forestdb -p doc-per-day=1 -p primary_indx_type=VIEW
|
|
Here is a sample rqg1.ini:
|
[global]
|
port:8091
|
username:root
|
password:couchbase
|
|
[servers]
|
1:SERVER_1
|
|
[SERVER_1]
|
ip:172.23.107.56
|
services=n1ql,kv,index
|
|
|
[membase]
|
rest_username:Administrator
|
rest_password:password
|
|
|
[tuq_client]
|
goroot:/root/n1ql/go
|
sherlock_path=/opt/couchbase/bin
|
|
Please replace ip address with your centos VM.
|
Please let me know if you need this machine to reproduce.
|
|
bucketflusherror.conf contains:
|
tuqquery.tuq_index.QueriesJoinViewsTests:
|
test_run_query,to_run=test_where_in_subquery_equal_more,index_field=join_day;join_mo
|
tuqquery.tuq_dml.DMLQueryTests:
|
test_with_hints,to_run=delete_where_clause_json_hints,index_field=job_title
|
test_with_hints,to_run=delete_where_clause_json_hints,index_field=job_title,index_type=GSI
|
test_with_hints,to_run=update_keys_clause_hints,index_field=name
|
test_with_hints,to_run=update_keys_clause_hints,index_field=name,index_type=GSI
|
test_with_hints,to_run=update_where_hints,index_field=join_day
|
test_with_hints,to_run=update_where_hints,index_field=join_day,index_type=GSI
|
test_multiple_indexes_query_attr
|
test_multiple_indexes_query_non_ind_attr
|
|
Error message seen:
|
[2016-03-23 12:16:43,269] - [basetestcase:364] INFO - sleep for 10 secs. wait for indexer ...
|
[2016-03-23 12:16:53,273] - [tuq:103] INFO - ----------------------------------------------------------------------------------------------------
|
----------------------------------------------------------------------------------------------------
|
Temp process shutdown to debug MB-16888
|
----------------------------------------------------------------------------------------------------
|
[2016-03-23 12:16:53,273] - [remote_util:2267] INFO - running command.raw on 172.23.107.56: ps aux | grep cbq
|
[2016-03-23 12:16:53,376] - [remote_util:2304] INFO - command executed successfully
|
(['498 1599 3.3 0.4 362360 17952 ? Sl 12:16 0:00 /opt/couchbase/bin/cbq-engine --datastore=http://127.0.0.1:8091 --http=:8093 --configstore=http://127.0.0.1:8091 --enterprise=true --https=:18093 --certfile=/opt/couchbase/var/lib/couchbase/config/ssl-cert-key.pem --keyfile=/opt/couchbase/var/lib/couchbase/config/ssl-cert-key.pem --ssl_minimum_protocol=tlsv1', 'root 1684 0.0 0.0 106120 2716 ? Ss 12:16 0:00 bash -c ps aux | grep cbq', 'root 1688 0.0 0.0 103368 1956 ? S 12:16 0:00 grep cbq'], [])
|
[2016-03-23 12:16:53,377] - [remote_util:2267] INFO - running command.raw on 172.23.107.56: ps aux | grep indexer
|
[2016-03-23 12:16:53,552] - [remote_util:2304] INFO - command executed successfully
|
(['498 1615 5.5 1.1 689560 48708 ? Sl 12:16 0:00 /opt/couchbase/bin/indexer -vbuckets=1024 -cluster=127.0.0.1:8091 -adminPort=9100 -scanPort=9101 -httpPort=9102 -streamInitPort=9103 -streamCatchupPort=9104 -streamMaintPort=9105 -storageDir=/opt/couchbase/var/lib/couchbase/data/@2i -diagDir=/opt/couchbase/var/lib/couchbase/crash -nodeUUID=b229a874bc0185b01b7ebf3e0ab88529 -storageMode=forestdb', 'root 1689 1.0 0.0 106120 2696 ? Ss 12:16 0:00 bash -c ps aux | grep indexer', 'root 1694 0.0 0.0 103364 2048 ? S 12:16 0:00 grep indexer'], [])
|
[2016-03-23 12:17:54,077] - [rest_client:757] ERROR - http://172.23.107.56:8091/pools/default/buckets/default/controller/doFlush error 503 reason: unknown {"_":"Unexpected error: {error,{flush_wait_failed,['ns_1@127.0.0.1'],[]}}"}
|
[('/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py', 783, '__bootstrap', 'self.__bootstrap_inner()'), ('/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py', 810, '__bootstrap_inner', 'self.run()'), ('lib/tasks/taskmanager.py', 31, 'run', 'task.step(self)'), ('lib/tasks/task.py', 71, 'step', 'self.execute(task_manager)'), ('lib/tasks/task.py', 3644, 'execute', 'self.set_exception(e)'), ('lib/tasks/future.py', 264, 'set_exception', 'print traceback.extract_stack()')]
|
Wed Mar 23 12:17:54 2016
|
[('/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py', 783, '__bootstrap', 'self.__bootstrap_inner()'), ('/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py', 810, '__bootstrap_inner', 'self.run()'), ('testrunner.py', 282, 'run', '**self._Thread__kwargs)'), ('/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/runner.py', 151, 'run', 'test(result)'), ('/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/case.py', 395, '__call__', 'return self.run(*args, **kwds)'), ('/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/case.py', 322, 'run', 'self.setUp()'), ('pytests/tuqquery/tuq_dml.py', 26, 'setUp', 'timeout=self.wait_timeout * 5)'), ('lib/couchbase_helper/cluster.py', 1003, 'bucket_flush', 'return _task.result(timeout)'), ('lib/tasks/future.py', 160, 'result', 'return self.__get_result()'), ('lib/tasks/future.py', 111, '__get_result', 'print traceback.extract_stack()')]
|
ERROR
|
|
======================================================================
|
ERROR: test_with_hints (tuqquery.tuq_dml.DMLQueryTests)
|
----------------------------------------------------------------------
|
Traceback (most recent call last):
|
File "pytests/tuqquery/tuq_dml.py", line 26, in setUp
|
timeout=self.wait_timeout * 5)
|
File "lib/couchbase_helper/cluster.py", line 1003, in bucket_flush
|
return _task.result(timeout)
|
File "lib/tasks/future.py", line 160, in result
|
return self.__get_result()
|
File "lib/tasks/future.py", line 112, in __get_result
|
raise self._exception
|
BucketFlushFailed: unable to flush bucket default on the host @ 172.23.107.56
|
|
----------------------------------------------------------------------
|
Ran 1 test in 78.246s
|
|
FAILED (errors=1)
|
summary so far suite tuqquery.tuq_index.QueriesJoinViewsTests , pass 1 , fail 0
|
|
Uploading logs from machine using cbcollectinfo.
|
Test Blocker