Details
-
Bug
-
Resolution: Fixed
-
Critical
-
3.0
-
Security Level: Public
-
None
-
3.0.0-1166-rel
-
Triaged
-
Ubuntu 64-bit
-
Yes
Description
Test to Reproduce:
./testrunner -i yourfile.ini -t view.createdeleteview.CreateDeleteViewTests.pending_removal_with_ddoc_ops,ddoc_ops=update,test_with_view=True,num_ddocs=3,num_views_per_ddoc=3,items=200000,nodes_out=1,active
_resident_threshold=10,dgm_run=True,eviction_policy=fullEviction,skip_cleanup=true
Steps to Reproduce:
1. Setup a 5-node cluster
2. Rebalance in all nodes
3. Load bucket to achieve dgm 10%
4. Failover 1 node
5. Create Views and perform ddoc update operations
6. Test exits with error during ddoc validation
014-08-18 04:00:37 | INFO | MainProcess | Cluster_Thread | [rest_client._query] index query url: http://10.3.5.90:8092/default/_design/ddoc_test1/_view/views0?stale=false&connection_timeout=60000&full_set=true
2014-08-18 04:15:37 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] socket error while connecting to http://10.3.5.90:8092/default/_design/ddoc_test1/_view/views0?stale=false&connection_timeout=60000&full_set=true error timed out
2014-08-18 04:15:37 | ERROR | MainProcess | Cluster_Thread | [task.execute] Unexpected Exception Caught
ERROR
[('/usr/lib/python2.7/threading.py', 524, '__bootstrap', 'self.__bootstrap_inner()'), ('/usr/lib/python2.7/threading.py', 551, '__bootstrap_inner', 'self.run()'), ('lib/tasks/taskmanager.py', 31, 'run', 'task.step(self)'), ('lib/tasks/task.py', 56, 'step', 'self.execute(task_manager)'), ('lib/tasks/task.py', 1525, 'execute', 'self.set_exception(e)'), ('lib/tasks/future.py', 264, 'set_exception', 'print traceback.extract_stack()')]
Mon Aug 18 04:15:37 2014
[('/usr/lib/python2.7/threading.py', 524, '__bootstrap', 'self.__bootstrap_inner()'), ('/usr/lib/python2.7/threading.py', 551, '__bootstrap_inner', 'self.run()'), ('testrunner.py', 262, 'run', '**self._Thread__kwargs)'), ('/usr/lib/python2.7/unittest/runner.py', 151, 'run', 'test(result)'), ('/usr/lib/python2.7/unittest/case.py', 391, '__call__', 'return self.run(*args, **kwds)'), ('/usr/lib/python2.7/unittest/case.py', 327, 'run', 'testMethod()'), ('pytests/view/createdeleteview.py', 626, 'pending_removal_with_ddoc_ops', 'self._verify_ddoc_data_all_buckets()'), ('pytests/view/createdeleteview.py', 274, '_verify_ddoc_data_all_buckets', 'result = self.cluster.query_view(self.master, ddoc_name, view.name, query, num_items, bucket)'), ('lib/couchbase/cluster.py', 464, 'query_view', 'return _task.result(timeout)'), ('lib/tasks/future.py', 160, 'result', 'return self.__get_result()'), ('lib/tasks/future.py', 111, '__get_result', 'print traceback.extract_stack()')]
2014-08-18 04:15:37 | WARNING | MainProcess | test_thread | [basetestcase.tearDown] CLEANUP WAS SKIPPED
======================================================================
ERROR: pending_removal_with_ddoc_ops (view.createdeleteview.CreateDeleteViewTests)
----------------------------------------------------------------------
Traceback (most recent call last):
File "pytests/view/createdeleteview.py", line 626, in pending_removal_with_ddoc_ops
self._verify_ddoc_data_all_buckets()
File "pytests/view/createdeleteview.py", line 274, in _verify_ddoc_data_all_buckets
result = self.cluster.query_view(self.master, ddoc_name, view.name, query, num_items, bucket)
File "lib/couchbase/cluster.py", line 464, in query_view
return _task.result(timeout)
File "lib/tasks/future.py", line 160, in result
return self.__get_result()
File "lib/tasks/future.py", line 112, in __get_result
raise self._exception
ServerUnavailableException: unable to reach the host @ 10.3.5.90
Logs:
[couchdb:error,2014-08-18T4:39:41.963,ns_1@10.3.5.90:<0.12717.6>:couch_log:error:44]Set view `default`, replica group `_design/ddoc_test0`, doc loader error
error: {timeout,{gen_server,call,
[<0.12658.6>,
,
60000]}}
stacktrace: [{gen_server,call,3,[
,
{line,188}]},
{couch_dcp_client,enum_docs_since,8,
[
,
]},
{couch_set_view_updater,'load_changes/8-fun-2',12,
[
{line,516}]},
{lists,foldl,3,[{file,"lists.erl"},{line,1248}]},
{couch_set_view_updater,load_changes,8,
[{file, "/home/buildbot/buildbot_slave/ubuntu-1004-x64-300-builder/build/build/couchdb/src/couch_set_view/src/couch_set_view_updater.erl"}
,
]},
{couch_set_view_updater,'update/8-fun-3',14,
[
,
]}]
[couchdb:error,2014-08-18T6:46:34.997,ns_1@10.3.5.90:<0.12648.6>:couch_log:error:44]dcp client (<0.12660.6>): vbucket-seqno stats timed out after 2.0 seconds. Waiting...
[couchdb:error,2014-08-18T6:46:39.608,ns_1@10.3.5.90:<0.21856.6>:couch_log:error:44]dcp client (<0.21861.6>): vbucket-seqno stats timed out after 2.0 seconds. Waiting...
[couchdb:error,2014-08-18T6:46:42.611,ns_1@10.3.5.90:<0.21856.6>:couch_log:error:44]dcp client (<0.21861.6>): vbucket-seqno stats timed out after 2.0 seconds. Waiting...
*Observed stacktraces and crashes in logs. Uploading logs.
Live Cluster:
1:10.3.5.90
2:10.3.5.91
3:10.3.5.92
4:10.3.5.93
5:10.3.4.75
Attachments
Issue Links
- relates to
-
MB-12111 Missing items in index after failover operation
- Resolved