Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-12005

vbucket-seqno stats getting timed out during Views DGM test

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Critical
    • 3.0.2
    • 3.0
    • Security Level: Public
    • None
    • 3.0.0-1166-rel
    • Triaged
    • Ubuntu 64-bit
    • Yes

    Description

      Test to Reproduce:
      ./testrunner -i yourfile.ini -t view.createdeleteview.CreateDeleteViewTests.pending_removal_with_ddoc_ops,ddoc_ops=update,test_with_view=True,num_ddocs=3,num_views_per_ddoc=3,items=200000,nodes_out=1,active
      _resident_threshold=10,dgm_run=True,eviction_policy=fullEviction,skip_cleanup=true

      Steps to Reproduce:
      1. Setup a 5-node cluster
      2. Rebalance in all nodes
      3. Load bucket to achieve dgm 10%
      4. Failover 1 node
      5. Create Views and perform ddoc update operations
      6. Test exits with error during ddoc validation

      014-08-18 04:00:37 | INFO | MainProcess | Cluster_Thread | [rest_client._query] index query url: http://10.3.5.90:8092/default/_design/ddoc_test1/_view/views0?stale=false&connection_timeout=60000&full_set=true
      2014-08-18 04:15:37 | ERROR | MainProcess | Cluster_Thread | [rest_client._http_request] socket error while connecting to http://10.3.5.90:8092/default/_design/ddoc_test1/_view/views0?stale=false&connection_timeout=60000&full_set=true error timed out
      2014-08-18 04:15:37 | ERROR | MainProcess | Cluster_Thread | [task.execute] Unexpected Exception Caught
      ERROR
      [('/usr/lib/python2.7/threading.py', 524, '__bootstrap', 'self.__bootstrap_inner()'), ('/usr/lib/python2.7/threading.py', 551, '__bootstrap_inner', 'self.run()'), ('lib/tasks/taskmanager.py', 31, 'run', 'task.step(self)'), ('lib/tasks/task.py', 56, 'step', 'self.execute(task_manager)'), ('lib/tasks/task.py', 1525, 'execute', 'self.set_exception(e)'), ('lib/tasks/future.py', 264, 'set_exception', 'print traceback.extract_stack()')]
      Mon Aug 18 04:15:37 2014
      [('/usr/lib/python2.7/threading.py', 524, '__bootstrap', 'self.__bootstrap_inner()'), ('/usr/lib/python2.7/threading.py', 551, '__bootstrap_inner', 'self.run()'), ('testrunner.py', 262, 'run', '**self._Thread__kwargs)'), ('/usr/lib/python2.7/unittest/runner.py', 151, 'run', 'test(result)'), ('/usr/lib/python2.7/unittest/case.py', 391, '__call__', 'return self.run(*args, **kwds)'), ('/usr/lib/python2.7/unittest/case.py', 327, 'run', 'testMethod()'), ('pytests/view/createdeleteview.py', 626, 'pending_removal_with_ddoc_ops', 'self._verify_ddoc_data_all_buckets()'), ('pytests/view/createdeleteview.py', 274, '_verify_ddoc_data_all_buckets', 'result = self.cluster.query_view(self.master, ddoc_name, view.name, query, num_items, bucket)'), ('lib/couchbase/cluster.py', 464, 'query_view', 'return _task.result(timeout)'), ('lib/tasks/future.py', 160, 'result', 'return self.__get_result()'), ('lib/tasks/future.py', 111, '__get_result', 'print traceback.extract_stack()')]
      2014-08-18 04:15:37 | WARNING | MainProcess | test_thread | [basetestcase.tearDown] CLEANUP WAS SKIPPED

      ======================================================================
      ERROR: pending_removal_with_ddoc_ops (view.createdeleteview.CreateDeleteViewTests)
      ----------------------------------------------------------------------
      Traceback (most recent call last):
      File "pytests/view/createdeleteview.py", line 626, in pending_removal_with_ddoc_ops
      self._verify_ddoc_data_all_buckets()
      File "pytests/view/createdeleteview.py", line 274, in _verify_ddoc_data_all_buckets
      result = self.cluster.query_view(self.master, ddoc_name, view.name, query, num_items, bucket)
      File "lib/couchbase/cluster.py", line 464, in query_view
      return _task.result(timeout)
      File "lib/tasks/future.py", line 160, in result
      return self.__get_result()
      File "lib/tasks/future.py", line 112, in __get_result
      raise self._exception
      ServerUnavailableException: unable to reach the host @ 10.3.5.90

      Logs:
      [couchdb:error,2014-08-18T4:39:41.963,ns_1@10.3.5.90:<0.12717.6>:couch_log:error:44]Set view `default`, replica group `_design/ddoc_test0`, doc loader error
      error: {timeout,{gen_server,call,
      [<0.12658.6>,

      {add_stream,706,0,0,5637,6}

      ,
      60000]}}
      stacktrace: [{gen_server,call,3,[

      {file,"gen_server.erl"}

      ,

      {line,188}

      ]},
      {couch_dcp_client,enum_docs_since,8,
      [

      {file, "/home/buildbot/buildbot_slave/ubuntu-1004-x64-300-builder/build/build/couchdb/src/couch_dcp/src/couch_dcp_client.erl"}

      ,

      {line,246}

      ]},
      {couch_set_view_updater,'load_changes/8-fun-2',12,
      [

      {file, "/home/buildbot/buildbot_slave/ubuntu-1004-x64-300-builder/build/build/couchdb/src/couch_set_view/src/couch_set_view_updater.erl"},
      {line,516}]},
      {lists,foldl,3,[{file,"lists.erl"},{line,1248}]},
      {couch_set_view_updater,load_changes,8,
      [{file, "/home/buildbot/buildbot_slave/ubuntu-1004-x64-300-builder/build/build/couchdb/src/couch_set_view/src/couch_set_view_updater.erl"}

      ,

      {line,589}

      ]},
      {couch_set_view_updater,'update/8-fun-3',14,
      [

      {file, "/home/buildbot/buildbot_slave/ubuntu-1004-x64-300-builder/build/build/couchdb/src/couch_set_view/src/couch_set_view_updater.erl"}

      ,

      {line,281}

      ]}]

      [couchdb:error,2014-08-18T6:46:34.997,ns_1@10.3.5.90:<0.12648.6>:couch_log:error:44]dcp client (<0.12660.6>): vbucket-seqno stats timed out after 2.0 seconds. Waiting...
      [couchdb:error,2014-08-18T6:46:39.608,ns_1@10.3.5.90:<0.21856.6>:couch_log:error:44]dcp client (<0.21861.6>): vbucket-seqno stats timed out after 2.0 seconds. Waiting...
      [couchdb:error,2014-08-18T6:46:42.611,ns_1@10.3.5.90:<0.21856.6>:couch_log:error:44]dcp client (<0.21861.6>): vbucket-seqno stats timed out after 2.0 seconds. Waiting...

      *Observed stacktraces and crashes in logs. Uploading logs.

      Live Cluster:
      1:10.3.5.90
      2:10.3.5.91
      3:10.3.5.92
      4:10.3.5.93
      5:10.3.4.75

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              harsha Harsha Havanur
              Meenakshi Meenakshi Goel
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty