Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-19522

500 error for queries + Timeout 1m15s exceeded in n1ql_thr_ryow_Q2_20M_moi.test

    XMLWordPrintable

Details

    • Bug
    • Resolution: Duplicate
    • Critical
    • 4.5.0
    • 4.5.0
    • secondary-index
    • 4.5.0-2477
    • Untriaged
    • Centos 64-bit
    • Unknown

    Description

      http://perf.jenkins.couchbase.com/job/hades-dev-ryow-from-src/93/parameters/

      in the client logs I see 500 error for queries:

      [2016-05-05 01:50:52,892: WARNING/PoolWorker-1] File "/tmp/worker-ryow-172.23.100.70-bucket-1/perfrunner/env/lib/python2.7/site-packages/spring/wgen.py", line 34, in with_sleep
      		 
      [2016-05-05 01:50:52,892: WARNING/PoolWorker-1] method(self)
      		 
      [2016-05-05 01:50:52,892: WARNING/PoolWorker-1] File "/tmp/worker-ryow-172.23.100.70-bucket-1/perfrunner/env/lib/python2.7/site-packages/spring/wgen.py", line 584, in do_batch
      		 
      [2016-05-05 01:50:52,892: WARNING/PoolWorker-1] len(list(self.cb.client.n1ql_query(nq)))
      		 
      [2016-05-05 01:50:52,892: WARNING/PoolWorker-1] File "/tmp/worker-ryow-172.23.100.70-bucket-1/perfrunner/env/lib/python2.7/site-packages/couchbase/n1ql.py", line 507, in __iter__
      		 
      [2016-05-05 01:50:52,893: WARNING/PoolWorker-1] for row in self._process_payload(raw_rows):
      		 
      [2016-05-05 01:50:52,893: WARNING/PoolWorker-1] File "/tmp/worker-ryow-172.23.100.70-bucket-1/perfrunner/env/lib/python2.7/site-packages/couchbase/n1ql.py", line 466, in _process_payload
      		 
      [2016-05-05 01:50:52,893: WARNING/PoolWorker-1] self._handle_meta(self.raw.value)
      		 
      [2016-05-05 01:50:52,893: WARNING/PoolWorker-1] Process Process-1473:
      		 
      [2016-05-05 01:50:52,893: WARNING/PoolWorker-1] File "/tmp/worker-ryow-172.23.100.70-bucket-1/perfrunner/env/lib/python2.7/site-packages/couchbase/n1ql.py", line 459, in _handle_meta
      		 
      [2016-05-05 01:50:52,893: WARNING/PoolWorker-1] Traceback (most recent call last):
      		 
      [2016-05-05 01:50:52,894: WARNING/PoolWorker-1] raise N1QLError.pyexc('N1QL Execution failed', err)
      		 
      [2016-05-05 01:50:52,894: WARNING/PoolWorker-1] File "/usr/local/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
      		 
      [2016-05-05 01:50:52,894: WARNING/PoolWorker-1] N1QLError: <N1QL Execution failed, OBJ={u'msg': u'Timeout 1m15s exceeded', u'code': 4080}>
      		 
      [2016-05-05 01:50:52,894: WARNING/PoolWorker-1] self.run()
      		 
      [2016-05-05 01:50:52,894: WARNING/PoolWorker-1] File "/usr/local/lib/python2.7/multiprocessing/process.py", line 114, in run
      		 
      [2016-05-05 01:50:52,895: WARNING/PoolWorker-1] self._target(*self._args, **self._kwargs)
      		 
      [2016-05-05 01:50:52,895: WARNING/PoolWorker-1] File "/tmp/worker-ryow-172.23.100.70-bucket-1/perfrunner/env/lib/python2.7/site-packages/spring/wgen.py", line 637, in run
      		 
      [2016-05-05 01:50:52,895: WARNING/PoolWorker-1] self.do_batch()
      		 
      [2016-05-05 01:50:52,896: WARNING/PoolWorker-1] File "<decorator-gen-12>", line 2, in do_batch
      		 
      [2016-05-05 01:50:52,896: WARNING/PoolWorker-1] File "/tmp/worker-ryow-172.23.100.70-bucket-1/perfrunner/env/lib/python2.7/site-packages/spring/wgen.py", line 34, in with_sleep
      		 
      [2016-05-05 01:50:52,896: WARNING/PoolWorker-1] method(self)
      		 
      [2016-05-05 01:50:52,896: WARNING/PoolWorker-1] File "/tmp/worker-ryow-172.23.100.70-bucket-1/perfrunner/env/lib/python2.7/site-packages/spring/wgen.py", line 584, in do_batch
      		 
      [2016-05-05 01:50:52,896: WARNING/PoolWorker-1] len(list(self.cb.client.n1ql_query(nq)))
      		 
      [2016-05-05 01:50:52,897: WARNING/PoolWorker-1] File "/tmp/worker-ryow-172.23.100.70-bucket-1/perfrunner/env/lib/python2.7/site-packages/couchbase/n1ql.py", line 506, in __iter__
      		 
      [2016-05-05 01:50:52,897: WARNING/PoolWorker-1] raw_rows = self.raw.fetch(self._mres)
      		 
      [2016-05-05 01:50:52,897: WARNING/PoolWorker-1] HTTPError: <RC=0x3B[HTTP Operation failed. Inspect status code for details], HTTP Request failed. Examine 'objextra' for full result, Results=1, C Source=(src/http.c,140), OBJ=ViewResult<rc=0x3B[HTTP Operation failed. Inspect status code for details], value={u'status': u'errors', u'errors': [{u'msg': u'queryport.connPoolTimeout from 172.23.100.45:9101 - cause: queryport.connPoolTimeout from 172.23.100.45:9101', u'code': 5000}], u'results': [], u'metrics': {u'elapsedTime': u'14.884455031s', u'executionTime': u'2.015051908s', u'resultSize': 0, u'resultCount': 0, u'errorCount': 1}, u'requestID': u'cbcaa399-1230-4b32-9754-7577366be119', u'signature': {u'*': u'*'}}, http_status=500>>

      and then Timeout 1m15s exceeded

       
      		 
      [2016-05-05 01:51:07,471: WARNING/PoolWorker-1] File "/tmp/worker-ryow-172.23.100.70-bucket-1/perfrunner/env/lib/python2.7/site-packages/spring/wgen.py", line 637, in run
      		 
      [2016-05-05 01:51:07,472: WARNING/PoolWorker-1] self.do_batch()
      		 
      [2016-05-05 01:51:07,472: WARNING/PoolWorker-1] File "<decorator-gen-12>", line 2, in do_batch
      		 
      [2016-05-05 01:51:07,472: WARNING/PoolWorker-1] File "/tmp/worker-ryow-172.23.100.70-bucket-1/perfrunner/env/lib/python2.7/site-packages/spring/wgen.py", line 34, in with_sleep
      		 
      [2016-05-05 01:51:07,472: WARNING/PoolWorker-1] method(self)
      		 
      [2016-05-05 01:51:07,472: WARNING/PoolWorker-1] File "/tmp/worker-ryow-172.23.100.70-bucket-1/perfrunner/env/lib/python2.7/site-packages/spring/wgen.py", line 584, in do_batch
      		 
      [2016-05-05 01:51:07,473: WARNING/PoolWorker-1] len(list(self.cb.client.n1ql_query(nq)))
      		 
      [2016-05-05 01:51:07,473: WARNING/PoolWorker-1] File "/tmp/worker-ryow-172.23.100.70-bucket-1/perfrunner/env/lib/python2.7/site-packages/couchbase/n1ql.py", line 507, in __iter__
      		 
      [2016-05-05 01:51:07,473: WARNING/PoolWorker-1] for row in self._process_payload(raw_rows):
      		 
      [2016-05-05 01:51:07,473: WARNING/PoolWorker-1] File "/tmp/worker-ryow-172.23.100.70-bucket-1/perfrunner/env/lib/python2.7/site-packages/couchbase/n1ql.py", line 466, in _process_payload
      		 
      [2016-05-05 01:51:07,473: WARNING/PoolWorker-1] self._handle_meta(self.raw.value)
      		 
      [2016-05-05 01:51:07,473: WARNING/PoolWorker-1] File "/tmp/worker-ryow-172.23.100.70-bucket-1/perfrunner/env/lib/python2.7/site-packages/couchbase/n1ql.py", line 459, in _handle_meta
      		 
      [2016-05-05 01:51:07,474: WARNING/PoolWorker-1] raise N1QLError.pyexc('N1QL Execution failed', err)
      		 
      [2016-05-05 01:51:07,474: WARNING/PoolWorker-1] N1QLError: <N1QL Execution failed, OBJ={u'msg': u'Timeout 1m15s exceeded', u'code': 4080}>
      		 
      [2016-05-05 01:51:07,561: WARNING/PoolWorker-1] Process Process-1024:
      		 
      [2016-05-05 01:51:07,561: WARNING/PoolWorker-1] Traceback (most recent call last):
      		 
      [2016-05-05 01:51:07,562: WARNING/PoolWorker-1] File "/usr/local/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
      		 
      [2016-05-05 01:51:07,562: WARNING/PoolWorker-1] self.run()
      		 
      [2016-05-05 01:51:07,562: WARNING/PoolWorker-1] File "/usr/local/lib/python2.7/multiprocessing/process.py", line 114, in run
      		 
      [2016-05-05 01:51:07,562: WARNING/PoolWorker-1] self._target(*self._args, **self._kwargs)
      		 
      [2016-05-05 01:51:07,562: WARNING/PoolWorker-1] File "/tmp/worker-ryow-172.23.100.70-bucket-1/perfrunner/env/lib/python2.7/site-packages/spring/wgen.py", line 637, in run
      		 
      [2016-05-05 01:51:07,563: WARNING/PoolWorker-1] self.do_batch()
      		 
      [2016-05-05 01:51:07,563: WARNING/PoolWorker-1] File "<decorator-gen-12>", line 2, in do_batch
      		 
      [2016-05-05 01:51:07,563: WARNING/PoolWorker-1] File "/tmp/worker-ryow-172.23.100.70-bucket-1/perfrunner/env/lib/python2.7/site-packages/spring/wgen.py", line 34, in with_sleep
      		 
      [2016-05-05 01:51:07,563: WARNING/PoolWorker-1] method(self)
      		 
      [2016-05-05 01:51:07,563: WARNING/PoolWorker-1] File "/tmp/worker-ryow-172.23.100.70-bucket-1/perfrunner/env/lib/python2.7/site-packages/spring/wgen.py", line 584, in do_batch
      		 
      [2016-05-05 01:51:07,564: WARNING/PoolWorker-1] len(list(self.cb.client.n1ql_query(nq)))
      		 
      [2016-05-05 01:51:07,564: WARNING/PoolWorker-1] File "/tmp/worker-ryow-172.23.100.70-bucket-1/perfrunner/env/lib/python2.7/site-packages/couchbase/n1ql.py", line 506, in __iter__
      		 
      [2016-05-05 01:51:07,564: WARNING/PoolWorker-1] raw_rows = self.raw.fetch(self._mres)
      		 
      [2016-05-05 01:51:07,564: WARNING/PoolWorker-1] _TimeoutError_0x17 (generated, catch TimeoutError): <RC=0x17[Client-Side timeout exceeded for operation. Inspect network conditions or increase the timeout], HTTP Request failed. Examine 'objextra' for full result, Results=1, C Source=(src/http.c,140), OBJ=ViewResult<rc=0x17[Client-Side timeout exceeded for operation. Inspect network conditions or increase the timeout], value=None, http_status=0>>

      Attachments

        1. pprof.txt
          50 kB
        2. pprof4.txt
          52 kB
        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            sarath Sarath Lakshman
            andreibaranouski Andrei Baranouski
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty