Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-33208

Queries failing with bulkget exceeded MaxBulkRetries for default(vbid:858,keys:<ud>[query-testemployee11295.7170176-0]</ud>) error

    XMLWordPrintable

Details

    • Untriaged
    • Yes

    Description

      Build : 6.5.0-2473 onwards
      Last known good build : 6.5.0-2468

      The queries in the N1QL GSI tests in sanity jobs are failing with the following errors :

      [2019-02-28 10:32:59,677] - [task:2557] INFO -  Query SELECT * FROM default WHERE  join_yr > 2010 and join_yr < 2014 ORDER BY _id  results leads to INCORRECT RESULT 
      [('/usr/lib64/python2.7/threading.py', 785, '__bootstrap', 'self.__bootstrap_inner()'), ('/usr/lib64/python2.7/threading.py', 812, '__bootstrap_inner', 'self.run()'), ('lib/tasks/taskmanager.py', 31, 'run', 'task.step(self)'), ('lib/tasks/task.py', 79, 'step', 'self.check(task_manager)'), ('lib/tasks/task.py', 2572, 'check', 'self.set_exception(e)'), ('lib/tasks/future.py', 264, 'set_exception', 'print traceback.extract_stack()')]
      Thu Feb 28 10:32:59 2019
      [2019-02-28 10:32:59,679] - [task:2557] INFO -  Query SELECT * FROM default where job_title IS NOT NULL ORDER BY job_title,_id results leads to INCORRECT RESULT 
      [('/usr/lib64/python2.7/threading.py', 785, '__bootstrap', 'self.__bootstrap_inner()'), ('/usr/lib64/python2.7/threading.py', 812, '__bootstrap_inner', 'self.run()'), ('lib/tasks/taskmanager.py', 31, 'run', 'task.step(self)'), ('lib/tasks/task.py', 79, 'step', 'self.check(task_manager)'), ('lib/tasks/task.py', 2572, 'check', 'self.set_exception(e)'), ('lib/tasks/future.py', 264, 'set_exception', 'print traceback.extract_stack()')]
      Thu Feb 28 10:32:59 2019
      [('/usr/lib64/python2.7/threading.py', 785, '__bootstrap', 'self.__bootstrap_inner()'), ('/usr/lib64/python2.7/threading.py', 812, '__bootstrap_inner', 'self.run()'), ('testrunner.py', 297, 'run', '**self._Thread__kwargs)'), ('/usr/lib64/python2.7/unittest/runner.py', 151, 'run', 'test(result)'), ('/usr/lib64/python2.7/unittest/case.py', 433, '__call__', 'return self.run(*args, **kwds)'), ('/usr/lib64/python2.7/unittest/case.py', 369, 'run', 'testMethod()'), ('pytests/2i/indexscans_2i.py', 48, 'test_multi_create_query_explain_drop_index', 'self._query_explain_in_async()'), ('pytests/2i/base_2i.py', 763, '_query_explain_in_async', 'task.result()'), ('lib/tasks/future.py', 160, 'result', 'return self.__get_result()'), ('lib/tasks/future.py', 111, '__get_result', 'print traceback.extract_stack()')]
      [2019-02-28 10:32:59,680] - [indexscans_2i:51] INFO - unable to get expected results for query CBQError: host 172.23.107.160: ERROR:{u'status': u'errors', u'errors': [{u'msg': u'Error performing bulk get operation  - cause: {1 errors, starting with bulkget exceeded MaxBulkRetries for default(vbid:947,keys:<ud>[query-testemployee11215.900264-0]</ud>)}', u'code': 12008}], u'results': [], u'metrics': {u'elapsedTime': u'1.170963955s', u'executionTime': u'1.170910687s', u'resultSize': 0, u'resultCount': 0, u'errorCount': 1}, u'requestID': u'b03dd3bb-62e0-485b-8171-022814f3ed5b', u'signature': {u'*': u'*'}}
      

      I tried to reproduce manually using the same steps as in the test, but couldnt reproduce it manually. But with the sanity jobs, it is 100% reproducible.

      cbcollectinfo is attached. 172.23.107.160 is the query node.

      Query logs have the following around the time of failure:
      2019-02-28T10:32:59.673-08:00 [INFO] Pool Get returned default: no connection pool
      2019-02-28T10:32:59.673-08:00 [INFO] Pool Get returned default: no connection pool
      2019-02-28T10:32:59.674-08:00 [INFO] Pool Get returned default: no connection pool
      2019-02-28T10:32:59.674-08:00 [ERROR] bulkget exceeded MaxBulkRetries for default(vbid:858,keys:<ud>[query-testemployee11295.7170176-0]</ud>)

      This could be caused by https://issues.couchbase.com/browse/MB-33185

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            People

              mihir.kamdar Mihir Kamdar (Inactive)
              mihir.kamdar Mihir Kamdar (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty