Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-62852

ANN query fails with InternalServerFailureException

    XMLWordPrintable

Details

    • Untriaged
    • 0
    • Unknown

    Description

      After running ANN query for a while I am getting error from indexer as follow:

      Query: SELECT RAW id FROM _default WHERE size IN $size AND brand IN $brand ORDER BY ANN(vec, $qvec, "L2_SQUARED", 3) ASC LIMIT 100
      Exception in thread Thread-1 (run_queries):
      Traceback (most recent call last):
        File "/opt/homebrew/Cellar/python@3.11/3.11.3/Frameworks/Python.framework/Versions/3.11/lib/python3.11/threading.py", line 1038, in _bootstrap_inner
          self.run()
        File "/opt/homebrew/Cellar/python@3.11/3.11.3/Frameworks/Python.framework/Versions/3.11/lib/python3.11/threading.py", line 975, in run
          self._target(*self._args, **self._kwargs)
        File "/Users/pierre.regazzoni/couchbase/testrunner/scripts/vector_query.py", line 219, in run_queries
          self.n1ql_query(cb_scope, xb, qdoc, gdocs[idx], search_function, collection, vector_field, is_xattr, is_base64, is_bigendian, nprobes)
        File "/Users/pierre.regazzoni/couchbase/testrunner/scripts/vector_query.py", line 231, in n1ql_query
          for row in result.rows():
        File "/opt/homebrew/lib/python3.11/site-packages/couchbase/n1ql.py", line 96, in __next__
          raise ex
        File "/opt/homebrew/lib/python3.11/site-packages/couchbase/n1ql.py", line 90, in __next__
          return self._get_next_row()
                 ^^^^^^^^^^^^^^^^^^^^
        File "/opt/homebrew/lib/python3.11/site-packages/couchbase/n1ql.py", line 81, in _get_next_row
          raise ErrorMapper.build_exception(row)
      couchbase.exceptions.InternalServerFailureException: InternalServerFailureException(<ec=5, category=couchbase.common, message=internal_server_failure (5), context=QueryErrorContext({'last_dispatched_to': '172.23.216.141:8093', 'last_dispatched_from': '192.168.101.149:50313', 'retry_attempts': 0, 'client_context_id': '44785b-30ee-5b49-b84d-62ff0654f23c5d', 'method': 'POST', 'path': '/query/service', 'http_status': 500, 'http_body': '{\n"requestID": "28d70c4e-934a-463a-898a-1d37f3d259bb",\n"clientContextID": "44785b-30ee-5b49-b84d-62ff0654f23c5d",\n"signature": "json",\n"results": [\n],\n"errors": [{"code":5000,"msg":" dial tcp 127.0.0.1:9101: connect: connection refused from [127.0.0.1:9101] - cause:  dial tcp 127.0.0.1:9101: connect: connection refused from [127.0.0.1:9101]","reason":{"_level":"exception","caller":"secondary_index:2306","cause":{"error":"dial tcp 127.0.0.1:9101: connect: connection refused from [127.0.0.1:9101]"},"code":4350,"key":"indexing.error","message":"GSI error"}}],\n"status": "fatal",\n"metrics": {"elapsedTime": "73.125719ms","executionTime": "72.979502ms","resultCount": 0,"resultSize": 0,"serviceLoad": 6,"errorCount": 1}\n}\n', 'first_error_code': 5000, 'first_error_message': ' dial tcp 127.0.0.1:9101: connect: connection refused from [127.0.0.1:9101] - cause:  dial tcp 127.0.0.1:9101: connect: connection refused from [127.0.0.1:9101]', 'statement': 'SELECT RAW id FROM _default WHERE size IN $size AND brand IN $brand ORDER BY ANN(vec, $qvec, "L2_SQUARED", 3) ASC LIMIT 100', 'parameters': '{"$brand":["nike"],"$brandidx":0,"$qvec":[0.0,0.0,1.0,7.0,3.0,11.0,29.0,0.0,0.0,0.0,7.0,29.0,5.0,34.0,68.0,0.0,3.0,0.0,14.0,91.0,7.0,24.0,57.0,20.0,25.0,17.0,26.0,44.0,3.0,0.0,1.0,5.0,0.0,0.0,0.0,1.0,3.0,143.0,143.0,1.0,0.0,0.0,0.0,1.0,3.0,110.0,143.0,12.0,143.0,2.0,4.0,3.0,0.0,34.0,125.0,139.0,81.0,16.0,43.0,10.0,0.0,0.0,0.0,18.0,24.0,2.0,0.0,8.0,9.0,80.0,46.0,25.0,8.0,0.0,0.0,12.0,20.0,21.0,35.0,26.0,143.0,0.0,0.0,0.0,0.0,0.0,3.0,66.0,143.0,2.0,3.0,3.0,0.0,0.0,0.0,18.0,3.0,0.0,0.0,0.0,0.0,6.0,17.0,10.0,8.0,0.0,0.0,0.0,1.0,8.0,17.0,22.0,143.0,5.0,0.0,0.0,0.0,0.0,0.0,43.0,117.0,0.0,0.0,0.0,0.0,0.0,2.0,24.0],"$size":[8],"$sizeidx":0,"client_context_id":"44785b-30ee-5b49-b84d-62ff0654f23c5d","query_context":"`default`.`_default`","statement":"SELECT RAW id FROM _default WHERE size IN $size AND brand IN $brand ORDER BY ANN(vec, $qvec, \\"L2_SQUARED\\", 3) ASC LIMIT 100","timeout":"299500ms"}', 'context_type': 'QueryErrorContext'}), C Source=/Users/couchbase/jenkins/workspace/python/sdk/python-packaging-pipeline/py-client/src/n1ql.cxx:281>) 

      to proceed I need to restart indexer and it will work again for a while.

      This is single node cluster with SIFT small data and following index:

      CREATE INDEX vector_index_L2_SQUARED IF NOT EXISTS ON _default(size, brand, vec VECTOR) WITH {'dimension': 128, 'train_list': 10000, 'description': 'IVF,PQ8x8', 'similarity': 'L2_SQUARED', 'scan_nprobes': 1} 

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            sai.teja Sai Krishna Teja
            pierre.regazzoni Pierre Regazzoni
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty