Uploaded image for project: 'Couchbase Python Client Library'
  1. Couchbase Python Client Library
  2. PYCBC-1230

Segfault and/or StopIteration error when executing queries in multithreading app

    XMLWordPrintable

Details

    • Bug
    • Resolution: Unresolved
    • Major
    • .backlog3.x
    • 3.2.5, 3.2.6
    • None
    • None
    • 1

    Description

      Running the following code sample can produce segmentation faults and/or StopIteration exceptions. It would appear the C-bindings are not probably handling an error so a SystemError is returned which then causes the seg fault/StopIteration error.

      Tasks:

      • Need to do some research into the bindings to determine what might be the cause.
      • Might need better locking around the query generators? Need to research.
      • Verify code snippet provided.

      Issue came from this forum post.

      import argparse
      import random
      from concurrent.futures import ThreadPoolExecutor, ProcessPoolExecutor, as_completed
      import time
      import traceback
       
       
      from couchbase.auth import PasswordAuthenticator
      from couchbase.cluster import Cluster, ClusterOptions, QueryOptions
      from couchbase.options import LockMode
       
       
      class CouchbaseConfig:
          uri = "couchbase://localhost"
          user = "YourUsername"
          password = "YourPassword"
          cert_path = None
       
      sql = 'SELECT * FROM `travel-sample`.inventory.airport WHERE airportname = $1 LIMIT 10'
       
       
      def get_couchbase():
          cb = Cluster(CouchbaseConfig.uri, ClusterOptions(
                  PasswordAuthenticator(CouchbaseConfig.user, CouchbaseConfig.password, CouchbaseConfig.cert_path),
                  # unlock_gil=True,
                  lockmode=LockMode.WAIT
              )
          )
          return cb
       
      # cb = get_couchbase()
      def run_query_loop(cnt=1000):
          cb = get_couchbase()
          print(id(cb))
          airportnames = ["Calais Dunkerque", "Les Loges", "Bray", "abc", "efg", "hjk"]
          for _ in range(cnt):
              try:
                  airportname = random.choice(airportnames)
                  result = cb.query(sql, QueryOptions(metrics=True, positional_parameters=(airportname,)))
                  for row in result:
                  # for row in result.rows():
                      # print("Found row: {}".format(row))
                      pass
              except:
                  print(f"sql => {sql}, args => {airportname}")
                  traceback.print_exc()
                  raise
          return "success"
       
       
       
      def get_args():
          parser = argparse.ArgumentParser()
          parser.add_argument('workers', metavar='int', type=int, help='number of thread/prcoess')
          parser.add_argument('q_count', metavar='int', type=int, help='run n1ql query times')
          parser.add_argument('use_thread', metavar='int', type=int, default=0, help='use thread to run task')
          return parser.parse_args()
       
       
      if __name__ == '__main__':
          args = get_args()
          bt = time.time()
          futs = []
          if args.use_thread:
              with ThreadPoolExecutor(args.workers) as pool:
                  for _ in range(args.workers):
                      futs.append(pool.submit(run_query_loop, args.q_count))
          else:
              with ProcessPoolExecutor(args.workers) as pool:
                  for _ in range(args.workers):
                      futs.append(pool.submit(run_query_loop, args.q_count))
       
          for fut in as_completed(futs):
              exc = fut.exception()
              if exc:
                  print("future error ===> ", exc)
              else:
                  print("future result ===> ", fut.result())
       
          print("main end: ", time.time() - bt)
          # run example:
          # python test_n1ql.py 4 1000 1
      
      

      Attachments

        No reviews matched the request. Check your Options in the drop-down menu of this sections header.

        Activity

          People

            jared.casey Jared Casey
            jared.casey Jared Casey
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:

              Gerrit Reviews

                There are no open Gerrit changes

                PagerDuty