Uploaded image for project: 'Couchbase Python Client Library'
  1. Couchbase Python Client Library
  2. PYCBC-1201

Memory Leak with Python SDK 3.2.3 (or 3.2.2)

    XMLWordPrintable

Details

    • Bug
    • Resolution: Resolved
    • Major
    • 4.0.3
    • 3.2.2, 3.2.3
    • library
    • None
    • Fedora 34 Python 3.9
    • 1

    Description

      I'm trying to migrate from Couchbase 2.5.12 into 3.2.2 (or 3.2.3) but I'm getting a mem leak. I faced this same Problem with 2.5.12 and I was able to sort that problem by disabling tracing in the connection string like this: 
      couchbase://localhost:8091?operation_timeout=15&bootstrap_on=http&detailed_errcodes=1&enable_tracing=false
      {{}}
      When moved to Python SDK 3.2.2 I noticed that I have the same (mem leak) problem, so I tried to disable tracing by:

      • Using same conn string (with enable_tracing=false) without luck, mem leak still persists
      • Pass a ClusterTracingOptions instance to Cluster init with all fields set to None
      • Pass a Tracer (dummy obj) to Cluster init
      • Tried pass both ^ to Cluster init (tracing_options and tracer)

      Here is a complete example:

      def multi_proc_insert(a_id, f_path):
          tracing_opts = ClusterTracingOptions(
              **dict(
                  tracing_threshold_kv=None,
                  tracing_threshold_view=None,
                  tracing_threshold_query=None,
                  tracing_threshold_search=None,
                  tracing_threshold_analytics=None,
                  tracing_threshold_queue_size=None,
                  tracing_threshold_queue_flush_interval=None,
                  tracing_orphaned_queue_size=None,
                  tracing_orphaned_queue_flush_interval=None,
              )
          )
          tracer = trace.get_tracer(__name__)
          clu = Cluster(
              COUCH_HOST,
              ClusterOptions(
                  PasswordAuthenticator(COUCH_USER, COUCH_PASS),
              ),
              tracing_options=tracing_opts,
              tracer=CouchbaseOtelTracer(otel_tracer=tracer),
          )
          cbu = clu.bucket(BUCKET_NAME)
          collection = cbu.default_collection()
          lines = mapcount(f_path)
          logger.info(f"Start to process {f_path}\t lines: {lines}")
          inserted = 0
       
          t1 = time.time()
          with open(f_path, "r") as fp:
              while True:
                  e_hash = fp.readline()
                  if not e_hash:
                      logger.info(f"e_hash not found (EOF)")
                      break
                  key = DOC_TYPE + e_hash.strip()
                  try:
                      collection.mutate_in(key, [SD.array_addunique("", a_id)])
                  except PathExistsException:
                      logger.debug(f"a_id {a_id} already exists on doc {key}")
                  except DocumentNotFoundException:
                      collection.insert(key, [a_id])
                  except Exception:
                      logger.exception(
                          f"Error processing {f_path}, inserted rows: {inserted} remaining: {lines - inserted}"
                      )
                  inserted += 1
          t2 = time.time()
          logger.info(
              f"Finished with {f_path}\t lines: {lines} inserted: {inserted} remaining: {lines - inserted} in seconds {t2 - t1}"
          )
      

      Attachments

        1. suppressor.log
          17 kB
        2. multiproc_tracing.py
          4 kB
        3. disable_tracing.py
          1 kB

        Issue Links

          Activity

            People

              jared.casey Jared Casey
              edvm Emiliano Dalla Verde Marcozzi
              Votes:
              2 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                PagerDuty