Description
I'm trying to migrate from Couchbase 2.5.12 into 3.2.2 (or 3.2.3) but I'm getting a mem leak. I faced this same Problem with 2.5.12 and I was able to sort that problem by disabling tracing in the connection string like this:
couchbase://localhost:8091?operation_timeout=15&bootstrap_on=http&detailed_errcodes=1&enable_tracing=false
{{}}
When moved to Python SDK 3.2.2 I noticed that I have the same (mem leak) problem, so I tried to disable tracing by:
- Using same conn string (with enable_tracing=false) without luck, mem leak still persists
- Pass a ClusterTracingOptions instance to Cluster init with all fields set to None
- Pass a Tracer (dummy obj) to Cluster init
- Tried pass both ^ to Cluster init (tracing_options and tracer)
Here is a complete example:
def multi_proc_insert(a_id, f_path):
|
tracing_opts = ClusterTracingOptions(
|
**dict(
|
tracing_threshold_kv=None,
|
tracing_threshold_view=None,
|
tracing_threshold_query=None,
|
tracing_threshold_search=None,
|
tracing_threshold_analytics=None,
|
tracing_threshold_queue_size=None,
|
tracing_threshold_queue_flush_interval=None,
|
tracing_orphaned_queue_size=None,
|
tracing_orphaned_queue_flush_interval=None,
|
)
|
)
|
tracer = trace.get_tracer(__name__)
|
clu = Cluster(
|
COUCH_HOST,
|
ClusterOptions(
|
PasswordAuthenticator(COUCH_USER, COUCH_PASS),
|
),
|
tracing_options=tracing_opts,
|
tracer=CouchbaseOtelTracer(otel_tracer=tracer),
|
)
|
cbu = clu.bucket(BUCKET_NAME)
|
collection = cbu.default_collection()
|
lines = mapcount(f_path)
|
logger.info(f"Start to process {f_path}\t lines: {lines}") |
inserted = 0 |
|
|
t1 = time.time()
|
with open(f_path, "r") as fp: |
while True: |
e_hash = fp.readline()
|
if not e_hash: |
logger.info(f"e_hash not found (EOF)") |
break |
key = DOC_TYPE + e_hash.strip()
|
try: |
collection.mutate_in(key, [SD.array_addunique("", a_id)]) |
except PathExistsException:
|
logger.debug(f"a_id {a_id} already exists on doc {key}") |
except DocumentNotFoundException:
|
collection.insert(key, [a_id])
|
except Exception:
|
logger.exception(
|
f"Error processing {f_path}, inserted rows: {inserted} remaining: {lines - inserted}" |
)
|
inserted += 1 |
t2 = time.time()
|
logger.info(
|
f"Finished with {f_path}\t lines: {lines} inserted: {inserted} remaining: {lines - inserted} in seconds {t2 - t1}" |
)
|
Attachments
Issue Links
- relates to
-
PYCBC-1382 Custom Python types should call tp_free in their destructor
-
- Resolved
-