Skip to:

  1. Skip to Jira Navigation
  2. Skip to Side Navigation
  3. Skip to Main Content

[BP 7.2.0] - XDCR - Prolonged TMPFAIL or ENOMEM could lead to memory bloat

Description

  1. XDCR receives ENOMEM and prepares the packet for resend (http://src.couchbase.org/source/xref/6.6.2/goproj/src/github.com/couchbase/goxdcr/parts/xmem_nozzle.go#2041)

  2. Leading to this call to get bytes: http://src.couchbase.org/source/xref/6.6.2/goproj/src/github.com/couchbase/goxdcr/parts/xmem_nozzle.go#2426-2427

  3. At first glance, the above looks to be using a slice reference, but digging into gomemcached…

  4. http://src.couchbase.org/source/xref/6.6.2/godeps/src/github.com/couchbase/gomemcached/mc_req.go#104-118 <- this call actually make copies of the data

So even if the stats do not move (i.e. the analysis I’ve shown), and yet target keeps returning ENOMEM (and potentially TMPFAIL), XDCR will keep retrying to resend…. but each resend actually allocates memory instead of reusing existing ones.

Components

Affects versions

Fix versions

Labels

Environment

None

Link to Log File, atop/blg, CBCollectInfo, Core dump

None

Release Notes Description

None

Activity

Show:

CB robot May 17, 2023 at 1:42 AM

Build couchbase-server-7.1.5-3804 contains gomemcached commit a6b8a5b with commit message:
: Add BytesPreallocated method

Ayush Nayyar April 12, 2023 at 8:41 AM

Replicated on 7.2.0-5090, validated fix on 7.2.0-5093.

CB robot January 21, 2023 at 12:53 AM

Build couchbase-server-7.2.0-5093 contains goxdcr commit 8d16124 with commit message:
: reuse allocated byte slice when retrying due to target temp errors

CB robot January 21, 2023 at 12:53 AM

Build couchbase-server-7.2.0-5093 contains gomemcached commit a6b8a5b with commit message:
: Add BytesPreallocated method

Fixed
Pinned fields
Click on the next to a field label to start pinning.

Details

Assignee

Reporter

Is this a Regression?

No

Triage

Untriaged

Story Points

Priority

Instabug

Open Instabug

PagerDuty

Sentry

Zendesk Support

Created July 21, 2022 at 5:27 PM
Updated May 17, 2023 at 1:42 AM
Resolved January 20, 2023 at 10:45 PM
Instabug