[MB-42819] Memory leak within gRPC's resetTransport(..) due to context.WithCancel(..) - Couchbase database

Abhinav Dangeti created issue - 18/Nov/20 10:39 AM

Abhinav Dangeti made changes - 18/Nov/20 10:55 AM

Field	Original Value	New Value
Description	N1QL/FTS use grpc version 1.17. There seems to be a leak within grpc's resetTransport as pointed out here - [https://github.com/grpc/grpc-go/issues/3667] and [https://github.com/etcd-io/etcd/issues/11371] which is addressed by [https://github.com/grpc/grpc-go/pull/2985] for version 1.24 Consider upgrading grpc within these projects to v1.24.0? * [https://github.com/couchbase/cbft\|http://github.com/couchbase/cbft] * [https://github.com/couchbase/cbftx] * [https://github.com/couchbase/query\|http://github.com/couchbase/query] * [https://github.com/couchbase/n1fty\|http://github.com/couchbase/n1fty]	N1QL/FTS use grpc version 1.17. There seems to be a leak within grpc's resetTransport as pointed out here - * [https://github.com/grpc/grpc-go/issues/3667] * [https://github.com/etcd-io/etcd/issues/11371] * [https://github.com/grpc/grpc-go/issues/2976] which is addressed by [https://github.com/grpc/grpc-go/pull/2985] for version 1.24 Consider upgrading grpc within these projects to v1.24.0? * [https://github.com/couchbase/cbft\|http://github.com/couchbase/cbft] * [https://github.com/couchbase/cbftx] * [https://github.com/couchbase/query\|http://github.com/couchbase/query] * [https://github.com/couchbase/n1fty\|http://github.com/couchbase/n1fty]

Abhinav Dangeti made changes - 18/Nov/20 10:56 AM

Link

This issue relates to CBSE-9208 [ CBSE-9208 ]

Sitaram Vemulapalli made changes - 18/Nov/20 11:11 AM

Fix Version/s

6.6.2 [ 17103 ]

Abhinav Dangeti added a comment - 18/Nov/20 1:45 PM

For 7.0 builds ..

Abhinav Dangeti added a comment - 18/Nov/20 1:45 PM For 7.0 builds .. http://review.couchbase.org/c/cbft/+/140595 http://review.couchbase.org/c/cbftx/+/140596 http://review.couchbase.org/c/n1fty/+/140597 http://review.couchbase.org/c/query/+/140598

Abhinav Dangeti made changes - 18/Nov/20 1:45 PM

Summary

Connection leak within gRPC's resetTransport

Memory leak within gRPC's resetTransport

Abhinav Dangeti made changes - 18/Nov/20 1:45 PM

Summary

Memory leak within gRPC's resetTransport

Memory leak within gRPC's resetTransport(..)

Couchbase Build Team added a comment - 18/Nov/20 2:00 PM

Build couchbase-server-7.0.0-3792 contains cbft commit b704118 with commit message:
~~MB-42819~~: Upgrade grpc version from 1.17.0 -> 1.24.0

Couchbase Build Team added a comment - 18/Nov/20 2:00 PM Build couchbase-server-7.0.0-3792 contains cbft commit b704118 with commit message: MB-42819 : Upgrade grpc version from 1.17.0 -> 1.24.0

Couchbase Build Team added a comment - 18/Nov/20 2:00 PM

Build couchbase-server-7.0.0-3792 contains cbftx commit 6ce92cd with commit message:
~~MB-42819~~: grpc 1.17.0 -> 1.24.0

Couchbase Build Team added a comment - 18/Nov/20 2:00 PM Build couchbase-server-7.0.0-3792 contains cbftx commit 6ce92cd with commit message: MB-42819 : grpc 1.17.0 -> 1.24.0

Couchbase Build Team added a comment - 18/Nov/20 2:00 PM

Build couchbase-server-7.0.0-3792 contains n1fty commit cc88733 with commit message:
~~MB-42819~~: grpc 1.17.0 -> 1.24.0

Couchbase Build Team added a comment - 18/Nov/20 2:00 PM Build couchbase-server-7.0.0-3792 contains n1fty commit cc88733 with commit message: MB-42819 : grpc 1.17.0 -> 1.24.0

Couchbase Build Team added a comment - 18/Nov/20 2:00 PM

Build couchbase-server-7.0.0-3792 contains query commit e8dcce7 with commit message:
~~MB-42819~~: grpc 1.17.0 -> 1.24.0

Couchbase Build Team added a comment - 18/Nov/20 2:00 PM Build couchbase-server-7.0.0-3792 contains query commit e8dcce7 with commit message: MB-42819 : grpc 1.17.0 -> 1.24.0

Abhinav Dangeti added a comment - 18/Nov/20 3:04 PM - edited

Here's a proposal for a manifest update for 6.6.2 builds .. http://review.couchbase.org/c/manifest/+/140601 to ..

grpc version to v1.24.0
protobuf version to v1.3.5
net version update to same as master

(Pending approval for 6.6.2)

Abhinav Dangeti added a comment - 18/Nov/20 3:04 PM - edited Here's a proposal for a manifest update for 6.6.2 builds .. http://review.couchbase.org/c/manifest/+/140601 to .. grpc version to v1.24.0 protobuf version to v1.3.5 net version update to same as master (Pending approval for 6.6.2)

Abhinav Dangeti made changes - 19/Nov/20 9:11 AM

Summary

Memory leak within gRPC's resetTransport(..)

Memory leak within gRPC's resetTransport(..) due to context without Timeout

Abhinav Dangeti made changes - 19/Nov/20 9:11 AM

Summary

Memory leak within gRPC's resetTransport(..) due to context without Timeout

Memory leak within gRPC's resetTransport(..)

Abhinav Dangeti made changes - 19/Nov/20 9:13 AM

Description

N1QL/FTS use grpc version 1.17.

There seems to be a leak within grpc's resetTransport as pointed out here -
* [https://github.com/grpc/grpc-go/issues/3667]
* [https://github.com/etcd-io/etcd/issues/11371]
* [https://github.com/grpc/grpc-go/issues/2976]

which is addressed by [https://github.com/grpc/grpc-go/pull/2985] for version 1.24

Consider upgrading grpc within these projects to v1.24.0?
* [https://github.com/couchbase/cbft|http://github.com/couchbase/cbft]
* [https://github.com/couchbase/cbftx]
* [https://github.com/couchbase/query|http://github.com/couchbase/query]
* [https://github.com/couchbase/n1fty|http://github.com/couchbase/n1fty]

N1QL/FTS use grpc version 1.17.

There seems to be a leak within grpc's resetTransport as pointed out here -
* [https://github.com/grpc/grpc-go/issues/3667]
* [https://github.com/etcd-io/etcd/issues/11371]
* [https://github.com/grpc/grpc-go/issues/2976]

which is addressed by [https://github.com/grpc/grpc-go/pull/2985] for version 1.24

Consider upgrading grpc within these projects to v1.24.0?
* [https://github.com/couchbase/cbft|http://github.com/couchbase/cbft]
* [https://github.com/couchbase/cbftx]
* [https://github.com/couchbase/query|http://github.com/couchbase/query]
* [https://github.com/couchbase/n1fty|http://github.com/couchbase/n1fty]

The leak is coming from context.WithCancel as pointed out here: [https://github.com/grpc/grpc-go/issues/2976].

Setting a timeout with context is a workaround for this - by setting a deadline for the datastore.IndexConnection which is handed to n1fty's Search(..).

Abhinav Dangeti made changes - 19/Nov/20 9:19 AM

Description

N1QL/FTS use grpc version 1.17.

There seems to be a leak within grpc's resetTransport as pointed out here -
* [https://github.com/grpc/grpc-go/issues/3667]
* [https://github.com/etcd-io/etcd/issues/11371]
* [https://github.com/grpc/grpc-go/issues/2976]

which is addressed by [https://github.com/grpc/grpc-go/pull/2985] for version 1.24

Consider upgrading grpc within these projects to v1.24.0?
* [https://github.com/couchbase/cbft|http://github.com/couchbase/cbft]
* [https://github.com/couchbase/cbftx]
* [https://github.com/couchbase/query|http://github.com/couchbase/query]
* [https://github.com/couchbase/n1fty|http://github.com/couchbase/n1fty]

The leak is coming from context.WithCancel as pointed out here: [https://github.com/grpc/grpc-go/issues/2976].

Setting a timeout with context is a workaround for this - by setting a deadline for the datastore.IndexConnection which is handed to n1fty's Search(..).

N1QL/FTS use grpc version 1.17.

There seems to be a leak within grpc's resetTransport as pointed out here -
* [https://github.com/grpc/grpc-go/issues/3667]
* [https://github.com/etcd-io/etcd/issues/11371]
* [https://github.com/grpc/grpc-go/issues/2976]

which is addressed by [https://github.com/grpc/grpc-go/pull/2985] for version 1.24

Consider upgrading grpc within these projects to v1.24.0?
* [https://github.com/couchbase/cbft|http://github.com/couchbase/cbft]
* [https://github.com/couchbase/cbftx]
* [https://github.com/couchbase/query|http://github.com/couchbase/query]
* [https://github.com/couchbase/n1fty|http://github.com/couchbase/n1fty]

The leak is coming from context.WithCancel as pointed out here: [https://github.com/grpc/grpc-go/issues/2976].

Setting a timeout with context is a workaround for this - by setting a deadline for the datastore.IndexConnection which is handed to n1fty's Search(..). But WithTimeout for context was introduced in 7.0, so this is not a viable workaround for older builds.

Abhinav Dangeti made changes - 20/Nov/20 3:58 PM

Summary

Memory leak within gRPC's resetTransport(..)

Memory leak within gRPC's resetTransport(..) due to context.WithCancel(..)

Wayne Siu made changes - 23/Nov/20 5:15 PM

Fix Version/s		6.6.1 [ 17002 ]
Fix Version/s	6.6.2 [ 17103 ]

Wayne Siu made changes - 23/Nov/20 5:15 PM

Labels

approved-for-6.6.1

Wayne Siu made changes - 23/Nov/20 5:16 PM

Link

This issue blocks MB-40528 [ MB-40528 ]

Abhinav Dangeti made changes - 23/Nov/20 6:36 PM

Resolution		Fixed [ 1 ]
Status	Open [ 1 ]	Resolved [ 5 ]

Wayne Siu made changes - 23/Nov/20 7:58 PM

Assignee

Abhinav Dangeti [ abhinav ]

Mihir Kamdar [ mihir.kamdar ]

Girish Benakappa added a comment - 01/Dec/20 12:11 PM

Closing as there were no regressions seen with build 6.6.1-9198 w.r.t gRPC.

Girish Benakappa added a comment - 01/Dec/20 12:11 PM Closing as there were no regressions seen with build 6.6.1-9198 w.r.t gRPC.

Girish Benakappa made changes - 01/Dec/20 12:11 PM

Status

Resolved [ 5 ]

Closed [ 6 ]

Lynn Straus made changes - 17/Jun/21 3:22 PM

Fix Version/s

7.0.0 [ 17233 ]

Lynn Straus made changes - 17/Jun/21 3:32 PM

Fix Version/s

Cheshire-Cat [ 15915 ]

Couchbase Server

Memory leak within gRPC's resetTransport(..) due to context.WithCancel(..)

Details

Description

Attachments

Gerrit Reviews

Activity

People

Dates

Gerrit Reviews

PagerDuty