Details
-
Bug
-
Status: Closed
-
Major
-
Resolution: Resolved
-
6.6.2, 7.0.0
-
Untriaged
-
1
-
Unknown
Description
Projector logs numDocsPending stats every 1min. To that extent, is spawns the go-routines to fetch seqnos from KV node. One go-routine queries ns_server every 10sec to validate the presence of a bucket in the cluster and clear the book-keeping if the bucket wa to be deleted.
In cases where there are large number of buckets, periodically querying all the bucket endpoints is adding significant load on ns_server i.e. in a 30 bucket setup, it would be 30*6 = 180 requests per min. These requests would be to "pools/default/buckets/<bucket_name>?uuid=<>" endpoint which is a relatively costly call.
We can safely increase the periodic poll to 10 min instead of 10 seconds as there will not be any functional impact to projector
Attachments
Issue Links
- is a backport of
-
-
- Closed
-
Activity
| Fix Version/s | 7.0.2 [ 18012 ] | |
| Fix Version/s | 6.6.4 [ 17614 ] | |
| Fix Version/s | Neo [ 17615 ] |
| Assignee | Varun Velamuri [ varun.velamuri ] | Sai Krishna Teja [ JIRAUSER25045 ] |
| Affects Version/s | 7.0.0 [ 17233 ] | |
| Affects Version/s | 7.0.1 [ 17104 ] | |
| Affects Version/s | Neo [ 17615 ] |
| Description |
Projector logs numDocsPending stats every 1min. To that extent, is spawns the go-routines to fetch seqnos from KV node. One go-routine queries ns_server every 10sec to validate the presence of a bucket in the cluster and clear the book-keeping if the bucket wa to be deleted.
In cases where there are large number of buckets, periodically querying all the bucket endpoints is adding significant load on ns_server i.e. in a 30 bucket setup, it would be 30*6 = 180 requests per min. These requests would be to "pools/default/buckets/<bucket_name>?uuid=<>" endpoint which is a relatively costly call. The goal of this ticket is to provide a configurable option to disable the logging of num_docs_pending stat. Logging this stat is disabled by default and it can be enabled only if some real need for it arises |
Projector logs numDocsPending stats every 1min. To that extent, is spawns the go-routines to fetch seqnos from KV node. One go-routine queries ns_server every 10sec to validate the presence of a bucket in the cluster and clear the book-keeping if the bucket wa to be deleted.
In cases where there are large number of buckets, periodically querying all the bucket endpoints is adding significant load on ns_server i.e. in a 30 bucket setup, it would be 30*6 = 180 requests per min. These requests would be to "pools/default/buckets/<bucket_name>?uuid=<>" endpoint which is a relatively costly call. We can safely increase the periodic poll to 10 min instead of 10 seconds as there will not be any functional impact to projector |
| Link | This issue blocks MB-46308 [ MB-46308 ] |
| Labels | approved-for-7.0.2 |
Build couchbase-server-7.0.2-6537 contains indexing commit d99d3b7 with commit message:
MB-47878 : [BP of MB-47666 to 7.0.2] Increase polling interval in pollForDeletedBuckets
| Resolution | Resolved [ 10400 ] | |
| Status | Open [ 1 ] | Resolved [ 5 ] |
Validated on 7.0.2-6632. Seeing the increased in duration between periodic poll.
127.0.0.1 - @projector-cbauth [03/Sep/2021:06:54:31 +0000] "GET /pools/default/buckets/default?bucket_uuid=9826e46ec494bc4ebf876c6487861816 HTTP/1.1" 200 12689 - "Go-http-client/1.1" 4
|
127.0.0.1 - @projector-cbauth [03/Sep/2021:07:04:31 +0000] "GET /pools/default/buckets/default?bucket_uuid=9826e46ec494bc4ebf876c6487861816 HTTP/1.1" 200 12687 - "Go-http-client/1.1" 5
|
127.0.0.1 - @projector-cbauth [03/Sep/2021:07:14:31 +0000] "GET /pools/default/buckets/default?bucket_uuid=9826e46ec494bc4ebf876c6487861816 HTTP/1.1" 200 12688 - "Go-http-client/1.1" 5
|
127.0.0.1 - @projector-cbauth [03/Sep/2021:07:24:31 +0000] "GET /pools/default/buckets/default?bucket_uuid=9826e46ec494bc4ebf876c6487861816 HTTP/1.1" 200 12688 - "Go-http-client/1.1" 5
|
127.0.0.1 - @projector-cbauth [03/Sep/2021:07:29:03 +0000] "GET /pools/default/buckets/default?bucket_uuid=9826e46ec494bc4ebf876c6487861816 HTTP/1.1" 200 12687 - "Go-http-client/1.1" 4
|
127.0.0.1 - @projector-cbauth [03/Sep/2021:07:29:06 +0000] "GET /pools/default/buckets/default?bucket_uuid=9826e46ec494bc4ebf876c6487861816 HTTP/1.1" 200 12687 - "Go-http-client/1.1" 5
|
127.0.0.1 - @projector-cbauth [03/Sep/2021:07:34:31 +0000] "GET /pools/default/buckets/default?bucket_uuid=9826e46ec494bc4ebf876c6487861816 HTTP/1.1" 200 12737 - "Go-http-client/1.1" 6
|
| Status | Resolved [ 5 ] | Closed [ 6 ] |
Jeelan Poola, Mihir Kamdar, I think we should take this issue for 7.0.2 as it is coming from CBSE's. Apologies for missing this out earlier. This issue reduces load on ns_server and would help in cases like rebalance.
The tentative fix is to increase the timeout for "pollForDeletedBuckets" method in dcp_seqnos_local.go from 10 seconds to 10 minutes. Functionally, there will be no impact as this method is used in the context of "numDocsPending" stat at projector. In the worst case, the stat will not be logged.
CC: Sai Krishna Teja