Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-47878

[BP of MB-47666 to 7.0.2]- CBSE - Increase the poll time in pollForDeletedBuckets at projector

    XMLWordPrintable

Details

    • Untriaged
    • 1
    • Unknown

    Description

      Projector logs numDocsPending stats every 1min. To that extent, is spawns the go-routines to fetch seqnos from KV node. One go-routine queries ns_server every 10sec to validate the presence of a bucket in the cluster and clear the book-keeping if the bucket wa to be deleted.

      In cases where there are large number of buckets, periodically querying all the bucket endpoints is adding significant load on ns_server i.e. in a 30 bucket setup, it would be 30*6 = 180 requests per min. These requests would be to "pools/default/buckets/<bucket_name>?uuid=<>" endpoint which is a relatively costly call.

      We can safely increase the periodic poll to 10 min instead of 10 seconds as there will not be any functional impact to projector

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            varun.velamuri Varun Velamuri created issue -
            varun.velamuri Varun Velamuri made changes -
            Field Original Value New Value
            Link This issue Clones MB-47666 [ MB-47666 ]
            varun.velamuri Varun Velamuri made changes -
            Fix Version/s 7.0.2 [ 18012 ]
            Fix Version/s 6.6.4 [ 17614 ]
            Fix Version/s Neo [ 17615 ]

            Jeelan Poola, Mihir Kamdar, I think we should take this issue for 7.0.2 as it is coming from CBSE's. Apologies for missing this out earlier. This issue reduces load on ns_server and would help in cases like rebalance. 

            The tentative fix is to increase the timeout for "pollForDeletedBuckets" method in dcp_seqnos_local.go from 10 seconds to 10 minutes. Functionally, there will be no impact as this method is used in the context of "numDocsPending" stat at projector. In the worst case, the stat will not be logged. 

            CC: Sai Krishna Teja

            varun.velamuri Varun Velamuri added a comment - Jeelan Poola , Mihir Kamdar , I think we should take this issue for 7.0.2 as it is coming from CBSE's. Apologies for missing this out earlier. This issue reduces load on ns_server and would help in cases like rebalance.  The tentative fix is to increase the timeout for "pollForDeletedBuckets" method in dcp_seqnos_local.go from 10 seconds to 10 minutes. Functionally, there will be no impact as this method is used in the context of "numDocsPending" stat at projector. In the worst case, the stat will not be logged.  CC: Sai Krishna Teja
            varun.velamuri Varun Velamuri made changes -
            Assignee Varun Velamuri [ varun.velamuri ] Sai Krishna Teja [ JIRAUSER25045 ]
            varun.velamuri Varun Velamuri made changes -
            Affects Version/s 7.0.0 [ 17233 ]
            Affects Version/s 7.0.1 [ 17104 ]
            Affects Version/s Neo [ 17615 ]
            varun.velamuri Varun Velamuri made changes -
            Summary [BP of MB-47666 to 7.0.2]- Provide configurable option to disbale logging numDocsPending stat for projector [BP of MB-47666 to 7.0.2]- Increase the poll time in pollForDeletedBuckets
            varun.velamuri Varun Velamuri made changes -
            Summary [BP of MB-47666 to 7.0.2]- Increase the poll time in pollForDeletedBuckets [BP of MB-47666 to 7.0.2]- Increase the poll time in pollForDeletedBuckets at projector
            varun.velamuri Varun Velamuri made changes -
            Description Projector logs numDocsPending stats every 1min. To that extent, is spawns the go-routines to fetch seqnos from KV node. One go-routine queries ns_server every 10sec to validate the presence of a bucket in the cluster and clear the book-keeping if the bucket wa to be deleted.

            In cases where there are large number of buckets, periodically querying all the bucket endpoints is adding significant load on ns_server i.e. in a 30 bucket setup, it would be 30*6 = 180 requests per min. These requests would be to "pools/default/buckets/<bucket_name>?uuid=<>" endpoint which is a relatively costly call.

            The goal of this ticket is to provide a configurable option to disable the logging of num_docs_pending stat. Logging this stat is disabled by default and it can be enabled only if some real need for it arises
            Projector logs numDocsPending stats every 1min. To that extent, is spawns the go-routines to fetch seqnos from KV node. One go-routine queries ns_server every 10sec to validate the presence of a bucket in the cluster and clear the book-keeping if the bucket wa to be deleted.

            In cases where there are large number of buckets, periodically querying all the bucket endpoints is adding significant load on ns_server i.e. in a 30 bucket setup, it would be 30*6 = 180 requests per min. These requests would be to "pools/default/buckets/<bucket_name>?uuid=<>" endpoint which is a relatively costly call.

            We can safely increase the periodic poll to 10 min instead of 10 seconds as there will not be any functional impact to projector
            jeelan.poola Jeelan Poola made changes -
            Summary [BP of MB-47666 to 7.0.2]- Increase the poll time in pollForDeletedBuckets at projector [BP of MB-47666 to 7.0.2]- CBSE - Increase the poll time in pollForDeletedBuckets at projector
            jeelan.poola Jeelan Poola added a comment -

            Mihir Kamdar FYI, the fix has no functional impact, as per Varun.

            jeelan.poola Jeelan Poola added a comment - Mihir Kamdar FYI, the fix has no functional impact, as per Varun.
            jeelan.poola Jeelan Poola made changes -
            Link This issue blocks MB-46308 [ MB-46308 ]
            jeelan.poola Jeelan Poola made changes -
            Labels approved-for-7.0.2

            Build couchbase-server-7.0.2-6537 contains indexing commit d99d3b7 with commit message:
            MB-47878 : [BP of MB-47666 to 7.0.2] Increase polling interval in pollForDeletedBuckets

            build-team Couchbase Build Team added a comment - Build couchbase-server-7.0.2-6537 contains indexing commit d99d3b7 with commit message: MB-47878 : [BP of MB-47666 to 7.0.2] Increase polling interval in pollForDeletedBuckets
            sai.teja Sai Krishna Teja made changes -
            Resolution Resolved [ 10400 ]
            Status Open [ 1 ] Resolved [ 5 ]

            Validated on 7.0.2-6632. Seeing the increased in duration between periodic poll.

             

            127.0.0.1 - @projector-cbauth [03/Sep/2021:06:54:31 +0000] "GET /pools/default/buckets/default?bucket_uuid=9826e46ec494bc4ebf876c6487861816 HTTP/1.1" 200 12689 - "Go-http-client/1.1" 4
            127.0.0.1 - @projector-cbauth [03/Sep/2021:07:04:31 +0000] "GET /pools/default/buckets/default?bucket_uuid=9826e46ec494bc4ebf876c6487861816 HTTP/1.1" 200 12687 - "Go-http-client/1.1" 5
            127.0.0.1 - @projector-cbauth [03/Sep/2021:07:14:31 +0000] "GET /pools/default/buckets/default?bucket_uuid=9826e46ec494bc4ebf876c6487861816 HTTP/1.1" 200 12688 - "Go-http-client/1.1" 5
            127.0.0.1 - @projector-cbauth [03/Sep/2021:07:24:31 +0000] "GET /pools/default/buckets/default?bucket_uuid=9826e46ec494bc4ebf876c6487861816 HTTP/1.1" 200 12688 - "Go-http-client/1.1" 5
            127.0.0.1 - @projector-cbauth [03/Sep/2021:07:29:03 +0000] "GET /pools/default/buckets/default?bucket_uuid=9826e46ec494bc4ebf876c6487861816 HTTP/1.1" 200 12687 - "Go-http-client/1.1" 4
            127.0.0.1 - @projector-cbauth [03/Sep/2021:07:29:06 +0000] "GET /pools/default/buckets/default?bucket_uuid=9826e46ec494bc4ebf876c6487861816 HTTP/1.1" 200 12687 - "Go-http-client/1.1" 5
            127.0.0.1 - @projector-cbauth [03/Sep/2021:07:34:31 +0000] "GET /pools/default/buckets/default?bucket_uuid=9826e46ec494bc4ebf876c6487861816 HTTP/1.1" 200 12737 - "Go-http-client/1.1" 6

            hemant.rajput Hemant Rajput added a comment - Validated on 7.0.2-6632. Seeing the increased in duration between periodic poll.   127.0.0.1 - @projector-cbauth [03/Sep/2021:06:54:31 +0000] "GET /pools/default/buckets/default?bucket_uuid=9826e46ec494bc4ebf876c6487861816 HTTP/1.1" 200 12689 - "Go-http-client/1.1" 4 127.0.0.1 - @projector-cbauth [03/Sep/2021:07:04:31 +0000] "GET /pools/default/buckets/default?bucket_uuid=9826e46ec494bc4ebf876c6487861816 HTTP/1.1" 200 12687 - "Go-http-client/1.1" 5 127.0.0.1 - @projector-cbauth [03/Sep/2021:07:14:31 +0000] "GET /pools/default/buckets/default?bucket_uuid=9826e46ec494bc4ebf876c6487861816 HTTP/1.1" 200 12688 - "Go-http-client/1.1" 5 127.0.0.1 - @projector-cbauth [03/Sep/2021:07:24:31 +0000] "GET /pools/default/buckets/default?bucket_uuid=9826e46ec494bc4ebf876c6487861816 HTTP/1.1" 200 12688 - "Go-http-client/1.1" 5 127.0.0.1 - @projector-cbauth [03/Sep/2021:07:29:03 +0000] "GET /pools/default/buckets/default?bucket_uuid=9826e46ec494bc4ebf876c6487861816 HTTP/1.1" 200 12687 - "Go-http-client/1.1" 4 127.0.0.1 - @projector-cbauth [03/Sep/2021:07:29:06 +0000] "GET /pools/default/buckets/default?bucket_uuid=9826e46ec494bc4ebf876c6487861816 HTTP/1.1" 200 12687 - "Go-http-client/1.1" 5 127.0.0.1 - @projector-cbauth [03/Sep/2021:07:34:31 +0000] "GET /pools/default/buckets/default?bucket_uuid=9826e46ec494bc4ebf876c6487861816 HTTP/1.1" 200 12737 - "Go-http-client/1.1" 6
            hemant.rajput Hemant Rajput made changes -
            Status Resolved [ 5 ] Closed [ 6 ]
            wayne Wayne Siu made changes -
            Link This issue is a backport of MB-47666 [ MB-47666 ]
            wayne Wayne Siu made changes -
            Link This issue Clones MB-47666 [ MB-47666 ]

            People

              sai.teja Sai Krishna Teja
              varun.velamuri Varun Velamuri
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty