Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-47878

[BP of MB-47666 to 7.0.2]- CBSE - Increase the poll time in pollForDeletedBuckets at projector

    XMLWordPrintable

Details

    • Untriaged
    • 1
    • Unknown

    Description

      Projector logs numDocsPending stats every 1min. To that extent, is spawns the go-routines to fetch seqnos from KV node. One go-routine queries ns_server every 10sec to validate the presence of a bucket in the cluster and clear the book-keeping if the bucket wa to be deleted.

      In cases where there are large number of buckets, periodically querying all the bucket endpoints is adding significant load on ns_server i.e. in a 30 bucket setup, it would be 30*6 = 180 requests per min. These requests would be to "pools/default/buckets/<bucket_name>?uuid=<>" endpoint which is a relatively costly call.

      We can safely increase the periodic poll to 10 min instead of 10 seconds as there will not be any functional impact to projector

      Attachments

        Issue Links

          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            Jeelan Poola, Mihir Kamdar, I think we should take this issue for 7.0.2 as it is coming from CBSE's. Apologies for missing this out earlier. This issue reduces load on ns_server and would help in cases like rebalance. 

            The tentative fix is to increase the timeout for "pollForDeletedBuckets" method in dcp_seqnos_local.go from 10 seconds to 10 minutes. Functionally, there will be no impact as this method is used in the context of "numDocsPending" stat at projector. In the worst case, the stat will not be logged. 

            CC: Sai Krishna Teja

            varun.velamuri Varun Velamuri added a comment - Jeelan Poola , Mihir Kamdar , I think we should take this issue for 7.0.2 as it is coming from CBSE's. Apologies for missing this out earlier. This issue reduces load on ns_server and would help in cases like rebalance.  The tentative fix is to increase the timeout for "pollForDeletedBuckets" method in dcp_seqnos_local.go from 10 seconds to 10 minutes. Functionally, there will be no impact as this method is used in the context of "numDocsPending" stat at projector. In the worst case, the stat will not be logged.  CC: Sai Krishna Teja
            jeelan.poola Jeelan Poola added a comment -

            Mihir Kamdar FYI, the fix has no functional impact, as per Varun.

            jeelan.poola Jeelan Poola added a comment - Mihir Kamdar FYI, the fix has no functional impact, as per Varun.

            Build couchbase-server-7.0.2-6537 contains indexing commit d99d3b7 with commit message:
            MB-47878 : [BP of MB-47666 to 7.0.2] Increase polling interval in pollForDeletedBuckets

            build-team Couchbase Build Team added a comment - Build couchbase-server-7.0.2-6537 contains indexing commit d99d3b7 with commit message: MB-47878 : [BP of MB-47666 to 7.0.2] Increase polling interval in pollForDeletedBuckets

            Validated on 7.0.2-6632. Seeing the increased in duration between periodic poll.

             

            127.0.0.1 - @projector-cbauth [03/Sep/2021:06:54:31 +0000] "GET /pools/default/buckets/default?bucket_uuid=9826e46ec494bc4ebf876c6487861816 HTTP/1.1" 200 12689 - "Go-http-client/1.1" 4
            127.0.0.1 - @projector-cbauth [03/Sep/2021:07:04:31 +0000] "GET /pools/default/buckets/default?bucket_uuid=9826e46ec494bc4ebf876c6487861816 HTTP/1.1" 200 12687 - "Go-http-client/1.1" 5
            127.0.0.1 - @projector-cbauth [03/Sep/2021:07:14:31 +0000] "GET /pools/default/buckets/default?bucket_uuid=9826e46ec494bc4ebf876c6487861816 HTTP/1.1" 200 12688 - "Go-http-client/1.1" 5
            127.0.0.1 - @projector-cbauth [03/Sep/2021:07:24:31 +0000] "GET /pools/default/buckets/default?bucket_uuid=9826e46ec494bc4ebf876c6487861816 HTTP/1.1" 200 12688 - "Go-http-client/1.1" 5
            127.0.0.1 - @projector-cbauth [03/Sep/2021:07:29:03 +0000] "GET /pools/default/buckets/default?bucket_uuid=9826e46ec494bc4ebf876c6487861816 HTTP/1.1" 200 12687 - "Go-http-client/1.1" 4
            127.0.0.1 - @projector-cbauth [03/Sep/2021:07:29:06 +0000] "GET /pools/default/buckets/default?bucket_uuid=9826e46ec494bc4ebf876c6487861816 HTTP/1.1" 200 12687 - "Go-http-client/1.1" 5
            127.0.0.1 - @projector-cbauth [03/Sep/2021:07:34:31 +0000] "GET /pools/default/buckets/default?bucket_uuid=9826e46ec494bc4ebf876c6487861816 HTTP/1.1" 200 12737 - "Go-http-client/1.1" 6

            hemant.rajput Hemant Rajput added a comment - Validated on 7.0.2-6632. Seeing the increased in duration between periodic poll.   127.0.0.1 - @projector-cbauth [03/Sep/2021:06:54:31 +0000] "GET /pools/default/buckets/default?bucket_uuid=9826e46ec494bc4ebf876c6487861816 HTTP/1.1" 200 12689 - "Go-http-client/1.1" 4 127.0.0.1 - @projector-cbauth [03/Sep/2021:07:04:31 +0000] "GET /pools/default/buckets/default?bucket_uuid=9826e46ec494bc4ebf876c6487861816 HTTP/1.1" 200 12687 - "Go-http-client/1.1" 5 127.0.0.1 - @projector-cbauth [03/Sep/2021:07:14:31 +0000] "GET /pools/default/buckets/default?bucket_uuid=9826e46ec494bc4ebf876c6487861816 HTTP/1.1" 200 12688 - "Go-http-client/1.1" 5 127.0.0.1 - @projector-cbauth [03/Sep/2021:07:24:31 +0000] "GET /pools/default/buckets/default?bucket_uuid=9826e46ec494bc4ebf876c6487861816 HTTP/1.1" 200 12688 - "Go-http-client/1.1" 5 127.0.0.1 - @projector-cbauth [03/Sep/2021:07:29:03 +0000] "GET /pools/default/buckets/default?bucket_uuid=9826e46ec494bc4ebf876c6487861816 HTTP/1.1" 200 12687 - "Go-http-client/1.1" 4 127.0.0.1 - @projector-cbauth [03/Sep/2021:07:29:06 +0000] "GET /pools/default/buckets/default?bucket_uuid=9826e46ec494bc4ebf876c6487861816 HTTP/1.1" 200 12687 - "Go-http-client/1.1" 5 127.0.0.1 - @projector-cbauth [03/Sep/2021:07:34:31 +0000] "GET /pools/default/buckets/default?bucket_uuid=9826e46ec494bc4ebf876c6487861816 HTTP/1.1" 200 12737 - "Go-http-client/1.1" 6

            People

              sai.teja Sai Krishna Teja
              varun.velamuri Varun Velamuri
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Gerrit Reviews

                  There are no open Gerrit changes

                  PagerDuty