Uploaded image for project: 'Couchbase Server'
  1. Couchbase Server
  2. MB-47666

Increase the poll time in pollForDeletedBuckets at projector

    XMLWordPrintable

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 6.6.2, 7.1.0, 7.0.1
    • 7.1.0
    • secondary-index
    • None
    • Untriaged
    • 1
    • Unknown

    Description

      Projector logs numDocsPending stats every 1min. To that extent, is spawns the go-routines to fetch seqnos from KV node. One go-routine queries ns_server every 10sec to validate the presence of a bucket in the cluster and clear the book-keeping if the bucket wa to be deleted.

      In cases where there are large number of buckets, periodically querying all the bucket endpoints is adding significant load on ns_server i.e. in a 30 bucket setup, it would be 30*6 = 180 requests per min. These requests would be to "pools/default/buckets/<bucket_name>?uuid=<>" endpoint which is a relatively costly call.

      Attachments

        Issue Links

          backports to

          Bug - A problem which impairs or prevents the functions of the product. MB-47674 [BP of MB-47666] Provide configurable option to disbale logging numDocsPending stat for projector

          • Major - Major loss of function.
          • Closed

          Bug - A problem which impairs or prevents the functions of the product. MB-47878 [BP of MB-47666 to 7.0.2]- CBSE - Increase the poll time in pollForDeletedBuckets at projector

          • Major - Major loss of function.
          • Closed
          No reviews matched the request. Check your Options in the drop-down menu of this sections header.

          Activity

            Ascending order - Click to sort in descending order
            build-team Couchbase Build Team added a comment -

            Build couchbase-server-7.0.2-6537 contains indexing commit d99d3b7 with commit message:
            MB-47878 : [BP of MB-47666 to 7.0.2] Increase polling interval in pollForDeletedBuckets

            build-team Couchbase Build Team added a comment - Build couchbase-server-7.0.2-6537 contains indexing commit d99d3b7 with commit message: MB-47878 : [BP of MB-47666 to 7.0.2] Increase polling interval in pollForDeletedBuckets
            build-team Couchbase Build Team added a comment -

            Build couchbase-server-7.1.0-1186 contains indexing commit f5acc40 with commit message:
            MB-47666 : Increase polling interval in pollForDeletedBuckets

            build-team Couchbase Build Team added a comment - Build couchbase-server-7.1.0-1186 contains indexing commit f5acc40 with commit message: MB-47666 : Increase polling interval in pollForDeletedBuckets
            jeelan.poola Jeelan Poola added a comment -

            Varun Velamuri Can this be resolved now?

            jeelan.poola Jeelan Poola added a comment - Varun Velamuri Can this be resolved now?
            hemant.rajput Hemant Rajput added a comment -

            Validated on Enterprise Edition 6.6.5 build 10067

             

            [root@node1-cb660-centos7 logs]# grep pools/default/buckets/travel-sample?bucket http_access_internal.log | grep @projector-cbauth |tail 
            		 
            127.0.0.1 - @projector-cbauth [22/Dec/2021:06:31:11 +0000] "GET /pools/default/buckets/travel-sample?bucket_uuid=d12bf81ec72a1b87d9e9f2bc3194bf9a HTTP/1.1" 200 10438 - Go-http-client/1.1
            		 
            127.0.0.1 - @projector-cbauth [22/Dec/2021:06:41:11 +0000] "GET /pools/default/buckets/travel-sample?bucket_uuid=d12bf81ec72a1b87d9e9f2bc3194bf9a HTTP/1.1" 200 10438 - Go-http-client/1.1
            		 
            127.0.0.1 - @projector-cbauth [22/Dec/2021:06:51:11 +0000] "GET /pools/default/buckets/travel-sample?bucket_uuid=d12bf81ec72a1b87d9e9f2bc3194bf9a HTTP/1.1" 200 10438 - Go-http-client/1.1
            		 
            127.0.0.1 - @projector-cbauth [22/Dec/2021:07:01:11 +0000] "GET /pools/default/buckets/travel-sample?bucket_uuid=d12bf81ec72a1b87d9e9f2bc3194bf9a HTTP/1.1" 200 10438 - Go-http-client/1.1
            		 
            127.0.0.1 - @projector-cbauth [22/Dec/2021:07:11:11 +0000] "GET /pools/default/buckets/travel-sample?bucket_uuid=d12bf81ec72a1b87d9e9f2bc3194bf9a HTTP/1.1" 200 10438 - Go-http-client/1.1
            		 
            127.0.0.1 - @projector-cbauth [22/Dec/2021:07:21:11 +0000] "GET /pools/default/buckets/travel-sample?bucket_uuid=d12bf81ec72a1b87d9e9f2bc3194bf9a HTTP/1.1" 200 10438 - Go-http-client/1.1
            		 
            127.0.0.1 - @projector-cbauth [22/Dec/2021:07:31:11 +0000] "GET /pools/default/buckets/travel-sample?bucket_uuid=d12bf81ec72a1b87d9e9f2bc3194bf9a HTTP/1.1" 200 10438 - Go-http-client/1.1
            		 
            127.0.0.1 - @projector-cbauth [22/Dec/2021:07:41:11 +0000] "GET /pools/default/buckets/travel-sample?bucket_uuid=d12bf81ec72a1b87d9e9f2bc3194bf9a HTTP/1.1" 200 10438 - Go-http-client/1.1

            hemant.rajput Hemant Rajput added a comment - Validated on Enterprise Edition 6.6.5 build 10067   [root@node1-cb660-centos7 logs]# grep pools/default/buckets/travel-sample?bucket http_access_internal.log | grep @projector-cbauth |tail  127.0.0.1 - @projector-cbauth [22/Dec/2021:06:31:11 +0000] "GET /pools/default/buckets/travel-sample?bucket_uuid=d12bf81ec72a1b87d9e9f2bc3194bf9a HTTP/1.1" 200 10438 - Go-http-client/1.1 127.0.0.1 - @projector-cbauth [22/Dec/2021:06:41:11 +0000] "GET /pools/default/buckets/travel-sample?bucket_uuid=d12bf81ec72a1b87d9e9f2bc3194bf9a HTTP/1.1" 200 10438 - Go-http-client/1.1 127.0.0.1 - @projector-cbauth [22/Dec/2021:06:51:11 +0000] "GET /pools/default/buckets/travel-sample?bucket_uuid=d12bf81ec72a1b87d9e9f2bc3194bf9a HTTP/1.1" 200 10438 - Go-http-client/1.1 127.0.0.1 - @projector-cbauth [22/Dec/2021:07:01:11 +0000] "GET /pools/default/buckets/travel-sample?bucket_uuid=d12bf81ec72a1b87d9e9f2bc3194bf9a HTTP/1.1" 200 10438 - Go-http-client/1.1 127.0.0.1 - @projector-cbauth [22/Dec/2021:07:11:11 +0000] "GET /pools/default/buckets/travel-sample?bucket_uuid=d12bf81ec72a1b87d9e9f2bc3194bf9a HTTP/1.1" 200 10438 - Go-http-client/1.1 127.0.0.1 - @projector-cbauth [22/Dec/2021:07:21:11 +0000] "GET /pools/default/buckets/travel-sample?bucket_uuid=d12bf81ec72a1b87d9e9f2bc3194bf9a HTTP/1.1" 200 10438 - Go-http-client/1.1 127.0.0.1 - @projector-cbauth [22/Dec/2021:07:31:11 +0000] "GET /pools/default/buckets/travel-sample?bucket_uuid=d12bf81ec72a1b87d9e9f2bc3194bf9a HTTP/1.1" 200 10438 - Go-http-client/1.1 127.0.0.1 - @projector-cbauth [22/Dec/2021:07:41:11 +0000] "GET /pools/default/buckets/travel-sample?bucket_uuid=d12bf81ec72a1b87d9e9f2bc3194bf9a HTTP/1.1" 200 10438 - Go-http-client/1.1
            hemant.rajput Hemant Rajput added a comment -

            Validated on 7.1.0-1986. Polling interval increased to 10mins

             

            127.0.0.1 - @projector-cbauth [04/Jan/2022:08:52:26 +0000] "GET /pools/default/buckets/test_bucket?bucket_uuid=5bd3353e5afc026ffc17f5c2061ea468 HTTP/1.1" 200 10706 - "Go-http-client/1.1" 2
            		 
            127.0.0.1 - @projector-cbauth [04/Jan/2022:08:56:38 +0000] "GET /pools/default/buckets/test_bucket?bucket_uuid=5bd3353e5afc026ffc17f5c2061ea468 HTTP/1.1" 200 10766 - "Go-http-client/1.1" 2
            		 
            127.0.0.1 - @projector-cbauth [04/Jan/2022:09:06:38 +0000] "GET /pools/default/buckets/test_bucket?bucket_uuid=5bd3353e5afc026ffc17f5c2061ea468 HTTP/1.1" 200 10766 - "Go-http-client/1.1" 3
            		 
            127.0.0.1 - @projector-cbauth [04/Jan/2022:09:16:38 +0000] "GET /pools/default/buckets/test_bucket?bucket_uuid=5bd3353e5afc026ffc17f5c2061ea468 HTTP/1.1" 200 10765 - "Go-http-client/1.1" 4
            		 
            [root@node1-cb660-centos7 logs]# 

            hemant.rajput Hemant Rajput added a comment - Validated on 7.1.0-1986. Polling interval increased to 10mins   127.0.0.1 - @projector-cbauth [04/Jan/2022:08:52:26 +0000] "GET /pools/default/buckets/test_bucket?bucket_uuid=5bd3353e5afc026ffc17f5c2061ea468 HTTP/1.1" 200 10706 - "Go-http-client/1.1" 2 127.0.0.1 - @projector-cbauth [04/Jan/2022:08:56:38 +0000] "GET /pools/default/buckets/test_bucket?bucket_uuid=5bd3353e5afc026ffc17f5c2061ea468 HTTP/1.1" 200 10766 - "Go-http-client/1.1" 2 127.0.0.1 - @projector-cbauth [04/Jan/2022:09:06:38 +0000] "GET /pools/default/buckets/test_bucket?bucket_uuid=5bd3353e5afc026ffc17f5c2061ea468 HTTP/1.1" 200 10766 - "Go-http-client/1.1" 3 127.0.0.1 - @projector-cbauth [04/Jan/2022:09:16:38 +0000] "GET /pools/default/buckets/test_bucket?bucket_uuid=5bd3353e5afc026ffc17f5c2061ea468 HTTP/1.1" 200 10765 - "Go-http-client/1.1" 4 [root@node1-cb660-centos7 logs]# 

            People

              varun.velamuri Varun Velamuri
              varun.velamuri Varun Velamuri
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                PagerDuty